The Domains page polls every 2 seconds while a DNS check runs. Each poll
parsed the compose file, and the parser queued one ServerFilesFromServerJob
per volume. The jobs starved the DNS check jobs and deployments.
- Skip the compose parse in pollDnsChecks and keep the known services.
- Make ServerFilesFromServerJob unique per resource until it runs.
Fixes#12098
The commit message was read only during the repository clone. Deployments
that skipped the build or failed before the clone showed only the commit
hash. They now reuse the message stored by an earlier deployment of the
same application and commit.
Closes#12116
Pull request deployments sent every non-railpack build pack to build_image(),
which no longer handles the static build pack since it was moved to
build_static_image(). Static preview deployments then failed with
"open Dockerfile: no such file or directory".
Closes#8238
Sentry self-hosted (getsentry/self-hosted 26.8.0) could not be deployed
as a Docker Compose application. Three separate problems blocked it:
- YAML collection aliases: the first fix raised the limit to 256 only in
Application::oldRawParser(). The safety check added in v4.4.0
(validateDockerComposeForInjection) and all other Compose parse calls
still used the Symfony default of 128, so the Sentry file (133 aliases)
failed with "Maximum number of collection aliases (128) exceeded".
Add parseDockerComposeYaml() and use it for every Compose parse.
- Argument list too long: the deployment job sent the generated Compose
file as base64 inside the SSH command. Linux limits one argument to
128 KiB, and the Sentry file is about 370 KB. execute_remote_command()
now accepts an 'input' key that streams content over SSH stdin, and
the job writes the Compose file with it.
- A list-style build arg without a value (build.args: [SENTRY_IMAGE])
merged into a map environment kept its numeric key, so Docker Compose
rejected the file with "non-string key in ...environment: 0". It now
becomes NAME with no value, as in Compose.
Closes#11725
Check enabled volume backup schedules hourly and notify teams when no
execution occurs within the configured number of days. Expose the alert
threshold in the UI and API, with alerts disabled by default.
Route volume backup results and warnings to the appropriate Telegram
backup threads.
Every released Sentinel with traffic analytics (1.0.0 and later) has
the dashboard and series routes and the requests, bytes_in and app
fields, so the fallbacks for builds without them are not needed.
- Keep "no data" handling for Sentinel without traffic routes (0.0.x)
- Keep the resource-scope fallback, which Sentinel 1.0.1 needs
The per-server schedules reused the cache keys of the old schedule, so
every server ran these checks at once on the first run after the
upgrade. New keys start fresh, so each server runs at its own slot.
- A dispatch failure for one additional server no longer skips the
other servers or the combined notification
- After a deployment fails to dispatch, the queue starts the next
queued deployment; nested queue advances run in a loop, not as
recursion, so repeated failures end
Require usage to fall five percentage points below the threshold before releasing the notification throttle. Reset it after Sentinel reports recovery so a new spike alerts immediately.
Add regression coverage for recovery and usage fluctuating near the threshold.
- A queued deployment on a deleted server no longer stops the queue;
one deployment that fails to start does not stop the others
- Cancelling a deployment starts the queued deployments of the same
application on all servers, as a finished deployment does
Since 17ca63ff4 every deployment except Docker image apps failed on a
build-time variable with a name such as MY-VAR, also restarts and
deployments that reuse an existing image. v4.3.23 failed only when it
built an image.
- Check build-time names right before the build steps; the start of
the deployment only logs a warning
- Names that break the .env file still fail every deployment
- Leave invalid names out of the build secrets helper container, so
restarts with build secrets work
The high disk usage throttle stayed for 24 hours after usage dropped,
so a new spike in that time sent no alert.
- Release the throttle when disk usage is at or below the threshold
When a container that a volume backup stopped no longer existed (for
example after a redeploy), recovery kept it as remaining. The execution
needed attention and every later run of the schedule was skipped.
- Treat "no such container" or "no such object" from docker inspect as
recovered; other failures still need attention
When the execution of the next database could not be created, the
error handler used the state of the previous database. It marked the
successful backup of that database as failed and could delete its file.
- Reset the execution, location, and output at the start of each database
ReconcileGithubRunnersJob connected over SSH every minute to every
server with a runner configuration, also when the runners were
disabled.
- Reconcile a server only when its runners are enabled or it still has
active runner executions, so jobs that run after disabling still end
failed() marked the latest running execution of the server as failed.
A manual run that never started, or a run that timed out, could mark a
different running cleanup as failed.
- Keep the execution id in the cache under the queue job uuid, because
failed() runs on a fresh copy of the job
- Change nothing when the job created no execution
A manual cleanup that started while a stop-triggered cleanup held the
server lock was dropped. The UI said it was queued, but it never ran.
- Release a manual run and try again after 60 seconds
- Stop the retries 780 seconds after dispatch (the lock expiry plus a
margin)
- Scheduled and stop-triggered runs are still dropped
CleanupInstanceStuffsJob, CleanupOrphanedPreviewContainersJob, and
CheckMissingDatabaseBackupsJob had no uniqueFor. A lock left by a
stopped worker blocked them for up to 24 hours.
- Set uniqueFor to 60, 600, and 1800 seconds
- Give CheckMissingDatabaseBackupsJob a 1800 second timeout, so a run
cannot outlive its lock
RevalidateUnusableS3StoragesJob had no uniqueFor. A lock left by a
stopped worker blocked the job for up to 24 hours, so unusable S3
storages were not checked again and their backups were skipped.
- Set uniqueFor to 1800 seconds
When a deployment on an additional server finished, only the queue of
the primary server advanced. Deployments queued on the additional server
could wait with no limit.
- Also advance the queue of the server of the finished deployment
Jobs that v4.3.23 put in the queue have no type property. The promoted
constructor property had no default, so these jobs failed with "must not
be accessed before initialization" after the upgrade.
- Declare type as a normal property with a null default
Scope resource permissions and related records to the owning team across
team switches, cloning, deployments, and resource operations.
Merge ClickHouse volume mounts, preserve anonymous data volumes on stop,
and require adopting current data before starting or restarting.
Fix file storage deletion checks and add regression coverage.
Require deploy or root token access when creating managed mounts outside
resource directories or changing their file contents through the API.
Synchronize KeyDB and Dragonfly passwords with local and shared
REDIS_PASSWORD values while preserving secret manager resolution.
Allow validated host file and directory mounts outside resource directories
while protecting external paths from deletion.
Clean up cancelled deployment containers and clear tracked process IDs.
Preserve legacy KeyDB and Dragonfly password parsing, accept server backup
paths without extension restrictions, and allow Compose variable proxy ports.
Add Application::isGithubAppSource(), which checks that the source is a
GithubApp, and use it in place of is_github_based() for pull request
status updates in ApplicationDeploymentJob and for the "Load pull
requests" action in the previews view.
Previews::load_prs now throws an error for non-GitHub App sources
instead of calling the GitHub API with a GitLab or other source. Also
adds a test checking that GitLab sources don't show or load GitHub pull
requests.
- Restart a running Sentinel that has not pushed for 10 minutes, at most
once per hour. The restart applies the current URL and token.
- Restart an unhealthy Sentinel. Do not restart while its health check
starts or while its API does not answer yet, which caused restart loops.
- Remember why pushes fail: the reason Coolify rejects a push, and the last
push error from Sentinel's /api/push-status. Show it on the Sentinel page
while Sentinel is out of sync.
- When the instance URL or public IP changes, move remote servers that use
the generated Sentinel URL to the new URL. Custom URLs stay.
- ServerCheckJob uses the stored reachability state instead of a second SSH
probe. One failed probe no longer marks the server unreachable without
UNREACHABLE_THRESHOLD or marks all its resources exited.
- Do not dispatch ServerCheckJob for build servers and Swarm workers.
- Check cloud provider state every 5 minutes instead of every minute.
- ServerPatchCheckJob uses isFunctional() and drops the unused retries.
- Delete the unused ServerCleanupMux job.
- Disable SSH multiplexing per command. The global config change stayed in
the queue worker, so later jobs (also deployments) ran without mux.
Remove the periodic ConnectProxyToNetworksJob dispatch from
PushServerUpdateJob, the synchronous connect in ServerCheckJob when the
proxy is found running, and the connect-on-add in the server
Destinations component. Drop the now-unused
proxy.connect_networks_interval_seconds config option.
Replace the rate-limit/interval tests with a test asserting that no
jobs are pushed when the proxy is running.
Add StandaloneDocker::proxyConnectCommand() and run it inline when a
destination is created, replacing the synchronous
ConnectProxyToNetworksJob dispatch. Deployments also run it before
docker compose up, so the proxy is attached to the destination network
even when it was missing.
- On cloud, RestartService and RestartServiceApplication now run on the
deployments queue instead of high.
- StartProxy and RestartProxyJob, and the remote processes they start,
now run on the deployments queue.
- Self-hosted still uses the high queue.
- Tests cover the new routing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- CheckAndStartSentinelJob returns early when the server is not
functional. It now reads the container state with `docker ps --filter`,
so an SSH failure throws and is no longer treated as a missing
container that needs a restart.
- ServerManagerJob only dispatches the Sentinel version check for
functional servers.
- On cloud, ServerPatchCheckJob runs on the maintenance queue and
ServerLimitCheckJob runs on the webhooks queue.
- The default number of cloud maintenance workers goes from 10 to 60.
- excludeCertainErrors drops its unused ignore list and is typed as
`never`.
- Docs and the Horizon config comments are updated, and tests cover the
new behavior.
On cloud, GitHub pull request webhooks and Stripe events now run on a
dedicated `webhooks` queue so a busy `high` queue cannot delay them.
Self-hosted instances keep these jobs on `high`.
- Add webhooks_queue() helper and use it in ProcessGithubPullRequestWebhook
and StripeProcessJob
- Provision a `webhooks` Horizon pool in cloud production, sized by
HORIZON_WEBHOOKS_PROCESSES (default 10)
- Document the new pool, rollout and rollback steps
- Cover pool provisioning and job queue routing in tests
Scheduled, manual and stop-triggered cleanups share one per-server lock.
Stop-triggered cleanups wait 60 seconds and are coalesced per server, so
a bulk stop queues one cleanup. Remote commands share a 540 second
deadline below the 600 second job timeout, and application images are
scanned with a fixed number of remote commands.
Recovery attempts now back off exponentially, stop after 8 attempts, and
mark the execution as needing attention with a failure category. The
scheduler dispatches at most 100 recovery jobs per run, and a pending S3
cleanup no longer blocks new backups. S3 delete errors now raise with
their cause.
Servers no longer all connect at minute 0. The hourly Sentinel check uses
a per-server minute, and the weekly patch check uses a per-server time on
Sunday between 04:00 and 23:59.
- Recover at most 500 open occurrences per scheduler run.
- Mark claims older than the worker timeout as failed.
- Treat long-pending occurrences like lost queued jobs, and do not run
backups late when they are older than 24 hours.
- Run fixed-time schedules once when daylight saving time repeats an hour.
- Do not run a scheduled task again when Redis hands it out after a killed
worker, and do not overwrite a finished execution in failed().
Queued GetContainersStatus runs are unique per server until a worker
starts them, and CleanupHelperContainersJob locks per server instead of
class-wide. Add UniqueUntilProcessingJobDecorator for laravel-actions.
After a start, the UI showed "exited" until the regular status check ran.
That check runs on the shared high queue, which can be busy (60 s on Cloud).
The start job (database) and the start task (service, on the deployment
queue) now inspect only the started containers and store their status
before they broadcast. While a health check reports "starting", they
inspect again for up to about 15 s.
ContainerStatusAggregator now builds the per-container status and the
Compose aggregation for both writers, so GetContainersStatus and the
targeted check store the same strings.
The SSH retry matched connection words in any error output. When `mc alias set`
in a backup upload could not reach S3, its "connection refused" output caused a
retry of the full batch, and the second `docker run --name backup-of-<uuid>`
failed with a name conflict that hid the real error.
- Retry only SSH transport failures: exit code 255, or errors without an exit code.
- Keep the remote exit code on deployment command failures.
- Remove a leftover backup helper container in the same upload batch.
Closes#12059
On cloud, StartService and DeployServiceApplication dispatched their
CoolifyTask on the high queue, so service starts competed with all other
high jobs. remote_process() and CoolifyTask now accept a queue (default
high); start paths pass deployment_queue(). The queued RestartDatabase
action also runs on deployment_queue() instead of the default queue.
- Manual webhooks (GitHub, GitLab, Gitea, Bitbucket) now reject all
deliveries for a locked-out scope before reading secrets or verifying
signatures, including correctly signed ones, until the window expires.
- ProvisionGithubRunnerJob checks whether the execution has already
finished before and after each provisioning step, and only sets
ready_at or Failed while it is still occupying/provisioning. A runner
that is cancelled, timed out or completed mid-provisioning is no longer
started, retried or overwritten.
- VolumeCloneJob creates bind-volume clones as plain local volumes
without the source driver options, so the clone never mounts the
source host directory.
- Image pulls for additional servers no longer ignore docker pull errors.
- Add tests for each change.
- Skip the occurrence of a backup whose previous run still holds the
overlap lock, so stale recovery does not run it later; do not rerun a
lost occurrence when a newer one already completed.
- Skip a database backup while a start, restart, or import of the database
is in progress, and record the skip in the execution list.
- Give every backup file a unique name per run and keep the service
database backup folder of earlier releases.
- Record missed scheduled tasks and Docker cleanups as failed executions
and notify the team.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Try to remove every volume and always delete the configuration
directory (it holds the .env with secrets), then report the volumes that
failed.
- Delete volume backup schedules also when a service is removed from
Coolify only.
- The delete API no longer claims the server is unreachable when the
caller asked to delete from Coolify only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Older versions accepted any static image through the API. Such a stored
value now stops the deployment with a message that lists the supported
web servers instead of a ValueError.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Andras Bacsai
Brandon BridgesandAndras Bacsai
