Each dump and S3 upload got the full backup timeout, so a long dump plus
an upload passed the worker timeout. The worker then killed the job
without cleanup: the execution stayed running, and the upload container
and the dump file stayed on the server.
- Give each step only the time that is left of the backup timeout, so a
run that takes too long fails inside the job and cleans up.
- Run the dump once. An SSH retry started a second dump into the same
file while the first could still run in the container.
- Explain exit code 124 and say in the timeout help that it covers the
complete run.
- Remove backup upload containers of executions that are not running
in the daily helper container cleanup.
Closes#8139
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The MinIO client rejects endpoints such as `/storage/v1/s3`. Transfers to
these endpoints now use curl with a presigned URL that Coolify signs, for
database backups, volume backups, and S3 restores.
Closes#8530
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
S3 storages with a self-signed or private CA certificate failed with
"cURL error 60", because the Coolify container and the backup helper do
not use the CA certificates of the host. The new opt-in setting turns off
the certificate check in the PHP S3 clients and sets MC_INSECURE for mc.
Closes#9762
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mc keeps one --resolve IP per host, and Coolify always pinned the IPv4
address of a dual-stack endpoint. On servers without IPv4, uploads failed
with "network is unreachable". A helper container on the upload network
now connects to each validated IP, IPv4 first, and the first IP that
connects is pinned. S3 restores are now pinned in the same way.
Closes#12397
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a "Use path-style URLs" setting to S3 storages. It is detected from
the endpoint (Alibaba OSS, Tencent COS, Volcengine/BytePlus TOS), and the
connection test switches to virtual-hosted style when the provider
rejects path-style requests. Backups pass `--path off` to mc.
Closes#10347
A user database named coolify-db was backed up as the instance database,
which targeted the wrong container. It was also hidden from the server
database list and from status updates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pg_dump ran with --no-acl --no-owner, so single-database backups lost
all GRANTs and a restored Supabase database did not work. The import
already skips owners and privileges unless "Keep owners and privileges"
is selected, so the backup now keeps them.
Closes#12244
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mc --resolve applies per invocation and is not stored in the alias, so
mc cp resolved the S3 host again inside the helper container. On hosts
with split DNS this failed with "no such host", and the upload did not
use the IP validated by SafeWebhookUrl.
Fixes#12129
When the execution of the next database could not be created, the
error handler used the state of the previous database. It marked the
successful backup of that database as failed and could delete its file.
- Reset the execution, location, and output at the start of each database
Allow validated host file and directory mounts outside resource directories
while protecting external paths from deletion.
Clean up cancelled deployment containers and clear tracked process IDs.
Preserve legacy KeyDB and Dragonfly password parsing, accept server backup
paths without extension restrictions, and allow Compose variable proxy ports.
The SSH retry matched connection words in any error output. When `mc alias set`
in a backup upload could not reach S3, its "connection refused" output caused a
retry of the full batch, and the second `docker run --name backup-of-<uuid>`
failed with a name conflict that hid the real error.
- Retry only SSH transport failures: exit code 255, or errors without an exit code.
- Keep the remote exit code on deployment command failures.
- Remove a leftover backup helper container in the same upload batch.
Closes#12059
- Skip the occurrence of a backup whose previous run still holds the
overlap lock, so stale recovery does not run it later; do not rerun a
lost occurrence when a newer one already completed.
- Skip a database backup while a start, restart, or import of the database
is in progress, and record the skip in the execution list.
- Give every backup file a unique name per run and keep the service
database backup folder of earlier releases.
- Record missed scheduled tasks and Docker cleanups as failed executions
and notify the team.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A plain pipe into the compressor only reported the compressor's exit
status, so a failed dump still produced a small, valid archive. Add
pipeToFileKeepingExitStatus() to propagate the source's exit status in
POSIX sh and use it for all compressed dump commands.
- Delete the partial local backup file on any failed backup, not only
for ClickHouse, since untracked files are never removed by retention
- Wait up to 10s for locks when running VACUUM INTO for SQLite backups
- Refuse SQLite restores unless the file has a SQLite header, so an
empty backup no longer wipes the database
- Add tests for dump exit status handling and SQLite restore validation
A non-root SSH user cannot open redirects, expand globs, or cd into
/data/coolify (9999:root 0700). Several remote commands relied on that
and failed on non-root servers.
- Write README files with `tee` instead of shell redirects in database
starts and application deployments
- Clear Postgres init scripts with `find` instead of a glob
- Split the proxy config backfill if/else so each branch gets sudo
- Start the log drain with `docker compose -f` instead of `cd`
- Wrap the preserved-repository compose start in `sh -c` for non-root
- Stage clone archives outside /data/coolify via StagesCloneArchives
- Extend the sudo parser and add non-root command tests
Horizon workers kept Server::findCached() results between jobs, so scheduled
database backups used stale reachability and subscription state. A backup that
was skipped with the stale state counted as done for that day. Flush the
Server identity map before each queue job.
Add debug logs for the full path of a scheduled run, so a multi-node cloud
instance shows where a run is lost:
- manager start delay, gap since the previous start, host, pid, memory, and
a warning when the start is later than the catch-up window
- skip logs with the server state that was used and the current database state
- occurrence enqueued, claimed, rejected, completed and failed, with host and
the delay since the due time; a warning when the previous run is still open
- count of occurrences still open after 15 minutes on each manager run
- database backup runs that end without a backup
- server connection state changes with the checking node and the SSH error
Also keep publishing pending occurrences when one of them fails.
Development mode rewrote /data/coolify paths to the dev Docker volume
for every server. That is only right for the testing-host backend,
which uses the host Docker daemon. The KVM localhost and other dev VMs
have their own /data/coolify, so bind mounts, Compose content files,
the Traefik and Caddy traffic mounts, Sentinel, and database proxies
pointed to wrong paths there.
- Server::sharesDevHostDocker() and devHostDockerPath() decide per
server; only the testing-host server (ip coolify-testing-host) gets
the volume path. Production is unchanged.
- The testing-host volume names come from the Compose project
(DEV_COOLIFY_DATA_VOLUME, DEV_COOLIFY_BACKUPS_VOLUME), with the
legacy names as fallback.
- In development, saved proxy configurations with the old path are
fixed when they load and ask for a proxy restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add database-backed schedule states and deliveries so distributed schedulers publish each occurrence once, queue jobs claim executions atomically, and stale occurrences are cleaned up.
instant_remote_process() applied its timeout only to Process::timeout(), while the shell-level timeout wrapper around ssh always used the global 3600s default, killing long backups and scheduled tasks at 1h with ssh exit code 255. Also applies the backup timeout to the S3 upload step and makes the default configurable via SSH_COMMAND_TIMEOUT.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Store s3_storage_id on scheduled volume backup executions so retention
and recovery use the S3 that received the upload. Extract
DeleteScheduledVolumeBackup for UI and resource deletion, and isolate
database backup retention failures so cleanup errors do not fail the job.
Enable scheduled ClickHouse backups across the job, API, and UI, and
guard unsupported database types via isBackupSolutionAvailable().
Convert Stripe subscription sync from a job to an action with clearer
discrepancy resolution, and add cloud:export-users plus
cloud:cleanup-unverified-users with tests.
Prevent scheduled database backups from enabling S3 uploads without a valid team-owned storage configuration, and preserve the previous S3 storage ID in missing-storage error messages.
Add coverage for backup edit/create validation and S3 upload failure messaging.
Centralize scheduled job queue selection with crons_queue() and use it for scheduler, task, and database backup jobs so cloud runs on crons while self-hosted stays on high.
Himanshu AhujaandClaude Fable 5