When the execution of the next database could not be created, the
error handler used the state of the previous database. It marked the
successful backup of that database as failed and could delete its file.
- Reset the execution, location, and output at the start of each database
Allow validated host file and directory mounts outside resource directories
while protecting external paths from deletion.
Clean up cancelled deployment containers and clear tracked process IDs.
Preserve legacy KeyDB and Dragonfly password parsing, accept server backup
paths without extension restrictions, and allow Compose variable proxy ports.
The SSH retry matched connection words in any error output. When `mc alias set`
in a backup upload could not reach S3, its "connection refused" output caused a
retry of the full batch, and the second `docker run --name backup-of-<uuid>`
failed with a name conflict that hid the real error.
- Retry only SSH transport failures: exit code 255, or errors without an exit code.
- Keep the remote exit code on deployment command failures.
- Remove a leftover backup helper container in the same upload batch.
Closes#12059
- Skip the occurrence of a backup whose previous run still holds the
overlap lock, so stale recovery does not run it later; do not rerun a
lost occurrence when a newer one already completed.
- Skip a database backup while a start, restart, or import of the database
is in progress, and record the skip in the execution list.
- Give every backup file a unique name per run and keep the service
database backup folder of earlier releases.
- Record missed scheduled tasks and Docker cleanups as failed executions
and notify the team.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A plain pipe into the compressor only reported the compressor's exit
status, so a failed dump still produced a small, valid archive. Add
pipeToFileKeepingExitStatus() to propagate the source's exit status in
POSIX sh and use it for all compressed dump commands.
- Delete the partial local backup file on any failed backup, not only
for ClickHouse, since untracked files are never removed by retention
- Wait up to 10s for locks when running VACUUM INTO for SQLite backups
- Refuse SQLite restores unless the file has a SQLite header, so an
empty backup no longer wipes the database
- Add tests for dump exit status handling and SQLite restore validation
A non-root SSH user cannot open redirects, expand globs, or cd into
/data/coolify (9999:root 0700). Several remote commands relied on that
and failed on non-root servers.
- Write README files with `tee` instead of shell redirects in database
starts and application deployments
- Clear Postgres init scripts with `find` instead of a glob
- Split the proxy config backfill if/else so each branch gets sudo
- Start the log drain with `docker compose -f` instead of `cd`
- Wrap the preserved-repository compose start in `sh -c` for non-root
- Stage clone archives outside /data/coolify via StagesCloneArchives
- Extend the sudo parser and add non-root command tests
Horizon workers kept Server::findCached() results between jobs, so scheduled
database backups used stale reachability and subscription state. A backup that
was skipped with the stale state counted as done for that day. Flush the
Server identity map before each queue job.
Add debug logs for the full path of a scheduled run, so a multi-node cloud
instance shows where a run is lost:
- manager start delay, gap since the previous start, host, pid, memory, and
a warning when the start is later than the catch-up window
- skip logs with the server state that was used and the current database state
- occurrence enqueued, claimed, rejected, completed and failed, with host and
the delay since the due time; a warning when the previous run is still open
- count of occurrences still open after 15 minutes on each manager run
- database backup runs that end without a backup
- server connection state changes with the checking node and the SSH error
Also keep publishing pending occurrences when one of them fails.
Development mode rewrote /data/coolify paths to the dev Docker volume
for every server. That is only right for the testing-host backend,
which uses the host Docker daemon. The KVM localhost and other dev VMs
have their own /data/coolify, so bind mounts, Compose content files,
the Traefik and Caddy traffic mounts, Sentinel, and database proxies
pointed to wrong paths there.
- Server::sharesDevHostDocker() and devHostDockerPath() decide per
server; only the testing-host server (ip coolify-testing-host) gets
the volume path. Production is unchanged.
- The testing-host volume names come from the Compose project
(DEV_COOLIFY_DATA_VOLUME, DEV_COOLIFY_BACKUPS_VOLUME), with the
legacy names as fallback.
- In development, saved proxy configurations with the old path are
fixed when they load and ask for a proxy restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add database-backed schedule states and deliveries so distributed schedulers publish each occurrence once, queue jobs claim executions atomically, and stale occurrences are cleaned up.
instant_remote_process() applied its timeout only to Process::timeout(), while the shell-level timeout wrapper around ssh always used the global 3600s default, killing long backups and scheduled tasks at 1h with ssh exit code 255. Also applies the backup timeout to the S3 upload step and makes the default configurable via SSH_COMMAND_TIMEOUT.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Store s3_storage_id on scheduled volume backup executions so retention
and recovery use the S3 that received the upload. Extract
DeleteScheduledVolumeBackup for UI and resource deletion, and isolate
database backup retention failures so cleanup errors do not fail the job.
Enable scheduled ClickHouse backups across the job, API, and UI, and
guard unsupported database types via isBackupSolutionAvailable().
Convert Stripe subscription sync from a job to an action with clearer
discrepancy resolution, and add cloud:export-users plus
cloud:cleanup-unverified-users with tests.
Prevent scheduled database backups from enabling S3 uploads without a valid team-owned storage configuration, and preserve the previous S3 storage ID in missing-storage error messages.
Add coverage for backup edit/create validation and S3 upload failure messaging.
Centralize scheduled job queue selection with crons_queue() and use it for scheduler, task, and database backup jobs so cloud runs on crons while self-hosted stays on high.
Add `WithoutOverlapping` middleware to `DatabaseBackupJob` keyed by backup ID
with timeout-based lock expiry to prevent concurrent runs.
Mark long-running backup executions as failed when they exceed the stale
time threshold, and add periodic retention enforcement in
`CleanupInstanceStuffsJob` with cache-based throttling.
Also add float casts for retention max-storage fields on
`ScheduledDatabaseBackup` and comprehensive feature tests covering
overlap middleware, stale detection, casts, and retention behavior.
Add strict validation for Docker network names using a regex pattern
that matches Docker's naming rules (alphanumeric start, followed by
alphanumeric, dots, hyphens, underscores).
Changes:
- Add DOCKER_NETWORK_PATTERN to ValidationPatterns with helper methods
- Validate network field in Destination creation and update Livewire components
- Add setNetworkAttribute mutator on StandaloneDocker and SwarmDocker models
- Apply escapeshellarg() to all network field usages in shell commands across
ApplicationDeploymentJob, DatabaseBackupJob, StartService, Init command,
proxy helpers, and Destination/Show
- Add comprehensive tests for pattern validation and model mutator
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Apply proper shell escaping to all user-controlled values interpolated into
backup shell commands (PostgreSQL username/password, MySQL/MariaDB root
password, MongoDB URI). Also URL-encode MongoDB credentials before embedding
in connection URI. Adds unit tests for escaping behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add validateDatabasesBackupInput() helper that properly parses all
database backup formats including MongoDB's "db:col1,col2|db2:col3"
and validates each component individually.
- Validate and escape collection names in DatabaseBackupJob
- Replace comma-only split in BackupEdit with format-aware validation
- Add input validation in API create_backup and update_backup endpoints
- Add unit tests for collection name and multi-format validation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add new Resources tab to storage show page displaying backup schedules using
that storage. Refactor storage show layout with navigation tabs for General
and Resources sections. Move delete action from form to show component.
Implement cascade deletion in S3Storage model to automatically disable S3
backups when storage is deleted. Improve error handling in DatabaseBackupJob
to throw exception when S3 storage is missing instead of silently returning.
- New Storage/Resources Livewire component
- Add resources.blade.php view
- Add storage.resources route
- Move delete() method from Form to Show component
- Add deleting event listener to S3Storage model
- Track backup count and current route in Show component
- Add #[On('submitStorage')] attribute to form submission
- Implement pagination for skipped jobs display with 20 items per page
- Add pagination controls (previous/next buttons) to the scheduled jobs view
- Exclude ScheduledJobManager "started" events from run logs, keeping only "completed" events
- Add ShouldBeEncrypted interface to ScheduledTaskJob for secure queue handling
- Update log filtering to fetch 500 recent skips and slice for pagination
- Use Log facade instead of fully qualified class name
Implement dedup key-based cron tracking to make scheduled jobs resilient to queue
delays. Even if a job is delayed by minutes, it will catch the missed cron window
by tracking previousRunDate in cache instead of relying on isDue() alone.
- Add dedupKey parameter to shouldRunNow() in ScheduledJobManager
- When provided, uses getPreviousRunDate() + cache tracking for resilience
- Falls back to isDue() for docker cleanups without dedup key
- Prevents double-dispatch within same cron window
- Optimize ServerConnectionCheckJob dispatch
- Skip SSH checks if Sentinel is healthy (enabled and live)
- Reduces redundant checks when Sentinel heartbeat proves connectivity
- Remove hourly Sentinel update checks
- Consolidate to daily CheckAndStartSentinelJob dispatch
- Crash recovery handled by sentinelOutOfSync → ServerCheckJob flow
- Add logging for skipped database backups with context (backup_id, database_id, status)
- Refactor skip reason methods to accept server parameter, avoiding redundant queries
- Add comprehensive test suite for scheduling with various delay scenarios and timezones
Helper containers are started with --rm flag which automatically removes the container when it stops. Removed redundant docker rm commands from graceful_shutdown_container in ApplicationDeploymentJob and replaced docker rm with docker stop in DatabaseBackupJob.
🤖 Generated with Claude Code
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When multiple scheduled tasks or database backups run concurrently on
the same server, they compete for the same SSH multiplexed connection
socket, causing race conditions and SSH exit code 255 errors.
This fix adds a `disableMultiplexing` parameter to bypass SSH
multiplexing for jobs that may run concurrently:
- Add `disableMultiplexing` param to `generateSshCommand()`
- Add `disableMultiplexing` param to `instant_remote_process()`
- Update `ScheduledTaskJob` to use `disableMultiplexing: true`
- Update `DatabaseBackupJob` to use `disableMultiplexing: true`
- Add debug logging to track execution without multiplexing
- Add unit tests for the new parameter
Each backup and scheduled task now gets an isolated SSH connection,
preventing contention on the shared multiplexed socket.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Allows user-configured backup timeouts > 3600 to be respected. Previously, the SSH process used a hardcoded 3600 second timeout regardless of the job timeout setting. Now the timeout is passed through to instant_remote_process() for all backup operations.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add retry configuration to CoolifyTask (3 tries, 600s timeout)
- Add retry configuration to ScheduledTaskJob (3 tries, configurable timeout)
- Add retry configuration to DatabaseBackupJob (2 tries)
- Implement exponential backoff for all jobs (30s, 60s, 120s intervals)
- Add failed() handlers with comprehensive error logging to scheduled-errors channel
- Add execution tracking: started_at, retry_count, duration (decimal), error_details
- Add configurable timeout field to scheduled tasks (60-3600s, default 300s)
- Update UI to include timeout configuration in task creation/editing forms
- Increase ScheduledJobManager lock expiration from 60s to 90s for high-load environments
- Implement safe queue cleanup with restart vs runtime modes
- Restart mode: aggressive cleanup (marks all processing jobs as failed)
- Runtime mode: conservative cleanup (only marks jobs >12h as failed, skips deployments)
- Add cleanup:redis --restart flag for system startup
- Integrate cleanup into Dev.php init() for development environment
- Increase scheduled-errors log retention from 7 to 14 days
- Create comprehensive test suite (unit and feature tests)
- Add TESTING_GUIDE.md with manual testing instructions
Fixes issues with jobs failing after single attempt and "attempted too many times" errors
Changed backup_log_uuid property to nullable and removed eager initialization in constructor. This allows the ID to be generated when actually needed rather than upfront.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Introduced a new notification class, BackupSuccessWithS3Warning, to alert users when local backups succeed but S3 uploads fail.
- Updated DatabaseBackupJob to track local backup success and handle S3 upload errors, improving error reporting and user notifications.
- Modified ScheduledDatabaseBackupExecution model to include a new s3_uploaded boolean field for tracking S3 upload status.
- Adjusted views and validation logic to reflect changes in backup execution status and S3 handling.
- Added tests to ensure the new s3_uploaded column is correctly implemented and validated.
- Refactored the DatabaseBackupJob to replace instances of backup->uuid with backup_log_uuid for consistency in container naming.
- Ensured that all related Docker commands and processes reference the updated backup_log_uuid, improving clarity and reducing potential errors during backup operations.
- Refactored the DatabaseBackupJob to generate a unique UUID for each database backup execution within the loop, improving clarity and ensuring uniqueness for each backup attempt.
- Removed redundant UUID generation logic from the initial part of the handle method.
Andras Bacsai
peaklabs-dev
Himanshu AhujaandClaude Fable 5