ServerManagerJob and the Docker cleanup step of ScheduledJobManager
load the settings of every server with SELECT *. That reads the
patch_check_result JSON, which holds the full update list of each
server. With 12,709 rows, the scan took about 4 ms, but the full fetch
took 390-536 ms and 294 MB of PHP memory. Without the column it took
74 ms and 111 MB.
Add ServerSetting::withoutPatchCheckResult() and use it in both loops.
Queued jobs reload the full settings through SerializesModels.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jobs that v4.3.23 put in the queue have no type property. The promoted
constructor property had no default, so these jobs failed with "must not
be accessed before initialization" after the upgrade.
- Declare type as a normal property with a null default
Require deploy or root token access when creating managed mounts outside
resource directories or changing their file contents through the API.
Synchronize KeyDB and Dragonfly passwords with local and shared
REDIS_PASSWORD values while preserving secret manager resolution.
Recovery attempts now back off exponentially, stop after 8 attempts, and
mark the execution as needing attention with a failure category. The
scheduler dispatches at most 100 recovery jobs per run, and a pending S3
cleanup no longer blocks new backups. S3 delete errors now raise with
their cause.
- Recover at most 500 open occurrences per scheduler run.
- Mark claims older than the worker timeout as failed.
- Treat long-pending occurrences like lost queued jobs, and do not run
backups late when they are older than 24 hours.
- Run fixed-time schedules once when daylight saving time repeats an hour.
- Do not run a scheduled task again when Redis hands it out after a killed
worker, and do not overwrite a finished execution in failed().
The scheduler dispatched VolumeBackupRecoveryJob every minute for each
pending execution, and each job retried 20 times with a 60 second backoff.
A stuck execution kept about 20 failing jobs active at a time.
- Run each recovery job once; the scheduler is the only retry path.
- Make the job unique per execution so duplicates do not pile up.
- Dispatch recovery for an execution at most once every 5 minutes.
- Clear the pending flag and record the reason when recovery cannot
succeed (server, resource, S3 storage, or filename no longer exists),
so it does not fail forever and block new backups.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scheduler ran one dispatcher for database backups, tasks, volume backups,
and Docker cleanups after each other. A slow type, for example many tasks due
at the same minute, made the whole run longer, and the shared overlap lock
then made all types skip the next minute.
Run `scheduled:dispatch --type=<type>` every minute for backups, tasks,
volume-backups, and docker-cleanups. Each type runs in its own background
process with its own scheduler and job overlap locks, and publishes and
recovers only the deliveries of its own job type. Running the command without
a type still processes all types.
The missed daily backups on Coolify Cloud came from Redis evicting queue keys
(allkeys-lfu on a full instance), not from the scan of all schedules. The
next_run_at design (452259aac, 5c5740533) did not prevent that, and it added
stored state that must be kept in sync on every frequency, timezone, and server
change. It was not released, so remove it:
- revert next_run_at, its model hooks, the "Next run" UI, and the diagnostics
changes; restore scheduled_job_states and the queue overlap lock
- a new migration removes the next_run_at columns and creates
scheduled_job_states again where the removed migration already ran
Keep the changes that fix real problems:
- run the manager every minute from the scheduler with `scheduled:dispatch`,
so a busy queue cannot delay it past the catch-up window
- load the server chain of scheduled backups with each chunk, which removes
about 7,000 queries per run on Coolify Cloud
- catch Throwable for each schedule, so one broken schedule does not stop the
others
- publish database and volume backups again when their queued job was not
started after 60 minutes (for example, lost in Redis); log lost tasks and
Docker cleanups as missed. A lost backup is not run again when a newer
occurrence of the same schedule exists.
A schedule with a frequency that no longer parses stayed due and logged
"invalid frequency" every minute. Its next run is now cleared (shown as
invalid frequency in the UI), the warning is logged at most once a day,
and fixing the frequency makes it run again. The enabled flag is not
changed.
A delivery stuck in "enqueued" because its queued job was lost was never
run or cleaned up. After 60 minutes, database and volume backups are
published again (late backups run once), and tasks and Docker cleanups
are marked missed with a log entry. The claim token makes a duplicate
job a no-op.
The scheduled job manager checked every enabled schedule each minute and
evaluated its cron expression in PHP. On Coolify Cloud one run took more than
60 seconds, so runs overlapped and minutes were skipped.
Each schedule now stores its next due time:
- next_run_at on scheduled database backups, scheduled tasks, and volume
backups, and docker_cleanup_next_run_at on server settings (UTC, indexed)
- model hooks calculate it when a schedule is created or its frequency or
enabled flag changes; a server timezone change or a move to another server
calculates it again for the affected schedules
The `scheduled:dispatch` command runs the manager every minute from the
scheduler, not from a queue. It selects only due rows and claims each
occurrence with an atomic update of next_run_at, together with its delivery
record, so parallel runs on several nodes never dispatch it twice.
- database and volume backups run once when they are late; tasks and Docker
cleanups more than 10 minutes late are logged as missed
- a local time that occurs two times when the clocks go back runs once
- the migration continues each schedule after its last recorded occurrence and
drops scheduled_job_states
Show the next run, in the server timezone, on the database backup, scheduled
task, volume backup, and Docker cleanup pages, or "Invalid frequency" when the
frequency cannot be calculated. scheduled:diagnostics now lists overdue
schedules and open deliveries.
Remove workarounds that are no longer needed: the queue overlap lock and its
stale-lock repair, the catch-up window, and debug-only logs.
Horizon workers kept Server::findCached() results between jobs, so scheduled
database backups used stale reachability and subscription state. A backup that
was skipped with the stale state counted as done for that day. Flush the
Server identity map before each queue job.
Add debug logs for the full path of a scheduled run, so a multi-node cloud
instance shows where a run is lost:
- manager start delay, gap since the previous start, host, pid, memory, and
a warning when the start is later than the catch-up window
- skip logs with the server state that was used and the current database state
- occurrence enqueued, claimed, rejected, completed and failed, with host and
the delay since the due time; a warning when the previous run is still open
- count of occurrences still open after 15 minutes on each manager run
- database backup runs that end without a backup
- server connection state changes with the checking node and the SSH error
Also keep publishing pending occurrences when one of them fails.
Add database-backed schedule states and deliveries so distributed schedulers publish each occurrence once, queue jobs claim executions atomically, and stale occurrences are cleaned up.
Start backup and task pagination without an id lower bound so a legacy coolify-db backup or scheduled task at id 0 is dispatched with later positive ids. Document instance sentinel ids so they are not treated as ordinary autoincrement rows.
Add polymorphic volume backup scheduling for persistent volumes and directories, expose schedule management via API, and reorganize backup configuration and execution views.
Centralize scheduled job queue selection with crons_queue() and use it for scheduler, task, and database backup jobs so cloud runs on crons while self-hosted stays on high.
Use config-based queue selection for deployment and scheduled jobs so cloud dispatches deployments to `deployments` and scheduled jobs to `crons`, while self-hosted keeps using `high`.
Add coverage for deployment queue helper, start action routing, and scheduled job manager routing.
Extract the shouldRunNow() method from ScheduledJobManager and ServerManagerJob into
a reusable shouldRunCronNow() helper function. This centralizes cron scheduling logic
and enables consistent deduplication behavior across all scheduled job types.
- Create shouldRunCronNow() helper in bootstrap/helpers/shared.php with timezone
and dedup support
- Refactor ScheduledJobManager and ServerManagerJob to use the shared helper
- Add ScheduledJobDiagnostics command for inspecting cache state and scheduling
decisions across all scheduled jobs
- Simplify shouldRunNow tests to directly test the helper function
- Add DockerCleanupJob test for error handling and execution tracking
- Increase scheduled log retention from 1 to 7 days
Implements getPreviousRunDate() + cache-based tracking in shouldRunNow()
to prevent duplicate dispatch of scheduled jobs when queue delays push
execution past the cron minute. This resilience ensures jobs catch missed
windows without double-dispatching within the same cron window.
Updated scheduled job dispatches to include dedupKey parameter:
- Docker cleanup operations
- Server connection checks
- Sentinel restart checks
- Server storage checks
- Server patch checks
DockerCleanupJob now dispatches on the 'high' queue for faster processing.
Includes comprehensive test coverage for dedup behavior across different
cron schedules and delay scenarios.
Move expensive runtime checks (service/application status) after cron
validation to avoid running them for tasks that aren't due. Critical
checks (orphans, infrastructure) remain in first phase.
Also fix database heading parameters to be built from the model.
- Refactor shouldRunNow() to only fire on first run (empty cache) if actually due by cron schedule, preventing spurious executions after cache loss or service restart
- Add enrichSkipLogsWithLinks() method to fetch and populate resource names and links for tasks, backups, and docker cleanup jobs in skip logs
- Update skip logs UI to display resource column with links to related resources, improving navigation and context
- Add fallback display when linked resources are deleted
- Expand tests to cover both restart scenarios: non-due jobs (should not fire) and due jobs (should fire)
Implement dedup key-based cron tracking to make scheduled jobs resilient to queue
delays. Even if a job is delayed by minutes, it will catch the missed cron window
by tracking previousRunDate in cache instead of relying on isDue() alone.
- Add dedupKey parameter to shouldRunNow() in ScheduledJobManager
- When provided, uses getPreviousRunDate() + cache tracking for resilience
- Falls back to isDue() for docker cleanups without dedup key
- Prevents double-dispatch within same cron window
- Optimize ServerConnectionCheckJob dispatch
- Skip SSH checks if Sentinel is healthy (enabled and live)
- Reduces redundant checks when Sentinel heartbeat proves connectivity
- Remove hourly Sentinel update checks
- Consolidate to daily CheckAndStartSentinelJob dispatch
- Crash recovery handled by sentinelOutOfSync → ServerCheckJob flow
- Add logging for skipped database backups with context (backup_id, database_id, status)
- Refactor skip reason methods to accept server parameter, avoiding redundant queries
- Add comprehensive test suite for scheduling with various delay scenarios and timezones
- Add retry configuration to CoolifyTask (3 tries, 600s timeout)
- Add retry configuration to ScheduledTaskJob (3 tries, configurable timeout)
- Add retry configuration to DatabaseBackupJob (2 tries)
- Implement exponential backoff for all jobs (30s, 60s, 120s intervals)
- Add failed() handlers with comprehensive error logging to scheduled-errors channel
- Add execution tracking: started_at, retry_count, duration (decimal), error_details
- Add configurable timeout field to scheduled tasks (60-3600s, default 300s)
- Update UI to include timeout configuration in task creation/editing forms
- Increase ScheduledJobManager lock expiration from 60s to 90s for high-load environments
- Implement safe queue cleanup with restart vs runtime modes
- Restart mode: aggressive cleanup (marks all processing jobs as failed)
- Runtime mode: conservative cleanup (only marks jobs >12h as failed, skips deployments)
- Add cleanup:redis --restart flag for system startup
- Integrate cleanup into Dev.php init() for development environment
- Increase scheduled-errors log retention from 7 to 14 days
- Create comprehensive test suite (unit and feature tests)
- Add TESTING_GUIDE.md with manual testing instructions
Fixes issues with jobs failing after single attempt and "attempted too many times" errors
## Problem
Scheduled tasks, backups, and auto-updates stopped working after 1-2 months
with error: MaxAttemptsExceededException: App\Jobs\ScheduledJobManager has
been attempted too many times.
Root cause: ScheduledJobManager used WithoutOverlapping with only
releaseAfter(60), causing locks without expiration (TTL=-1) that persisted
indefinitely when jobs hung or processes crashed.
## Solution
### Part 1: Prevention (Future Locks)
- Added expireAfter(60) to ScheduledJobManager middleware
- Lock now auto-expires after 60 seconds (matches everyMinute schedule)
- Changed from releaseAfter(60) to expireAfter(60)->dontRelease()
- Follows Laravel best practices and matches other Coolify jobs
### Part 2: Recovery (Existing Locks)
- Enhanced cleanup:redis command with --clear-locks flag
- Scans Redis for stale locks (TTL=-1) and removes them
- Called automatically during app:init on startup/upgrade
- Provides immediate recovery for affected instances
## Changes
- app/Jobs/ScheduledJobManager.php: Added expireAfter(60)->dontRelease()
- app/Console/Commands/CleanupRedis.php: Added cleanupCacheLocks() method
- app/Console/Commands/Init.php: Auto-clear locks on startup
- tests/Unit/ScheduledJobManagerLockTest.php: Test to prevent regression
- STALE_LOCK_FIX.md: Complete documentation
## Testing
- Unit tests pass (2 tests, 8 assertions)
- Code formatted with Pint
- Matches pattern used by CleanupInstanceStuffsJob
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Andras BacsaiandClaude Opus 5.5