- Add a per-task overlap lock to ScheduledTaskJob that outlives the worker
timeout, so a long or killed run cannot overlap the next cron run.
- Mark the dropped occurrence as skipped instead of releasing the job.
Closes#11648
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Check enabled volume backup schedules hourly and notify teams when no
execution occurs within the configured number of days. Expose the alert
threshold in the UI and API, with alerts disabled by default.
Route volume backup results and warnings to the appropriate Telegram
backup threads.
Allow validated host file and directory mounts outside resource directories
while protecting external paths from deletion.
Clean up cancelled deployment containers and clear tracked process IDs.
Preserve legacy KeyDB and Dragonfly password parsing, accept server backup
paths without extension restrictions, and allow Compose variable proxy ports.
- Recover at most 500 open occurrences per scheduler run.
- Mark claims older than the worker timeout as failed.
- Treat long-pending occurrences like lost queued jobs, and do not run
backups late when they are older than 24 hours.
- Run fixed-time schedules once when daylight saving time repeats an hour.
- Do not run a scheduled task again when Redis hands it out after a killed
worker, and do not overwrite a finished execution in failed().
When queue workers are down, an every-minute task or Docker cleanup
missed a run every minute, and each missed run sent a notification.
Each missed run still gets its own failed execution, but the team now
gets at most one missed-run notification per schedule each hour,
through NotificationThrottle.
- Skip the occurrence of a backup whose previous run still holds the
overlap lock, so stale recovery does not run it later; do not rerun a
lost occurrence when a newer one already completed.
- Skip a database backup while a start, restart, or import of the database
is in progress, and record the skip in the execution list.
- Give every backup file a unique name per run and keep the service
database backup folder of earlier releases.
- Record missed scheduled tasks and Docker cleanups as failed executions
and notify the team.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scheduler ran one dispatcher for database backups, tasks, volume backups,
and Docker cleanups after each other. A slow type, for example many tasks due
at the same minute, made the whole run longer, and the shared overlap lock
then made all types skip the next minute.
Run `scheduled:dispatch --type=<type>` every minute for backups, tasks,
volume-backups, and docker-cleanups. Each type runs in its own background
process with its own scheduler and job overlap locks, and publishes and
recovers only the deliveries of its own job type. Running the command without
a type still processes all types.
The missed daily backups on Coolify Cloud came from Redis evicting queue keys
(allkeys-lfu on a full instance), not from the scan of all schedules. The
next_run_at design (452259aac, 5c5740533) did not prevent that, and it added
stored state that must be kept in sync on every frequency, timezone, and server
change. It was not released, so remove it:
- revert next_run_at, its model hooks, the "Next run" UI, and the diagnostics
changes; restore scheduled_job_states and the queue overlap lock
- a new migration removes the next_run_at columns and creates
scheduled_job_states again where the removed migration already ran
Keep the changes that fix real problems:
- run the manager every minute from the scheduler with `scheduled:dispatch`,
so a busy queue cannot delay it past the catch-up window
- load the server chain of scheduled backups with each chunk, which removes
about 7,000 queries per run on Coolify Cloud
- catch Throwable for each schedule, so one broken schedule does not stop the
others
- publish database and volume backups again when their queued job was not
started after 60 minutes (for example, lost in Redis); log lost tasks and
Docker cleanups as missed. A lost backup is not run again when a newer
occurrence of the same schedule exists.
A schedule with a frequency that no longer parses stayed due and logged
"invalid frequency" every minute. Its next run is now cleared (shown as
invalid frequency in the UI), the warning is logged at most once a day,
and fixing the frequency makes it run again. The enabled flag is not
changed.
A delivery stuck in "enqueued" because its queued job was lost was never
run or cleaned up. After 60 minutes, database and volume backups are
published again (late backups run once), and tasks and Docker cleanups
are marked missed with a log entry. The claim token makes a duplicate
job a no-op.
The scheduled job manager checked every enabled schedule each minute and
evaluated its cron expression in PHP. On Coolify Cloud one run took more than
60 seconds, so runs overlapped and minutes were skipped.
Each schedule now stores its next due time:
- next_run_at on scheduled database backups, scheduled tasks, and volume
backups, and docker_cleanup_next_run_at on server settings (UTC, indexed)
- model hooks calculate it when a schedule is created or its frequency or
enabled flag changes; a server timezone change or a move to another server
calculates it again for the affected schedules
The `scheduled:dispatch` command runs the manager every minute from the
scheduler, not from a queue. It selects only due rows and claims each
occurrence with an atomic update of next_run_at, together with its delivery
record, so parallel runs on several nodes never dispatch it twice.
- database and volume backups run once when they are late; tasks and Docker
cleanups more than 10 minutes late are logged as missed
- a local time that occurs two times when the clocks go back runs once
- the migration continues each schedule after its last recorded occurrence and
drops scheduled_job_states
Show the next run, in the server timezone, on the database backup, scheduled
task, volume backup, and Docker cleanup pages, or "Invalid frequency" when the
frequency cannot be calculated. scheduled:diagnostics now lists overdue
schedules and open deliveries.
Remove workarounds that are no longer needed: the queue overlap lock and its
stale-lock repair, the catch-up window, and debug-only logs.
Horizon workers kept Server::findCached() results between jobs, so scheduled
database backups used stale reachability and subscription state. A backup that
was skipped with the stale state counted as done for that day. Flush the
Server identity map before each queue job.
Add debug logs for the full path of a scheduled run, so a multi-node cloud
instance shows where a run is lost:
- manager start delay, gap since the previous start, host, pid, memory, and
a warning when the start is later than the catch-up window
- skip logs with the server state that was used and the current database state
- occurrence enqueued, claimed, rejected, completed and failed, with host and
the delay since the due time; a warning when the previous run is still open
- count of occurrences still open after 15 minutes on each manager run
- database backup runs that end without a backup
- server connection state changes with the checking node and the SSH error
Also keep publishing pending occurrences when one of them fails.
Add database-backed schedule states and deliveries so distributed schedulers publish each occurrence once, queue jobs claim executions atomically, and stale occurrences are cleaned up.
Start backup and task pagination without an id lower bound so a legacy coolify-db backup or scheduled task at id 0 is dispatched with later positive ids. Document instance sentinel ids so they are not treated as ordinary autoincrement rows.
Andras BacsaiandClaude Opus 5.5