Check enabled volume backup schedules hourly and notify teams when no
execution occurs within the configured number of days. Expose the alert
threshold in the UI and API, with alerts disabled by default.
Route volume backup results and warnings to the appropriate Telegram
backup threads.
The missed daily backups on Coolify Cloud came from Redis evicting queue keys
(allkeys-lfu on a full instance), not from the scan of all schedules. The
next_run_at design (452259aac, 5c5740533) did not prevent that, and it added
stored state that must be kept in sync on every frequency, timezone, and server
change. It was not released, so remove it:
- revert next_run_at, its model hooks, the "Next run" UI, and the diagnostics
changes; restore scheduled_job_states and the queue overlap lock
- a new migration removes the next_run_at columns and creates
scheduled_job_states again where the removed migration already ran
Keep the changes that fix real problems:
- run the manager every minute from the scheduler with `scheduled:dispatch`,
so a busy queue cannot delay it past the catch-up window
- load the server chain of scheduled backups with each chunk, which removes
about 7,000 queries per run on Coolify Cloud
- catch Throwable for each schedule, so one broken schedule does not stop the
others
- publish database and volume backups again when their queued job was not
started after 60 minutes (for example, lost in Redis); log lost tasks and
Docker cleanups as missed. A lost backup is not run again when a newer
occurrence of the same schedule exists.
The scheduled job manager checked every enabled schedule each minute and
evaluated its cron expression in PHP. On Coolify Cloud one run took more than
60 seconds, so runs overlapped and minutes were skipped.
Each schedule now stores its next due time:
- next_run_at on scheduled database backups, scheduled tasks, and volume
backups, and docker_cleanup_next_run_at on server settings (UTC, indexed)
- model hooks calculate it when a schedule is created or its frequency or
enabled flag changes; a server timezone change or a move to another server
calculates it again for the affected schedules
The `scheduled:dispatch` command runs the manager every minute from the
scheduler, not from a queue. It selects only due rows and claims each
occurrence with an atomic update of next_run_at, together with its delivery
record, so parallel runs on several nodes never dispatch it twice.
- database and volume backups run once when they are late; tasks and Docker
cleanups more than 10 minutes late are logged as missed
- a local time that occurs two times when the clocks go back runs once
- the migration continues each schedule after its last recorded occurrence and
drops scheduled_job_states
Show the next run, in the server timezone, on the database backup, scheduled
task, volume backup, and Docker cleanup pages, or "Invalid frequency" when the
frequency cannot be calculated. scheduled:diagnostics now lists overdue
schedules and open deliveries.
Remove workarounds that are no longer needed: the queue overlap lock and its
stale-lock repair, the catch-up window, and debug-only logs.
Add polymorphic volume backup scheduling for persistent volumes and directories, expose schedule management via API, and reorganize backup configuration and execution views.
Andras Bacsai