The missed daily backups on Coolify Cloud came from Redis evicting queue keys
(allkeys-lfu on a full instance), not from the scan of all schedules. The
next_run_at design (452259aac, 5c5740533) did not prevent that, and it added
stored state that must be kept in sync on every frequency, timezone, and server
change. It was not released, so remove it:
- revert next_run_at, its model hooks, the "Next run" UI, and the diagnostics
changes; restore scheduled_job_states and the queue overlap lock
- a new migration removes the next_run_at columns and creates
scheduled_job_states again where the removed migration already ran
Keep the changes that fix real problems:
- run the manager every minute from the scheduler with `scheduled:dispatch`,
so a busy queue cannot delay it past the catch-up window
- load the server chain of scheduled backups with each chunk, which removes
about 7,000 queries per run on Coolify Cloud
- catch Throwable for each schedule, so one broken schedule does not stop the
others
- publish database and volume backups again when their queued job was not
started after 60 minutes (for example, lost in Redis); log lost tasks and
Docker cleanups as missed. A lost backup is not run again when a newer
occurrence of the same schedule exists.
High disk usage notifications were sent every ~10 minutes on Coolify Cloud
because the cache rate limiter did not hold across Sentinel-triggered checks.
- Add a notification_throttles table and NotificationThrottle helper that
atomically records when a notification was sent for a resource.
- Add the ThrottledNotification contract; Team::notify() skips a throttled
notification that was already sent within its interval.
- High disk usage: configurable per-server interval
(server_disk_usage_notification_interval_hours, default 24h) in
Server > Advanced and the servers API.
- Container restarted (coolify-proxy, database TCP proxy): at most one
notification per hour per server/database during a restart loop.
- Move the unreachable and missing-backup markers into the new table and
migrate existing data (servers.unreachable_notification_sent and
scheduled_database_backups.missing_backup_notification_sent_at).
- Regenerate OpenAPI files.
The scheduled job manager checked every enabled schedule each minute and
evaluated its cron expression in PHP. On Coolify Cloud one run took more than
60 seconds, so runs overlapped and minutes were skipped.
Each schedule now stores its next due time:
- next_run_at on scheduled database backups, scheduled tasks, and volume
backups, and docker_cleanup_next_run_at on server settings (UTC, indexed)
- model hooks calculate it when a schedule is created or its frequency or
enabled flag changes; a server timezone change or a move to another server
calculates it again for the affected schedules
The `scheduled:dispatch` command runs the manager every minute from the
scheduler, not from a queue. It selects only due rows and claims each
occurrence with an atomic update of next_run_at, together with its delivery
record, so parallel runs on several nodes never dispatch it twice.
- database and volume backups run once when they are late; tasks and Docker
cleanups more than 10 minutes late are logged as missed
- a local time that occurs two times when the clocks go back runs once
- the migration continues each schedule after its last recorded occurrence and
drops scheduled_job_states
Show the next run, in the server timezone, on the database backup, scheduled
task, volume backup, and Docker cleanup pages, or "Invalid frequency" when the
frequency cannot be calculated. scheduled:diagnostics now lists overdue
schedules and open deliveries.
Remove workarounds that are no longer needed: the queue overlap lock and its
stale-lock repair, the catch-up window, and debug-only logs.
Add `WithoutOverlapping` middleware to `DatabaseBackupJob` keyed by backup ID
with timeout-based lock expiry to prevent concurrent runs.
Mark long-running backup executions as failed when they exceed the stale
time threshold, and add periodic retention enforcement in
`CleanupInstanceStuffsJob` with cache-based throttling.
Also add float casts for retention max-storage fields on
`ScheduledDatabaseBackup` and comprehensive feature tests covering
overlap middleware, stale detection, casts, and retention behavior.
Replace all uses of `forceFill`, `forceCreate`, and `forceFill` with their
non-force equivalents across models, actions, controllers, and Livewire
components. Add explicit `$fillable` arrays to all affected Eloquent models
to enforce mass assignment protection.
Add ModelFillableCreationTest and ModelFillableRegressionTest to verify that
model creation respects fillable constraints and prevent regressions.
Replace $guarded usage with explicit $fillable arrays across all models.
Sync fillable definitions with current database schema and add tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Modified backup retrieval logic in DatabasesController to utilize the new ownedByCurrentTeamAPI method for improved access control based on team ID.
- Enhanced code consistency and maintainability by centralizing team-based filtering in the ScheduledDatabaseBackup model.
- Updated backup retrieval logic in DatabasesController to utilize the new ownedByCurrentTeam method for improved access control.
- Enhanced code readability and maintainability by centralizing team-based filtering in the ScheduledDatabaseBackup model.
- Modified backup configuration queries in the DatabasesController to filter by team ID, ensuring proper access control.
- Enhanced S3 storage retrieval to use the current team context for better data integrity.
- Added a relationship method in ScheduledDatabaseBackup model to associate backups with teams.
Andras BacsaiandClaude Opus 5.5
Andras Bacsai
Vahor
ayntk-ai
Thijmen