The missed daily backups on Coolify Cloud came from Redis evicting queue keys
(allkeys-lfu on a full instance), not from the scan of all schedules. The
next_run_at design (452259aac, 5c5740533) did not prevent that, and it added
stored state that must be kept in sync on every frequency, timezone, and server
change. It was not released, so remove it:
- revert next_run_at, its model hooks, the "Next run" UI, and the diagnostics
changes; restore scheduled_job_states and the queue overlap lock
- a new migration removes the next_run_at columns and creates
scheduled_job_states again where the removed migration already ran
Keep the changes that fix real problems:
- run the manager every minute from the scheduler with `scheduled:dispatch`,
so a busy queue cannot delay it past the catch-up window
- load the server chain of scheduled backups with each chunk, which removes
about 7,000 queries per run on Coolify Cloud
- catch Throwable for each schedule, so one broken schedule does not stop the
others
- publish database and volume backups again when their queued job was not
started after 60 minutes (for example, lost in Redis); log lost tasks and
Docker cleanups as missed. A lost backup is not run again when a newer
occurrence of the same schedule exists.
The scheduled job manager checked every enabled schedule each minute and
evaluated its cron expression in PHP. On Coolify Cloud one run took more than
60 seconds, so runs overlapped and minutes were skipped.
Each schedule now stores its next due time:
- next_run_at on scheduled database backups, scheduled tasks, and volume
backups, and docker_cleanup_next_run_at on server settings (UTC, indexed)
- model hooks calculate it when a schedule is created or its frequency or
enabled flag changes; a server timezone change or a move to another server
calculates it again for the affected schedules
The `scheduled:dispatch` command runs the manager every minute from the
scheduler, not from a queue. It selects only due rows and claims each
occurrence with an atomic update of next_run_at, together with its delivery
record, so parallel runs on several nodes never dispatch it twice.
- database and volume backups run once when they are late; tasks and Docker
cleanups more than 10 minutes late are logged as missed
- a local time that occurs two times when the clocks go back runs once
- the migration continues each schedule after its last recorded occurrence and
drops scheduled_job_states
Show the next run, in the server timezone, on the database backup, scheduled
task, volume backup, and Docker cleanup pages, or "Invalid frequency" when the
frequency cannot be calculated. scheduled:diagnostics now lists overdue
schedules and open deliveries.
Remove workarounds that are no longer needed: the queue overlap lock and its
stale-lock repair, the catch-up window, and debug-only logs.
Replace nested null checks with nullsafe operator chains, add ?Server
return type, drop unreachable database branch, and cover all paths with
feature tests.
Replace all uses of `forceFill`, `forceCreate`, and `forceFill` with their
non-force equivalents across models, actions, controllers, and Livewire
components. Add explicit `$fillable` arrays to all affected Eloquent models
to enforce mass assignment protection.
Add ModelFillableCreationTest and ModelFillableRegressionTest to verify that
model creation respects fillable constraints and prevent regressions.
Replace $guarded usage with explicit $fillable arrays across all models.
Sync fillable definitions with current database schema and add tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enable `HasFactory` on `Environment`, `Project`, `ScheduledTask`, and
`StandaloneDocker`, and add dedicated factories for related models to
stabilize feature/unit tests.
Also bump `visus/cuid2` to `^6.0` and refresh `composer.lock` with the
resulting dependency updates.
- Add authorization checks ($this->authorize) for all read/write operations
- Use customApiValidator() instead of Validator::make() to match codebase patterns
- Add extra field rejection to prevent mass assignment
- Use Application::ownedByCurrentTeamAPI() for consistent query patterns
- Remove non-existent standalone_postgresql_id from hidden fields
- Add execution listing endpoints for both applications and services
- Add ScheduledTaskExecution OpenAPI schema
- Use $request->only() instead of $request->all() for safe updates
- Add ScheduledTaskFactory and feature tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add retry configuration to CoolifyTask (3 tries, 600s timeout)
- Add retry configuration to ScheduledTaskJob (3 tries, configurable timeout)
- Add retry configuration to DatabaseBackupJob (2 tries)
- Implement exponential backoff for all jobs (30s, 60s, 120s intervals)
- Add failed() handlers with comprehensive error logging to scheduled-errors channel
- Add execution tracking: started_at, retry_count, duration (decimal), error_details
- Add configurable timeout field to scheduled tasks (60-3600s, default 300s)
- Update UI to include timeout configuration in task creation/editing forms
- Increase ScheduledJobManager lock expiration from 60s to 90s for high-load environments
- Implement safe queue cleanup with restart vs runtime modes
- Restart mode: aggressive cleanup (marks all processing jobs as failed)
- Runtime mode: conservative cleanup (only marks jobs >12h as failed, skips deployments)
- Add cleanup:redis --restart flag for system startup
- Integrate cleanup into Dev.php init() for development environment
- Increase scheduled-errors log retention from 7 to 14 days
- Create comprehensive test suite (unit and feature tests)
- Add TESTING_GUIDE.md with manual testing instructions
Fixes issues with jobs failing after single attempt and "attempted too many times" errors
- Introduced `ValidationPatterns` class to standardize validation rules and messages for name and description fields across the application.
- Updated various components and models to utilize the new validation patterns, ensuring consistent sanitization and validation logic.
- Replaced the `HasSafeNameAttribute` trait with `HasSafeStringAttribute` to enhance attribute handling and maintain consistency in name sanitization.
- Enhanced the `CleanupNames` command to align with the new validation rules, allowing for a broader range of valid characters in names.
- Introduced `CleanupNames` command to sanitize name fields by removing invalid characters, ensuring only letters, numbers, spaces, dashes, underscores, and dots are retained.
- Implemented options for dry run, model-specific cleaning, database backup, and forced execution.
- Updated `Init` command to call the new `cleanup:names` command.
- Enhanced project and environment validation to enforce name sanitization rules.
- Added `HasSafeNameAttribute` trait to relevant models for consistent name handling.
Andras Bacsai
Ahmed
Lucas Michot
Andras Bacsai
Vahor
Valentin Schabschneider
ayntk-ai
Thijmen
Stuart Rowlands