230 Commits
Author SHA1 Message Date
Andras Bacsai 407c384b09 feat(backups): add configurable missing volume backup alerts
Check enabled volume backup schedules hourly and notify teams when no
execution occurs within the configured number of days. Expose the alert
threshold in the UI and API, with alerts disabled by default.

Route volume backup results and warnings to the appropriate Telegram
backup threads.
2026-10-07 09:37:35 +02:00
Andras Bacsai d54c03730d feat(scheduler): make scheduled job dispatch mode configurable
Default to sequential dispatch on self-hosted instances and concurrent
per-type dispatch on Coolify Cloud. Allow SCHEDULED_JOBS_DISPATCH_MODE
to override the default, falling back for invalid values.

Cover defaults, overrides, and fallback behavior in scheduler tests.
2026-10-06 23:16:40 +02:00
Andras BacsaiandClaude Opus 5.5 476ba3c6ab fix(database-import): delete staged API uploads that were never imported
Add an hourly cleanup:database-import-uploads command that removes staged
import uploads older than a day, while holding the import lock of the
database so a running import keeps its file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 16:00:29 +02:00
Andras Bacsai f575b2d179 feat(scheduler): run one scheduled job dispatcher for each schedule type
The scheduler ran one dispatcher for database backups, tasks, volume backups,
and Docker cleanups after each other. A slow type, for example many tasks due
at the same minute, made the whole run longer, and the shared overlap lock
then made all types skip the next minute.

Run `scheduled:dispatch --type=<type>` every minute for backups, tasks,
volume-backups, and docker-cleanups. Each type runs in its own background
process with its own scheduler and job overlap locks, and publishes and
recovers only the deliveries of its own job type. Running the command without
a type still processes all types.
2026-10-02 09:19:54 +02:00
Andras Bacsai 50be16be8f revert(scheduler): go back to the cron scan and keep the reliability fixes
The missed daily backups on Coolify Cloud came from Redis evicting queue keys
(allkeys-lfu on a full instance), not from the scan of all schedules. The
next_run_at design (452259aac, 5c5740533) did not prevent that, and it added
stored state that must be kept in sync on every frequency, timezone, and server
change. It was not released, so remove it:
- revert next_run_at, its model hooks, the "Next run" UI, and the diagnostics
  changes; restore scheduled_job_states and the queue overlap lock
- a new migration removes the next_run_at columns and creates
  scheduled_job_states again where the removed migration already ran

Keep the changes that fix real problems:
- run the manager every minute from the scheduler with `scheduled:dispatch`,
  so a busy queue cannot delay it past the catch-up window
- load the server chain of scheduled backups with each chunk, which removes
  about 7,000 queries per run on Coolify Cloud
- catch Throwable for each schedule, so one broken schedule does not stop the
  others
- publish database and volume backups again when their queued job was not
  started after 60 minutes (for example, lost in Redis); log lost tasks and
  Docker cleanups as missed. A lost backup is not run again when a newer
  occurrence of the same schedule exists.
2026-10-02 08:05:18 +02:00
Andras BacsaiandClaude Opus 5.5 fd85b740f5 feat(github-runners): add ephemeral self-hosted runners on build servers (#12054)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 12:13:11 +02:00
Andras BacsaiandClaude Opus 5.5 639a07e4cb feat(s3): revalidate unusable S3 storages hourly
Re-test S3 storages marked unusable every hour so a storage that becomes
reachable again is usable without manual action. Failures keep the
storage unusable and reuse the existing once-per-outage email.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 10:16:09 +02:00
Andras Bacsai 452259aacc feat(scheduler): dispatch scheduled jobs by their stored next run time
The scheduled job manager checked every enabled schedule each minute and
evaluated its cron expression in PHP. On Coolify Cloud one run took more than
60 seconds, so runs overlapped and minutes were skipped.

Each schedule now stores its next due time:
- next_run_at on scheduled database backups, scheduled tasks, and volume
  backups, and docker_cleanup_next_run_at on server settings (UTC, indexed)
- model hooks calculate it when a schedule is created or its frequency or
  enabled flag changes; a server timezone change or a move to another server
  calculates it again for the affected schedules

The `scheduled:dispatch` command runs the manager every minute from the
scheduler, not from a queue. It selects only due rows and claims each
occurrence with an atomic update of next_run_at, together with its delivery
record, so parallel runs on several nodes never dispatch it twice.
- database and volume backups run once when they are late; tasks and Docker
  cleanups more than 10 minutes late are logged as missed
- a local time that occurs two times when the clocks go back runs once
- the migration continues each schedule after its last recorded occurrence and
  drops scheduled_job_states

Show the next run, in the server timezone, on the database backup, scheduled
task, volume backup, and Docker cleanup pages, or "Invalid frequency" when the
frequency cannot be calculated. scheduled:diagnostics now lists overdue
schedules and open deliveries.

Remove workarounds that are no longer needed: the queue overlap lock and its
stale-lock repair, the catch-up window, and debug-only logs.
2026-09-29 10:07:23 +02:00
Andras BacsaiandClaude Opus 5.5 2c6e221db5 feat(dns): release orphaned managed DNS records on a schedule
An hourly command releases managed DNS records whose references all point
to missing or deleted resources, or that have no references after a grace
period. Each record goes through the per-hostname lock and re-check, so a
hostname that a user adds back is kept. An empty reference list no longer
deletes all references of a record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 21:09:13 +02:00
Andras Bacsai 19949a6f6f feat(scheduling): persist and claim scheduled job occurrences
Add database-backed schedule states and deliveries so distributed schedulers publish each occurrence once, queue jobs claim executions atomically, and stale occurrences are cleaned up.
2026-09-18 13:56:40 +02:00
Andras Bacsai 7d11bc92f7 feat(logs): accept all for log lines and keep -1
API log endpoints and the GetLogs UI now treat `all` as unbounded
output, with `-1` remaining as a compatibility alias. MCP still
falls back to a positive default. Traefik version checks run from
CheckForUpdatesJob instead of a weekly schedule.
2026-09-13 15:25:20 +02:00
Andras Bacsai 0b462f6a58 feat(backups): alert when scheduled backups are missing (#11433) 2026-09-07 14:09:42 +02:00
Andras Bacsai 1d0d6b4a79 fix(scheduler): run stuck resource cleanup in background at 03:17
Bump the Coolify release version to 4.3.16 and update schedule coverage.
2026-09-03 19:23:08 +02:00
Andras Bacsai 45f9a33985 fix(resources): schedule stuck cleanup and delete metadata before jobs
Soft-delete API resources before dispatching cleanup jobs and run stuck-resource cleanup daily on one server.
2026-08-25 20:25:40 +02:00
Andras Bacsai 76030a30d4 chore(v5): archive V5 implementation and remove runtime integration
Move V5 source, migrations, UI, scripts, and tests into documentation, then remove V5 routes, models, jobs, configuration, dependencies, and application hooks.
2026-08-15 19:13:30 +02:00
Andras Bacsai d0247d3b09 feat(v5): gate V5 to development environments only
Introduce V5Feature and config so V5 routes, jobs, commands, morph maps,
and model queries run only when enabled. Move V5 migrations to
migrations-v5 (loaded only when enabled), remove Flux from production
Docker/install paths, and add isolation tests.
2026-07-19 11:44:12 +02:00
Andras Bacsai fbc6fd5ce5 fix(v5): rotate agent tokens every fifteen minutes 2026-07-06 22:35:17 +02:00
Andras Bacsai 6ae45684f9 feat(v5): add server reconciliation and canvas APIs
Split V5 dashboard behavior into domain controllers and policies,
add agent token rotation/revocation, status reconciliation jobs,
ingress firewall syncing, and canvas connection APIs.

Add migrations for V5 status tracking, server capabilities, resource
connection aliases, and revoked agent tokens.
2026-07-06 17:40:37 +02:00
Andras Bacsai 40294bc3b3 chore: inspect staged changes 2026-06-02 14:05:26 +02:00
Andras Bacsai 43884823c6 chore(ssh): remove stale mux cleanup job
Drop the scheduled stale multiplexed connection cleanup job, its SSH mux
health/orphan config, and the tests that covered that cleanup path.
2026-05-26 14:40:38 +02:00
Andras Bacsai ed3780b2a7 fix(schedule): run stale multiplex cleanup on crons queue
Dispatch CleanupStaleMultiplexedConnections through the crons queue and
cover the scheduled job queue assignment with a feature test.
2026-05-26 13:51:22 +02:00
Andras Bacsai 57d879263d fix(ssh): prevent orphaned multiplexed connections
Serialize multiplexed SSH master creation per server to avoid concurrent workers spawning orphaned processes. Enable scheduled cleanup for stale mux connections and add guarded orphan process reaping with tests.
2026-05-22 17:31:38 +02:00
Andras Bacsai b124397613 fix(schedule): prevent duplicate SSL certificate regeneration
Run RegenerateSslCertJob on one server only and add coverage to ensure scheduled production jobs use onOneServer.
2026-05-21 19:19:43 +02:00
Andras BacsaiandClaude Opus 4.7 90ddbb3572 feat(security): support expiration on API tokens with warning notifications
Add optional expiration to personal API tokens. Users pick a duration
(1/7/30/60/90 days or Never) at creation time. Expired tokens are
rejected by Sanctum, pruned hourly by sanctum:prune-expired, and a
team notification fires ~24h before expiry so owners can rotate
before API calls start failing.

- ApiTokens Livewire component stores expires_at from expiresInDays
- Rework issued-tokens UI from card grid to table (matches other views)
- New ApiTokenExpirationWarningJob scheduled hourly (idempotent via RateLimiter)
- New ApiTokenExpiringNotification (email/discord/telegram/slack/pushover)
- api_token_expiring added to alwaysSendEvents so users cannot silence
  expiry warnings from the per-event notification toggle UI
- sanctum:prune-expired cadence moved from daily to hourly

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-20 14:28:38 +02:00
Andras Bacsai b88f9fca67 chore: prepare for PR 2026-02-25 12:07:29 +01:00
Andras BacsaiandClaude 3962f1a5b9 perf: Remove dead server filtering code from Kernel scheduler
Remove unused server filtering logic in Kernel.php that was querying servers
but never using the results. Simplify Sentinel update checks in ServerManagerJob
by reusing the $isSentinelEnabled variable and removing unnecessary timezone
parameter for hourly cron execution.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-11 10:33:55 +01:00
Andras BacsaiandClaude 945cce9587 feat: Add scheduled job to cleanup orphaned PR containers
Add CleanupOrphanedPreviewContainersJob that runs daily to find and remove any PR preview containers that weren't properly cleaned up when their PR was closed.

The job:
- Scans all functional servers for containers with coolify.pullRequestId label
- Checks if the corresponding ApplicationPreview record exists in the database
- Removes containers where the preview record no longer exists (truly orphaned)
- Acts as a safety net for webhook failures or race conditions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-08 17:15:52 +01:00
Andras Bacsai 4002044877 Refactor: Move sentinel update checks to ServerManagerJob and add tests for hourly dispatch 2025-12-04 14:58:18 +01:00
Andras Bacsai 8c77c63043 feat(proxy): add Traefik version tracking with notifications and dismissible UI warnings
- Add automated Traefik version checking job running weekly on Sundays
- Implement version detection from running containers and comparison with versions.json
- Add notifications across all channels (Email, Discord, Slack, Telegram, Pushover, Webhook) for outdated versions
- Create dismissible callout component with localStorage persistence
- Display cross-branch upgrade warnings (e.g., v3.5 -> v3.6) with changelog links
- Show patch update notifications within same branch
- Add warning icon that appears when callouts are dismissed
- Prevent duplicate notifications during proxy restart by adding restarting parameter
- Fix notification spam with transition-based logic for status changes
- Enable system email settings by default in development mode
- Track last saved/applied proxy settings to detect configuration drift
2025-11-14 11:35:22 +01:00
Andras Bacsai d2ed68e605 refactor(jobs): pull github changelogs from cdn instead of github 2025-09-01 16:13:55 +02:00
Andras Bacsai 7d10711a65 fix(servercheck): Properly check server statuses with and without Sentinel 2025-08-22 11:50:56 +02:00
Andras Bacsai a2ef545b6b feat(changelog): implement automated changelog fetching from GitHub and enhance changelog read tracking 2025-08-12 10:07:11 +02:00
Andras Bacsai fbe98cfd11 refactor(scheduling): replace deprecated job checks with ScheduledJobManager and ServerResourceManager for improved scheduling efficiency 2025-07-12 14:44:52 +02:00
Andras Bacsai 36fe235bea feat(logging): implement scheduled logs command and enhance backup/task scheduling with cron checks 2025-07-11 15:10:55 +02:00
Andras Bacsai ecbfaa9a8a fix(scheduling): change redis cleanup command frequency from hourly to weekly for better resource management 2025-07-08 10:57:09 +02:00
Andras Bacsai 6ea6d2742b feat(server): implement server patch check notifications
- Added a new job, ServerPatchCheckJob, to handle server patch checks and notifications.
- Introduced a new notification class, ServerPatchCheck, for sending updates via email, Discord, Slack, Pushover, and Telegram.
- Updated notification settings models to include server patch notification options for email, Discord, Slack, Pushover, and Telegram.
- Created a migration to add server patch notification fields to the respective settings tables.
- Enhanced the UI to allow users to enable/disable server patch notifications across different channels.
2025-05-26 14:03:59 +02:00
Péter Gyarmati 9939b10e5a feat(service): add Marimo service (#5559) 2025-05-14 14:43:39 +02:00
Andras Bacsai 9d56d06352 feat(redis): add scheduled cleanup command for Redis keys and enhance cleanup logic 2025-05-07 22:09:25 +02:00
🏔️ Peak 9557957f18 Merge branch 'next' into feat-db-ssl 2025-02-19 17:57:59 +01:00
Andras Bacsai 694b0ac249 feat(core): add error logging and cron parsing to docker/server schedules 2025-02-15 13:21:23 +01:00
peaklabs-dev 852be5fd93 feat(ssl): check for SSL renewal twice daily 2025-02-05 22:11:10 +01:00
Andras Bacsai 0e94a4a472 fix: add own servers on cloud 2025-01-13 09:17:54 +01:00
Andras Bacsai a396d51452 Merge branch 'next' into hotfix 2025-01-10 20:23:29 +01:00
Andras Bacsai 5c8b26d446 fix 2025-01-10 12:31:20 +01:00
Andras Bacsai b09f0043d1 fix: restrict jobs on cloud
fix: restrict sentinel endpoint
2025-01-10 11:54:45 +01:00
Andras Bacsai 1fe4dd722b Revert "rector: arrrrr"
This reverts commit 16c0cd10d8.
2025-01-07 15:31:43 +01:00
Andras Bacsai da07b4fdcf refactor: rename parameter in DatabaseBackupJob for clarity 2025-01-07 15:21:14 +01:00
Andras Bacsai 16c0cd10d8 rector: arrrrr 2025-01-07 14:52:08 +01:00
Andras Bacsai 4d18e3372d fix: scheduled tasks and backups are executed by server timezone. 2025-01-03 20:32:56 +01:00
peaklabs-dev f2d0f46248 fix: always use docker cleanup frequency 2024-12-17 15:20:06 +01:00