- Retry CDN requests only for transient errors and honor Retry-After
(Cloudflare error 1200 sends 60 s), with jitter.
- Read helper and Sentinel versions from the local versions.json, which
the hourly update check refreshes, instead of fetching per job/server.
- Replace the random 0-600 s delay with a stable per-instance offset
inside the cron interval (max 1 h).
- Manual update checks and upgrades make one CDN attempt only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All instances share the same default update cron, so they request the
CDN at the same second and Cloudflare rejects the burst (error 1200).
Queue the CDN fetching jobs with a random 0-600s delay to spread the load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Check enabled volume backup schedules hourly and notify teams when no
execution occurs within the configured number of days. Expose the alert
threshold in the UI and API, with alerts disabled by default.
Route volume backup results and warnings to the appropriate Telegram
backup threads.
Default to sequential dispatch on self-hosted instances and concurrent
per-type dispatch on Coolify Cloud. Allow SCHEDULED_JOBS_DISPATCH_MODE
to override the default, falling back for invalid values.
Cover defaults, overrides, and fallback behavior in scheduler tests.
The per-server schedules reused the cache keys of the old schedule, so
every server ran these checks at once on the first run after the
upgrade. New keys start fresh, so each server runs at its own slot.
/settings/backup returned 500 when the coolify-db resource existed without
a scheduled backup. Configure backup now reuses the existing coolify-db and
creates only the missing schedule for the root team, without forcing id 0.
Closes#12074
Servers no longer all connect at minute 0. The hourly Sentinel check uses
a per-server minute, and the weekly patch check uses a per-server time on
Sunday between 04:00 and 23:59.
Add an hourly cleanup:database-import-uploads command that removes staged
import uploads older than a day, while holding the import lock of the
database so a running import keeps its file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a --purge option that purges the synced CDN files (and the legacy
/coolify-nightly/ paths for nightly) from the Cloudflare cache, and remind
the operator to run it after the CDN pull request is deployed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CDN files are synced through the coollabs-cdn GitHub repository only.
Remove the --bunny direct upload path, the BUNNY_API_KEY and
BUNNY_STORAGE_API_KEY settings, and rename the command to sync:cdn.
The scheduler ran one dispatcher for database backups, tasks, volume backups,
and Docker cleanups after each other. A slow type, for example many tasks due
at the same minute, made the whole run longer, and the shared overlap lock
then made all types skip the next minute.
Run `scheduled:dispatch --type=<type>` every minute for backups, tasks,
volume-backups, and docker-cleanups. Each type runs in its own background
process with its own scheduler and job overlap locks, and publishes and
recovers only the deliveries of its own job type. Running the command without
a type still processes all types.
The command called shouldRunCronNow() with the dedup keys, and that helper
writes the key it checks. For server-check, server-storage-check, and
server-patch-check these are the keys of ServerManagerJob, so running the
command in a minute when a check was due made the real check skip that run.
It also read cache keys for backups, tasks, and Docker cleanups that the
scheduled job manager no longer uses. Read scheduled_job_states instead, and
list open and failed deliveries, where an old "enqueued" row shows a queued
job that was lost.
The missed daily backups on Coolify Cloud came from Redis evicting queue keys
(allkeys-lfu on a full instance), not from the scan of all schedules. The
next_run_at design (452259aac, 5c5740533) did not prevent that, and it added
stored state that must be kept in sync on every frequency, timezone, and server
change. It was not released, so remove it:
- revert next_run_at, its model hooks, the "Next run" UI, and the diagnostics
changes; restore scheduled_job_states and the queue overlap lock
- a new migration removes the next_run_at columns and creates
scheduled_job_states again where the removed migration already ran
Keep the changes that fix real problems:
- run the manager every minute from the scheduler with `scheduled:dispatch`,
so a busy queue cannot delay it past the catch-up window
- load the server chain of scheduled backups with each chunk, which removes
about 7,000 queries per run on Coolify Cloud
- catch Throwable for each schedule, so one broken schedule does not stop the
others
- publish database and volume backups again when their queued job was not
started after 60 minutes (for example, lost in Redis); log lost tasks and
Docker cleanups as missed. A lost backup is not run again when a newer
occurrence of the same schedule exists.
GitLab, Authentik, Zitadel, Clerk and Azure identities were keyed by the
provider name only, so a user with the same subject on another instance
could sign in as the linked user after the base URL or tenant changed.
The issuer column now stores the normalized instance, existing rows are
backfilled from the current settings, and oauth:rekey handles host renames.
High disk usage notifications were sent every ~10 minutes on Coolify Cloud
because the cache rate limiter did not hold across Sentinel-triggered checks.
- Add a notification_throttles table and NotificationThrottle helper that
atomically records when a notification was sent for a resource.
- Add the ThrottledNotification contract; Team::notify() skips a throttled
notification that was already sent within its interval.
- High disk usage: configurable per-server interval
(server_disk_usage_notification_interval_hours, default 24h) in
Server > Advanced and the servers API.
- Container restarted (coolify-proxy, database TCP proxy): at most one
notification per hour per server/database during a restart loop.
- Move the unreachable and missing-backup markers into the new table and
migrate existing data (servers.unreachable_notification_sent and
scheduled_database_backups.missing_backup_notification_sent_at).
- Regenerate OpenAPI files.
Import now always claims the server and rebinds Sentinel. Remove the
SSH-written ownership file, the write_remote/rebind_sentinel/claim
options, and the POST /servers/{uuid}/claim endpoint. The migrator now
fails when the target cannot take management. Disabling management
records transfer.management_disabled so the server shows as
transferable, and the transfer and import pages are simplified.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-test S3 storages marked unusable every hour so a storage that becomes
reachable again is usable without manual action. Failures keep the
storage unusable and reuse the existing once-per-outage email.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scheduled job manager checked every enabled schedule each minute and
evaluated its cron expression in PHP. On Coolify Cloud one run took more than
60 seconds, so runs overlapped and minutes were skipped.
Each schedule now stores its next due time:
- next_run_at on scheduled database backups, scheduled tasks, and volume
backups, and docker_cleanup_next_run_at on server settings (UTC, indexed)
- model hooks calculate it when a schedule is created or its frequency or
enabled flag changes; a server timezone change or a move to another server
calculates it again for the affected schedules
The `scheduled:dispatch` command runs the manager every minute from the
scheduler, not from a queue. It selects only due rows and claims each
occurrence with an atomic update of next_run_at, together with its delivery
record, so parallel runs on several nodes never dispatch it twice.
- database and volume backups run once when they are late; tasks and Docker
cleanups more than 10 minutes late are logged as missed
- a local time that occurs two times when the clocks go back runs once
- the migration continues each schedule after its last recorded occurrence and
drops scheduled_job_states
Show the next run, in the server timezone, on the database backup, scheduled
task, volume backup, and Docker cleanup pages, or "Invalid frequency" when the
frequency cannot be calculated. scheduled:diagnostics now lists overdue
schedules and open deliveries.
Remove workarounds that are no longer needed: the queue overlap lock and its
stale-lock repair, the catch-up window, and debug-only logs.
cleanup:database used orderBy()->skip(10)->delete(). Laravel drops the offset
in a DELETE, so it deleted all old rows anyway, and the count query with the
offset printed 0 on PostgreSQL. Count without the offset, delete in batches of
1000 rows so a large table is not locked by one long DELETE, and write the
output through the command so it is captured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a Lima backend to scripts/dev. Auto mode picks it on macOS when
limactl (Lima >= 2.0) is installed, and COOLIFY_DEV_SERVER_BACKEND=lima
forces it. Lima VMs are named coolify-dev-<slot>-<profile>. Coolify
connects to them through Lima's forwarded SSH port on
host.docker.internal, and only guest ports 80/443 are forwarded to
the host.
dev:qemu:seed now takes --ip and --port options, so the seeded server
can use a forwarded SSH address instead of the profile IP.
Add tests for the Lima start flow and for seeding with an overridden
address, and document the macOS setup in AGENTS.md.
An hourly command releases managed DNS records whose references all point
to missing or deleted resources, or that have no references after a grace
period. Each record goes through the per-hostname lock and re-check, so a
hostname that a user adds back is kept. An empty reference list no longer
deletes all references of a record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A Coolify restart now fails queued or running database imports, so
they no longer block all later imports with 409.
- An import without progress for the SSH command timeout plus 30
minutes (at least 2 hours) is stale and no longer blocks.
- The import sets its operation property when the activity is created,
so a worker cannot save the activity without it.
- Start and restart refuse a second operation while a start, restart,
or import is in progress, in the UI, the API (409), and actions.
- DatabaseStartJob runs only while its activity is still queued and
the newest one, and holds a lock per database during the commands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Proxy: list and delete Traefik ACME certificates from the server proxy
page via new TraefikAcmeService and Get/DeleteTraefikCertificate actions
- DNS: track ownership and cross-resource references for managed DNS
records so records are only deleted when no longer referenced; release
records asynchronously on resource deletion via ReleaseManagedDnsRecordsJob
and ManagedDnsRecordCleanup; harden Cloudflare provider deletion results
- Databases: fail closed on start when prerequisites or the CA certificate
are missing (DatabaseStartException, Server::ensureCaCertificate) and
clean up stale start activities via ResourceStartActivity
- Webhooks: throttle repeated manual webhook signature failures for
GitHub, GitLab, Gitea and Bitbucket
- Deployments: improve compose build-context handling and compose file
load error reporting
- Install scripts: rework terminal UI output in install.sh (stable and
nightly)
- Misc: settings sidebar accordion fixes, log drain toggle rollback,
add Serverside to README sponsors
- Add migrations and tests covering the above
Merge scripts/dev-instances into scripts/dev. Each branch gets its own
Compose project, container, volumes, port block, libvirt network, and
KVM VMs, so stop and start reuse the same data and worktrees do not
collide.
- Name instances after the branch; keep slots and APP_KEYs in the main
checkout's .dev-instances/ so they survive worktree removal
- Reuse existing QEMU VMs instead of recreating them; add dev:qemu --fresh
- Give each instance an isolated libvirt network (10.221.<slot>.0/24)
and VM names coolify-dev-<branch>--<profile>
- Publish per-instance browser ports and publish them with tailscale
serve when APP_URL is a tailnet host
- Add urls, exec, logs, container, destroy, and teardown commands; run
teardown from jean.json before Jean deletes a worktree
- Fix Reverb/terminal browser ports and the testing-host alias in the
instance compose file
- Keep fresh worktree instances working: pre-create laravel.log, log the
host dev:qemu call to stderr, and stop Vite from watching vendor/
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Andras BacsaiandClaude Opus 5.5
🏔️ Peak
