27 Commits
Author SHA1 Message Date
Andras BacsaiandClaude Opus 5.5 9a6cc8316e fix(horizon): route Sentinel push jobs to a dedicated cloud sentinel queue
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:49:56 +01:00
Andras BacsaiandClaude Opus 5.5 63153aa8b4 fix(cloud): rework unsubscribed team cleanup and account deletion (#12210)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 17:32:16 +02:00
Andras Bacsai b081673c98 fix(server): skip Sentinel checks on unreachable servers
- CheckAndStartSentinelJob returns early when the server is not
  functional. It now reads the container state with `docker ps --filter`,
  so an SSH failure throws and is no longer treated as a missing
  container that needs a restart.
- ServerManagerJob only dispatches the Sentinel version check for
  functional servers.
- On cloud, ServerPatchCheckJob runs on the maintenance queue and
  ServerLimitCheckJob runs on the webhooks queue.
- The default number of cloud maintenance workers goes from 10 to 60.
- excludeCertainErrors drops its unused ignore list and is typed as
  `never`.
- Docs and the Horizon config comments are updated, and tests cover the
  new behavior.
2026-10-03 23:39:38 +02:00
Andras Bacsai 083e71c481 fix(horizon): route webhook jobs to a dedicated cloud webhooks queue
On cloud, GitHub pull request webhooks and Stripe events now run on a
dedicated `webhooks` queue so a busy `high` queue cannot delay them.
Self-hosted instances keep these jobs on `high`.

- Add webhooks_queue() helper and use it in ProcessGithubPullRequestWebhook
  and StripeProcessJob
- Provision a `webhooks` Horizon pool in cloud production, sized by
  HORIZON_WEBHOOKS_PROCESSES (default 10)
- Document the new pool, rollout and rollback steps
- Cover pool provisioning and job queue routing in tests
2026-10-03 21:37:09 +02:00
Andras Bacsai e144b990c3 feat(horizon): add fixed-size per-queue worker pools on Coolify Cloud
Cloud production now runs one supervisor per queue (deployments, crons,
high, default, maintenance), so a busy queue cannot starve the others.
Add maintenance_queue() for slow Docker cleanups and a shared
horizon.worker_timeout config value. Self-hosted keeps the s6 supervisor.
2026-10-03 21:24:54 +02:00
Andras Bacsai bef08d84de chore(horizon): lower job trim times
Keep completed and recent jobs for 10 minutes, pending for 30 minutes,
and failed and monitored jobs for 1 day.
2026-10-02 07:19:39 +02:00
Andras Bacsai dd597d1527 fix(backups): support long-running volume backups 2026-08-18 14:10:23 +02:00
peaklabs-dev 5da6c5e118 feat(horizon): read allowed emails from config and allow root user by id
- read allowed emails from config instead of env(), which returns null once configs are cached in production
- allow the root user by id instead of matching their email
- clean up gate code
2026-08-17 13:07:49 +02:00
Andras Bacsai 76030a30d4 chore(v5): archive V5 implementation and remove runtime integration
Move V5 source, migrations, UI, scripts, and tests into documentation, then remove V5 routes, models, jobs, configuration, dependencies, and application hooks.
2026-08-15 19:13:30 +02:00
Andras Bacsai d0247d3b09 feat(v5): gate V5 to development environments only
Introduce V5Feature and config so V5 routes, jobs, commands, morph maps,
and model queries run only when enabled. Move V5 migrations to
migrations-v5 (loaded only when enabled), remove Flux from production
Docker/install paths, and add isolation tests.
2026-07-19 11:44:12 +02:00
Andras Bacsai 6ae45684f9 feat(v5): add server reconciliation and canvas APIs
Split V5 dashboard behavior into domain controllers and policies,
add agent token rotation/revocation, status reconciliation jobs,
ingress firewall syncing, and canvas connection APIs.

Add migrations for V5 status tracking, server capabilities, resource
connection aliases, and revoked agent tokens.
2026-07-06 17:40:37 +02:00
Andras Bacsai b7b0dfeddd chore: prepare for PR 2026-02-23 13:24:49 +01:00
🏔️ Peak d736b3e8b7 feat(queues): improve Horizon config to reduce CPU and RAM usage (#6212) 2025-07-18 19:17:28 +02:00
Andras Bacsai b08c6b5b33 fix(horizon): update queue configuration to use environment variable for dynamic queue management 2025-07-12 14:04:01 +02:00
Andras Bacsai c2e076be54 add min process for horizon 2024-11-02 18:32:45 +01:00
Andras Bacsai f8d6ce2730 Update memory limit to 128MB in horizon configuration 2024-09-02 10:22:10 +02:00
Andras Bacsai eebb99ca22 chore: Update memory limit to 64MB in horizon configuration 2024-08-30 21:24:09 +02:00
Andras Bacsai fcb3d71cb4 feat: add high priority queue 2024-06-20 14:52:12 +02:00
Thijmen d86274cc37 Fix styling 2024-06-10 20:43:34 +00:00
Andras Bacsai 24bd4da94b Update horizon.php and docker-compose.prod.yml 2024-04-13 15:16:57 +02:00
Andras Bacsai 117f1d4155 fix: horizon 2023-12-30 15:30:57 +01:00
Andras Bacsai 075f3ce930 remove unnecessary queue 2023-12-30 15:29:26 +01:00
Andras Bacsai 94960d96a9 add max horizon processes 2023-12-20 11:47:51 +01:00
Andras Bacsai 76aab722b8 fix: limit horizon processes to 2 by default 2023-10-18 15:07:04 +02:00
Andras Bacsai 80af200c9f feat: long running queue with 1 hour of timeout 2023-06-28 18:20:01 +02:00
Andras Bacsai 8677b1d85d fix 2023-05-23 15:48:05 +02:00
Andras Bacsai 48e2ce2ca4 horizon + redis 2023-05-10 09:26:25 +02:00