Sending SSH command batches on stdin changes how every remote command
reaches a server, which is too risky for a patch release. Move it to
next and restore the heredoc transport from before 1a2a5362a. The
shared test helpers stay because they work with both transports.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ServerCheckJob uses the stored reachability state instead of a second SSH
probe. One failed probe no longer marks the server unreachable without
UNREACHABLE_THRESHOLD or marks all its resources exited.
- Do not dispatch ServerCheckJob for build servers and Swarm workers.
- Check cloud provider state every 5 minutes instead of every minute.
- ServerPatchCheckJob uses isFunctional() and drops the unused retries.
- Delete the unused ServerCleanupMux job.
- Disable SSH multiplexing per command. The global config change stayed in
the queue worker, so later jobs (also deployments) ran without mux.
A single failed SSH attempt or any unrelated exception in
ServerConnectionCheckJob marked the server unreachable and unusable, so
deployments failed with "Server is not functional" while the server was
online.
- Retry a failed SSH attempt once after 3 seconds within the same check.
- Mark the server unreachable only after UNREACHABLE_THRESHOLD (2)
consecutive failed checks; notifications use the same threshold.
- Keep the stored state when the job fails for a reason other than SSH or
Docker (database, cache, bugs).
- Raise the job timeout to 60 seconds and the overlap lock above it. The
old 15 second timeout killed checks before they wrote a result.
- Run the Docker and Compose version commands once with a 15 second
timeout. A failed Compose check no longer marks the server unusable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Horizon workers kept Server::findCached() results between jobs, so scheduled
database backups used stale reachability and subscription state. A backup that
was skipped with the stale state counted as done for that day. Flush the
Server identity map before each queue job.
Add debug logs for the full path of a scheduled run, so a multi-node cloud
instance shows where a run is lost:
- manager start delay, gap since the previous start, host, pid, memory, and
a warning when the start is later than the catch-up window
- skip logs with the server state that was used and the current database state
- occurrence enqueued, claimed, rejected, completed and failed, with host and
the delay since the due time; a warning when the previous run is still open
- count of occurrences still open after 15 minutes on each manager run
- database backup runs that end without a backup
- server connection state changes with the checking node and the SSH error
Also keep publishing pending occurrences when one of them fails.
Store each server's Docker version and build stop commands via
dockerStopCommand() so newer Docker uses --timeout instead of
deprecated --time/-t. Also show Docker version in server details.
Move reachability notification triggering out of isReachableChanged into
a dedicated ServerReachabilityChanged event dispatched by
ServerConnectionCheckJob. Remove the blocking 3-attempt sleep loop from
isReachableChanged — unreachable_count threshold alone now gates the
Unreachable notification. Add feature and unit tests covering all
notification dispatch paths.
Reduce load on unreachable servers by implementing exponential backoff
during connectivity failures. Check frequency decreases based on
consecutive failure count:
0-2: every cycle
3-5: ~15 min intervals
6-11: ~30 min intervals
12+: ~60 min intervals
Uses server ID hash to distribute checks across cycles and prevent
thundering herd.
ServerCheckJob and ServerConnectionCheckJob increment unreachable_count
on failures. ServerManagerJob applies backoff logic before dispatching
checks. Includes comprehensive test coverage.
Andras BacsaiandClaude Opus 5.5
Kevin Dench
Cornelis Terblanche