Horizon workers kept Server::findCached() results between jobs, so scheduled
database backups used stale reachability and subscription state. A backup that
was skipped with the stale state counted as done for that day. Flush the
Server identity map before each queue job.
Add debug logs for the full path of a scheduled run, so a multi-node cloud
instance shows where a run is lost:
- manager start delay, gap since the previous start, host, pid, memory, and
a warning when the start is later than the catch-up window
- skip logs with the server state that was used and the current database state
- occurrence enqueued, claimed, rejected, completed and failed, with host and
the delay since the due time; a warning when the previous run is still open
- count of occurrences still open after 15 minutes on each manager run
- database backup runs that end without a backup
- server connection state changes with the checking node and the SSH error
Also keep publishing pending occurrences when one of them fails.
Andras Bacsai
Kevin Dench