Each dump and S3 upload got the full backup timeout, so a long dump plus
an upload passed the worker timeout. The worker then killed the job
without cleanup: the execution stayed running, and the upload container
and the dump file stayed on the server.
- Give each step only the time that is left of the backup timeout, so a
run that takes too long fails inside the job and cleans up.
- Run the dump once. An SSH retry started a second dump into the same
file while the first could still run in the container.
- Explain exit code 124 and say in the timeout help that it covers the
complete run.
- Remove backup upload containers of executions that are not running
in the daily helper container cleanup.
Closes#8139
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Queued GetContainersStatus runs are unique per server until a worker
starts them, and CleanupHelperContainersJob locks per server instead of
class-wide. Add UniqueUntilProcessingJobDecorator for laravel-actions.
Andras BacsaiandClaude Opus 5.5