Each dump and S3 upload got the full backup timeout, so a long dump plus
an upload passed the worker timeout. The worker then killed the job
without cleanup: the execution stayed running, and the upload container
and the dump file stayed on the server.
- Give each step only the time that is left of the backup timeout, so a
run that takes too long fails inside the job and cleans up.
- Run the dump once. An SSH retry started a second dump into the same
file while the first could still run in the container.
- Explain exit code 124 and say in the timeout help that it covers the
complete run.
- Remove backup upload containers of executions that are not running
in the daily helper container cleanup.
Closes#8139
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Queued GetContainersStatus runs are unique per server until a worker
starts them, and CleanupHelperContainersJob locks per server instead of
class-wide. Add UniqueUntilProcessingJobDecorator for laravel-actions.
- Fix container filtering to properly distinguish base deployments (pullRequestId=0) from PR deployments
- Add deployment cancellation when PR closes via webhook to prevent race conditions
- Prevent CleanupHelperContainersJob from killing active deployment containers
- Enhance error messages with exit codes and actual errors instead of vague "Oops" messages
- Protect status transitions in finally blocks to ensure proper job failure handling
kashik0i
Andras Bacsai
Lucas Michot
andrasbacsai
Thijmen