mirror of
https://github.com/bckelley/tconnectsync.git
synced 2026-08-24 03:34:12 -05:00
Harden autoupdate against transient errors instead of exiting
get() retries only HTTP 401 and 500, so any other API error propagates out of the autoupdate loop and exits the process. When Tandem retired the reportsfacade endpoints and pumpeventmetadata began returning 404 (#146), a container with a restart policy would crash-loop. That is the worst possible response to an API outage: the credentials cache dies with the process, so every restart performs a full login against sso.tandemdiabetes.com. In my EU deployment that was a fresh login roughly every two minutes for hours from a single IP, which seems a good way to earn a WAF ban while already broken. Transient network errors (DNS failures, timeouts, mid-stream disconnects, urllib3 retry-budget exhaustion) have the same problem. This keeps both failure families inside the loop and backs off exponentially: 30s doubling to a cap of AUTOUPDATE_DEFAULT_SLEEP_SECONDS (300s default), reset on any successful poll. The cap reuses the existing poll interval, so a failing API is never contacted more often than a healthy one. After three consecutive failures the log escalates from WARNING to ERROR. Staying alive forever would make a real outage silent on deployments whose only alarm is the container dying, so after AUTOUPDATE_API_FAILURE_MINUTES (default 45) of unbroken failure the process gives up and exits non-zero. That is roughly one restart per hour during a genuine outage instead of one every two minutes, while short blips stay silent. Set 0 to disable. This is deliberately not gated on AUTOUPDATE_RESTART_ON_FAILURE, which covers the pump-not-uploading watchdog where restarting achieves nothing (as the existing TODO notes) and which many users therefore disable. An unreachable API is a different failure and gets its own knob. ApiLoginException stays fatal: bad credentials are not transient, and retrying them in-process would hammer the login endpoint with attempts that cannot succeed. Also included: - A defensive clamp so a negative rolling-average entry can never reach time.sleep() and crash with ValueError. - Tests covering the backoff sequence, reset-on-success, the sustained-failure exit, the opt-out, and that login failures and programming errors still propagate. - README documentation for all nine AUTOUPDATE_* variables, none of which were documented outside secret.py.
This commit is contained in:
@@ -381,6 +381,36 @@ An example `run.sh` if you built tconnectsync locally:
|
||||
docker run tconnectsync --auto-update
|
||||
```
|
||||
|
||||
#### Tuning Auto-Update
|
||||
|
||||
These optional environment variables control how `--auto-update` polls and how
|
||||
it behaves when things go wrong. The defaults are sensible; you generally only
|
||||
need these if you are seeing too many (or too few) restarts.
|
||||
|
||||
| Variable | Default | What it does |
|
||||
| --- | --- | --- |
|
||||
| `AUTOUPDATE_DEFAULT_SLEEP_SECONDS` | `300` | Poll interval when no better estimate is available. Also the ceiling for the retry backoff below. |
|
||||
| `AUTOUPDATE_MAX_SLEEP_SECONDS` | `1500` | Upper bound on the adaptive poll interval, regardless of how rarely new data appears. |
|
||||
| `AUTOUPDATE_UNEXPECTED_NO_INDEX_SLEEP_SECONDS` | `60` | How long to wait when new data is overdue based on the pump's previous cadence. |
|
||||
| `AUTOUPDATE_USE_FIXED_SLEEP` | `false` | Set true to always sleep `AUTOUPDATE_DEFAULT_SLEEP_SECONDS` instead of adapting to the pump's observed upload cadence. |
|
||||
| `AUTOUPDATE_API_FAILURE_MINUTES` | `45` | Exit with a non-zero code after this many minutes of unbroken API/network failure, so your container platform restarts tconnectsync and can alert you. Set `0` to never exit. |
|
||||
| `AUTOUPDATE_NO_DATA_FAILURE_MINUTES` | `180` | Log an error if the pump has not reported new events for this long. Usually means the pump simply is not uploading. |
|
||||
| `AUTOUPDATE_FAILURE_MINUTES` | `75` | Log an error if events are appearing but no data has synced successfully for this long. |
|
||||
| `AUTOUPDATE_RESTART_ON_FAILURE` | `false` | Whether the two watchdogs above also exit non-zero. Independent of `AUTOUPDATE_API_FAILURE_MINUTES`. |
|
||||
| `AUTOUPDATE_MAX_LOOP_INVOCATIONS` | `-1` | Stop after this many poll cycles. `-1` means run forever; mainly useful for testing. |
|
||||
|
||||
**On failures and restarts.** Transient errors (DNS blips, timeouts, HTTP 404/502/503
|
||||
from Tandem) do not crash tconnectsync. It retries with a growing backoff — 30s,
|
||||
60s, 120s, 240s, then holding at `AUTOUPDATE_DEFAULT_SLEEP_SECONDS` — and resets
|
||||
as soon as a poll succeeds. Staying in-process matters: an exit discards the
|
||||
cached credentials, so a restart loop means a fresh login on every attempt,
|
||||
which risks tripping Tandem's rate limiting.
|
||||
|
||||
Only once the API has been failing continuously for `AUTOUPDATE_API_FAILURE_MINUTES`
|
||||
does tconnectsync give up and exit, so that a genuine outage surfaces (roughly one
|
||||
restart per hour) instead of disappearing into an endless quiet retry. Invalid
|
||||
credentials are never retried — they exit immediately, since retrying cannot help.
|
||||
|
||||
### Running with Cron
|
||||
|
||||
If you choose not to run tconnectsync with `--auto-update` continuously,
|
||||
|
||||
Reference in New Issue
Block a user