starred/Foundry
1
0
Fork 0
mirror of https://github.com/Misterio77/Foundry.git synced 2026-08-24 02:14:13 -05:00
Code Issues Packages Projects Releases Wiki Activity

docs(merope): correct disk plans, record boot fix

Browse Source 
Device letters on merope follow USB enumeration order and are not stable;
they changed three times on 2026-08-08/09 as drives were re-plugged. Both
documents now address disks by label throughout.

Corrections to the migration plan:

- The ~28 MiB/s attributed to the media disk was a saturated USB 2.0 link,
  not a seek-bound spindle. On USB 3 it sustains 183 MB/s, 6.5x faster.
  Sizing arguments were built on the wrong number.
- The 04:42 reset was a UAS device reset on the M.2 bridge (174c:2362),
  not swap: swap stayed flat at 2.27 GiB with 4.4 GiB available. The
  watchdog held it to 105s against 48min the previous day.
- Both drives now sit on the VL805 SuperSpeed side, so they share one
  PCIe lane rather than having independent paths.

Boot, resolved 2026-08-09 in two stages, both firmware-level and
therefore invisible to nixos-rebuild:

- EEPROM walks USB in port order and looped when the drive in 2-1 had no
  bootable partition. Excluded both data drives by VID:PID. Note
  'rpi-eeprom-config --apply' is unusable on NixOS, since the wrapper
  points FIRMWARE_ROOT at a nonexistent path; build the image with
  --config and stage pieeprom.upd/.sig by hand.
- U-Boot 2021.04 predates bootstd and declares USB as a single instance,
  so boot_targets held one usb0 and only devnum=0 was ever tried.
  Replaced with 2026.04, which enumerates every bootdev. Old binary kept
  as u-boot-rpi4.bin.bak.

Backups doc records that the M.2 has 2233 corruption errors and has never
been scrubbed, so they were found passively rather than by verification.
Music (92 GB, the only irreplaceable data there) has been copied to the
backups disk; a scrub is now the open item.

Assisted-by: pi (claude-opus-5)
This commit is contained in:
Gabriel Fontes
2026-08-09 12:16:39 -03:00
parent 3cfb0d440a
commit b5e059f96b
2 changed files with 209 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/todo/backups.md
+63 -10
View File
@@ -1,13 +1,48 @@
# Backups
There are none. No `restic`, `borg`, `btrbk` or anything else appears anywhere
in this flake, and sdc1 — a 932 GB disk mounted at `/srv/backups` — currently
holds 5.9 MB.
in this flake, and the 932 GB disk labelled `backups`, mounted at
`/srv/backups`, currently holds 5.9 MB.
> **Device letters are not stable on merope.** They follow USB enumeration
> order and changed on 2026-08-08 when drives were re-plugged. Always address
> disks by label (`merope`, `media`, `backups`) or UUID.
The impermanence setup makes this easier than it would otherwise be: every
`environment.persistence` entry is, by construction, an exact declaration of
what state matters. That list is the starting inventory.
## Why this stopped being theoretical
On 2026-08-08, investigating an overnight reset, three things surfaced about the
M.2 that holds every service's state:
```
[merope].corruption_errs 2233
```
1. **2233 checksum failures** logged by btrfs on the root filesystem.
2. **A scrub has never run** — `btrfs scrub status` reports "no stats
available" against 267 GiB. So those failures were not found by verification;
they were found *passively*, when something read those blocks and got back
garbage. Each one returned an I/O error to whatever was reading.
3. On a single-device btrfs there is no second copy, so that data is **gone**,
and has been for an unknown length of time.
Separately, the USB bridge in front of that disk (ASMedia ASM2362) dropped off
the bus entirely at 04:42 and needed a device reset, and on the following boot
the SuperSpeed link failed to train at all.
The counter is cumulative and undated — it may all be one bad episode from long
ago. But the honest position is that the only copy of firefly, radicale, immich
and deluge state currently lives on a disk with confirmed silent corruption,
behind a bridge with demonstrated link instability, and nobody has ever checked
whether the rest of it reads back.
**This moves ahead of `merope-disk-reorganize.md`.** A scrub is the diagnostic
that answers whether the corruption is historical or ongoing — but run it
*after* there is a copy, not before, because the answer might be "ongoing".
## What is actually at risk
### Tier 1 — small and irreplaceable (~5 GB)
@@ -124,15 +159,21 @@ the next snapshot, not a broken library.
3-2-1, honestly applied:
```
alcyone ──restic over tailnet──> merope sdc1 /backups
merope ──restic local────────> merope sdc1 /backups
alcyone ──restic over tailnet──> merope `backups` disk, /backups
merope ──restic local────────> merope `backups` disk, /backups
merope ──tier 1 + immich─────> offsite object storage (~25 GB)
```
Backing merope up to a disk inside merope covers the common cases: accidental
deletion, a bad rebuild, sda or sdb failing. It does **not** cover fire, theft,
or the fact that all three disks hang off a single VL805 controller on one PCIe
lane. Hence offsite.
deletion, a bad rebuild, the M.2 or the media disk failing. It does **not** cover
fire, theft, or the fact that all three disks hang off a single VL805 controller
on one PCIe lane. Hence offsite.
One point in its favour: the `backups` disk reports **0 corruption errors**,
against 2233 on the M.2, and it is the only drive still on the USB 2.0 bus — so
it shares neither the SuperSpeed link nor the bridge that has been misbehaving.
480 Mbps is slow for the initial ~115 GB seed (call it two hours) and irrelevant
for incrementals.
Excluding music keeps the offsite set at roughly 25 GB, which is small enough
that cost stops being a consideration. Music can be added later if desired; it
@@ -176,7 +217,19 @@ went wrong in January.
5. Add monitoring and the restore drill. Not optional; it is the step that turns
the rest into a backup rather than a cron job.
Step 1 is worth doing **today**, in whatever crude form works:
```bash
restic -r /srv/backups/restic init
restic -r /srv/backups/restic backup /persist /var/lib
```
Not declarative, no dumps, no retention — and still strictly better than the
current state, which is a disk with 2233 checksum failures and no copy of
anything. Replace it with the real thing once it exists.
Note the interaction with `merope-disk-reorganize.md`: that plan moves `/srv` to
sdb1 and remounts sdc1 at `/backups`. Paths here assume the post-migration
layout, so it is worth doing the migration first, or writing the repository path
so it survives the move.
the media disk and remounts the backups disk at `/backups`. Paths here assume
the post-migration layout, so either write the repository path so it survives
the move, or accept re-pointing it once. Given the corruption findings, do not
reorder these — backups first, migration second.
docs/todo/merope-disk-reorganize.md
+146 -29
View File
@@ -1,40 +1,141 @@
# merope disk reorganization
Consolidate all media onto sdb in one btrfs subvolume so every import becomes a
rename or hardlink instead of a cross-device copy, and sda — which holds the
swapfile — sees no media I/O at all.
Consolidate all media onto the **media** disk in one btrfs subvolume so every
import becomes a rename or hardlink instead of a cross-device copy, and the M.2
— which holds the swapfile — sees no media I/O at all.
## Device naming: use labels, never letters
Kernel device letters on this host are **not stable**. They are assigned in USB
enumeration order, which changes when a drive is re-plugged, moved between
ports, or hot-plugged after boot. On 2026-08-08 the 12 TB and the 932 GB swapped
letters simply by being plugged in a different order.
| label | size | role | filesystem UUID prefix |
|------------|--------|-------------------------|------------------------|
| `merope` | 465 GB | M.2 NVMe, root + swap | `1660ec93` |
| `media` | 12 TB | tv, movies, photos | `b8126efa` |
| `backups` | 932 GB | backup target | `6d8471ca` |
Everything below refers to these labels. Use `/dev/disk/by-label/<label>` or
`/dev/disk/by-uuid/` in commands. Any instruction naming `sdb` or `sdc` is a bug.
## Background
On 2026-08-07 merope hung for ~48 minutes under a single 1.1 GB movie import.
No OOM, no panic, no thermal event: `/swap/swapfile` shares sda with the rootfs,
and a Radarr cross-device import saturated that disk at ~85 MiB/s, so page-ins
queued behind bulk I/O indefinitely.
No OOM, no panic, no thermal event: `/swap/swapfile` shares the M.2 with the
rootfs, and a Radarr cross-device import saturated that disk, so page-ins queued
behind bulk I/O indefinitely.
That was mitigated in `e5ae4001` (watchdog, ondemand governor, swappiness, PSI)
and `e5e1048a` (cgroup weights, BFQ), which turned the failure from a hang into
graceful degradation. This document removes the underlying cause.
graceful degradation.
Measured during a four-season download the same evening:
### What 2026-08-08 changed
| device | queue depth | throughput | note |
|--------|-------------|------------|-------------------------------|
| sda | ~3 | 70–82 MiB/s| SSD, coping |
| sdb | 20–29 | ~28 MiB/s | 95% utilised, seek-bound |
At 04:42 the host reset again, and the cause was **not** swap. The USB bridge
backing the M.2 (ASMedia ASM2362, `174c:2362`) stopped answering SCSI commands:
Network ingest was only 26 MiB/s. Every byte moved roughly three times: written
to sda, read back off sda for the import, written to sdb — and both disks share
a single VL805 controller on one PCIe lane.
```
sda uas_eh_abort_handler ... inflight: CMD OUT
scsi host0: uas_eh_device_reset_handler start
usb 2-2: reset SuperSpeed USB device number 2
```
Swap sat flat at 2.27 GiB throughout and 4.4 GiB stayed available — memory was
never involved. The watchdog from `e5ae4001` caught it: **105 seconds of
downtime, self-recovered**, against 48 minutes and a manual power cycle the day
before.
On the reboot that followed, the SuperSpeed link failed to train
(`usb2-port2: Cannot enable. Maybe the USB cable is bad?`) and all three drives
enumerated on the USB 2.0 bus at 480 Mbps. A cold power cycle — full power
removal, not a reboot — restored it.
### The 28 MiB/s figure was wrong
Measured during a four-season download on 2026-08-07:
| device | queue depth | throughput | note |
|--------|-------------|-------------|--------------------------|
| M.2 | ~3 | 70–82 MiB/s | SuperSpeed, coping |
| media | 20–29 | ~28 MiB/s | 95% utilised |
That was read as a seek-bound spinning disk. It was not. The 12 TB was plugged
into a **USB 2.0 port**, and 28 MiB/s is a saturated 480 Mbps link. On USB 3,
measured 2026-08-09 with the boot problems resolved:
```
1258291200 bytes (1.3 GB) copied, 6.88015 s, 183 MB/s
```
**6.5× faster**, and the drive was never the constraint. Every capacity argument
in this document was originally sized against the wrong number; the migration is
now clearly worth doing rather than a careful trade.
### New caveat: shared PCIe lane
With the media disk on USB 3, it and the M.2 both sit on the SuperSpeed side of
the VL805 — one PCIe Gen2 x1 lane, roughly 440 MB/s usable **shared**. Before,
the M.2 had that lane to itself. This is still vastly better than 40 MB/s, but
the premise is *separate spindles, shared path*, not fully independent I/O.
### Boot order — resolved 2026-08-09
Putting the media disk in USB 3 broke booting, in two separate stages. Both are
fixed; both live in firmware, outside this flake, so they survive a reinstall
and will not reappear in a `nixos-rebuild`.
**Stage 1 — the EEPROM bootloader.** It walks USB devices in port order and, on
finding no bootable partition on the drive in 2-1, falls through to the SD slot
and then loops (the default `BOOT_ORDER` ends in `f`). Fixed by excluding the two
data drives by VID:PID so it never considers them:
```
USB_MSD_EXCLUDE_VID_PID=174c:55aa,152d:0580
```
Applied by rebuilding the EEPROM image with `rpi-eeprom-config --config`, then
copying it to `/firmware/pieeprom.upd` with a matching `pieeprom.sig` from
`rpi-eeprom-digest`. Note `--apply` does **not** work on NixOS: the nixpkgs
wrapper points `FIRMWARE_ROOT` at a directory that does not exist.
**Stage 2 — U-Boot.** The EEPROM then handed off cleanly, but U-Boot failed the
same way for a different reason. The installed binary was **U-Boot 2021.04**,
which predates `bootstd` and uses legacy `distro_bootcmd`. There, `rpi.h`
declares USB as a single instance:
```c
#define BOOT_TARGET_USB(func) func(USB, usb, 0)
```
so `boot_targets` contains exactly one `usb0`, and `usb_boot` only ever tries
`devnum=0`. With the media disk in 2-1 it enumerated first, had no
`extlinux.conf`, and U-Boot moved on to `mmc0`/`pxe`/`dhcp` without ever looking
at the M.2.
Fixed by replacing `/firmware/u-boot-rpi4.bin` with a current build
(`nixpkgs#ubootRaspberryPi4_64bit`, U-Boot 2026.04), which uses `bootstd` and
enumerates every bootdev rather than a single hardcoded index. The 2021.04
binary is kept beside it as `u-boot-rpi4.bin.bak`.
> **Recovery note.** `/firmware` is on the M.2 *inside* the Argon case. If a
> future U-Boot or EEPROM change fails to boot, recovery means opening the case
> and mounting that partition elsewhere — there is no SD card in the slot. Keep
> a copy of `u-boot-rpi4.bin.bak` off-machine before touching either again.
Result: both drives negotiate 5000 Mbps and the host boots unattended, so the
watchdog is safe again.
## Target layout
```
sda2 ephemeral root + /persist (service state) + swapfile
sdb1 subvolume "srv" -> /srv
├── media/{tv,movies,photos,music}
├── torrents/{downloading,completed}
└── incoming/{downloading,complete} <- sabnzbd
sdc1 -> /backups
merope ephemeral root + /persist (service state) + swapfile
media subvolume "srv" -> /srv
├── media/{tv,movies,photos,music}
├── torrents/{downloading,completed}
└── incoming/{downloading,complete} <- sabnzbd
backups -> /backups
```
Every app-visible path is unchanged: `/srv/media/tv`, `/srv/media/music`,
@@ -86,7 +187,7 @@ handles**, which block the unmounts below.
Leave `postgresql`, `redis-immich` and `meilisearch` running — all on sda, none
touch `/srv`, and immich returns without a database restart.
### 2. Back music up to sdc1
### 2. Back music up to the backups disk
92 GB onto a 932 GB disk currently holding 5.9 MB. Makes the whole migration
reversible.
@@ -94,7 +195,7 @@ reversible.
### 3. Build the new subvolume (reflink — no data moves)
```bash
mount -o subvolid=5 /dev/sdb1 /mnt # fs root, not the default subvol
mount -o subvolid=5 /dev/disk/by-label/media /mnt # fs root, not the default subvol
btrfs subvolume create /mnt/srv
mkdir -p /mnt/srv/media
cp -a --reflink=always /mnt/{tv,movies,photos} /mnt/srv/media/
@@ -144,7 +245,8 @@ find /mnt/srv -links +1 | wc -l # must match the source count
### 6. Repoint mounts
Subvolume `srv` mounts directly at `/srv` — no bind mount and no `/persist/srv`,
since nothing about sdb is ephemeral. sdc1 moves to `/backups`.
since nothing about the media disk is ephemeral. The backups disk moves to
`/backups`.
Before unmounting the old `/srv`:
@@ -176,7 +278,7 @@ btrfs subvolume delete /mnt/{tv,movies,photos} # old sdb subvolumes
## Config changes
```nix
# sabnzbd.nix — both on sdb, so the final move is a rename
# sabnzbd.nix — both on the media disk, so the final move is a rename
download_dir = /srv/incoming/downloading
complete_dir = /srv/incoming/complete
```
@@ -192,7 +294,8 @@ max_connections_global = 200; # was -1
```
Eight concurrent torrents was survivable on flash and is a seek generator on
rust. Also in SABnzbd: enable pause-downloading-during-post-processing, and cut
rust. Note this is about *seeks*, not bandwidth — at 175 MB/s the link is no
longer the limit, but random access on a 12 TB drive still is. Also in SABnzbd: enable pause-downloading-during-post-processing, and cut
usenet connections from 225 to ~50 — a Pi 4 cannot use anywhere near that, and
each is a TLS socket with its own buffers and softirq cost.
@@ -209,15 +312,29 @@ imported files after the migration and only revisit if it is genuinely bad.
## Rollback
Nothing is deleted until step 8, music is backed up on sdc1, and reverting is a
disko change plus a generation switch.
Nothing is deleted until step 8, music is backed up on the backups disk, and
reverting is a disko change plus a generation switch.
## Verify afterwards
- `stat` a freshly imported episode in both staging and library: same inode,
link count 2. If it is still copying, that shows up immediately rather than in
three weeks.
- Watch `qd_sdb` during the next import. The 26–29 spikes should stop existing.
- Watch queue depth on the media disk during the next import. The 26–29 spikes
should stop existing — though note they were largely a USB 2.0 artifact and
may already be gone before any of this work happens.
## Prerequisites
- [x] **The host boots unattended** with both drives on USB 3 — EEPROM exclusion
plus U-Boot 2026.04, done 2026-08-09.
- [x] **Music is copied off the M.2** — 92 GB to the `backups` disk, verified.
That was the only irreplaceable data on the disk with 2233 checksum
failures; everything else there is regenerable service state.
- [ ] **A scrub has run** on both the M.2 and the media disk. 4.6 TB should not
be reflinked into a new subvolume on the assumption it is intact, and the
2233 errors on the M.2 are still unexplained — cumulative, undated, and
never verified. At 183 MB/s this is now affordable.
## Known trade