Device letters on merope follow USB enumeration order and are not stable;
they changed three times on 2026-08-08/09 as drives were re-plugged. Both
documents now address disks by label throughout.
Corrections to the migration plan:
- The ~28 MiB/s attributed to the media disk was a saturated USB 2.0 link,
not a seek-bound spindle. On USB 3 it sustains 183 MB/s, 6.5x faster.
Sizing arguments were built on the wrong number.
- The 04:42 reset was a UAS device reset on the M.2 bridge (174c:2362),
not swap: swap stayed flat at 2.27 GiB with 4.4 GiB available. The
watchdog held it to 105s against 48min the previous day.
- Both drives now sit on the VL805 SuperSpeed side, so they share one
PCIe lane rather than having independent paths.
Boot, resolved 2026-08-09 in two stages, both firmware-level and
therefore invisible to nixos-rebuild:
- EEPROM walks USB in port order and looped when the drive in 2-1 had no
bootable partition. Excluded both data drives by VID:PID. Note
'rpi-eeprom-config --apply' is unusable on NixOS, since the wrapper
points FIRMWARE_ROOT at a nonexistent path; build the image with
--config and stage pieeprom.upd/.sig by hand.
- U-Boot 2021.04 predates bootstd and declares USB as a single instance,
so boot_targets held one usb0 and only devnum=0 was ever tried.
Replaced with 2026.04, which enumerates every bootdev. Old binary kept
as u-boot-rpi4.bin.bak.
Backups doc records that the M.2 has 2233 corruption errors and has never
been scrubbed, so they were found passively rather than by verification.
Music (92 GB, the only irreplaceable data there) has been copied to the
backups disk; a scrub is now the open item.
Assisted-by: pi (claude-opus-5)
No backup tooling exists anywhere in the flake today, and sdc1 is a
932GB disk holding 5.9MB.
Covers what is actually irreplaceable -- firefly, radicale, deluge
state, mail, git, immich, music -- rather than the 4.6TB of re-acquirable
video, and specifies database dumps rather than live file copies, which
would otherwise produce backups that silently will not restore.
Mail and calendars are already replicated across three hosts, so they are
included for history rather than redundancy: sync propagates deletion,
and the useful protection there is retention depth, not another copy.
Assisted-by: pi (claude-opus-5)
Consolidates all media onto sdb in a single btrfs subvolume mounted at
/srv, so every import becomes a rename or hardlink rather than a
cross-device copy, and the disk holding the swapfile sees no media I/O.
Written up after the 2026-08-07 stall, whose root cause was swap
contending with a Radarr import on the same device.
Assisted-by: pi (claude-opus-5)
Fold back UI edits from the hosts dashboard:
- RAM Usage: rename the used-memory series to "Memory", add a "Swap"
series (SwapTotal - SwapFree), and retarget the configFromData
transformation from byFrameRefID to byType so Total Memory applies as
max to both series. Show the legend, since there are now two series.
- Rename the Storage gauge to "Disk Usage".
- New "Disk IO" timeseries: rate of node_disk_io_time_weighted_seconds_total,
joined against node_btrfs_info via label_replace to strip the partition
suffix, so series are labelled by filesystem label.
The UI export predated d9a6c23e and would have reverted its Storage
query and threshold fixes; those hunks were dropped rather than folded.
Assisted-by: pi (claude-opus-5)
The top threshold step rendered a 100%-full filesystem in plain text
colour rather than red, so the alarm state was unreachable -- red only
covered 90-99.99%.
Both targets pulled a full range while groupingToMatrix keeps one value
per cell, so switch them to instant queries.
Join on (instance, uuid) rather than uuid alone; with uuid-only matching,
a multi-value $hosts could cross-match filesystems between hosts.
Assisted-by: pi (claude-opus-5)
The 2026-08-07 stall presented as a CPU problem but was I/O underneath:
a Radarr cross-disk import saturated the USB SSD holding both the rootfs
and the swapfile, and everything else queued behind it.
The existing Nice values were aimed at the wrong thing twice over. They
only ever addressed CPU scheduling, which was the symptom -- renicing a
process blocked in D-state achieves nothing. They were also a layer too
low: with the cgroup v2 cpu controller enabled, CPU is split between
service cgroups by weight, and every service on this host sat at an
identical 100. Nice only reorders tasks within a cgroup, so a lone
service being polite to itself changed nothing.
Replace them with CPUWeight/IOWeight, which is the layer that actually
arbitrates between services. Nice and IOSchedulingPriority are dropped
rather than kept alongside: nothing here escapes the cpu controller, and
ioprio depends on BFQ exactly as IOWeight does, so it is not a fallback
for anything -- if BFQ is absent both are inert together.
deluged is included despite never having had a Nice; it writes to the
same device continuously and was the other half of the load that day.
recyclarr loses its Nice without replacement -- it only makes a handful
of API calls to the *arrs and never contends for anything.
Requires BFQ. All three disks were on mq-deadline, which has no notion of
I/O weights at all and would have made IOWeight a silent no-op.
Assisted-by: pi (claude-opus-5)
Almost everything that requests activation is being rude about it, and with
follow_mouse on it turns into a focus fight. Default to off and opt browsers
back in, so opening a link from another app still raises the browser.
Makes the LinOffice focus rule redundant; the maximize suppression stays,
since that one is Excel being Excel rather than a focus policy.
Assisted-by: pi (claude-opus-5)
FreeRDP RemoteApp windows mirror the Windows-side focus state, so Office
requests activation constantly. With focus_on_activate and follow_mouse both
on, that ping-pongs: 22 unprompted focus flips in 30s between Excel and the
terminal under the cursor, which also made clicks land on a window that was
mid-flip. Each activation carries a maximize request too, which read as the
window spontaneously fullscreening whenever it regained focus.
Assisted-by: pi (claude-opus-5)
merope hung for ~48min (16:59-17:56 on 2026-08-07) with all four cores
pinned. No OOM, no panic, no thermal event -- it simply stopped making
progress and needed a manual power cycle.
Root cause: /swap/swapfile lives on sda, the same USB SSD as the rootfs.
swappiness=60 had ratcheted ~4.2G of cold anonymous pages onto it over
days (monotonically -- it never came back). A torrent write plus a Radarr
cross-disk import then saturated that device at ~85MiB/s, so page-ins
queued behind bulk I/O indefinitely. Nothing exceeded a memory limit, so
the kernel OOM killer never engaged.
All four USB ports share one VL805 controller on a single PCIe lane, so
relocating swap to another disk would not have given it an independent
path.
- watchdog: bound any future stall to a reboot instead of a silent hang
- ondemand: merope never exceeded 600MHz of its 1500MHz in 4h of 100% load
- swappiness=10: MemAvailable never fell below 3.6G, swap is not
load-bearing here; keep the swapfile as a backstop, not a habit
- psi=1 + earlyoom: the kernel was not even measuring stall time
Assisted-by: pi (claude-opus-5)
The setuid /run/wrappers/bin/unix_chkpwd wrapper is generic plumbing that
nixpkgs' pam_unix requires (system-manager, unlike NixOS, doesn't provide
it), not host policy. Move it to modules/system-manager/unix-chkpwd.nix,
applied unconditionally like NixOS does; hosts/.../pam.nix keeps only the
pam.d stacks.
Assisted-by: pi (claude-opus-4-8)
Port nixpkgs' nix.registry option (config/nix-flakes.nix) into
modules/system-manager/nix-registry.nix, giving system-manager the same
interface NixOS has. common/global/nix.nix now just sets
'nix.registry = mapAttrs (_: flake: {inherit flake;}) flakeInputs' like
the NixOS side, instead of hand-writing /etc/nix/registry.json.
Assisted-by: pi (claude-opus-4-8)
system-manager's nix module has no nix.registry option (NixOS does), so
generate /etc/nix/registry.json directly, mirroring nixpkgs' generator.
The hydra auto-upgrade's 'nix flake metadata self' needs 'self' resolvable
from the system registry; this also registers the other inputs (nixpkgs,
etc.) for ad-hoc use, matching the NixOS side.
Assisted-by: pi (claude-opus-4-8)
build.gradle used 'latest.release', a dynamic version, so the Gradle
deps lockfile captured an ephemeral net.runelite snapshot
(client 1.12.34-20260721…-SNAPSHOT). repo.runelite.net prunes old
snapshots, so fetchDeps 404'd on re-fetch. Pin to the current concrete
release (1.12.35) and regenerate deps.json — releases aren't pruned, so
the FOD stays reproducible.
Assisted-by: pi (claude-opus-4-8)
Its old nebula-cyan-yellow (seed #0a2a4a, blue 210) clashed with
merope's towers-ice (blue 213). Switch to samurai-cyberpunk-pink-green
(seed #a63ee1), a vivid violet-magenta that fills the empty ~280 gap and
sits clear of every other home's seed color.
Stop reusing maia's deer-lunar-fantasy; switch to
cubist-crystal-brown-teal (seed #1acdaf), a vivid teal that fills the
empty teal/cyan gap and is distinct from every other home's seed color.
hydra.m7.rs is tailnet-gated (write-enabled, so locked down), so mgc needs
Tailscale to reach it for the hosts.mgc auto-upgrade. Add Tailscale's
official apt repo + signing key and install it in the Ubuntu-prereqs
stage, keeping it apt-managed (updates via apt, up on boot, independent of
the nix config). Joining the tailnet (tailscale up) stays a manual step.
Assisted-by: pi (claude-opus-4-8)
The CLI now ships on the system PATH (environment.systemPackages), so the
apps.system-manager passthrough is redundant. install.sh's bootstrap (the
one caller) instead builds the systemConfig and runs its own
bin/register-profile + bin/activate, needing no flake app.
Assisted-by: pi (claude-opus-4-8)
Mirror hosts/nixos/common/global/auto-upgrade.nix for the system-manager
side: enable system.hydraAutoUpgrade against hydra.m7.rs (foundry/main),
job hosts.mgc. Disable the timer when the flake is dirty.
Assisted-by: pi (claude-opus-4-8)
Fold outputs.systemConfigs into the hydra 'hosts' jobs alongside
nixosConfigurations, so Hydra builds mgc's system-manager generation as
hosts.mgc (matching the hosts.<name> convention the auto-upgrade uses).
Assisted-by: pi (claude-opus-4-8)
Port modules/nixos/hydra-auto-upgrade.nix to the system-manager side as
modules/system-manager/hydra-auto-upgrade.nix, exposed via a new
outputs.systemManagerModules and imported by the common/global base.
The script (cached-system-manager) mirrors cached-nixos-rebuild, adapted
to system-manager's model: there is no /run/current-system active pointer
and no boot-time reactivation, so the registered profile
(/nix/var/nix/profiles/system-manager-profiles/system-manager) is the
single source of truth. Only 'diff' and 'switch' actions; switch does
register-profile + activate (the CLI's switch = both, atomically).
Not enabled anywhere yet.
Assisted-by: pi (claude-opus-4-8)
greetd had Requires=userborn.service, but System Manager restarts the
oneshot userborn.service on every activation. The Requires propagates
that restart to greetd, which tears down the running Wayland session
(then Restart=always bounces it back to the greeter) on every switch.
Relax to Wants=userborn.service, keeping After= for ordering: userborn
still runs before greetd at boot, but its per-activation restart no
longer drags greetd (and the live session) down. X-RestartIfChanged=false
already prevented greetd's own restart; this closes the dependency path.
Assisted-by: pi (claude-opus-4-8)
Put the CLI on the system PATH (/run/system-manager/sw/bin), like
nixos-rebuild is on NixOS, instead of 'nix run .#system-manager'. Pinned
to the flake's system-manager input, matching what builds the host.
Assisted-by: pi (claude-opus-4-8)
Move the unix_chkpwd security wrapper and the greetd/hyprlock pam.d
stacks out of greetd.nix into a dedicated pam.nix (greetd.nix keeps just
the greeter service/user/session-wrapper). Factor the shared auth block
into one binding. The generated pam.d/* and the wrapper are byte-identical
before and after.
Also drop two no-op nix.settings: build-users-group (already Nix's
default 'nixbld') and the @wheel trusted-user (Ubuntu has no wheel group).
Assisted-by: pi (claude-opus-4-8)
hyprlock's pam.d did 'include login' too, which nix's linux-pam can't
parse (Debian @include). Give it a minimal self-contained auth stack on
nix's pam_unix, matching greetd. hyprlock runs unprivileged, so pam_unix
verifies via the setuid /run/wrappers/bin/unix_chkpwd added alongside the
greetd fix.
Assisted-by: pi (claude-opus-4-8)
Our pam.d/greetd did 'include login', chaining into Ubuntu's
/etc/pam.d/{login,common-*}, which is built on Debian's @include
extension. greetd links nix's linux-pam, whose config parser only knows
'include'/'substack' (no @include), so it errored 'illegal module type:
@include' and greetd never authenticated on tty1.
nix's libpam also can't dlopen Ubuntu's modules (pamtester reports
'Module is unknown'), so the stack uses nix's own modules by absolute
path, mirroring Ubuntu's expanded common-auth/account/session: pam_unix
for auth/account/session, pam_loginuid/pam_env, and nix pam_systemd to
register the logind session (XDG_RUNTIME_DIR, seat) the Wayland session
needs.
nixpkgs' pam_unix execs its verification helper from the fixed path
/run/wrappers/bin/unix_chkpwd; without it, pam_unix auth silently fails
even though the store binary works. Add a setuid unix_chkpwd security
wrapper (as NixOS does), which fixes pam_unix for both root callers
(greetd) and unprivileged ones (hyprlock).
Assisted-by: pi (claude-opus-4-8)
greetd 0.10's command tokenizer rejects the POSIX '\'' quoting that
lib.escapeShellArgs emits for the inline `bash --login -c 'exec "$@"' --`
session-wrapper argument, failing at config parse time with
'configuration error: illegal character'. greetd then crash-loops and
tty1 stays blank (login only worked from another tty).
Move the wrapper into a writeScript with a `bash --login` shebang and
pass its path to --session-wrapper. Same semantics (session runs in a
login shell so /etc/profile is sourced), but the command string no
longer contains nested shell quoting greetd can't parse.
Assisted-by: pi (claude-opus-4-8)
System Manager and the upstream Nix installer only ship POSIX
/etc/profile.d fragments to set PATH/XDG_DATA_DIRS (SM fish support is an
unfinished upstream TODO), so a bare fish shell misses /run/wrappers/bin,
/run/system-manager/sw/bin, the nix default profile, and
/run/system-manager/sw/share. Translate the two relevant fragments with
babelfish and source them; guard on the SM bin dir so it is a no-op when
the env was already inherited (e.g. the bash-wrapped graphical session).
Assisted-by: pi (claude-opus-4-8)
A prior System Manager activation can leave our /etc/pam.d/{greetd,hyprlock}
symlinks on disk without recording them in its etc-files state, so a
re-activation sees them as unmanaged and aborts. Set replaceExisting so it
backs the stale file up to <path>.system-manager-backup and relinks instead.
Scoped to the two files we declare: Ubuntu ships no greetd/hyprlock, and
System Manager's own generated entries (nix.conf already carries upstream
replaceExisting; profile.d, environment.d) don't need our intervention.
Assisted-by: pi (claude-opus-4-8)
The environment.d approach set XDG_DATA_DIRS in the user manager environment
(visible in show-environment) but not in time to affect its unit search path,
which systemd fixes when `systemd --user` is exec'd -- so home-manager's
xdg-desktop-portal units under share/systemd/user still were not found. Set it
on the user@.service process instead via an /etc drop-in, which does reach the
search path (and its user D-Bus and services), and drop the now-redundant
environment.d file. Takes effect on next login.
Assisted-by: pi (gpt-5.6-sol)
Ubuntu's logind starts the systemd user manager (and user D-Bus) without the
nix profiles in XDG_DATA_DIRS, so package-shipped units like the
xdg-desktop-portal services under share/systemd/user are never discovered.
targets.genericLinux only exports XDG_DATA_DIRS to login shells, too late for
the user manager. Set it through environment.d, which systemd reads at manager
start. Takes effect on next login.
Assisted-by: pi (gpt-5.6-sol)
Download each integrity tarball to a file with --fail, connect/max timeouts,
and retries instead of piping curl straight into openssl. A stalled connection
now times out and retries rather than hanging the FOD build forever, and a
failed fetch aborts instead of yielding an empty digest. Output is unchanged,
so npmDepsHash values stay the same.
Assisted-by: pi (gpt-5.6-sol)
pi-mcp-adapter does have integrity-less deps, so its FOD content changed with
the real-integrity fix. The old output was reused from the store (FOD paths
derive from name+hash), masking the mismatch on hosts with a warm cache; a
clean build (e.g. mgc) failed. Update to the deterministic hash.
Assisted-by: pi (gpt-5.6-sol)
Gabriel Fontes