The honest full-tree compat baseline (all 1148 integration suites through
the real in-process lane) and its analysis, preserved under
sandbox/reports/2026-07-08/ so a re-run can't clobber it:
- COMPAT.csv / COMPAT_LATEST.md — baseline at 9c0d69d8f3 (60.5%
test-level pass across the 763 engaged suites).
- COMPAT-postfix.csv / .md — re-sweep after the reauth/reconfigure fix
(402e7987bd): 60.5% -> 62.4% pass, errors -887.
- clusters/ — failure clustering (clusters.md + clusters.json) over the
--tb=line dumps.
- FINDINGS.md — the diagnosis: eight cross-cutting root-cause clusters
ranked by leverage, with the fix each suggests. Cluster 1 (reauth/
reconfigure) is the one already fixed; its writeback residual, the
SETUP_RETRY gap (333 integrations), and the snapshot refresh are the
next levers. runtime_data (50 integrations) is documented as an
inherent white-box divergence, not a bug.
Tooling: sandbox/cluster_failures.py (reusable clusterer, reads
$SANDBOX_ERRORS_DIR), run_compat.py gains --jobs parallelism, --tb=line
+ wide columns for full failure reasons, and the tagged-vs-engaged
'main'/'no_op' bucket split. reports/README.md documents the convention;
the generated report archive is excluded from codespell/prettier.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCotUYum6AoisyrxshoiJJ
The biggest single failure cluster in the compat baseline (~298 suites,
282 reauth + 176 reconfigure failures): a reauth / reconfigure / reset
flow calls ConfigFlow._get_reauth_entry() / _get_reconfigure_entry(),
which resolve the entry via async_get_known_entry on the flow's hass.
That flow runs in the sandbox, whose private hass has never seen the
entry main owns — so every such flow raised UnknownEntry on its first
step and the proxy aborted it as 'sandbox_flow_error'.
FlowInit gains an optional EntrySetup 'entry' field. When the flow
context references an entry_id main owns, the proxy serialises that
entry (shared entry_to_setup_proto builder, also now used by the
entry_setup payload) and the sandbox flow runner seeds a copy into its
private config_entries before async_init — so the reauth/reconfigure
entry lookups resolve. A plain user/discovery flow (no entry_id) is
unchanged.
This unblocks those flows from erroring on step one; the terminal
async_update_reload_and_abort still mutates the sandbox's private entry
copy rather than main's — that entry-writeback crossing is the next
lever (see reports/2026-07-08/FINDINGS.md), tracked separately.
Regression tests both sides: the proxy attaches main's entry to
FlowInit; the runner resolves _get_reconfigure_entry from the seeded
copy (stash-verified to fail without the seed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCotUYum6AoisyrxshoiJJ
setup.py's component-setup path (the path HA boot uses for every
existing entry, and the path async_setup_component-style tests use)
calls entry.async_setup_locked directly — it never went through
ConfigEntries.async_setup, so the router hook there was skipped and a
sandboxed entry would quietly run its integration locally on main at
every restart.
The router consult moves onto ConfigEntry.async_setup — the single
funnel every setup path uses (manager, boot/component setup, the
SETUP_RETRY timer) — so a routed entry can never slip into local setup;
the manager-level helpers are deleted. ConfigEntries.async_setup keeps
a shortcut for tagged entries so a routed custom (HACS) integration
with no code on main doesn't fail component setup before reaching the
router. Regression test drives async_setup_component end-to-end and
asserts the RPC crossed and local async_setup_entry never ran
(stash-verified to fail on the unfixed tree).
The compat lane now also reports how many entries were TAGGED next to
how many engaged, and run_compat classifies tagged==0 suites as 'main'
(camera/tts/system/ALWAYS_MAIN integrations legitimately measure
vanilla behavior) instead of lumping them into the suspicious no_op
bucket — no_op now precisely means 'tagged but never routed'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCotUYum6AoisyrxshoiJJ
Follow-ups from running the now-real compat lane against sun (8 -> 4
failures, and no more silent gaps):
- Sensor proxies rebuild native_value as datetime/date for
timestamp/date device classes — SensorEntity.state required a real
datetime and an ISO string killed the entity add ('str' object has no
attribute 'tzinfo'), leaving timestamp sensors unbridgeable.
- An integration's own-domain entities (sun.sun) now bridge: the bridge
falls back to a bare main-side EntityComponent when the domain's
component only exists inside the (sandboxed) integration setup —
restricted to the entry's own domain so a compromised sandbox cannot
mint entities in arbitrary made-up domains — and a GenericSandboxEntity
passes the pushed state + attributes through verbatim (typed domain
proxies keep their contract). The client attributes such entities to
the sole entry owning their domain when platform/registry linkage is
absent.
- Live core-config propagation: entry_setup's CoreConfig snapshot went
stale when the user changed the home location/units/language on main.
The bridge now pushes sandbox/core_config on EVENT_CORE_CONFIG_UPDATE;
the runtime applies it and re-fires the event on the private bus so
integrations recompute exactly as they would locally.
- The in-proc compat lane settles the whole sandbox round-trip inside
async_block_till_done (private hass + entity-bridge queue + channel
dispatch), giving vanilla tests their local synchronous semantics.
Both the settle loop and EntityBridge.async_drain are wall-clock
bounded: py-spy showed a clock-jump test's self-rescheduling entity
storm refilling the queue every tick, turning one teardown into 190s
of drain-spin (196s -> 12s for the sun suite).
- run_compat.py gains --jobs (thread pool of pytest subprocesses) so a
full-tree baseline doesn't take all night.
An earlier attempt forwarded tests.common.async_fire_time_changed to
the private hass; reverted within this change after it broke sun's
trigger tests — the in-proc lane shares one event loop, so main-hass
time firing already reaches the private hass's timers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCotUYum6AoisyrxshoiJJ
B5 leftovers: _owned_domains() is cached with dirty-flag invalidation
(proxy/platform/teardown hooks + SIGNAL_CONFIG_ENTRY_CHANGED) instead of
rescanning every config entry per inbound fire_event; StreamTransport
writes header+body without concatenating a full-frame copy;
sandbox_apply_state takes ownership of the freshly-decoded attributes
dict instead of re-copying it; climate's per-read inline import moved to
module top.
C3: bridge.py (1167 lines) sheds three verbatim regions — store.py
(SandboxStoreServer + quotas + key validation), service_forwarder.py
(forwarder build + error translation; the bridge's remember_context /
async_raw_call_service become declared interface, dropping the SLF001
pokes), description.py (SandboxEntityDescription + device-info
deserialise, dissolving the bridge<->entity type-import cycle). Down to
830 lines, no logic changes.
E3: EntrySetup now carries a typed CoreConfig snapshot
(lat/long/elevation/tz/unit-system/language/country/currency/location
name) filled from main's hass.config; the sandbox applies it before
async_setup (direct writes + async_set_time_zone — no Store persist, no
core-config event on the bare hass). Sandboxed integrations stop
computing sun times and unit conversions against HA defaults.
B4: ARCHITECTURE §5 documents the measured per-group boot cost (~0.9s /
~117MB; the upstream lazy-zone trim is tracked separately).
Also logs a newly-discovered compat gap in the plan doc: timestamp
sensor proxies push str state where SensorEntity.state needs datetime.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCotUYum6AoisyrxshoiJJ
C1 — the wire-type constants lived in protocol.py, hand-mirrored but
excluded from the drift guard and missing four types (ping + the three
flow ops appeared as string literals in six files). All MSG_* constants
now live in the guarded, mirrored messages.py, REGISTRY is keyed by
them, production literals are gone, and both protocol.py copies are
deleted. The drift guard (script + pre-commit files pattern) now also
byte-diffs the checked-in _proto gencode pair.
C5 — dead code, each re-verified before deletion: the unused
async_update_entry(sandbox=...) plumbing (the field is now in
FROZEN_CONFIG_ENTRY_ATTRS — stronger immutability, smaller core diff);
SandboxEntityDescription.device_id (write-only);
_parse_supports_response's unreachable branches;
_translate_remote_error's duplicate branches; SandboxData.channels (no
production reader — consumers use bridges[group].channel);
drain_inflight's dead reader-task filter; Channel.from_transport
(test-only duplicate of __init__(transport=...)); _serialize_body's
None branch; uint32 wrap for _next_id on a months-lived channel;
defensive getattr on always-present dataclass fields.
E6 — sources.py: the global fetch lock serialized downloads of
different repos and _TARBALL_CACHE pinned whole tarballs for process
lifetime; per-key single-flight tasks + a completed-key set replace
both. service_mirror does one services_for_domain lookup per
registration instead of two. event_mirror's docstring described an
event_filter fast path that cannot exist (core filters receive only
event.data, and data-less events skip filtered listeners) — the
docstring now tells the truth instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCotUYum6AoisyrxshoiJJ
_build_registry() ran at module import time despite its docstring
promising laziness — the first register_entity RPC imported all 31
domain proxy modules (each pulling its whole component package) on
main's event loop: measured ~384 ms / +67 MB cold. proxy_class_for()
now imports just the requested domain's module on first use (memoized;
introspects the module's single SandboxProxyEntity subclass, so the
irregular names like SandboxDateTimeEntity need no table), and the
bridge warms it via the import executor so the one-time import doesn't
block the loop either. Measured: 27 ms / +7 MB for the first light
entity, 0.7 us memoized, unknown domains fall back to the generic
proxy. Deletes the 75-line hand-rolled registry.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCotUYum6AoisyrxshoiJJ
Channel (both mirrors): register_push_inline() runs a sync push handler
directly in the read loop — no task, no semaphore, nothing queued (so it
also bypasses the max_queued shed). Per-entity in-order delivery becomes
a guarantee instead of an accident of FIFO task scheduling. Calls stay
on the async task path (they must write replies); the overload shed now
logs dropped pushes instead of silently returning. Main's
_handle_state_changed and _handle_fire_event have no awaits — both are
now sync callbacks registered inline. Measured push throughput 105k ->
156k msg/s over a socketpair (and ~55k -> 156k cumulative with the
orjson wire change).
EntityBridge: the task-per-state-change model (asyncio.create_task per
EVENT_STATE_CHANGED) is replaced by a per-entity latest-state slot
drained by one writer task. Rapid bursts coalesce to the newest state
(test: 5 events -> 1 push), removal/register races are ordered by
construction (a _writing marker keeps a removal-during-register from
being misclassified), and the last update in a burst can no longer be
shed into a permanently stale proxy. Deletes the _removed_while_pending
/ _state_differs reconciliation machinery this replaces (deferred
plan-review-simplification Phase 6, now justified by measurements).
Also (plan §E5): entities whose _describe fails are skipped stickily
(cleared on their entity-registry update) instead of re-attempting a
register on every state write, and the unregister path pops _last_hash.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCotUYum6AoisyrxshoiJJ
Every google.protobuf.Struct / ListValue field on the wire becomes a
bytes field holding orjson-encoded JSON (same names + numbers; fields
whose None-vs-{} presence is load-bearing stay optional). One mirrored
encoder/decoder pair (encode_json / decode_json[_dict] in messages.py,
built on HA's json_encoder_default with a str fallback) replaces
struct_to_dict / dict_to_struct / _value_to_py / listvalue helpers and
every scattered pre-coercion (_wire_safe, flow_runner._to_json_safe,
entry_runner._json_safe).
Why (measured, plan-review-overhead §B1):
- state_changed serialize round-trip 26.3us -> 4.1us (6.5x), frames
~20% smaller; Struct.update alone cost 40x orjson.dumps.
- Struct stores every number as a double: the is_integer() restoration
hack corrupted genuine floats (2.0 -> 2) and silently lost precision
above 2^53. JSON bytes preserve int/float natively — the whole
"Numbers note" class of workarounds is deleted, with a fidelity test
(2.0 stays float, 255 stays int, 2**53+1 exact).
- The store/restore-state paths stop quadruple-converting: the
prepare_save_json bytes now ride the wire directly and main decodes
once before its unchanged validation/quota/write logic.
- Producers coerce by construction (datetime/enum/set attributes can't
crash a push), so the A1 hot-fix json_safe wrapping folds into the
encoder.
Both _pb2 mirrors regenerated; wire-module drift guard green; both
suites green; sun compat probe failure set unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCotUYum6AoisyrxshoiJJ
Fixes the correctness findings from plan-review-overhead §A (+E4), each
with a regression test:
- A2+A5: core now owns entry state on both router paths. The
ConfigEntryRouter contract changes to raise ConfigEntryError on
failure; core maps it to SETUP_ERROR (setup) or leaves the entry
state untouched (refused unload — previously a refused unload was
wrongly marked NOT_LOADED while the remote entry stayed loaded).
Every SLF001 state poke moves out of router.py into
config_entries.py, which owns ConfigEntry.
- A4: the router setup/unload branches now honor entry.setup_lock, so
a direct async_setup during a reload can no longer run two
entry_setup RPCs concurrently for one entry.
- A3: SandboxBridge.async_teardown sweeps _mirrored_services off
hass.services — a respawned sandbox's re-registration was skipped by
the has_service guard, leaving every call on a dead channel forever.
- A6: the sandbox translation overlay merges inside
_async_get_component_strings before the title fallback, so a provider
dict without "title" gets the catalog/integration.name fill-in;
_TranslationCache._async_load returns to dev-identical.
- A7: proxy init + upsert share one _apply_description with symmetric
clearing — a dropped device_class/device_info no longer sticks.
- A8: the frame-size cap is enforced on write too (both channel
mirrors); an oversize payload fails its own call with
FrameTooLargeError (oversize replies come back as error frames)
instead of making the peer abort the whole channel.
- A1: state-attribute pushes run through json_safe — a datetime
attribute raised ValueError inside the fire-and-forget push task and
left main's proxy permanently stale. Main-side service_data/query
args get the same coercion (_wire_safe) before dict_to_struct.
- E4: FlowRunner.async_stop now stops the private hass (executor +
timer leak per in-process sandbox), and InProcessSandbox.stop runs
the graceful sandbox/shutdown first, like production.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCotUYum6AoisyrxshoiJJ
Two critical review findings (plan-review-overhead E1+E2), each the test
that would have caught the other:
E1 — FlowRunner.create built a bare HomeAssistant and never ran the
registry loads bootstrap does, so er.async_get returned an unloaded
EntityRegistry with no .entities and every real EntityPlatform add died
with AttributeError while entry_setup still ACKed ok — the sandbox
bridged zero entities for real integrations. Load the
area/category/device/entity/floor/issue/label registries in create(),
before the channel opens, so their Stores bind to the local tempdir.
Regression-tested by driving the real sun integration through
EntryRunner end-to-end.
E2 — both compat pytest plugins only installed the entry autotag; the
sandbox_inprocess/sandbox_subprocess fixtures were opt-in and no vanilla
test requests them, so hass.config_entries.router stayed None and every
tagged entry set up locally: COMPAT.md's 99.97% measured vanilla tests,
not sandbox compatibility. The plugins now inject their sandbox fixture
into every hass-using test at collection, count router-driven
entry_setups via a test-side wrapper, and report the count in the
terminal summary; run_compat.py records the count per integration,
marks green-but-never-engaged rows no_op, and fails the run outright if
nothing engaged. Probe: sun now runs 10 routed entry setups, 98/101
passing (3 real compat gaps for the lane to report).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCotUYum6AoisyrxshoiJJ
Upstream dev now enables mypy's explicit-override error code across
homeassistant/ (#174488); add the 267 missing @override decorators in
the sandbox component. No mirrored wire module is affected, so the
drift guard stays green.
Also adds sandbox/plans/plan-review-overhead.md — the 2026-07-07
perf/simplification review findings + measurements whose execution
order drives the following commits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCotUYum6AoisyrxshoiJJ
Add PRESENTATION.md Slide 11 enumerating the five core-HA seams (from
ARCHITECTURE.md §12) and the no-monkey-patching discipline. Update §12 +
the slide to include the inverse `async_unregister_remote_platform` hook
that the crash-recovery plan added alongside `async_register_remote_platform`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the three drifted client coercers — event_mirror._to_json_safe
(raw set order), entity_bridge._serialise (sorted sets via _iter), and
entry_runner._json_safe — with a single hass_client._json.json_safe built
on HA's json_encoder_default (the as_dict/set/enum-aware single source of
truth), plus a str() fallback so best-effort event data still degrades a
single odd field to a string instead of raising in the bus callback.
entry_runner keeps its empty-result dict guard and delegates the coercion;
the capabilities hash stays stable (compared within one process, so raw
list(set) order suffices). Drops the now-unused Iterable/json_bytes/
json_loads imports. Adds tests/test_json.py round-tripping sets, enums,
as_dict objects, datetimes, the str() fallback, and non-str keys.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Channel.__init__ / from_transport no longer default `codec` to JsonCodec —
it is now a required keyword. Every production construction site already
passes ProtobufCodec; a forgotten codec is now a construction-time error
instead of silently speaking JSON at a protobuf peer.
JsonCodec leaves the production channel.py (both mirrors) and moves to each
side's test helpers: tests/components/sandbox/_helpers.py (HA) and
hass_client/testing/_jsoncodec.py (client). Channel-core tests import it
from there; the two channel.py mirrors stay byte-identical. The now-unused
`import json` is dropped from channel.py and the protocol.py docstring is
updated for the required codec + relocated JSON codec.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the callerless raise_not_proxied helper (zero callers since the
query RPCs landed), its __all__ entry, and the now-unused NoReturn /
HomeAssistantError imports.
Remove the vestigial `if old_state is not None and entity_id not in
self._pending: pass` no-op block in the client entity_bridge — control
fell through it unchanged — and the now-dead old_state assignment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a `_features_flag: type[IntFlag] | None` class-attr hook on
SandboxProxyEntity and a `_coerce_supported_features` helper that wraps
`description.supported_features` in the domain's IntFlag once, used by both
`__init__` and `sandbox_update_description`. Replace the 17 identical
~10-line `__init__` overrides (light, fan, lock, cover, climate,
media_player, notify, …) with a single `_features_flag = <Domain>EntityFeature`
line each, dropping the now-unused TYPE_CHECKING imports.
The four @final mangled-attribute `sandbox_apply_state` overrides
(button, event, notify, scene) are untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make channel.py byte-identical across both mirrors (it already matched
modulo docstrings/comments/log capitalization; codec_protobuf.py and
messages.py were already identical). Add check_mirror_drift.sh asserting
all three pairs are byte-identical, wired as a regular prek hook that
fires whenever either copy of a mirrored file changes. Document the
edit-both rule in-file next to the guard.
This retires the ad-hoc 'apply to both mirrors' discipline the earlier
review-follow-up plans had to carry by hand.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plan-4 commits accidentally added generated sandbox store fixtures
(tests/testing_config/.storage/sandbox/built-in/*) written by the test
runtime. origin tracks none of testing_config/.storage; the prior
c3f0abc53cf 'stop tracking generated test .storage' untracked a sibling
file but added no ignore rule, so it recurred. Remove these and add a
targeted .gitignore so the test store dir can't be re-committed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Discovery flows feed the proxy non-JSON objects — a *ServiceInfo dataclass
(IPv4Address fields, sets) as the first-step user_input and a DiscoveryKey in
context — which dict_to_struct couldn't hold, crashing the flow unhandled
(the except only caught channel errors). The router routes discovery-sourced
flows to the sandbox with no source filter, so this was reachable.
Main side: _to_jsonable walks context + first-step payload into Struct-safe
primitives (dataclasses -> field dicts, IPs -> str, sets -> lists) before
dict_to_struct. Broadened the except to abort cleanly on any unmapped payload
that still trips the marshaller.
Sandbox side: _rehydrate_discovery rebuilds the real DiscoveryKey + the
source's BaseServiceInfo (zeroconf/homekit, ssdp, dhcp, usb, hassio, mqtt) so
async_step_<source> receives the type it expects. Unmapped sources (bluetooth)
leave a dict; reconstruction failures degrade to the dict with the proxy's
clean abort as the outer backstop.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two fixes in the flow proxy:
1. MENU support end-to-end. async_show_menu is common and marshals like a
form. Added FlowResult.menu_options (ListValue) + sort to the proto, marshal
it sandbox-side, and re-issue async_show_menu on main. A dict-form menu
(id -> label) crosses as ordered [id, label] pairs so labels survive; a
menu selection is forwarded as the sandbox flow's {"next_step_id": <chosen>}
navigation choice.
2. Leak fix. The unsupported-result-type branch set _terminated=True before
aborting, which made async_remove skip the flow_abort RPC — leaving the
sandbox-side flow in progress and wedging retries on already_in_progress if
it had set a unique_id. The non-terminal result types (external-step,
progress) no longer set _terminated, so async_remove reaps the sandbox flow.
Regenerated proto gencode for both mirrors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The proxy CREATE_ENTRY path read only data/title/description and called
async_create_entry, which stamps the proxy class defaults VERSION=1 /
MINOR_VERSION=1 / options={}. A sandboxed flow with VERSION>1 lost its
schema version (spurious migration on next setup) and dropped options.
async_create_entry reads self.VERSION/self.MINOR_VERSION off the *instance*
(config_entries.py async_create_entry), so override the proxy instance's
values from the wire result before the call, and plumb options= through.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records the 7-phase client-runtime + wire-fidelity batch: per-phase
summaries, the messages.py both-mirrors byte-identical note, the Phase 7
no-narrowing decision, the Phase 3 correctness-fix approach + plan-5
writer-queue handoff, tests added, and final green verification.
_handle_shutdown set the shutdown event via call_soon so the reply lands
on the wire first, but call_soon only buys one loop turn — if the reply
write suspended (write-lock contention from unload pushes, drain
backpressure on a large restore_state), run()'s finally closed the channel
and cancelled the in-flight reply task, so main lost restore_state.
Add Channel.drain_inflight(timeout): wait for in-flight inbound handler
tasks (the shutdown reply included) to finish before close, without
cancelling anything. run()'s finally now drains before close(), so the
reply completes its write instead of being cancelled mid-flush.
Write-lock check (Phase 4 bullet 2): _write_lock is a plain mutex held
only across one frame write, so the shutdown reply and the unload-driven
_push_unregister writes serialize — no circular wait, no deadlock.
drain_inflight added to both hand-mirrored channel.py copies. Regression
test stalls the reply's drain and asserts main still receives it (verified
it fails with ChannelClosedError when the drain step is removed).
protobuf.Struct stores all numbers as double, so _value_to_py returned
entry.data ints, service_data ints, and the store envelope's version/
minor_version as floats — breaking socket()/isinstance(int)/range(...).
_value_to_py now coerces whole-number floats back to int
(int(v) if v.is_integer() else v); genuinely fractional values (0.5) keep
their float type. Reviewed every struct_to_dict / listvalue_to_list
consumer (flow/entry data, service_data, target, entity_query args/result,
store data, state attributes, capabilities, event_data, translations) —
none needs a whole-number float to stay float, so the global coercion is
safe; no field narrowing required.
Applied byte-identically to BOTH hand-mirrored copies:
homeassistant/components/sandbox/messages.py
sandbox/hass_client/hass_client/messages.py
(verified identical via diff). The explicit int32 proto fields (version,
minor_version, supported_features on EntrySetup) bypass the Struct and are
unaffected.
A failed async_setup left the rebuilt ConfigEntry in the sandbox's
config_entries (only unload popped it), so main's later retry of the same
entry_id was rejected with 'entry already loaded'. On both failure paths
(async_setup raised / returned False) the entry is now popped before
returning ok=False, so a re-sent entry_setup starts clean.
Sandbox-side half of plan 1's SETUP_RETRY decision: main shipped honest
SETUP_ERROR + manual reload and remains the only retry driver. The
sandbox-side ConfigEntryNotReady timer is deliberately NOT enabled (the
sandbox hass is never async_started).
EntityBridge._register added an approval refcount per registered entity
(approved.add(domain)) but nothing ever decremented it, so a platform
domain stayed approved for the process lifetime — the service/event gate
stayed open for domains with zero owning entities.
Track each entity's contributed domain (_approved_domain) and release it
(_release_approval -> approved.remove) on both unregister paths: the
new_state-is-None removal and the Phase 3 removal-while-pending flush.
Symmetric with the per-entity add; the refcount drops to absent when the
last entity of a domain unregisters. EntryRunner's per-entry
approved.remove(entry.domain) on entry_unload already balances the
per-entry add, so it is unchanged.
_on_state_changed dropped any state_changed while an entity sat in
_pending (register RPC in flight), and the register task pushed only the
snapshot captured at task-creation. A fast second update was lost; a
removal in the window left a ghost proxy on main (the entity wasn't in
_registered yet, so the removal was dropped).
After the register RPC resolves, the task now:
- if a removal raced (new _removed_while_pending flag), unregisters the
entity it just registered, so main keeps no ghost proxy;
- otherwise re-reads hass.states and pushes a state_changed when the live
state differs from the registered snapshot, flushing the coalesced gap.
This is the correctness fix only. Plan 5 (simplification) builds a
single-writer queue on top of the entity push path; when it lands it
should subsume this flush into the queue's ordering guarantees (noted in
code).
SandboxRuntime.run pushed Ready (manager flips to running, router sends
entry_setup), awaited the warm-load store_load, and only then registered
MSG_ENTRY_SETUP et al. An entry_setup arriving in that window hit
ChannelUnknownType -> SETUP_ERROR.
Now: register every inbound handler first, run the (outbound) warm-load
store_load, then push Ready as the last frame. This both removes the
no-handler race and preserves the warm-load-before-entry_setup invariant
(Ready timing still gates entry_setup until the restore cache is warm).
EVENT_SERVICE_REGISTERED fires synchronously while a service is
registered inside async_setup_entry, but EntryRunner only approves the
domain after async_setup returns, so ServiceMirror dropped those early
registrations with a warning and never replayed them.
ApprovedDomains now fires approve-listeners on the first (absent->present)
add; ServiceMirror subscribes async_sync_domain, which re-mirrors every
already-registered service of the freshly-approved domain (skipping any
already in _mirrored). Covers both the entry-runner approve path and the
entity-bridge per-entity approve path.
EventMirror is left as-is: owned events are transient, so a past event
cannot be replayed (noted in plan Phase 1).
All trust-boundary gates this plan proposed shipped (Phases 1-6, none
deferred), so the malicious-sandbox guarantees the architecture asserts now
match enforced reality. Added a one-line 'enforced on main in
bridge.py/channel.py' note for each gate, traceable to code:
* §4 channel read-backpressure shedding (both mirrors)
* §8 register_entity entry ownership + foreign-device-merge refusal
* §8 register_service / fire_event owned-domain gates + core deny-list
* §8 context-cache eviction bound on the resolve path
* §9 store key-length + value/total/key-count quotas
* §11 translation overlay narrowed to requested ∩ returned
No 'Known trust-boundary gaps' subsection is needed — nothing was deferred.
Added a changelog row. README.md/CLAUDE.md make no overstated boundary
claims (only 'isolated subprocesses', accurate), so no softening needed.
One forged-frame test per trust-boundary gate:
* fire_event: core (homeassistant_stop/call_service/state_changed) and
unowned-domain (zha_event/hue_event) events are dropped, never reach the bus.
* register_service: an unowned domain (persistent_notification) is rejected.
* register_entity: a foreign entry_id (entry.sandbox != group) is rejected;
a device_info colliding with a foreign entry's device is refused (no merge).
* translation: a forged foreign domain returned alongside the owned one is
dropped; only the requested ∩ returned survives.
* store_save: an overlong key and an oversized value are rejected; nothing
hits disk.
* context cache: a flood of distinct unknown context_ids stays bounded by
_CONTEXT_CACHE_MAX.
* channel backpressure: over the max_queued cap, inbound calls are shed with
a ChannelOverloaded error and the inflight set stops growing.
make_channel_pair gained max_queued_a/b passthrough for the backpressure test.
Two unbounded-growth vectors closed:
1. Context cache on resolve. _resolve_context minted a fresh Context per
unknown context_id but never enforced _CONTEXT_CACHE_MAX (only
_remember_context did), so a sandbox flooding distinct unknown ids grew
the cache without bound. Factor a single _store_context() helper used by
both paths so the cap + expiry-ordering apply uniformly.
2. Channel read backpressure (BOTH mirrors). The reader create_task'd a
handler per inbound frame; the inflight semaphore caps *running*
handlers but queued tasks — each pinning a decoded payload up to
MAX_FRAME_SIZE — grew without bound under a flood. _dispatch now sheds
over a DEFAULT_MAX_QUEUED cap on inflight handler tasks: inbound calls
are rejected with a ChannelOverloaded error frame, pushes dropped.
Responses are always handled inline above the gate, so backpressure
never starves a reply.
The channel.py edit is applied byte-identically to both hand-mirrored
copies (homeassistant/components/sandbox/channel.py and
sandbox/hass_client/hass_client/channel.py), rebased on top of plan
#1's Channel.close() fix, in this separate commit.
Design note: shed (reject/drop over a bounded cap) rather than block the
reader on the semaphore. Blocking the shared reader would deadlock the
documented nested-call pattern — a handler that issues channel.call() and
awaits its reply through the same reader would stall it once all slots are
held by such handlers (real on the client mirror: a call_service handler
doing a store_save round-trip to main). Shedding bounds memory without
that liveness hazard and stays safe in both mirrors.
_validate_key now caps key length (128, well under NAME_MAX). async_save
caps each value (4 MB) and enforces a per-group dir quota (32 MB total,
256 keys) via _enforce_group_quota before the atomic write, so a
compromised sandbox can no longer exhaust the host disk through the
store-routing channel. Limits are commented, generous-but-finite
constants.
Over-quota writes raise HomeAssistantError → remote-error frame; the
sandbox-side async_store_save already catches ChannelRemoteError and logs
(keeping its in-memory data), so a rejected flush degrades, not crashes.
The provider overlay (_TranslationCache._async_overlay_sandbox_strings)
splices every domain a sandbox returns, narrowed only by the broad
requested components set — so a compromised sandbox could return strings
for a co-requested victim domain (hue, http) and poison its frontend
strings. async_get_translations now keeps only the requested ∩ returned
intersection (domains & strings.keys()), the set this group was actually
asked to resolve, before handing strings to the cache.
_handle_register_entity now requires entry.sandbox == self.group, not
just that the entry_id resolves: a compromised sandbox may only register
entities for entries main routed to *this* group. Without this it could
attach entities and pre-create devices against a victim integration's
config entry.
Also reject a device pre-create that would merge (via shared
identifiers/connections) into a device already owned by a config entry
outside this group — _reject_foreign_device_merge — closing the
device-registry hijack vector. entry.sandbox is set by main at flow
completion, never by the sandbox.
entry_setup/entry_unload are main-initiated (main supplies the entry_id),
and the store server is scoped one-dir-per-channel by construction, so no
further entry_id trust points need gating.
Test entry fixtures across bridge/entity_query/domain_proxies/
crash_recovery/proto_transport updated to tag their entries sandbox="built-in".
_handle_register_service now requires the service domain to be one this
group owns (same main-side _owned_domains() derivation as the fire_event
gate), so a compromised sandbox can no longer squat
persistent_notification.* or any unclaimed domain.service. Unowned
domains are rejected with a HomeAssistantError → remote-error frame; the
existing refuse-to-clobber-an-existing-handler check is kept.
Existing register_service tests updated to own their mock domains via a
MockConfigEntry(sandbox="built-in").
Enforce on main the same <owned_domain>_ rule the sandbox claimed to:
_handle_fire_event now drops any event that is not in the <domain>_
namespace of a domain this group owns, plus a hard deny-list of core
control-plane events (homeassistant_*, call_service, state_changed, ...)
so an owned domain can never alias a core event.
Owned-domain trust is derived purely from main-side state via the new
_owned_domains() helper (entries with entry.sandbox == self.group plus
registered proxy platform domains) — never from a sandbox-supplied
identifier. The helper is the linchpin reused by the register_service
gate in the next phase.
Drops (never raises) on a forged push, so the dispatch loop is unaffected.
Documents all six phases shipped, the /phx:work→intent deviation, the
SETUP_RETRY→SETUP_ERROR fallback (true retry not feasible from the router
seam), the new EntityComponent.async_unregister_remote_platform public
hook, the channel.py both-mirrors note, test coverage, and final
verification (224 + 87 + 30 passed; drift clean; prek clean).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Guards every fix in the cluster:
- test_crash_recovery.py (new): crash→respawn→re-register (Phase 1),
sandbox-dies-availability + recovery (Phase 2), unload-while-down
releases the platform (Phase 4).
- test_manager.py: stop()-during-spawn completes without hang (Phase 3) —
a _PausingProcess lands stop() while _spawn is suspended so
self._process is still None.
- test_channel.py: Channel.close() after EOF still closes the transport
and awaits inflight exactly once, and is idempotent (Phase 5).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Channel.close() early-returned on `if self._closed: return`. But the read
loop's EOF `finally` already sets `_closed=True` (and cancels, never
awaits, inflight tasks), so a close() after EOF returned immediately —
transport.close() and the inflight gather never ran, leaking the stdin
pipe / unix connection every restart cycle. Split "already closed" (set
_closed, fail pending — idempotent) from "teardown not yet done" (close
transport + await inflight, guarded by a new _close_done flag that runs
exactly once regardless of who set _closed first).
channel.py is hand-mirrored — the identical fix is applied to BOTH copies
(homeassistant/components/sandbox/channel.py and
sandbox/hass_client/hass_client/channel.py).
SETUP_RETRY non-retry: the router runs outside ConfigEntry.async_setup,
so the SETUP_RETRY timer (async_call_later) is never armed for a sandbox
entry — a router-set SETUP_RETRY wedged the entry in a retry state that
never fires (and a later async_setup raised OperationNotAllowed). The
ChannelClosedError-during-entry_setup case now reports SETUP_ERROR
honestly (recoverable via manual reload); ARCHITECTURE.md §5 updated and
a router-driven true retry flagged as a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
router.async_unload_entry returned True without any cleanup when the
sandbox/channel was down, leaking the proxy entities and the
EntityComponent platform registration → a later re-setup hit "has
already been setup!".
Extract a shared _async_unload_main_side helper (delegates to
bridge.async_unload_entry, which now uses the Phase 1 public-hook
teardown) and call it on every exit that should release main-side state:
the sandbox-down early return and a ChannelClosedError mid-unload both
skip the (impossible) remote RPC but still remove the proxies + platform.
A live sandbox that refuses the unload (ChannelRemoteError) still returns
False with the proxies left in place.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two paths could hang HA shutdown or wedge a sandbox forever:
- stop() spawn-in-progress race: a stop() that landed inside _spawn read
self._process as None, terminated nothing, then awaited the supervisor
forever while the freshly-spawned healthy child ran unsupervised. Fix:
a post-spawn _stopping check in _run_one_{stdio,unix} kills the child
stop() missed, and stop()'s await is bounded with a SIGKILL+cancel
backstop. Terminate logic is extracted into a shared _terminate helper.
- Respawn had no ready-timeout: ready_timeout was only applied in the
first start(). _supervise_until_exit now bounds the ready handshake on
every attempt; a child that opens its channel but never signals ready
is killed and counts against the restart budget instead of leaving the
sandbox 'starting' forever.
Also: ensure_started no longer hands back a 'starting' zombie — it awaits
readiness (bounded) via the new async_wait_until_ready and raises
SandboxFailedError if the in-flight spawn never becomes running.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sandbox_set_available() had zero callers, so a dead sandbox's proxies
kept serving their last state — a crashed integration's light read "on"
forever to automations and the UI.
- Add SandboxBridge.async_mark_all_unavailable() (flips every owned proxy
to unavailable via the existing sandbox_set_available).
- Add a manager on_channel_closed callback fired right after the control
channel is closed on process exit; __init__ wires it to mark the
group's live bridge unavailable. Proxies flip back to available on
respawn through the normal register/state_changed round-trip.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Keystone fix for crash/restart recovery: a respawn used to overwrite
data.bridges with a fresh SandboxBridge but never released the old one's
proxy entities or its EntityComponent platform slots, so the first
register_entity after respawn raised ValueError("… has already been
setup!") and every entity for the entry failed permanently.
- Add the public inverse hook EntityComponent.async_unregister_remote_platform
(mirrors async_register_remote_platform; no private _platforms poke —
the bridge's old SLF001 poke in async_unload_entry is replaced by it).
- Add SandboxBridge.async_teardown() + a shared _async_teardown_entry
helper; async_unload_entry now routes through it.
- __init__: on restart, stash the displaced bridge in
SandboxData.pending_teardown and, when the fresh process goes ready,
tear it down and re-drive entry_setup (async_schedule_reload) for the
group's loaded entries. A new manager on_ready callback is the trigger
(fires on every (re)spawn after MSG_READY); capturing loaded entries
synchronously keeps a first start from being treated as a respawn.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Capture the 2026-06-12 sandbox code review as actionable artifacts:
- plans/plan-review-INDEX.md + five phased fix plans covering every verified
finding — crash/restart recovery, trust-boundary hardening, client-side
races + int->float wire fidelity, config-flow forwarding fidelity, and
simplification/dedup. Index records execution order and a finding->plan
coverage map; plans cross-reference each other by name.
- PRESENTATION.md — a 10-slide intro to the sandbox concept (what data
represents an integration in another instance -> entity/registry/device,
service control, event listening, action forwarding, entity RPC for
query-shaped APIs, then sandboxes as ephemeral one-domain isolates).
The documentation findings from the review were fixed directly (prior commit);
the remaining code findings are planned here, not yet implemented.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
