From d049ab23f953cfbc8a79de76441b321601cd1354 Mon Sep 17 00:00:00 2001 From: Paulus Schoutsen Date: Fri, 5 Jun 2026 07:04:08 -0400 Subject: [PATCH] sandbox/docs: mention translations in the unified view; drop architecture.html MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Translation forwarding (live pull-RPC + catalog provider) now puts the sandboxed integration's translations on main alongside its entities, services, and events — note it in the OVERVIEW + ARCHITECTURE goals. Remove the generated architecture.html; the architecture is published to a gist instead of carrying a rendered artifact in the tree. Co-Authored-By: Claude Opus 4.8 (1M context) --- sandbox/ARCHITECTURE.md | 4 +- sandbox/OVERVIEW.md | 4 +- sandbox/architecture.html | 2780 ------------------------------------- 3 files changed, 4 insertions(+), 2784 deletions(-) delete mode 100644 sandbox/architecture.html diff --git a/sandbox/ARCHITECTURE.md b/sandbox/ARCHITECTURE.md index 3f0b8bf4b1f0..e91c5f9f60c7 100644 --- a/sandbox/ARCHITECTURE.md +++ b/sandbox/ARCHITECTURE.md @@ -19,8 +19,8 @@ running everything locally. A user who adds a light integration through the frontend ends up with a device plus entities in main's registries, working area targeting (`light.turn_on` against an area resolves the sandboxed lights like any other light), and the -integration's services and events available on main — with the integration code -only ever executing inside the sandbox. +integration's services, events, and translations available on main — with the +integration code only ever executing inside the sandbox. The sandbox is **stateless**: it holds no persistent state of its own. Its storage and restore-state route to main (§9), and even the integration's *code* diff --git a/sandbox/OVERVIEW.md b/sandbox/OVERVIEW.md index 5cbc80a7c19a..ef8c14943d33 100644 --- a/sandbox/OVERVIEW.md +++ b/sandbox/OVERVIEW.md @@ -27,8 +27,8 @@ Run a Home Assistant integration's setup, config flow, entities, services, and events fully inside an **isolated subprocess** ("sandbox"), while the main HA instance keeps a **single, unified view** of devices, -entities, services, and events that looks identical to running -everything locally. +entities, services, events, and translations that looks identical to +running everything locally. A user adding a light integration through the frontend should end up with a device + entities in the main instance's registries, area diff --git a/sandbox/architecture.html b/sandbox/architecture.html deleted file mode 100644 index e217727b85de..000000000000 --- a/sandbox/architecture.html +++ /dev/null @@ -1,2780 +0,0 @@ - - - - - - Home Assistant Sandbox — Architecture - - - -
-
-

Home Assistant Sandbox

-

- A backend rewrite that runs each integration's setup, config flow, - entities, services, and events inside an isolated subprocess — while - main keeps a single, unified view of devices, entities, services, and - events that looks identical to running everything locally. -

-

- Architectural overview · Phases 0–17 landed · - 99.67 % compat-lane pass rate -

-
- -
-
-
807
-
Integrations swept
-
-
-
711
-
Pass cleanly
-
-
-
99.67 %
-
- Tests passing (34 266 / 34 378) -
-
-
-
3
-
HA Core files touched
-
-
-
3
-
Sandbox groups out of the box
-
-
- -
- Contents -
    -
  1. - The goal — what user-facing problem this solves -
    2. -
  2. The high-level shape
    3. -
  3. - The classifier — how routing decisions get made -
    4. -
  4. - Lifecycle — spawn, supervise, shut down -
    5. -
  5. - The channel — protobuf wire over pluggable transports -
    6. -
  6. Config-flow forwarding
    7. -
  7. - The entity bridge (Option B) -
    8. -
  8. Service & event mirroring
    9. -
  9. Sandbox auth & opt-in data sharing
    10. -
  10. - Store routing — current_sandbox -
    11. -
  11. - Test infrastructure & the compat lane -
    12. -
  12. The phase timeline
    13. -
  13. Open follow-ups
    14. -
  14. Where to look in the code
    15. -
-
- - -

- 1. The goal — what user-facing problem this solves -

- -

- Home Assistant runs a large universe of integrations — built-ins - for first-party platforms (Z-Wave, Hue, MQTT…), plus a long tail - of custom integrations installed via HACS. They all share a single - Python process. That means one misbehaving integration can leak memory - into the host, mishandle exceptions and bring down the event loop, - monkey-patch shared internals, or read state from unrelated - integrations. -

- -

- Sandbox puts each integration's code in an isolated subprocess (a - "sandbox"). From the user's perspective, nothing changes: they add a - light integration through the frontend, devices and entities show up in - the registries, area targeting works (light.turn_on against - an area resolves the sandboxed lights like any other light), the - integration's services and events are available on main — with the - integration's code only ever running inside the subprocess. -

- -
- The thesis. Routing is computed at runtime from the - integration's manifest and platform list. There's no user-facing config - knob, no manifest annotation, no per-integration migration. From the - perspective of a user, the frontend, and a downstream integration like - automation, the system looks exactly the same as if - everything ran locally. -
- -
- The Iron Law. No monkey-patching of private internals. - Anywhere the bridge needs to reach into HA Core's machinery, it does so - through a public, deliberately-added hook — - EntityComponent.async_register_remote_platform, - ConfigEntries.router, ConfigEntry.sandbox, - current_sandbox. The PR diff is slightly larger; the - maintenance story is dramatically smaller. -
- - -

2. The high-level shape

- -

- Main holds the canonical view of the world. Each sandbox group is a - separate Python process spawned lazily on first need. The two sides - communicate over a protobuf channel that rides a pluggable transport - (stdio by default, unix socket opt-in). Inside the sandbox, a private - HomeAssistant instance hosts the integration's real - async_setup_entry, ConfigFlow, entities, - services, and events. -

- -
- - - - - HOME ASSISTANT CORE (main process) - - - - - - Entity / Device / Area - - - Registries (canonical) - - - - - State machine - - - hass.states (canonical) - - - - - - components/sandbox - - - - - - SandboxFlowRouter - - - plugged into hass.config_ - - - entries.router; routes - - - flows + entry setup + unload - - - - - - SandboxManager - - - dict[group, SandboxProcess] - - - lazy spawn per group; - - - restart-on-crash w/ budget - - - - - - classify(integration) - - - pure function → group - - - - - - SandboxBridge (per group) - - - proxy-entity registry — light, switch, sensor, … - - - _CallServiceBatcher — coalesces per loop tick - - - service / event re-fire on main's bus - - - _SandboxStoreServer pinned to .storage/sandbox/<group>/ - - - - - - websocket dispatcher - - - no scope enforcement - - - deferred until the - - - WS transport lands - - - - - - sandbox/auth.py - - - System user per group - - - plain system-user - - - access token - - - - - - ConfigEntry.sandbox: str | None — routing tag (Phase 17) - - - set via ConfigFlowResult["sandbox"] at construction; persisted in - core.config_entries - - - - - - stdio protobuf - - - Channel - - - - - - - - - - - - SANDBOX SUBPROCESS (per group) - - - python -m hass_client.sandbox --name … --url … --token - … - - - - - - SandboxRuntime - - - private HomeAssistant instance - - - current_sandbox.set(bridge) — routes Store IO to main - - - - - - FlowRunner - - - drives integration's - - - ConfigFlow inside sandbox - - - - - - EntryRunner - - - runs async_setup_entry - - - against the sandbox's hass - - - - - - EntityBridge - - - pushes register_entity - - - + state_changed to main - - - - - - ServiceMirror - - - pushes register_service - - - for approved domains - - - - - - EventMirror - - - re-fires <approved>_* - - - events to main's bus - - - - - - ApprovedDomains - - - refcounted set; gates - - - ServiceMirror + EventMirror - - - - - - ChannelSandboxBridge (current_sandbox) - - - Store._async_load_data / _async_write_data / async_remove - - - → sandbox/store_{load,save,remove} on main - - - - - - channel.py (sandbox side) - - - protobuf framing, request/response, - - - push notifications, graceful close - - - - - - integration code (e.g. light platform) - - - runs unmodified: async_setup_entry, ConfigFlow, - - - add_entities, async_register, Store(…), bus.fire(…) - - - — sees a normal hass with a private states/bus - - -
- - -

- 3. The classifier — how routing decisions get made -

- -

- The classifier is the brain of the routing layer: a - pure function from a loaded Integration to - a SandboxAssignment. It runs in two places — from - SandboxFlowRouter.async_create_flow when a new flow starts, - and from SandboxFlowRouter.async_setup_entry when an - existing entry is being set up — and never imports the integration - to make its call. The check uses - Integration.platforms_exists() for the same reason. -

- -

Rule order (first match wins)

- -
    -
  1. - integration_type == "system" → main. - System integrations are part of the runtime; sandboxing them is - meaningless. -
    2. -
  2. - domain in ALWAYS_MAIN → main. A - small deny-list: script, automation, - scene, cloud, ai_task, - image. Each entry has an inline "why" in - const.py. ai_task and - image were added by the Phase 1 spike because their - service handlers do non-idempotent pre-dispatch work that neither - bridge option intercepts cleanly. -
    3. -
  3. - Any platform in SANDBOX_INCOMPATIBLE_PLATFORMS → - main. - Audio / byte-stream platforms the control channel can't ferry: - stt, tts, conversation, - assist_satellite, wake_word, - camera. -
    4. -
  4. - Custom (non-built-in) integration → - Sandbox("custom"). -
    5. -
  5. - Otherwise → Sandbox("built-in"). -
    6. -
- -

The three groups

- - - - - - - - - - - - - - - - - - - - - - - - - - -
GroupHostsDefault sharing
main - Nothing — matches above route here, no sandbox process - n/a
built-inEvery other built-in integration - share_states, share_entity_registry, - share_areas all True -
customEvery custom (HACS / user) integrationAll sharing False (locked down)
- -
- Why custom defaults to locked down. A - custom integration is the most likely vector for an attacker (HACS pulls - code from arbitrary GitHub repos). A custom integration that - needs to see main's state can opt in explicitly — the - platform refuses to give it a default pass. -
- -

Platforms that force routing to main

-
- stt - tts - conversation - assist_satellite - wake_word - camera -
- - -

4. Lifecycle — spawn, supervise, shut down

- -

Lazy spawn

-

- SandboxManager.ensure_started(group) creates the subprocess - only when the first flow or entry routes to it. The subprocess command - is: -

- - 
python -m hass_client.sandbox \
-    --name <name> \
-    --url stdio:// \
-    --token <sandbox access token>
- -

- --url selects the control-channel transport: - stdio:// (the default — frames over the subprocess's - stdin/stdout) or unix://<path> (the manager opens a - unix socket and the runtime dials back). ws:// / - wss:// are reserved for the deferred websocket transport - and rejected for now. The runtime opens the channel and sends a - Ready frame (sandbox/ready) as its first - message; the manager treats its arrival as “running” — - there is no stdout text marker, so stdout carries nothing but channel - frames. Frames are protobuf and length-prefixed (4-byte big-endian - length + body) on the stream transports. -

- -

Health and crash recovery

-

- SandboxProcess._supervise watches the subprocess for - unexpected exits. Restart-on-crash is bounded: - 3 attempts within a 60 s sliding window, with a - small backoff sleep between attempts. Exceeding the budget transitions - the sandbox to failed and - ensure_started raises - SandboxFailedError — the router surfaces this as - SETUP_RETRY on the affected entries. -

- -

- A sandbox/ping handler is registered and exercised by the - subprocess test; the periodic 30 s ping loop is wired through but - currently disabled (process-exit detection covers the hard-crash case). -

- -

Graceful shutdown

- -

- On EVENT_HOMEASSISTANT_STOP the integration runs the - following dance — the goal is that every sandboxed entity's last - state survives into the next boot's RestoreEntity: -

- -
    -
  1. - manager.async_graceful_shutdown_all(timeout=manager.shutdown_grace) - fans out sandbox/shutdown to every running sandbox. -
    2. -
  2. - Each sandbox unloads its entries via - config_entries.async_unload, snapshots - RestoreStateData.async_get_stored_states() into a - JSON-safe wrapped dict (round-tripped through orjson's HA-aware - encoder), returns it in the reply, then schedules its own shutdown - event via call_soon after the reply is queued so - the subprocess exits 0 on its own. -
    3. -
  3. - The reply lands in SandboxData's - on_shutdown_reply callback, which writes - restore_state to - <config>/.storage/sandbox/<group>/core.restore_state - via the bridge's store server. -
    4. -
  4. - manager.async_stop_all() falls through to SIGTERM, then - SIGKILL, for any sandbox that didn't ack the graceful round-trip. -
    5. -
- -

- On the next boot the runtime warm-loads - core.restore_state against a vanilla - Store before any handler registers, so the first - RestoreEntity.async_get_last_state() sees the previous - run's state. -

- -
- No store swap needed. The runtime sets - current_sandbox before the warm-load, and - Store's IO methods read the contextvar at call time, so the - load routes to main even though restore_state.py captured - the original Store reference at import. (Phase 8's - module-level storage rebinding couldn't reach that captured reference, - so it needed a dedicated sandbox-backed Store instance here - as a workaround — the contextvar made that unnecessary.) -
- - -

- 5. The channel — protobuf wire over pluggable transports -

- -

- The channel is split into three layers so the wire format and the byte - transport can each change without touching the concurrency-critical - dispatch core: -

- -
    -
  • - Channel — the dispatch core - (pending-id map, inflight semaphore, register / call / push / close). - It speaks Frame objects and never touches raw bytes. -
    • -
  • - Codec — Frame ↔ - bytes. ProtobufCodec is the production wire (a typed - protobuf Frame envelope; the codec owns the - type → message registry). The - line-oriented JsonCodec is kept only as the channel-core - test/debug wire. -
    • -
  • - Transport — moves whole frame - blobs. StreamTransport length-prefixes each frame (4-byte - big-endian length + body) over a reader/writer pair (stdio, unix - socket); a future WebSocketTransport drops in via - Channel.from_transport. -
    • -
- -

- Both sides of the bridge share an identical - protocol.py (type-string constants, mirrored verbatim in - homeassistant/components/sandbox/protocol.py and - sandbox/hass_client/hass_client/protocol.py) and an - identical messages.py / generated _pb2 pair. A - call_service request, for example, is a - CallService message (domain, - service, plus Struct target / - service_data for the dynamic fields) wrapped in the - Frame envelope. -

- -

Three message shapes ride the channel:

- -
    -
  • - RPC request / reply. Each request carries a - monotonically increasing id; the reply echoes it. The - originator awaits on a future keyed by id. -
    • -
  • - Push notification. No reply expected. Used for - register_entity, state_changed, - fire_event, register_service, etc. (the - recipient may still ack via a notification of its own for - backpressure, but there's no per-message future). -
    • -
  • - Graceful close. Either end can flush a final reply - and then close the channel cleanly. -
    • -
- -
- Known limitation — concurrent dispatcher. A - handler that issues channel.call(…) re-enters the - reader and can deadlock against its own pending reply. Phase 9 - worked around the specific case (restore_state in the shutdown reply); - the general fix is one task per inbound call so the reader can keep - draining the wire. This also unblocks firing - EVENT_HOMEASSISTANT_FINAL_WRITE on sandbox shutdown. -
- - -

6. Config-flow forwarding

- -

- This is where the user-facing magic happens. From the frontend's point - of view, adding a sandboxed integration looks identical to adding a - local one — the same form fields, the same error states, the same - final "Created entry" toast. -

- -

The router hook

- -

- HA Core's ConfigEntries grows a single - router attribute (introduced in Phase 4), consulted - from three call sites: -

- -
    -
  • - ConfigEntriesFlowManager.async_create_flow — when a - new flow starts. -
    • -
  • - ConfigEntries.async_setup — when an existing entry - is being set up. -
    • -
  • - ConfigEntries.async_unload — consults - router.async_unload_entry before falling through to - entry.async_unload(hass) (Phase 14). -
    • -
- -

- SandboxFlowRouter.async_create_flow runs the routing logic - in order: look up any existing entry for the handler key, fall back to - classify(integration), then either return - None (let HA handle it locally) or hand back a - SandboxFlowProxy ConfigFlow. The proxy issues - sandbox/flow_init, sandbox/flow_step, and - sandbox/flow_abort RPCs against the matching sandbox's - runtime; each RPC returns a marshalled FlowResult that the - proxy re-issues as async_show_form / - async_create_entry / async_abort so the - framework treats the result as native. -

- -

Where the routing tag lives

- -

- When the sandbox returns a final create_entry result, main - creates the ConfigEntry in its own store with - entry.sandbox = "<group>". On the next - ConfigEntries.async_setup(entry_id), the router sees that - tag, ensures the sandbox is running, and round-trips an - entry_setup RPC. -

- -
- Phase 17 pivot. The original plan was to set - sandbox by calling - async_update_entry(entry, sandbox=…) right after the - framework creates the entry. That didn't survive contact with the - framework: async_add invokes async_setup - inside its own body, before any after-hook fires — so - setup would run with sandbox=None and the router would - never route. The fix was to extend ConfigFlowResult with a - sandbox key (the same plumbing as - minor_version, options, - subentries) and read it at the entry constructor. Atomic, - smaller, and reuses the framework's own convention. -
- -

Inside the sandbox: _SandboxFlowManager

- -

- The integration's real ConfigFlow runs inside a - _SandboxFlowManager — a - ConfigEntriesFlowManager subclass that short-circuits the - CREATE_ENTRY path. Main is the canonical owner of the - ConfigEntry, so the sandbox never tries to add an entry to - its own private store. The framework still validates the flow, validates - form data, and computes the final result — only the persistence - step is suppressed. -

- -

What's deferred

- -
    -
  • - data_schema is stripped on the wire. The - flow runner sets _has_data_schema: True when it removed a - schema; the proxy logs a debug message when it sees the flag. A - voluptuous_serialize-based bridge is the obvious - follow-up — without it, frontend forms for sandboxed - integrations can't render argument hints. -
    • -
  • - unique_id is not propagated from the - sandbox's flow.context back to the proxy's - flow.context. The framework's duplicate detection on main - can miss this until that small marshalling extension lands. -
    • -
- - -

7. The entity bridge (Option B)

- -

- This was the riskiest design choice in the whole rewrite — how do - entity service calls (light.turn_on, - switch.toggle, - climate.set_temperature…) cross the sandbox - boundary? Phase 1 was a spike that compared two candidate designs - head-to-head with measured numbers. -

- -

The two candidates

- - - - - - - - - - - - - - - - - - - - - -
OptionWire shapeSandbox-side handler
A — method-forward RPC - sandbox/entity_method_call with - (entity_id, method, kwargs) - - getattr(entity, method)(**kwargs) — bypasses - the service layer -
B — action-call forwarding - Generic sandbox/call_service with - (domain, service, target, service_data) - - hass.services.async_call(…) — the - sandbox's normal dispatcher validates and invokes the entity - method on the (already-resolved) entity ids -
- -

The numbers

-

- 100-entity area light.turn_on, in-process transport, 5 - iterations median: -

- - - - - - - - - - - - - - - - - - - - - - - -
OptionMedian (ms)Per entity (ms)Glue LOC per domain
A~46~0.4642
B~64~0.6448
- -

Why B won

- -

- Option B is ~0.18 ms slower per entity (the cost is HA's full - service handler — target resolution, schema validation, per-entity - dispatch). Per-domain glue is essentially identical. The deciding factor - wasn't performance: -

- -
    -
  • - Phase 6 was going to need - sandbox/call_service anyway - for service mirroring. Option B reuses that one channel; Option A - would have added a parallel entity-only RPC alongside it. Smaller - protocol surface beats a couple-tenths of a millisecond. -
    • -
  • - Schema validation actually runs. In Option A, the - service-handler's schema validation was bypassed — if a caller - on main passed bad arguments, the sandbox-side entity method received - them straight, with no validation layer. Option B runs the real - dispatcher and rejects bad payloads at the boundary. -
    • -
  • - The integration's real service handler runs in its native - environment. - The call lands on the sandbox's own services.async_call, - so any per-entity dispatch logic, response data - (supports_response), and integration-specific behaviour - executes exactly as it would locally — not a reimplementation on - main. -
    • -
- -

Sandbox side

- -

- EntryRunner rebuilds a ConfigEntry from the - sandbox/entry_setup payload, fetches the integration's code - if needed (see below), drops the entry into the sandbox's - ConfigEntries, and runs the integration's - async_setup_entry. The integration adds entities the normal - way — EntityBridge listens for - EVENT_STATE_CHANGED on the sandbox's bus and, on each - entity's first appearance, pushes - sandbox/register_entity to main with: -

- -
    -
  • - entry_id, domain, - sandbox_entity_id -
    • -
  • - unique_id, name, icon, - has_entity_name -
    • -
  • - entity_category, device_class, - supported_features -
    • -
  • - capability_attributes - (supported_color_modes, color temp range, …) -
    • -
  • the initial state + attributes
    • -
- -

- Subsequent updates push sandbox/state_changed — state - + attributes only, no re-registration. -

- -

Integration source — fetch before setup (stateless)

- -

- Sandboxes hold no persistent state: config arrives on - entry_setup, storage routes to main, and the last stateful - bit — the integration code — is fetched at startup. - EntrySetup.integration_source carries - {kind: "builtin"} (a no-op; the bundled - homeassistant package provides it) or - {kind: "git", url, ref, …} for custom (HACS) - integrations. Main pins ref to an exact commit sha; the - sandbox downloads the codeload tarball before async_setup - and extracts it into - <config>/custom_components/<domain>, cached for - the process lifetime by (url, ref). Core stays - HACS-agnostic via a registered resolver hook - (async_register_sandbox_source_resolver); a custom domain - with no resolver fails loudly. -

- -

Main side

- -

- SandboxBridge receives register_entity, - instantiates a domain-specific proxy from entity/, and - attaches it to the matching EntityComponent via the new - EntityComponent.async_register_remote_platform core hook - (Phase 5's sole core change). The proxy holds a cached state + - attributes dict fed by state_changed; state, - available, and per-domain typed properties - (is_on, brightness, hs_color, - …) read from the cache. -

- -

- Main always resolves the target. The proxy entities - live in main's registries, so when a user calls - light.turn_on against an area or label, it is main's - service layer that expands it to a concrete entity-id list and invokes - async_turn_on on each proxy. Those proxy calls translate - into sandbox/call_service RPCs via a per-loop-tick batcher - (_CallServiceBatcher) that coalesces matching - (domain, service, service_data) calls into one multi-entity - RPC — so a 200-light area call pays one RPC, not - 200. The sandbox never re-resolves areas or labels; it only ever - receives concrete entity ids and dispatches to its real entities. -

- -

- Exception translation maps sandbox-side vol.Invalid → - TypeError and ServiceNotFound / - ServiceValidationError → - HomeAssistantError, so callers on main see the local-entity - error shape rather than a raw remote error. -

- -

The non-idempotent service handler problem

- -
- The spike surfaced a wrinkle neither option handles. - Some integrations have non-idempotent service handlers — the - handler does significant work before calling the entity method - (e.g., ai_task._resolve_attachments fetches bytes and - writes temp files), and by the time it dispatches into the entity, the - kwargs no longer satisfy the original service schema. Both bridge - options are intercepting at the wrong layer. - Resolution: ai_task and - image join ALWAYS_MAIN; a - service-handler-level interception story is queued for a future spec. -
- -

Domains shipped

- -

- Phase 5 shipped four canonical proxies (light, - switch, sensor, binary_sensor) as - the reference implementations of SandboxProxyEntity; - Phase 13 followed up with the remaining 28 supported domains using - the same pattern. Unknown-domain registrations fall back to the generic - SandboxProxyEntity — state + attributes work; - domain-typed properties don't. -

- - -

8. Service & event mirroring

- -

- Beyond entity service calls, integrations register their own domain - services (media_player.snapshot, zha.permit, - …) and fire their own domain events (zha_event, - mqtt_message_received). Both have to reach main, but only - for integrations that have actually been set up. -

- -

The approval gate

- -

- Once a sandboxed integration's async_setup_entry succeeds, - EntryRunner adds the entry's domain to a refcounted - ApprovedDomains set. EntityBridge also adds - the domain of each registered entity (so a sandbox that hosts a - light integration approves the light domain by - virtue of registering light entities). ServiceMirror and - EventMirror consult this set before forwarding anything - — everything else stays inside the sandbox. -

- -

ServiceMirror

-

- Listens on the sandbox bus for EVENT_SERVICE_REGISTERED / - EVENT_SERVICE_REMOVED and pushes - sandbox/register_service / - unregister_service (with supports_response) to - main. Schemas aren't serialised — the sandbox - owns the real schema and runs validation when the call lands on its - services.async_call. Main installs a thin forwarder that - ships each call back over the shared - sandbox/call_service channel, reusing the Phase 5 - exception translator. -

- -
- The forwarder refuses to clobber an existing handler - — so the light.turn_on registered by the host - light EntityComponent for our proxy entities - keeps its dispatch role for entity services. The mirror only installs - handlers for genuinely new domain-services the sandbox owns. -
- -

EventMirror

-

- Uses a MATCH_ALL listener with an internal-events deny-list - and forwards only <approved_domain>_* events (e.g. - zha_event, mqtt_message_received) via - sandbox/fire_event. Main re-fires each on its own bus so - automation listeners react as if the integration ran - locally. -

- -

- The sandbox's context_id is on the wire but main does - not honour it on the re-fire (a fresh local - Context is used) — carrying a richer - Context shape across the bridge is future work. -

- - -

9. Sandbox auth & opt-in data sharing

- -

- Each sandbox group runs against a dedicated system user. The manager - hands the subprocess a plain system-user access token, freshly minted - from that user's refresh token on every spawn. There is - no scope restriction on the token today. -

- -
- Scope enforcement is deferred. Phase 7 originally - shipped a RefreshToken.scopes field plus a - websocket-dispatcher _scope_allows check that rejected any - command outside {"sandbox/", "auth/current_user"} with - ERR_UNAUTHORIZED. But the sandbox never opened a websocket - back to main, so no code path ever exercised the check end-to-end - — it guarded a non-existent attack surface. - plan-strip-auth-scopes reverted the whole mechanism from - core HA (four files: auth/models.py, - auth/__init__.py, auth/auth_store.py, - websocket_api/connection.py). With no WS path open in - either direction, the sandbox token's reach is the same as v1's; the - posture is unchanged in practice. -
- -

- When the sandbox→main websocket transport actually lands, scope - enforcement is a green-field redesign with a real consumer in hand. The - prior thinking — the optional-field-on-RefreshToken - decision, the prefix-grant + exact-match grammar, and why a token - subclass was rejected — is preserved in - docs/auth-scoping-decision.md (marked SUPERSEDED) as the - starting point for that work. -

- -

Data sharing — the positive opt-in

- -

- Independently of the token's reach, data sharing into the - sandbox is a positive opt-in. SandboxGroupConfig ships - three knobs: -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Groupshare_statesshare_entity_registryshare_areas
mainTrueTrueTrue
built-inTrueTrueTrue
customFalseFalseFalse
- -

- The CLI accepts matching --share-* flags; the runtime - stores them on a SharingConfig dataclass. -

- -
- The subscription consumer isn't shipped yet. The config - flag is wired through to the runtime and the locked-down posture is - enforced trivially today (no subscription code exists). When the - sandbox→main websocket lands, opening the subscription (gated on - sharing.share_states) and the matching filtering on main's - emit path are owed in the same PR. -
- - -

10. Store routing — current_sandbox

- -

- Integrations persist state via - homeassistant.helpers.storage.Store. If a sandboxed - integration writes my_integration.config, where does that - file actually end up — in the sandbox's tempdir, or in main's - .storage/? -

- -

- The answer: always main. The sandbox owns code; main - owns state. -

- -

The contextvar

-

- Store reads a current_sandbox - ContextVar (declared in - homeassistant/helpers/sandbox_context.py) at IO time. When - it is set, Store._async_load_data, - Store._async_write_data, and - Store.async_remove delegate to the contextvar's - SandboxBridge instead of touching local disk — - talking to main via sandbox/store_load, - sandbox/store_save, sandbox/store_remove. - Branching at _async_write_data (rather than - async_save) is deliberate: async_save, - async_delay_save, and the - EVENT_HOMEASSISTANT_FINAL_WRITE flush all funnel through - _async_handle_write_data → - _async_write_data, so one branch covers every write path. - The migration loop in _async_load_data runs unchanged - whether the wrapped envelope came from disk or the bridge. -

- -

The bridge

-

- ChannelSandboxBridge - (hass_client/sandbox_bridge.py) implements the three - SandboxBridge store methods over the channel. - SandboxRuntime.run does - current_sandbox.set(ChannelSandboxBridge(channel)) right - after the channel opens and before the warm-load or any per-runner - handler registers, so every coroutine the runtime spawns inherits it - (asyncio copies the context at create_task time). One - sandbox process hosts one sandbox group, so a single bridge per runtime - is correct. This replaced the Phase 8 module-level - Store rebinding — a declared core HA hook rather than - a monkey-patch, and it reaches helpers like - restore_state that captured the original - Store reference at import. -

- -

Main side

-

- Each SandboxBridge owns a - _SandboxStoreServer pinned to - <config>/.storage/sandbox/<group>/. Writes use - util.file.write_utf8_file_atomic — the same primitive - Store itself uses, so atomicity and durability are - unchanged. Scope isolation is by construction: each bridge owns one - channel for one group; forging a cross-group call would require forging - the channel. Key validation (_require_key) rejects - /, \, NUL, ., .., - and any ..-prefixed key before any path is constructed. -

- -

What stays local

-

- Registries (entity/device/area/auth) that load during the sandbox's - startup before the channel is up keep their local - tempdir backing. Routing the HA-internals Stores too is a - larger decision that hasn't been made yet. -

- - -

11. Test infrastructure & the compat lane

- -

- The proof that the bridge doesn't regress integration behaviour is the - compat lane: run HA Core's existing per-integration test suites with - sandbox wired in and watch the pass rate. -

- -

Two pytest plugins

- - - - - - - - - - - - - - - - - - - - - -
PluginWireWhen to use
hass_client.testing.pytest_plugin - In-memory channel pair; SandboxRuntime as an asyncio - task - Fast feedback; freezer-safe
hass_client.testing.conftest_sandbox - Real stdio protobuf channel (python -m hass_client.sandbox) - Pins the subprocess boundary; freezer tests auto-skip
- -

- Both share the same manager-side SandboxBridge code path; - the only thing that differs is how the channel pair is materialised. The - in-memory plugin is the default for fast feedback; the subprocess plugin - is what gives us confidence the boundary really is a boundary. -

- -

The compat lane runner

- -

- run_compat.py drives either plugin against a list of - integration test directories, parses pytest's summary line, and writes - COMPAT.csv + COMPAT.md. Per-failure output - lands in ${SANDBOX_ERRORS_DIR:-/tmp/sandbox_errors}. - run_compat_full.py drives it across all 807 - classifier-routable, config-entry-based integrations and captures JUnit - per-test; categorize_failures.py buckets every failure into - a category for triage. -

- -

The headline numbers

- -
-
-
807
-
Integrations swept
-
-
-
711
-
Fully passing
-
-
-
99.67 %
-
Test-level pass rate
-
-
-
0
-
Bridge bucket failures
-
-
- -

- What's left in the failure bucket is almost entirely test-side - residuals: ~30 diagnostic snapshots showing - + 'sandbox': 'built-in' (fix: - pytest --snapshot-update per integration) and ~70 - created_at snapshot drifts (fix: integration-side - freezegun, or an optional clock-pinning fixture on the - compat plugin). Zero bridge bugs remain in the swept - set. -

- - -

12. The phase timeline

- -

- The bridge landed in 17 phases over several months. Each phase had a - single concrete deliverable, a status doc - (status/STATUS-phase-N.md) explaining what it actually - shipped, and explicit notes on what it deferred forward. The status - files are the authoritative record — this is the abridged tour. -

- -
-
-
Phase 0
-

Skeletons

-

- Empty HA integration loads; subprocess entrypoint exists; CI green. - Nothing functional yet — just the scaffolding both sides can - build into. -

-
- -
-
Phase 1
-

Entity-bridge spike

-

- Two in-process HomeAssistant instances joined by a JSON - transport; benchmarked Option A vs Option B head-to-head - on a 100-light area call. Recommendation: Option B. - Side-effect: discovered the non-idempotent service handler problem, - which pushed ai_task and image into - ALWAYS_MAIN. -

-
- -
-
Phase 2
-

Runtime classifier

-

- classify(integration). Pure function from manifest + - platform inspection to a group assignment — no user config, no - per-integration migration. -

-
- -
-
Phase 3
-

Sandbox lifecycle

-

- SandboxManager spawns one subprocess per group lazily; - restart-on-crash with a 3/60 s budget; Ready-frame - handshake. -

-
- -
-
Phase 4
-

Config-flow forwarding

-

- New flows run inside the sandbox; main owns the canonical - ConfigEntry store. Introduced - ConfigEntries.router and the - SandboxFlowProxy. -

-
- -
-
Phase 5
-

Entity bridge end-to-end

-

- Four canonical proxies ship (light, - switch, sensor, - binary_sensor); per-loop-tick fan-out batching; - exception translation. The other 28 domain proxies deferred to - Phase 13 as mechanical follow-ups. Sole core change: - EntityComponent.async_register_remote_platform. -

-
- -
-
Phase 6
-

Service & event mirroring

-

- Sandbox-side ServiceMirror + EventMirror; - refcounted ApprovedDomains set; main-side forwarder - reuses Phase 5's call_service channel and - exception translator. -

-
- -
-
Phase 7
-

Scoped auth + opt-in sharing

-

- RefreshToken.scopes + dispatcher enforcement; per-group - system user; SandboxGroupConfig with the three sharing - knobs. The riskiest phase from a security-review angle — - written up at length in docs/auth-scoping-decision.md. - The scopes mechanism was later reverted - (plan-strip-auth-scopes) because no consumer ever - opened the connection it guarded; the per-group system-user token - stays. -

-
- -
-
Phase 8
-

Store routing

-

- The current_sandbox contextvar routes every - Store(…) in the sandbox to - <config>/.storage/sandbox/<group>/<key> - on main. Store reads the contextvar at IO time — - no module-level rebinding. (Phase 8 shipped this as a storage-module - monkey-patch; plan-sandbox-context replaced it with the declared - hook.) -

-
- -
-
Phase 9
-

Graceful shutdown + restore-state hand-off

-

- Sandboxes unload entries and dump RestoreEntity state - into the shutdown reply; main persists it for the next boot's - warm-load. -

-
- -
-
Phase 10
-

Test infrastructure

-

- Two pytest plugins (in-process + real-subprocess) plus - run_compat.py. -

-
- -
-
Phase 11
-

Docs & cleanup

-

- OVERVIEW.md, the auth scoping decision write-up, the - per-phase STATUS files, this site's predecessor. -

-
- -
-
Phase 12
-

Concurrent dispatcher (partial)

-

- Worked around the specific reader-deadlock in the shutdown reply - path. Full "one task per inbound call" fix is still owed — - remains an open follow-up. -

-
- -
-
Phase 13
-

Remaining domain proxies

-

- The 28 mechanical wrappers around - SandboxProxyEntity — climate, - cover, media_player, lock, - fan, vacuum, … Query-shaped RPCs - (calendar, todo, weather) - return empty lists pending a follow-up. -

-
- -
-
Phase 14
-

Schema / unique_id / unload-hook / perf

-

- ConfigEntries.async_unload consults - router.async_unload_entry; assorted marshalling - extensions and perf passes on the batcher. -

-
- -
-
Phase 15
-

Focused compat lane

-

- Triaged the 33-integration smoke list end-to-end, every non-pass row - investigated. Result: 99.97 % on the focused lane — the - bridge is behaviour-equivalent on the integrations users hit first. -

-
- -
-
Phase 16
-

Cross-integration sweep + categorised backlog

-

- Ran every classifier-routable, config-entry-based integration (807 - of them, 34 378 tests). Bucketed every failure with - categorize_failures.py. Headline at this point: - 98.07 % test pass rate, with 552 of the 664 failures traceable - to a single root cause — the autotag's - __sandbox_group key in - entry.data perturbing fixtures and snapshots. -

-
- -
-
Phase 17
-

ConfigEntry.sandbox first-class field

-

- Moved the routing tag off entry.data onto a new - first-class ConfigEntry.sandbox: str | None field. The - pivot: original plan was to - async_update_entry(sandbox=…) after the - framework created the entry — didn't work because - async_add invokes async_setup inside its - own body, before any after-hook fires. Real fix: extend - ConfigFlowResult with a sandbox key (same - plumbing as minor_version / options / - subentries) and read it at the entry constructor. - Single fix; 552 of the 664 known failures closed. - New headline: 99.67 % test pass rate. -

-
-
- - -

13. Open follow-ups

- -

- These are the items the per-phase STATUS files flagged forward as - explicit non-goals for the first pass. They're tracked separately so the - core bridge stayed reviewable. -

- -
-
-

- share_states=True subscription consumer + main-side - filtering -

-

- The config knob is wired and the locked-down posture is enforced - trivially today (no subscription code exists). The consumer that - opens a subscription back to main and the filtering on main's emit - path are owed in the same PR. -

-
- -
-

Concurrent channel dispatcher

-

- A handler that issues channel.call(…) re-enters - the reader and can deadlock. Phase 9 worked around the specific - case; the general fix is one task per inbound call so the reader can - keep draining the wire. Also unblocks firing - EVENT_HOMEASSISTANT_FINAL_WRITE on sandbox shutdown. -

-
- -
-

data_schema serialisation across the flow wire

-

- Without it, frontend forms for sandboxed flows can't render argument - hints. voluptuous_serialize-based bridge is the obvious - shape. Pair it with the service-mirror schema serialisation in the - same PR. -

-
- -
-

- unique_id propagation through - SandboxFlowProxy -

-

- Small marshalling extension to _marshal_result. The - framework's duplicate detection on main misses sandboxed flows until - this lands. -

-
- -
-

- calendar / todo / - weather query-shaped RPCs -

-

- The Phase 13 proxies return empty lists for - async_get_events, todo_items, and - weather.async_forecast_* because the action-call - channel can't express server-side queries. Add a query-shaped RPC if - the compat sweep ever surfaces an integration that needs them. -

-
- -
-

- Non-idempotent service handlers (ai_task, - image, …) -

-

- Punted to ALWAYS_MAIN for now. A future spec on - service-handler-level interception or sandbox-aware integration - hooks is the long-term fix. The Phase 1 spike doc has the full - write-up. -

-
- -
-

Diagnostic snapshot drift / clock-pinning fixture

-

- Phase 17's BACKLOG.md documents two test-side - residuals: ~30 diagnostic snapshots that show - + 'sandbox': 'built-in' (fix: - --snapshot-update per integration) and ~70 - created_at snapshot drifts (fix: integration-side - freezegun, or a ~30 LOC clock-pinning fixture on the - compat plugin). -

-
-
- - -

14. Where to look in the code

- -

- Per-phase status/STATUS-phase-N.md files are the - authoritative record of what each phase actually built. For a quick map - of the codebase: -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ConcernHA Core sideSandbox side
Classifiercomponents/sandbox/classifier.py—
Lifecyclecomponents/sandbox/manager.py - hass_client/sandbox.py, - hass_client/sandbox/__main__.py -
Channelcomponents/sandbox/channel.pyhass_client/channel.py
Config flow - components/sandbox/router.py, - proxy_flow.py - hass_client/flow_runner.py
Entity bridge - components/sandbox/bridge.py, entity/ - - hass_client/entry_runner.py, - entity_bridge.py -
Service / event mirrorcomponents/sandbox/bridge.py - hass_client/service_mirror.py, - event_mirror.py, approved_domains.py -
Auth - components/sandbox/auth.py - (plain system-user token) - —
Store routing - components/sandbox/bridge.py - (_SandboxStoreServer), - helpers/sandbox_context.py, - helpers/storage.py - hass_client/sandbox_bridge.py
Shutdown - components/sandbox/__init__.py - (_on_stop), manager.py - - hass_client/sandbox.py - (_run_graceful_shutdown) -
Test infra— - hass_client/testing/, run_compat.py -
- -

The three Core files modified

-

- The bridge touches three HA Core files. Each is intentional, small, and - was introduced by a specific phase: -

-
    -
  • - homeassistant/config_entries.py — - ConfigEntries.router attribute and - ConfigEntryRouter protocol (Phase 4); - async_unload consults - router.async_unload_entry (Phase 14); - ConfigEntry.sandbox: str | None field plumbed through - ConfigFlowResult and - async_finish_flow (Phase 17). -
    • -
  • - homeassistant/helpers/entity_component.py - — - EntityComponent.async_register_remote_platform - (Phase 5). -
    • -
- -

Running the tests

- - 
# HA Core side
-uv run pytest tests/components/sandbox/ --no-cov -q
-
-# Client side (separate uv env — does NOT accept --no-cov)
-uv run pytest sandbox/hass_client/ -q
-
-# Compat lane
-cd sandbox && python run_compat.py
-
- - -
- -