* perf: stop the Socket.IO session pool blocking the websocket event loop
With WEBSOCKET_MANAGER=redis the session pool is a synchronous Redis client, so every call into it blocks the whole worker's event loop, not just the caller. Two paths did it constantly: the orphan reaper walked the pool one round trip per session with no await anywhere, freezing the loop for the entire sweep every cycle, and nearly every socket event re-read the sender's session back out of Redis. Other users' events and every in-flight generation on that pod wait behind both.
The reaper now walks the pool in HSCAN batches and deletes in bulk, yielding between batches, and no longer sleeps past half the lock TTL, which previously guaranteed a failed renew every cycle. The per-event reads are gone: Socket.IO events only reach the worker holding the connection, and that worker already saved the same session dict locally when the user authenticated, so it was asking Redis for its own data. The writes stay, since those are what other pods read.
Measured at 4000 users / 16 containers, Redis 1.1 ms away:
| | before | after |
|---|---|---|
| reaper sweep, 5k sessions | 5.6 s, loop frozen throughout | 62 ms, 4.2 ms worst block |
| same, crash recovery with every session expired | 11.5 s | 96 ms |
| heartbeat / usage ping / disconnect | 2 / 3 / 2 round trips | 1 / 2 / 1 |
| 50-member channel post | 50 round trips, 57.2 ms block | 0 round trips, 0.02 ms |
| loop time per wall second at rest | 103 ms (10.3%) | 67 ms (6.7%) |
The alternative, converting RedisDict to the async client, fixes the same paths with a far larger blast radius (every call site gains await, and `in`/`[]`/`del` cannot be awaited so the dict interface goes) and still round-trips for data already in memory. Two deliberate behaviour changes: a heartbeat re-adds a session the reaper already removed, so a tab that survives a stall recovers instead of staying out of the pool until it reconnects; and disconnect no longer skips Yjs document cleanup when the pool entry is already gone, which previously leaked that document's update log forever.
Closes#28172
* perf: cut disconnect and user session lookup pool round trips, harden the session reaper
Follow-up on top of the session pool reaper branch. With WEBSOCKET_MANAGER=redis two paths still blocked the worker's event loop on synchronous Redis calls. Every disconnect listed all models in use cluster-wide and fetched each one individually, one blocking round trip per model. Disconnecting all sessions of a user (admin role change or deletion) pulled the entire session pool in one HGETALL and decoded every entry in a single uninterrupted block.
Disconnect now fetches the usage pool once with items(), going from 2+N+M round trips to 2+M (N models in use cluster-wide, M models the session used), and its delete of an emptied model entry is KeyError-guarded because another node can remove the same key between snapshot and delete; unguarded, that race aborted the handler and skipped its Yjs document cleanup. The user session lookup reuses the reaper's HSCAN batches and yields to the loop between pages. The reaper previously died permanently on the first Redis connection error, on every node at once during an outage; it now logs, releases the lock and returns to retrying acquisition.
* refac: keep the socket pool perf work to the round trips
A review pass on this branch turned up four changes riding along with the round-trip work without belonging to it, so they are backed out here. The `disconnect` handler keeps its `if sid in SESSION_POOL:` guard, so USAGE_POOL and ydoc cleanup stay off the path for sockets that never authenticated. `RedisDict.set()` keeps its own inline HDEL. `get_session_ids_by_user_id` stays synchronous over one HGETALL, since it runs on user delete and role change rather than per message. The crash-resilience wrapper around the reaper loop is dropped; if that guard is worth having, it belongs in its own change.
What stays is the perf part. The reaper now sweeps the pool in bounded HSCAN batches and deletes expired sids with one HDEL per batch, down from HKEYS plus an HGET and a per-sid HDEL across the whole pool. The `disconnect` handler reads USAGE_POOL with a single HGETALL, down from HKEYS plus one HGET per model in use. Session lookups in the socket handlers come from the local Socket.IO store, which removes one Redis GET from every heartbeat, usage, channel and ydoc event.
Naming and annotations follow the file: `get_session_pool_batches` for the module's `get_` prefix, `RedisDict.pop_many` so both reaper branches use one word for removing keys, a named `SCAN_BATCH_SIZE`, and types on the new helpers.
* fix: invalidate the RedisDict write signature on batch delete
RedisDict.set() skips the write when the payload fingerprint matches the last one this process wrote, so a mutation that goes around set() has to clear that fingerprint. The new batch delete did not, leaving a stale fingerprint behind: the next refresh with identical content is treated as already written and silently skipped, so the hash stays empty.
Renamed pop_many to delete_many. In a dict emulation pop removes and returns; this returns nothing and cannot without an extra HMGET, so the name promised something it does not do. delete_many matches __delitem__ and the HDEL underneath. Its only call site is the session pool reaper, whose behaviour is unchanged: same fields deleted, same batching, same return.
Swap the JSON encoder/decoder used across the backend from stdlib json to
orjson when ENABLE_ORJSON is set — HTTP request bodies, JSONResponse
bodies, upstream provider responses, SSE chunks, and socket.io/Redis
payloads.
The flag defaults to off, in which case the app uses stdlib json and
engineio's codec verbatim, so default behaviour is unchanged.
- json_codec exports JSONCodec (stdlib json or the orjson codec) and
SOCKETIO_JSON (engineio's codec or the orjson codec); call sites import
JSONCodec and stay implementation-agnostic
- apply_orjson_http_json() is a no-op when the flag is off, leaving
starlette's Request.json / JSONResponse.render untouched
- the orjson codec falls back to the stdlib for inputs orjson rejects
(non-str dict keys, ints beyond 64 bits, NaN literals)
- orjson is imported only when the flag is on
- FastAPI(default_response_class=...) is deliberately not used: an
explicit default disables the Pydantic direct-to-bytes fast path for
response_model routes
get_all_models runs on every models refresh and, without the base-models cache (off by default), on every /api/models request. Several of its costs multiplied by the model count for no reason:
- The active action and filter id sets were derived from get_functions_by_type, which loads full function rows including plugin source and validates them, only for the ids and is_global flags. A generalized column-only query now returns (id, is_global) tuples; the existing filter-specific helper delegates to it.
- Action priorities were computed inside the per-model sort key, constructing a pydantic Valves object per action per model; with global actions in every model's list that was models x actions constructions per refresh. Priorities are now memoized per action.
- Global action and filter item dicts were rebuilt per model from the same modules. The item lists are now built once per function and shallow-copied per model, keeping per-model dicts independent exactly as before (nested values were already shared).
- Deactivated base-model overrides were dropped with models.remove, a linear scan and shift per removal; removals are now collected and filtered out in one identity-based pass, preserving list.remove's exact object semantics.
- RedisDict.set fingerprinted the payload by serializing the already-serialized mapping a second time plus a sha256; a direct dict comparison against the last written mapping has the same skip semantics without re-serializing anything.
- /api/models did tag normalization and profile-image stripping for every model before access filtering discarded the invisible ones, and always evaluated a json.dumps debug f-string; the work now runs only on visible models and the debug line is gated on the log level. The duplicate-id dedup keeps its position before filtering so the effective-model semantics are unchanged.
Benchmark:
| metric | before | after |
| --- | --- | --- |
| model-cache fingerprint, 200 models | 45 us | 1.4 us |
| action priority Valves builds, 200 models x 4 global actions | 0.37 ms (800 builds) | 0.002 ms (4 builds) |
| function-table payload for id sets | full rows incl. source | (id, is_global) tuples |
Functionally verified: the column-only id query matches the full-row query for actions and filters including inactive exclusion, and the fingerprint skip logic writes on first set, skips identical payloads, updates plus deletes stale keys on change and clears on empty, against a scripted fake Redis.
Classic298andGitHub
Timothy Jaeryang BaekandClassic298
okamototkandGitHub
Joseph LowandGitHub
Taylor Wilsdon
Jan Kessler
Jason Kidd