mirror of
https://github.com/Chia-Network/chia-blockchain.git
synced 2026-08-24 02:24:23 -05:00
Add Bugbot review context, update context (#21113)
* Add Bugbot consensus review context Document the validation-state and sub-epoch-summary invariant so automated PR review has the same consensus context as local agents. * Route Bugbot to repository context docs Keep Bugbot guidance focused on subsystem context routing so automated reviews use the same module map as local agents. * Add generated repository context docs Include the expanded module context map and route Bugbot to the same subsystem docs used by local agents. * Add EDC agent context pointer Include the generated EDC agent guidance alongside the Cursor agent context docs. * Fold EDC guidance into AGENTS Keep agent context entrypoints consolidated in AGENTS.md and remove the separate EDC pointer. * Remove generator name from agent guidance Keep AGENTS.md focused on repository context locations rather than the tooling used to produce them. * Consolidate context corpus into one source-verified canonical set - Normalize all subsystem docs to canonical names without chia- prefix, removing the split between old and new-dominant corpora. - Merge new-dominant content into consensus.md, full-node.md, wallet.md; split networking.md into server.md, protocols.md, apis.md; add types.md. - Rename 15 genuinely-new subsystem docs; fold chia-root.md into architecture-overview.md. - Remove filter-branch leakage (filter_challenge, predictable filter, protocol 0.0.38) from types.md, protocols.md, harvester.md. - Correct factual inaccuracies: /push_tx vs wallet queue semantics, Err sign/ban policy, protocol import-time coverage, server peer-serving, address-manager mutation rule, wallet tx-store rollback, ValidationState speculative advancement, block-creation default, MempoolItem fallback. - Replace unreachable verification SHA with PR-head commit. - Consolidate test guidance into one small router plus 12 on-demand testing/ docs; remove duplicate auto-attached test guides. - Make INDEX.md the authoritative routing manifest; update BUGBOT.md, AGENTS.md, and context-routing.mdc to reference one canonical corpus. - Replace context-chia-*.mdc rules with canonical narrow rules; remove recursive .cursor/** rule and consolidate duplicate tooling rules. - Tighten clvm-execution.md: remove stale cost_calculator.py reference, fix block-creation default, replace opcode catalogue with pointer. * Trim redundant context and testing docs after value audit - Trim benchmarks.md: remove per-script Workload Groups narration that is obvious from opening each script; keep fidelity, coupling, and fragility sections. - Trim testing/patterns.md: slim to the module-by-module setup map and layered-assertion list; remove block/tx/assertion how-to that duplicates per-area testing docs. - Trim testing/full-node.md: slim to starter template and sync-convergence tip; remove fixture list and assertion patterns duplicated by patterns.md and architecture.md. - Fold testing/service-wiring.md per-cluster correlation details into testing/architecture.md; delete service-wiring.md (high overlap with architecture.md). - Fix protocols.md: correct NewSignagePointHarvester2 message-ID wording (ID reassigned to 66, not literally the same ID). - Update INDEX.md and testing-guide.mdc router to reflect service-wiring.md removal. * Fix verification SHA to reference the main source baseline The docs were verified against source at PR base24db9ad390(origin/main), nota5647a9327(the original PR head, which only adds context docs and is not a meaningful source baseline). The chia/ source tree is identical across both commits since the PR changes no production code. * Fix remaining V2 filter-branch leakage in harvester.md and types.md - harvester.md: replace calculate_effective_plot_filter_bits() and calculate_min_plot_strength() (filter-branch-only) with the actual main source path: calculate_prefix_bits() with NUMBER_ZERO_BITS_PLOT_FILTER_V2 and height adjustments, then passes_plot_filter(); strength bounds checked by check_plot_param() against MIN_PLOT_STRENGTH/MAX_PLOT_STRENGTH. - harvester.md: use meta_group (actual PartialProofsData field name) instead of "meta group". - types.md: correct that candidate height feeds prefix-bit reductions for both V1 and V2, not just V1; calculate_prefix_bits() branches on V1/V2 constants and height thresholds.
This commit is contained in:
Almog De Paz
@@ -0,0 +1,41 @@
|
||||
# Bugbot Review Guidance
|
||||
|
||||
Use the repository context map before reporting issues that depend on
|
||||
cross-file invariants. Prefer the relevant context document over reasoning from
|
||||
a narrow diff alone. Start with `.cursor/context/INDEX.md` when unsure.
|
||||
|
||||
## Context Routing
|
||||
|
||||
Before reviewing a changed file, identify the subsystem and read the matching
|
||||
context document:
|
||||
|
||||
| Changed area | Read first |
|
||||
| -------------------------------------------------------------- | -------------------------------------- |
|
||||
| `chia/consensus/**`, block validation, difficulty, SSI, reorgs | `.cursor/context/consensus.md` |
|
||||
| `chia/full_node/**`, sync, batch validation, node state | `.cursor/context/full-node.md` |
|
||||
| `chia/full_node/mempool*.py`, fee logic, spend admission | `.cursor/context/mempool.md` |
|
||||
| `chia/server/**`, peer connections, rate limits | `.cursor/context/server.md` |
|
||||
| `chia/protocols/**`, wire messages | `.cursor/context/protocols.md` |
|
||||
| `chia/apis/**`, API stub metadata | `.cursor/context/apis.md` |
|
||||
| `chia/types/**`, shared types, serialization boundary | `.cursor/context/types.md` |
|
||||
| `chia/wallet/**` | `.cursor/context/wallet.md` |
|
||||
| CLVM, generators, puzzles, conditions | `.cursor/context/clvm-execution.md` |
|
||||
| `chia/farmer/**` | `.cursor/context/farmer.md` |
|
||||
| `chia/harvester/**` | `.cursor/context/harvester.md` |
|
||||
| `chia/timelord/**` | `.cursor/context/timelord.md` |
|
||||
| `chia/plotting/**`, `chia/plot_sync/**` | `.cursor/context/plotting.md` |
|
||||
| `chia/pools/**` | `.cursor/context/pools.md` |
|
||||
| `chia/daemon/**` | `.cursor/context/daemon.md` |
|
||||
| `chia/data_layer/**` | `.cursor/context/data-layer.md` |
|
||||
| `chia/rpc/**` | `.cursor/context/rpc.md` |
|
||||
| `chia/ssl/**` | `.cursor/context/ssl.md` |
|
||||
| `chia/simulator/**` | `.cursor/context/simulator.md` |
|
||||
| `chia/solver/**` | `.cursor/context/solver.md` |
|
||||
| `chia/cmds/**` | `.cursor/context/cmds.md` |
|
||||
| `chia/util/**` | `.cursor/context/util.md` |
|
||||
| Root config, build scripts, workflows, tooling | `.cursor/context/repo-tooling.md` |
|
||||
| Cross-cutting or security-sensitive changes | `.cursor/context/global-invariants.md` |
|
||||
|
||||
If multiple areas are touched, read each matching context document and check
|
||||
the interaction between their invariants. If a suspected issue contradicts the
|
||||
context, verify the full path before reporting it.
|
||||
@@ -16,17 +16,60 @@
|
||||
|
||||
| File | Covers | When to attach |
|
||||
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
|
||||
| [architecture-overview.md](architecture-overview.md) | Module map, actors, entrypoints, key types, `chia_rs` boundary | Starting any unfamiliar work; first-time orientation |
|
||||
| [consensus.md](consensus.md) | Block validation, difficulty adjustment, fork choice, VDF iterations, rewards | Touching `chia/consensus/`, block acceptance, reorgs |
|
||||
| [architecture-overview.md](architecture-overview.md) | Module map, actors, entrypoints, key types, `chia_rs` boundary, package root | Starting any unfamiliar work; first-time orientation |
|
||||
| [consensus.md](consensus.md) | Block validation, difficulty adjustment, fork choice, VDF iterations, rewards, reorg contract | Touching `chia/consensus/`, block acceptance, reorgs |
|
||||
| [mempool.md](mempool.md) | Transaction admission, eviction, fee logic, conflict detection, FF/DEDUP | Touching `chia/full_node/mempool*.py`, `eligible_coin_spends.py`, fee estimation |
|
||||
| [full-node.md](full-node.md) | FullNode orchestration, sync, block processing pipeline, FullNodeStore, FullNodeAPI | Touching `chia/full_node/full_node.py`, `full_node_api.py`, `full_node_store.py` |
|
||||
| [networking.md](networking.md) | WebSocket connections, rate limiting, peer discovery, protocol state machine | Touching `chia/server/`, `chia/protocols/`, connection handling |
|
||||
| [wallet.md](wallet.md) | Coin selection, wallet state manager, wallet node sync, sub-wallets | Touching `chia/wallet/` |
|
||||
| [clvm-execution.md](clvm-execution.md) | CLVM execution, condition processing, canonical serialization, cost metering | Touching puzzle execution, spend validation, generator logic |
|
||||
| [server.md](server.md) | WebSocket connections, rate limiting, peer discovery, TLS, address-manager behavior | Touching `chia/server/`, connection handling |
|
||||
| [protocols.md](protocols.md) | Wire protocol message schemas, numeric IDs, sender authorization, reply maps, capabilities | Touching `chia/protocols/`, message definitions |
|
||||
| [apis.md](apis.md) | API stub metadata, request/reply declarations, protocol-visible method names | Touching `chia/apis/*_stub.py`, API decorators |
|
||||
| [types.md](types.md) | Shared blockchain-format types, CLVM `Program` helpers, `Coin`/condition contracts, Rust boundary | Touching `chia/types/`, serialization compatibility |
|
||||
| [wallet.md](wallet.md) | Coin selection, wallet state manager, wallet node sync, sub-wallets, persistence, offers | Touching `chia/wallet/` |
|
||||
| [clvm-execution.md](clvm-execution.md) | CLVM execution paths, resource limits, canonical serialization, generator resolution, AGG_SIG | Touching puzzle execution, spend validation, generator logic |
|
||||
| [farmer.md](farmer.md) | Farmer proof flow, pool partial submission, reward targets, plot-sync receiver, solver management | Touching `chia/farmer/` |
|
||||
| [harvester.md](harvester.md) | Plot file management, PoS lookups, signage-point filter, plot sync | Touching `chia/harvester/` |
|
||||
| [timelord.md](timelord.md) | VDF scheduling, peak/unfinished-block selection, compact proof production | Touching `chia/timelord/` |
|
||||
| [plotting.md](plotting.md) | Plot creation, plot format, plot keys | Touching `chia/plotting/` |
|
||||
| [plot-sync.md](plot-sync.md) | Plot sync protocol, delta sync, sender/receiver state | Touching `chia/plot_sync/` |
|
||||
| [pools.md](pools.md) | Pool NFT / pool singleton state, pool wallet transitions, pool protocol payloads | Touching `chia/pools/` |
|
||||
| [daemon.md](daemon.md) | Daemon routing, keychain/process authority, service launch | Touching `chia/daemon/` |
|
||||
| [data-layer.md](data-layer.md) | DataLayer store mutations, root publication, mirror sync, proof verification | Touching `chia/data_layer/` |
|
||||
| [rpc.md](rpc.md) | RPC transport, error shapes, daemon websocket envelopes | Touching `chia/rpc/` |
|
||||
| [ssl.md](ssl.md) | Certificate generation, public/private CA material, SSL file permissions | Touching `chia/ssl/` |
|
||||
| [simulator.md](simulator.md) | Simulator block farming, reorg/revert helpers, BlockTools, service test harnesses | Touching `chia/simulator/` |
|
||||
| [solver.md](solver.md) | V2 plot partial proof solving, solver service, farmer coupling | Touching `chia/solver/` |
|
||||
| [seeder.md](seeder.md) | Crawler peer discovery, DNS seed responses, bootstrap-peer publication | Touching `chia/seeder/` |
|
||||
| [introducer.md](introducer.md) | Introducer peer collection, TCP vetting, DNS fallback | Touching `chia/introducer/` |
|
||||
| [cmds.md](cmds.md) | CLI command handlers, service start/stop wiring | Touching `chia/cmds/` |
|
||||
| [util.md](util.md) | DB wrapper, streamable, keychain, bech32m, error enum, config, caching | Touching `chia/util/` |
|
||||
| [benchmarks.md](benchmarks.md) | Benchmark harness, benchmark scripts, performance measurement | Touching `benchmarks/` |
|
||||
| [repo-tooling.md](repo-tooling.md) | Packaging, install scripts, CI workflows, build/release, GUI submodule, developer tools | Touching root config, `build_scripts/`, `.github/`, `tools/`, install scripts |
|
||||
| [global-invariants.md](global-invariants.md) | Cross-module invariants, state dependencies, trust boundaries, workflow traces, fragility clusters | Cross-cutting changes, security review, reorg-related work |
|
||||
|
||||
---
|
||||
|
||||
## Test guidance
|
||||
|
||||
Test harness selection and patterns live under `.cursor/context/testing/` and
|
||||
are routed by `.cursor/rules/testing-guide.mdc`. Read that rule when working in
|
||||
`chia/_tests/`.
|
||||
|
||||
| File | Covers |
|
||||
| ------------------------------------------------------ | -------------------------------------------------------------------------------------- |
|
||||
| [testing/architecture.md](testing/architecture.md) | Test architecture, consensus modes, fixture authority, CI, cross-subsystem correlation |
|
||||
| [testing/patterns.md](testing/patterns.md) | Block creation, transaction submission, assertion patterns |
|
||||
| [testing/blockchain.md](testing/blockchain.md) | Consensus/blockchain tests, reorg, overflow, fork invariants |
|
||||
| [testing/full-node.md](testing/full-node.md) | Full node sync, propagation, mempool-to-block, reorg tests |
|
||||
| [testing/mempool.md](testing/mempool.md) | Mempool acceptance/rejection, replacement, eviction |
|
||||
| [testing/data-layer.md](testing/data-layer.md) | DataStore logic, wallet-backed RPC, singleton lifecycle |
|
||||
| [testing/server.md](testing/server.md) | Connection lifecycle, API errors, DoS/ban, rate limiting |
|
||||
| [testing/wallet.md](testing/wallet.md) | Wallet fixtures, `wallet_environments`, wallet RPC tests |
|
||||
| [testing/clvm.md](testing/clvm.md) | Direct CLVM execution vs SpendSim |
|
||||
| [testing/cmds.md](testing/cmds.md) | CLI harness, mock RPC boundaries |
|
||||
| [testing/infrastructure.md](testing/infrastructure.md) | Shared harness, setup_nodes, SpendSim, convergence helpers |
|
||||
|
||||
---
|
||||
|
||||
## Quick reference — key files by size/complexity
|
||||
|
||||
| File | Lines | Role |
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
# Chia APIs Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/apis/` is the protocol metadata surface for service APIs. The classes are non-functional `Protocol` stubs, but their `ApiMetadata` decorators are runtime-relevant: outbound `WSChiaConnection.call_api()` uses remote stub metadata to decide whether a request can be sent, and local stub metadata to decode typed responses.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this when changing `chia/apis/*_stub.py`, `stub_protocol_registry.py`, API decorator metadata, request/reply declarations, or protocol-visible method names. For message schemas and numeric IDs, read `protocols.md`; for connection lifecycle and dispatch enforcement, read `server.md`.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- Stub decorators must mirror the concrete service API decorators for protocol-visible behavior. Drift can reject valid outbound calls or decode a valid response with the wrong local handler metadata.
|
||||
- The decorator derives the message type from the method name unless `request_type=` is set. Method names are therefore wire-contract inputs when no explicit request type is provided.
|
||||
- The first non-`self`/`peer` type hint is treated as the streamable payload class. Signature shape controls deserialization.
|
||||
- `peer_required`, `bytes_required`, and `execute_task` affect handler invocation, raw-byte preservation, and timeout behavior; they are not documentation-only flags.
|
||||
- `reply_types` on stubs are metadata for callable APIs, but runtime response validity is still controlled by `chia/protocols/protocol_state_machine.py`.
|
||||
- Sender authorization is separate. Allowed inbound node types are enforced by `ProtocolMessageTypeToNodeType`, not by stub presence.
|
||||
|
||||
## Change Guidance
|
||||
|
||||
- Adding a protocol method usually requires lockstep updates across payload class, `ProtocolMessageTypes`, sender map, state-machine reply/no-reply map, concrete API decorator, matching stub decorators, rate limits, and protocol tests.
|
||||
- Prefer `request_type=` when compatibility requires a Python method name that differs from the enum name.
|
||||
- Response-only messages still need local stub entries when they can be returned from `call_api()`, because the caller looks up response message metadata before deserializing.
|
||||
- When adding list-bearing payloads, check the concrete API/decorator path for `list_limits=` needs instead of relying on streamable types to bound adversarial input.
|
||||
- Do not infer consensus, wallet, farmer/harvester, timelord, solver, or DataLayer semantics from stubs. They declare wire-call shape only; validation and side effects live in concrete APIs and downstream services.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
For metadata machinery and runtime dispatch, read `chia/server/api_protocol.py` and `chia/server/ws_connection.py`. For stub registration, read `chia/apis/stub_protocol_registry.py`. For exact message IDs, sender authorization, and request/reply validity, read `chia/protocols/protocol_message_types.py`, `chia/protocols/protocol_message_type_to_node_type.py`, and `chia/protocols/protocol_state_machine.py`. For concrete behavior, read the matching service API under `chia/full_node/`, `chia/wallet/`, `chia/farmer/`, `chia/harvester/`, `chia/timelord/`, `chia/introducer/`, or `chia/solver/`.
|
||||
@@ -44,6 +44,17 @@ Other Chia packages use minimum-version pins. See `pyproject.toml` for current v
|
||||
| `chia/data_layer/` | DataLayer (data-storage singleton) | Medium |
|
||||
| `chia/cmds/` | CLI command handlers | Low |
|
||||
|
||||
## Package root
|
||||
|
||||
`chia/__init__.py`, `chia/__main__.py`, and `chia/py.typed` are process-wide entrypoint and namespace glue, not authorities for consensus, wallet state, P2P semantics, daemon privileges, or RPC behavior.
|
||||
|
||||
- `chia/__init__.py` resolves `__version__` from installed package metadata (falls back to `"unknown"` when unavailable). That value is visible in CLI output, daemon/RPC responses, peer handshakes, farmer pool headers, and logs.
|
||||
- Import-time runtime gates are process-wide: Python assertions are required and CPython free-threading is rejected, because consensus, networking, store, native-extension, async, and DB assumptions rely on those properties.
|
||||
- `chia/__main__.py` is a thin bridge to `chia.cmds.chia:main`; CLI behavior belongs in `chia/cmds/`.
|
||||
- `chia/py.typed` declares the package as typed for downstream consumers.
|
||||
- Console scripts in `pyproject.toml` are compatibility surfaces that must stay aligned with `chia.util.service_groups`, `chia start`, PyInstaller executable names, installer payloads, and GUI expectations. See `repo-tooling.md` for the full tooling contract.
|
||||
- Avoid adding package-root imports from heavy service modules; root imports run before root path, keys root, logging, config, and SSL checks are established.
|
||||
|
||||
## `chia_rs` boundary (largest external dependency)
|
||||
|
||||
Nearly all core consensus types live in Rust via `chia_rs`:
|
||||
@@ -115,7 +126,7 @@ Node roles are defined by `NodeType` in `chia/protocols/outbound_message.py`:
|
||||
|
||||
## Wire protocol overview
|
||||
|
||||
109 message types in `ProtocolMessageTypes` enum. Key flows:
|
||||
110 message types in `ProtocolMessageTypes` enum. Key flows:
|
||||
|
||||
- **Full Node ↔ Full Node**: `new_peak`, `new_transaction`, `request_block(s)`,
|
||||
`new_signage_point_or_end_of_sub_slot`, `request_compact_vdf`
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# Benchmarks Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`benchmarks/` is a set of standalone executable workloads for measuring performance of production Chia subsystems. It is not imported by node runtime code and it is not a pytest suite. Its value is in preserving representative pressure on hot paths: SQLite stores, mempool admission/rebuilds, streamable serialization, full-block JSON conversion, generator-reference lookup, test-chain materialization, and peer address persistence.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for changes to benchmark scripts, benchmark data setup, performance workload fidelity, benchmark output artifacts, or production API assumptions encoded by benchmarks.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- Benchmark scripts are consumers of production APIs, not alternate implementations. They should not contain benchmark-only behavior that production code depends on.
|
||||
- Most workloads intentionally bypass full service orchestration. They create fake or synthetic inputs directly for `BlockStore`, `CoinStore`, `MempoolManager`, `Blockchain`, `Streamable`, and `AddressManager`. This makes runs cheap and targeted, but it also means the benchmark is only as realistic as the synthetic state it builds.
|
||||
- The module depends heavily on `chia._tests.util.*` helpers for randomized blocks, reward coins, persistent test-chain data, and CLVM generator fixtures. Those helpers are test data factories, not stable public APIs.
|
||||
- Run benchmark scripts through the repository Python wrapper, for example `tools/py -m benchmarks.streamable`, to use the repo environment and pinned native dependencies.
|
||||
|
||||
## Data And Fidelity Assumptions
|
||||
|
||||
- Randomness is usually seeded to make comparisons reproducible. Preserve seeds unless the benchmark is deliberately being reshaped; unseeded randomness makes before/after comparisons noisy.
|
||||
- Synthetic block data in `block_store.py` is structurally valid enough for store serialization and lookup, but it does not pass through full consensus validation. Do not use its results to infer consensus validation cost.
|
||||
- `mempool.py` uses real wallet-generated signed transactions and reward coins, but its `BenchBlockRecord` is only the subset of `BlockRecord` consumed by `MempoolManager`. If mempool peak requirements change, this benchmark must be updated with the new fields or behavior.
|
||||
- `mempool-long-lived.py` uses an identity puzzle and empty aggregate signature to focus on mempool lifecycle cost. It is intentionally lower-fidelity for wallet signing and signature cost than `mempool.py`.
|
||||
- `block_ref.py` depends on an external full-node DB path and the sidecar `transaction_height_delta` fixture. It is the closest benchmark here to production chain shape, but it is read-only and measures generator reference lookup, not block validation.
|
||||
- `address_manager_store.py` mutates `AddressManager` internals directly to build a large address book quickly. This is useful for serialization cost, but fragile because `AddressManager` counts, matrices, and random-position lists are one consistency domain.
|
||||
|
||||
## Shared Infrastructure
|
||||
|
||||
- `benchmarks.utils.setup_db()` deletes the target DB before each run, enables WAL and `synchronous=full`, optionally logs SQL with `--sql-logging`, and yields a managed `DBWrapper2`. Store benchmarks depend on these settings for comparable write behavior.
|
||||
- `benchmarks.utils.get_commit_hash()` annotates streamable results with the short git hash and `-dirty` suffix. It changes process cwd to `benchmarks/`, so avoid calling it from code that assumes the original working directory remains stable.
|
||||
- Several scripts optionally emit cProfile artifacts through `gprof2dot` and Graphviz `dot`. These external tools are not Python dependencies; missing binaries break profiling output, not the measured production APIs.
|
||||
- Benchmarks write local artifacts such as `*-benchmark.db`, `.profile`, `.dot`, `.png`, `sql.log`, and optional JSON output. Treat these as generated files and keep them out of source changes unless the user explicitly asks for recorded results.
|
||||
|
||||
## Coupling To Production Contracts
|
||||
|
||||
- `BlockStore.add_full_block()` inserts blocks before canonical-chain state is set; benchmarks that call `set_in_chain()` and `set_peak()` are modeling the post-acceptance persistence sequence, not just raw insertion.
|
||||
- `CoinStore.new_block()` expects additions and removals to represent one block at a specific height/timestamp. Query benchmarks depend on the source-defined spent-index semantics for unspent and spent-at-height states.
|
||||
- `MempoolManager.new_peak()` has fast paths for simple transaction-block extension and slower paths for reorg or missing spent-coin information. Benchmarks should preserve both simple-extension and reorg-like paths because regressions often appear in different code.
|
||||
- `MempoolManager.pre_validate_spendbundle()` can use an executor and caches in-flight/seen bundle state; comparing inline and priority-thread-pool runs is part of the intended coverage.
|
||||
- `AddressManager.serialize_bytes()` is the current peers file format; service startup also supports an older migration path outside this benchmark. Serialization-only benchmarks should not be read as full peer-discovery coverage.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Changing benchmark constants can dominate results more than code changes. Keep `NUM_ITERS`, batch sizes, transaction-block cadence, and add/remove ratios stable when comparing branches.
|
||||
- Direct construction of `chia_rs` consensus types is version-sensitive. When native type constructors change, update benchmarks close to the production data model rather than adding compatibility wrappers around stale shapes.
|
||||
- SQLite benchmarks are sensitive to journal mode, synchronous setting, cache warmth, filesystem, and DB version. Preserve `setup_db()` settings when the goal is regression detection.
|
||||
- Avoid broad refactors that make benchmarks prettier but less targeted. These scripts intentionally duplicate setup so each workload can be run independently and fail close to the subsystem it measures.
|
||||
- If production APIs gain stricter invariants, prefer making benchmark synthetic state satisfy those invariants over weakening production code for benchmark convenience.
|
||||
|
||||
## Verification Guidance
|
||||
|
||||
- Syntax/import smoke checks: run the streamable and address-manager benchmark modules with their smallest practical iteration settings for non-DB subsystem coverage.
|
||||
- Store benchmarks are heavier and write DB files: `tools/py -m benchmarks.coin_store` and `tools/py -m benchmarks.block_store`.
|
||||
- `benchmarks.block_ref` requires a full-node database path argument and the sidecar transaction-height fixture; use it only when generator-reference lookup against realistic chain history is relevant.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Benchmark workloads: `benchmarks/`.
|
||||
- Shared benchmark helpers: `benchmarks/utils.py`.
|
||||
- Production consumers commonly exercised here: `chia/full_node/`, `chia/types/`, `chia/server/address_manager.py`.
|
||||
@@ -10,7 +10,6 @@
|
||||
| `chia/consensus/condition_tools.py` | `pkm_pairs()` — extract AGG_SIG conditions |
|
||||
| `chia/consensus/generator_tools.py` | `get_block_header()` — strip generator from block |
|
||||
| `chia/consensus/get_block_generator.py` | `get_block_generator()` — resolve generator refs |
|
||||
| `chia/consensus/cost_calculator.py` | `NPCResult` — name/puzzle/conditions result |
|
||||
| `chia/consensus/condition_costs.py` | Condition opcode costs |
|
||||
| `chia/types/blockchain_format/program.py` | CLVM program wrapper |
|
||||
| `chia/types/blockchain_format/serialized_program.py` | Lazy deserialization |
|
||||
@@ -57,9 +56,7 @@ Mempool items → solution_generator_backrefs() [Rust] → generator program
|
||||
|
||||
- Combines spend bundles into a single block generator
|
||||
- Uses back-references for compression
|
||||
- Legacy path: `create_block_generator()` serializes with `solution_generator_backrefs()`
|
||||
- Alternative path: `create_block_generator2()` uses Rust `BlockBuilder`; opt-in today via
|
||||
`full_node.config["block_creation"] = 1` (see TODO in `full_node_api.py` to make it default)
|
||||
- Default path: `create_block_generator2()` uses Rust `BlockBuilder`, selected by `full_node.config["block_creation"] = 1` (the default). Legacy path `create_block_generator()` is opt-in via `block_creation = 0`. See `full_node_api.py` block-version selection.
|
||||
|
||||
---
|
||||
|
||||
@@ -123,51 +120,9 @@ length has a minimum atom size threshold.
|
||||
|
||||
## Condition opcodes
|
||||
|
||||
**Location**: `chia/types/condition_opcodes.py`
|
||||
Condition opcode byte values are a CLVM output ABI contract owned by `chia/types/condition_opcodes.py`; see `types.md` for the compatibility invariant. Read the enum directly for exact values rather than relying on a copied table, since renumbering or reusing a byte value changes consensus and wallet behavior.
|
||||
|
||||
Key conditions (from CLVM spend output):
|
||||
|
||||
| Opcode | Name | Effect |
|
||||
| ------ | ---------------------------- | ----------------------------------------- |
|
||||
| 43 | `AGG_SIG_PARENT` | Require signature with parent data |
|
||||
| 44 | `AGG_SIG_PUZZLE` | Require signature with puzzle hash data |
|
||||
| 45 | `AGG_SIG_AMOUNT` | Require signature with amount data |
|
||||
| 46 | `AGG_SIG_PUZZLE_AMOUNT` | Require signature with puzzle+amount data |
|
||||
| 47 | `AGG_SIG_PARENT_AMOUNT` | Require signature with parent+amount data |
|
||||
| 48 | `AGG_SIG_PARENT_PUZZLE` | Require signature with parent+puzzle data |
|
||||
| 49 | `AGG_SIG_UNSAFE` | Require signature (no domain separation) |
|
||||
| 50 | `AGG_SIG_ME` | Require signature with coin data |
|
||||
| 51 | `CREATE_COIN` | Create a new coin |
|
||||
| 52 | `RESERVE_FEE` | Declare minimum fee |
|
||||
| 60 | `CREATE_COIN_ANNOUNCEMENT` | Create announcement |
|
||||
| 61 | `ASSERT_COIN_ANNOUNCEMENT` | Assert announcement exists |
|
||||
| 62 | `CREATE_PUZZLE_ANNOUNCEMENT` | Create puzzle announcement |
|
||||
| 63 | `ASSERT_PUZZLE_ANNOUNCEMENT` | Assert puzzle announcement |
|
||||
| 64 | `ASSERT_CONCURRENT_SPEND` | Assert another coin is spent |
|
||||
| 65 | `ASSERT_CONCURRENT_PUZZLE` | Assert puzzle hash is spent |
|
||||
| 66 | `SEND_MESSAGE` | Cross-coin messaging |
|
||||
| 67 | `RECEIVE_MESSAGE` | Cross-coin messaging |
|
||||
| 70 | `ASSERT_MY_COIN_ID` | Assert own coin ID |
|
||||
| 71 | `ASSERT_MY_PARENT_ID` | Assert parent coin ID |
|
||||
| 72 | `ASSERT_MY_PUZZLEHASH` | Assert own puzzle hash |
|
||||
| 73 | `ASSERT_MY_AMOUNT` | Assert own amount |
|
||||
| 74 | `ASSERT_MY_BIRTH_SECONDS` | Assert creation timestamp |
|
||||
| 75 | `ASSERT_MY_BIRTH_HEIGHT` | Assert creation height |
|
||||
| 76 | `ASSERT_EPHEMERAL` | Assert coin is ephemeral |
|
||||
| 90 | `SOFTFORK` | Future-proof softfork condition |
|
||||
|
||||
### Timelock conditions
|
||||
|
||||
| Opcode | Name | Effect |
|
||||
| ------ | -------------------------------- | ------------------------------ |
|
||||
| 80 | `ASSERT_SECONDS_RELATIVE` | Min seconds since confirmation |
|
||||
| 81 | `ASSERT_SECONDS_ABSOLUTE` | Min timestamp |
|
||||
| 82 | `ASSERT_HEIGHT_RELATIVE` | Min blocks since confirmation |
|
||||
| 83 | `ASSERT_HEIGHT_ABSOLUTE` | Min block height |
|
||||
| 84 | `ASSERT_BEFORE_SECONDS_RELATIVE` | Max seconds since confirmation |
|
||||
| 85 | `ASSERT_BEFORE_SECONDS_ABSOLUTE` | Max timestamp |
|
||||
| 86 | `ASSERT_BEFORE_HEIGHT_RELATIVE` | Max blocks since confirmation |
|
||||
| 87 | `ASSERT_BEFORE_HEIGHT_ABSOLUTE` | Max block height |
|
||||
Timelock conditions (`ASSERT_SECONDS_*`, `ASSERT_HEIGHT_*`, `ASSERT_BEFORE_*`) and `AGG_SIG_*` variants are interpreted by Rust validation and full-node/wallet code, not by the `ConditionWithArgs` dataclass. Semantic argument counts and domain-separation rules live with the validators.
|
||||
|
||||
---
|
||||
|
||||
@@ -175,17 +130,7 @@ Key conditions (from CLVM spend output):
|
||||
|
||||
### `AGG_SIG_ME_ADDITIONAL_DATA`
|
||||
|
||||
Mainnet: `ccd5bb71183532bff220ba46c268991a3ff07eb358e8255a65c30a2dce0e5fbb`
|
||||
|
||||
Each AGG_SIG variant appends different additional data:
|
||||
|
||||
- `AGG_SIG_PARENT`: `hash(data + 43)`
|
||||
- `AGG_SIG_PUZZLE`: `hash(data + 44)`
|
||||
- `AGG_SIG_AMOUNT`: `hash(data + 45)`
|
||||
- etc.
|
||||
|
||||
This provides replay protection across forks. Forks MUST change
|
||||
`AGG_SIG_ME_ADDITIONAL_DATA` to prevent cross-chain replays.
|
||||
Each network defines its `AGG_SIG_ME_ADDITIONAL_DATA` value (read it from the active constants, not a hard-coded mainnet value). Each AGG_SIG variant appends a different opcode-derived suffix to the additional data before signing, providing per-condition domain separation. Forks MUST change `AGG_SIG_ME_ADDITIONAL_DATA` to prevent cross-chain replay attacks.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
# Chia Cmds Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/cmds/` is the user-facing CLI adapter layer. It owns command-line parsing,
|
||||
local config/keychain setup, service/RPC client selection, terminal formatting,
|
||||
and orchestration of daemon/service commands. It should not be treated as a
|
||||
business-logic authority: most durable state, spend construction, consensus
|
||||
validation, network behavior, and service lifecycles live in sibling modules and
|
||||
are reached through config, keychain, daemon, and RPC boundaries.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for CLI command parsing, Click/dataclass command wiring, config/keychain setup from commands, RPC-client selection, transaction-output files, service start/stop commands, and terminal-facing behavior. For endpoint side effects, also read the concrete service context.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `chia.py` is the root Click command tree. It creates `ChiaCliContext`, records
|
||||
`root_path` and `keys_root_path`, configures the keyring root, optionally
|
||||
reads/caches a keyring passphrase, runs SSL permission checks, and registers
|
||||
every top-level command group.
|
||||
- `ChiaCliContext` is the in-process CLI context authority. Commands read root
|
||||
paths, key paths, expected address prefix caches, RPC ports, and selected
|
||||
fingerprints from this object through Click's `ctx.obj`; avoid side channels
|
||||
for values that belong there.
|
||||
- `cmd_classes.py` is the newer dataclass-command DSL. `@chia_command` freezes a
|
||||
command class, turns `option()` fields into Click options, recursively expands
|
||||
`@command_helper` fields, injects `ChiaCliContext` only for fields named
|
||||
`context`, and runs sync or async `run()` methods via `asyncio.run()`.
|
||||
- Plain Click modules remain common. Many commands are thin parser wrappers that
|
||||
import their `*_funcs` implementation lazily and dispatch with `asyncio.run()`.
|
||||
This split is intentional for import cost and dependency isolation, but it
|
||||
means behavior can be split between a parser file and a sibling funcs file.
|
||||
- `cmds_util.get_any_service_client()` is the generic RPC-client boundary for
|
||||
farmer, wallet, full node, harvester, data layer, simulator, and solver RPC
|
||||
clients. It loads config, chooses default ports, opens SSL or non-SSL clients,
|
||||
calls `healthz()`, and normalizes common connection/RPC errors for CLI output.
|
||||
- `start_funcs.py` and `stop.py` cross into the daemon websocket authority. The
|
||||
CLI launches `chia run_daemon --wait-for-unlock`, unlocks the daemon keyring
|
||||
when needed, and starts/stops service names resolved from
|
||||
`chia.util.service_groups`.
|
||||
|
||||
## Command Execution Model
|
||||
|
||||
- Top-level options are applied before subcommand execution. Any code that needs
|
||||
the selected root path, key root, or cached address prefix should read
|
||||
`ChiaCliContext.set_default(ctx)` rather than re-resolving defaults.
|
||||
- Async commands assume no running event loop in the CLI process. Both plain
|
||||
Click commands and dataclass commands generally use `asyncio.run()` at the
|
||||
command boundary; nested event-loop ownership belongs below RPC/service code,
|
||||
not inside command parsing.
|
||||
- Parser modules should do CLI-specific validation and conversion only:
|
||||
command-line shape, Click option types, basic argument consistency, user
|
||||
prompts, file input/output, and terminal messages. Wallet, full-node, pool,
|
||||
data-layer, DB, and keychain semantics should stay in sibling modules or the
|
||||
`*_funcs.py` command implementation files.
|
||||
- Several commands deliberately lazy-import heavy dependencies inside functions.
|
||||
Moving imports to module scope can change CLI startup time, optional dependency
|
||||
failures, and command help behavior.
|
||||
- `rpc.py` is an escape-hatch client. It dynamically adds one command per
|
||||
service, accepts raw endpoint names plus JSON request data, and bypasses typed
|
||||
service-specific command helpers. Treat it as an operational tool, not the
|
||||
canonical API contract.
|
||||
|
||||
## Data Conversion Contracts
|
||||
|
||||
- `param_types.py` is the CLI conversion authority for fees, wallet amounts,
|
||||
addresses, `bytes32`, and `uint64`. Fees are decimal XCH strings capped at
|
||||
a source-defined XCH cap and converted to mojos; wallet amounts are stored as `CliAmount`
|
||||
until the target wallet unit is known.
|
||||
- `AddressParamType` validates the selected network prefix for XCH/TXCH
|
||||
addresses by consulting `ChiaCliContext.expected_prefix` or loading
|
||||
`config.yaml`. Non-XCH address families are mapped through `AddressType`.
|
||||
Reusing it outside Click must account for possible config reads.
|
||||
- `CMDTXConfigLoader` and `CMDCoinSelectionConfigLoader` bridge CLI amount/coin
|
||||
filters to wallet `TXConfig` and `CoinSelectionConfig`, autofilling with
|
||||
consensus constants and wallet config/fingerprint data.
|
||||
- `tx_out_cmd()` and `TransactionEndpoint` are the transaction-output boundary.
|
||||
They add `--push/--no-push`, optional transaction-file output, coin-selection
|
||||
options, and absolute timelocks, then serialize `TransactionRecord` lists as a
|
||||
`TransactionBundle`. Commands that create wallet transactions should either use
|
||||
this path or explicitly document why their RPC shape is incompatible.
|
||||
- `TransactionEndpoint.__post_init__()` enforces that subclasses decorate `run()`
|
||||
with `@transaction_endpoint_runner`; this keeps file-output handling from being
|
||||
silently skipped in dataclass-style transaction commands.
|
||||
|
||||
## Service And State Coupling
|
||||
|
||||
- Wallet commands are the widest surface. `wallet.py`, `coins.py`, `plotnft.py`,
|
||||
and `signer.py` parse user intent, while `wallet_funcs.py`, `coin_funcs.py`,
|
||||
and `plotnft_funcs.py` call `WalletRpcClient`, format balances/transactions,
|
||||
resolve wallet units, and handle offer/NFT/DID/VC/notification flows.
|
||||
- `NeedsWalletRPC.wallet_rpc()` centralizes wallet RPC selection and login. If no
|
||||
fingerprint is supplied, `get_wallet()` may query the keychain, inspect the
|
||||
wallet's logged-in key and sync state, prompt the user, then call wallet
|
||||
`log_in()`. This is observable behavior, not a trivial client factory.
|
||||
- Data-layer commands use `DataLayerRpcClient` through `data_funcs.get_client()`
|
||||
and can also log into the wallet by fingerprint. Some DataLayer transaction
|
||||
endpoints are not wired through the generic transaction-output decorators due
|
||||
to API peculiarities; observer-only/offline transaction behavior is therefore
|
||||
uneven across command groups.
|
||||
- `init_funcs.py`, `configure.py`, and DB commands mutate local files. Config
|
||||
edits must use `lock_and_load_config()` for read-modify-write safety, and DB
|
||||
upgrade/backup/validate commands derive default paths from `full_node`
|
||||
selected-network config.
|
||||
- Key commands operate on local key custody. `keys_funcs.py` unlocks the keyring,
|
||||
adds private or observer keys, displays derived keys/addresses, signs and
|
||||
verifies messages, and updates config reward targets through `init_funcs`.
|
||||
Treat `--show-mnemonic-seed` and file-based key input/output as sensitive
|
||||
terminal/file surfaces.
|
||||
- Pool/plot NFT commands cross three authorities: wallet RPC, farmer RPC, and
|
||||
external pool HTTPS endpoints. Mainnet pool joins enforce HTTPS and pool
|
||||
protocol/relative-lock-height checks before wallet RPC submission.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Mixed command frameworks create duplication: old Click decorators, helper
|
||||
decorators in `cmds_util.py`, and dataclass helpers in `cmd_helpers.py` can
|
||||
express similar options with subtly different defaults, names, and timelock
|
||||
visibility.
|
||||
- `get_any_service_client()` consumes many errors by printing and not re-raising
|
||||
unless `consume_errors=False`. Tests rely on this distinction; callers that
|
||||
need programmatic failure must opt out of consumption.
|
||||
- CLI-facing validation is often user-experience oriented rather than complete
|
||||
domain validation. Do not rely on parser checks as wallet, consensus, or DB
|
||||
safety checks.
|
||||
- Several commands use `sys.exit()`, `print()`, `input()`, and `click.Abort`
|
||||
directly. This is expected for terminal UX but makes library reuse and tests
|
||||
sensitive to stdout/stderr and prompting behavior.
|
||||
- Address-prefix caching in `ChiaCliContext` is per Click invocation. Changes
|
||||
that bypass context can re-read config repeatedly or validate against the
|
||||
wrong network.
|
||||
- DB upgrade and config commands perform irreversible-looking filesystem
|
||||
operations, but usually leave original DBs untouched or require explicit
|
||||
output paths. Preserve those guardrails and user confirmations when changing
|
||||
operational commands.
|
||||
|
||||
## Test And Audit Strategy
|
||||
|
||||
- CLI tests usually use `click.testing.CliRunner`, temporary Chia roots, and
|
||||
monkeypatched RPC client factories from `chia/_tests/cmds/cmd_test_utils.py`.
|
||||
Prefer mocked RPC clients for command parsing/dispatch tests over simulator
|
||||
harnesses unless the behavior truly depends on chain state.
|
||||
- `test_cmd_framework.py` protects dataclass parsing, helper recursion, context
|
||||
injection, optional/sequence type handling, and transaction-endpoint
|
||||
invariants. Extend these when changing `cmd_classes.py` or `cmd_helpers.py`.
|
||||
- `test_click_types.py` protects amount/fee/address/bytes32/uint64 conversion,
|
||||
including network-prefix validation and decimal precision. Any change to
|
||||
`param_types.py` should update these cases before broad wallet command tests.
|
||||
- `test_cmds_util.py` protects RPC-client creation and error-consumption
|
||||
semantics. Changes to connection handling should test both consumed and
|
||||
propagated error modes.
|
||||
- Wallet command tests assert RPC request shapes and terminal output using test
|
||||
RPC clients. When adding transaction-producing wallet commands, include both
|
||||
`--push/--no-push` and transaction-file behavior if the command participates in
|
||||
the standard transaction-output path.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Root CLI and command context: `chia/cmds/chia.py`, `chia/cmds/cmd_classes.py`, `chia/cmds/cmd_helpers.py`.
|
||||
- Common RPC/config/keychain helpers: `chia/cmds/cmds_util.py`, `chia/cmds/init_funcs.py`, `chia/cmds/configure.py`.
|
||||
- Wallet and transaction command surfaces: `chia/cmds/wallet.py`, `chia/cmds/wallet_funcs.py`, `chia/cmds/coin_funcs.py`, `chia/cmds/param_types.py`.
|
||||
- Service control commands: `chia/cmds/start_funcs.py`, `chia/cmds/stop.py`, `chia/cmds/rpc.py`.
|
||||
+132
-214
@@ -1,280 +1,198 @@
|
||||
# Consensus Layer — Deep Context
|
||||
# chia-consensus
|
||||
|
||||
> Attach when touching `chia/consensus/`, block acceptance, reorgs, difficulty,
|
||||
> or VDF iteration logic.
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
## File map
|
||||
Scope: `chia/consensus/`. This is distilled architectural context for future audit or implementation agents. It intentionally omits file inventory and obvious helper summaries.
|
||||
|
||||
| File | Lines | Role |
|
||||
| ------------------------------- | ----- | ------------------------------------------------------------------------------------- |
|
||||
| `blockchain.py` | ~1090 | `Blockchain` class: chain state, `add_block()`, `_reconsider_peak()` |
|
||||
| `blockchain_interface.py` | ~54 | Protocol definitions: `BlockRecordsProtocol`, `BlocksProtocol`, `BlockchainInterface` |
|
||||
| `augmented_chain.py` | ~169 | `AugmentedBlockchain`: in-memory overlay for parallel validation |
|
||||
| `block_header_validation.py` | ~1060 | `validate_unfinished_header_block()`: all header checks |
|
||||
| `block_body_validation.py` | ~580 | `validate_block_body()`: transaction/coin checks, `ForkInfo` |
|
||||
| `block_creation.py` | ~650 | `create_unfinished_block()`, `unfinished_block_to_full_block()` |
|
||||
| `difficulty_adjustment.py` | ~410 | Difficulty & sub-slot-iters recalculation per epoch |
|
||||
| `pot_iterations.py` | ~101 | SP/IP iteration math, quality → required_iters |
|
||||
| `pos_quality.py` | ~29 | Expected plot size calculation |
|
||||
| `block_rewards.py` | ~54 | Pool/farmer reward schedule (halving) |
|
||||
| `coinbase.py` | ~25 | Pool/farmer coin creation from block height |
|
||||
| `full_block_to_block_record.py` | ~170 | `block_to_block_record()` |
|
||||
| `find_fork_point.py` | ~109 | Fork point search between two chains |
|
||||
| `get_block_challenge.py` | ~170 | Challenge computation per block |
|
||||
| `make_sub_epoch_summary.py` | ~250 | Sub-epoch summary creation |
|
||||
| `deficit.py` | ~55 | Deficit calculation for sub-epoch boundaries |
|
||||
| `multiprocess_validation.py` | ~300 | `PreValidationResult`, parallel block validation |
|
||||
| `condition_tools.py` | ~200 | `pkm_pairs()` for AGG_SIG conditions |
|
||||
| `signage_point.py` | ~10 | `SignagePoint` dataclass |
|
||||
| `vdf_info_computation.py` | ~180 | VDF info reconstruction from block records |
|
||||
| `default_constants.py` | ~119 | All consensus constant values |
|
||||
| `constants.py` | ~50 | `replace_str_to_bytes()` for config overrides |
|
||||
## When To Read This
|
||||
|
||||
---
|
||||
Read this for block validation, fork choice, reorg coin-state replay, difficulty/sub-slot logic, reward validation, VDF/POS validation, and hard-fork commitment behavior. If the task is peer sync, mempool admission, wallet state, or RPC shape, start with that module's context and return here only for consensus authority.
|
||||
|
||||
## `Blockchain.add_block()` — Core block acceptance
|
||||
## Landmarks
|
||||
|
||||
**Location**: `consensus/blockchain.py:294`
|
||||
| file | owns |
|
||||
| ------------------------------------------- | --------------------------------------------- |
|
||||
| `chia/consensus/blockchain.py` | add_block() commit authority, peak/fork state |
|
||||
| `chia/consensus/block_body_validation.py` | coin/tx state transition checks, ForkInfo use |
|
||||
| `chia/consensus/block_header_validation.py` | proof/timing/slot/header checks |
|
||||
| `chia/consensus/augmented_chain.py` | speculative overlay for batch validation |
|
||||
| `chia/consensus/blockchain_mmr.py` | post-HF2 header MMR state |
|
||||
| `chia/consensus/pot_iterations.py` | iteration/difficulty math |
|
||||
| `chia/consensus/default_constants.py` | consensus constants |
|
||||
|
||||
### Purpose
|
||||
## Implementation Authority
|
||||
|
||||
Single entry point for adding a validated block. Determines if a block
|
||||
becomes the new peak, an orphan, or is rejected.
|
||||
`chia.consensus` is the local authority for deciding whether a block is valid and whether it becomes the canonical peak. It does not own networking, sync orchestration, or wallet notifications; those live in full-node modules. It does own the consensus-facing view of chain state: block records, height-to-hash mapping, fork-local additions/removals, difficulty and sub-slot iteration transitions, VDF/POS validation, reward/fee/body checks, and post-HF2 header commitments.
|
||||
|
||||
### Inputs & assumptions
|
||||
Most public paths split validation into two stages:
|
||||
|
||||
- `block: FullBlock` — the full block to add
|
||||
- `pre_validation_result: PreValidationResult` — must have valid
|
||||
`required_iters` and no error (pre-validation already ran in parallel)
|
||||
- `sub_slot_iters: uint64` — pre-computed for this block's epoch
|
||||
- `fork_info: ForkInfo` — must correctly describe the fork context
|
||||
- **Caller holds the blockchain lock** (`priority_mutex`)
|
||||
- Header validation already passed via `validate_unfinished_header_block()`
|
||||
- Pre-validation computes `required_iters`, runs generator/signature work, validates finished headers, and builds an in-memory `BlockRecord`.
|
||||
- `Blockchain.add_block()` revalidates body state under the blockchain lock, writes durable stores, applies fork choice, updates height map and MMR, then advances `_peak_height`.
|
||||
|
||||
### Return
|
||||
That split is a core safety boundary. Pre-validation is allowed to be parallel and speculative. `add_block()` is the sequential commit point.
|
||||
|
||||
`(AddBlockResult, Err | None, StateChangeSummary | None)`
|
||||
## Consensus State And Ownership
|
||||
|
||||
### Block-by-block logic
|
||||
`Blockchain` coordinates four state stores that must agree after a committed peak change:
|
||||
|
||||
1. **Genesis check** (L326): `height == 0` requires `prev_header_hash ==
|
||||
GENESIS_CHALLENGE`.
|
||||
- `BlockStore`: full block bytes, block records, `in_main_chain`, peak pointer, sub-epoch challenge segments.
|
||||
- `CoinStore`: UTXO state for the current canonical peak only.
|
||||
- `BlockHeightMap`: dense canonical height -> hash cache plus canonical sub-epoch summaries.
|
||||
- `BlockchainMMRManager`: post-HF2 header-MMR state over finalized canonical block hashes.
|
||||
|
||||
2. **Extending main chain?** (L331): Fast path when `prev_header_hash ==
|
||||
peak.header_hash`.
|
||||
The update ordering in `Blockchain.add_block()` is deliberate:
|
||||
|
||||
3. **Disconnected block** (L337-343): If prev block not in cache →
|
||||
`DISCONNECTED_BLOCK`. Invariant: we only accept blocks connected to known
|
||||
chain.
|
||||
- full block and peak-related DB writes happen inside `block_store.transaction()`;
|
||||
- in-memory block record cache is updated after async DB work inside that transaction;
|
||||
- `height_map` and MMR are updated only after the DB transaction commits;
|
||||
- `_peak_height` is assigned last, after dependent stores can answer lookups for the new peak.
|
||||
|
||||
4. **Pre-validation error** (L345-348): Reject immediately on any error.
|
||||
If this ordering changes, readers can observe a peak whose block, height map entry, MMR state, or coin records are not yet available.
|
||||
|
||||
5. **ForkInfo assertions** (L354-366): Multiple assertions verify fork_info
|
||||
consistency. Incorrect fork_info → assertion failure (crash, not silent
|
||||
corruption).
|
||||
## Fork Choice And Reorg Contract
|
||||
|
||||
6. **Already-have-block** (L372-380): Even known blocks update fork_info
|
||||
(important for parallel batch validation).
|
||||
Fork choice is purely:
|
||||
|
||||
7. **Body validation** (L392-403): `validate_block_body()` checks coins,
|
||||
merkle roots, rewards, timelocks.
|
||||
- greater weight wins;
|
||||
- equal weight with lower `total_iters` wins;
|
||||
- otherwise current peak remains.
|
||||
|
||||
8. **Block record creation** (L415-423): `block_to_block_record()` computes
|
||||
lightweight record.
|
||||
`ForkInfo` is the bridge between validation and reorg application. It is not just metadata. It must contain every addition/removal/reward coin across the candidate fork range, with `block_hashes` ordered by height. `_reconsider_peak()` replays coin-store state from this object when a fork becomes peak. If `ForkInfo` is incomplete, the reorg can validate but apply the wrong coin set.
|
||||
|
||||
9. **Atomic DB transaction** (L432-476):
|
||||
- `add_full_block()` → `_reconsider_peak()` → `add_block_record()`
|
||||
- On success: update `_peak_height` and height map
|
||||
- On failure: rollback in-memory state, fork_info, block store cache
|
||||
Important sequencing rules:
|
||||
|
||||
### Invariants
|
||||
- `fork_info.peak_height` must line up with the candidate block's parent height before body validation.
|
||||
- Genesis and non-genesis candidates have different previous-hash expectations before body validation.
|
||||
- `fork_info.block_hashes` must cover the expected post-fork range exactly.
|
||||
- For main-chain extension, `fork_info.reset(...)` clears fork history before validating the new block.
|
||||
- For known/orphan/fork blocks, `advance_fork_info()` may replay intermediate full blocks from `BlockStore` to rebuild additions/removals before the current block is handled.
|
||||
|
||||
- `fork_info.peak_height == block.height - 1` before body validation
|
||||
- `block.height == 0 or fork_info.peak_hash == block.prev_header_hash`
|
||||
- Database transaction atomicity: no partial state updates
|
||||
- `_peak_height` only updated after commit
|
||||
The code uses assertions for many of these contracts. Treat assertion failures here as consensus-state corruption or caller misuse, not as normal invalid-block handling.
|
||||
|
||||
---
|
||||
## Validation Layering
|
||||
|
||||
## `_reconsider_peak()` — Fork choice rule
|
||||
Header validation answers “is the block’s proof/timing/slot/header structure valid?” Body validation answers “does this block correctly transform transaction and coin state?”
|
||||
|
||||
**Location**: `consensus/blockchain.py:486`
|
||||
Header validation depends on `ValidationState` (`ssi`, difficulty, previous SES block). `pre_validate_block()` mutates the caller-provided `ValidationState` while queuing batch validation, then passes a copy into the worker. Full node batch sync relies on this: the mutable `ValidationState` speculatively advances `ssi`/`difficulty` from raw `new_*` fields for scheduling, while `add_block()` later receives the preserved state needed for commit. Safety comes from `validate_finished_header_block()` rejecting those raw fields unless a valid `subepoch_summary_hash` accompanies them.
|
||||
|
||||
### Fork choice criteria (in order)
|
||||
Difficulty and SSI transitions must flow through a validated sub-epoch summary. `validate_finished_header_block()` rejects raw `new_difficulty` or `new_sub_slot_iters` fields unless the finished sub-slot also carries a valid `subepoch_summary_hash`. `block_to_block_record()` converts that validated hash into `block_rec.sub_epoch_summary_included`, and that computed block-record field is the authoritative source for committed `ValidationState` advancement; do not treat peer-provided `block.finished_sub_slots[0].challenge_chain.new_*` fields as trusted state on their own.
|
||||
|
||||
1. Higher weight wins
|
||||
2. On equal weight: lower `total_iters` wins
|
||||
3. Otherwise: no change (current peak stays)
|
||||
Body validation relies on header validation already being complete. It checks:
|
||||
|
||||
### Reorg handling
|
||||
- transaction-block vs non-transaction-block field presence;
|
||||
- reward claim exactness for skipped non-transaction blocks since the previous transaction block;
|
||||
- generator root/ref roots, ref count limits, and post-SF9 generator-ref ban;
|
||||
- CLVM cost and canonical generator encoding after SF9;
|
||||
- addition/removal Merkle roots and BIP158 transaction filter;
|
||||
- duplicate additions/removals within the block;
|
||||
- coin existence and double-spend rules across DB state plus `ForkInfo`;
|
||||
- value conservation, reserve fee, farmer reward overflow bound, and declared fee equality;
|
||||
- puzzle hash consistency for removals;
|
||||
- Rust time-lock evaluation using previous transaction block height/timestamp;
|
||||
- presence of an aggregate signature after pre-validation has already verified it.
|
||||
|
||||
- `coin_store.rollback_to_block(fork_info.fork_height)` removes coins above
|
||||
fork point
|
||||
- Replays all additions/removals from `fork_info` for the new chain
|
||||
- `block_store.rollback()` clears sub-epoch summaries above fork point
|
||||
- `block_store.set_in_chain()` marks new chain blocks
|
||||
- `block_store.set_peak()` updates stored peak
|
||||
The subtle part is coin lookup on forks. A removal may be:
|
||||
|
||||
### Assumption
|
||||
- ephemeral, created earlier in the same block;
|
||||
- from canonical DB state at or before the fork point;
|
||||
- created inside `fork_info.additions_since_fork`;
|
||||
- invalid because it only exists on the old canonical branch after the fork.
|
||||
|
||||
`fork_info.additions_since_fork` and `fork_info.removals_since_fork` contain
|
||||
ALL coin operations from `fork_height + 1` to the new peak. Incomplete data →
|
||||
inconsistent coin store.
|
||||
Do not simplify this into “query coin store and check spent”. That loses fork semantics.
|
||||
|
||||
---
|
||||
## Parallel Validation Overlay
|
||||
|
||||
## `ForkInfo` — Fork tracking state
|
||||
`AugmentedBlockchain` lets batch validation see blocks that are not committed yet. It is a chain overlay, not a passive cache:
|
||||
|
||||
**Location**: `consensus/block_body_validation.py:62`
|
||||
- extra blocks must be added contiguously;
|
||||
- height-to-hash lookups prefer the overlay;
|
||||
- fork ancestry is populated when the batch starts on a non-canonical branch;
|
||||
- generator references first walk overlay blocks, then delegate to the underlying chain;
|
||||
- the MMR manager is deep-copied so speculative MMR roots do not mutate canonical MMR state;
|
||||
- worker jobs receive `read_only_snapshot()` to prevent mutation from the executor path.
|
||||
|
||||
### Fields
|
||||
After a block successfully commits, full-node batch code removes it from the overlay. If the overlay and canonical chain disagree about height mapping, pre-validation can validate against a different ancestor path than `add_block()` later commits.
|
||||
|
||||
- `fork_height: int` — last block shared by fork and main chain
|
||||
- `peak_height: int` — height of the fork tip (-1 for genesis validation)
|
||||
- `peak_hash: bytes32`
|
||||
- `additions_since_fork: dict[bytes32, ForkAdd]` — all coin additions since fork
|
||||
- `removals_since_fork: dict[bytes32, ForkRem]` — all coin removals since fork
|
||||
- `block_hashes: list[bytes32]` — ordered header hashes from fork_height+1
|
||||
## Difficulty, Slots, And Time Coupling
|
||||
|
||||
### Critical methods
|
||||
Difficulty and sub-slot iterations are recomputed only at eligible epoch boundaries. Both use transaction-block timestamps around previous/current epoch reference points, are clamped by `DIFFICULTY_CHANGE_MAX_FACTOR`, then truncated to `SIGNIFICANT_BITS`; SSI is additionally rounded down to a multiple of `NUM_SPS_SUB_SLOT`.
|
||||
|
||||
- `reset()` — clear all fork state (used when extending main chain)
|
||||
- `update_fork_peak()` — advance peak, append header hash
|
||||
- `include_spends()` — record additions/removals from `SpendBundleConditions`
|
||||
- `include_reward_coins()` — record coinbase additions
|
||||
- `rollback()` — undo to a previous height
|
||||
Epoch/sub-epoch completion depends on:
|
||||
|
||||
### Invariant
|
||||
- sufficient height;
|
||||
- zero deficit;
|
||||
- whether a sub-epoch summary was already included;
|
||||
- whether the next height can be first in an epoch;
|
||||
- lookback through previous records, sometimes through fork paths rather than canonical height lookups.
|
||||
|
||||
`len(block_hashes) == peak_height - fork_height` — always.
|
||||
Overflow block handling changes both challenge selection and the “previous transaction block at signage point” calculation. Several hard-fork gates use `pre_sp_tx_block_height(...)`, not candidate height. This matters especially near HF2/SF9 boundaries.
|
||||
|
||||
---
|
||||
## Hard Fork Commitments
|
||||
|
||||
## `validate_block_body()` — Block body validation
|
||||
The hard-fork commitment rules introduce two commitments in this module:
|
||||
|
||||
**Location**: `consensus/block_body_validation.py:190`
|
||||
- sub-epoch summary `challenge_root` via `make_sub_epoch_summary(..., make_challenge_root=True)`;
|
||||
- `reward_chain_block.header_mmr_root`, validated in finished-header validation.
|
||||
|
||||
### Checks performed
|
||||
Both are gated by pre-signage-point transaction height. `block_creation.unfinished_block_to_full_block_with_mmr()` and `validate_finished_header_block()` must compute the same MMR root for the same candidate context, including overflow/new-slot behavior.
|
||||
|
||||
1. Non-tx blocks: foliage_transaction_block, transactions_info, generator all None
|
||||
2. Tx blocks: foliage_transaction_block and transactions_info must exist
|
||||
3. `transactions_info_hash` matches foliage commitment
|
||||
4. `foliage_transaction_block_hash` matches foliage commitment
|
||||
5. Reward claims valid (pool + farmer coins for all blocks since last tx block)
|
||||
6. Previous transaction block reference is correct
|
||||
7. Timestamp: `> prev_tx_block_timestamp` and `< max_future_time`
|
||||
8. Transaction filter matches additions/removals
|
||||
9. Generator cost ≤ `MAX_BLOCK_COST_CLVM`
|
||||
10. Generator ref list size ≤ `MAX_GENERATOR_REF_LIST_SIZE` (512)
|
||||
11. Merkle roots (additions and removals) match
|
||||
12. `check_time_locks()` (Rust) validates absolute/relative height/seconds
|
||||
13. Fees = `sum(removals) - sum(additions)` matches declared fees
|
||||
14. Coins not double-spent (checked against fork_info and coin store)
|
||||
15. Additions don't collide with existing coins
|
||||
MMR root semantics:
|
||||
|
||||
---
|
||||
- genesis has no MMR root;
|
||||
- at a new slot, all blocks through previous block are finalized;
|
||||
- within a slot, only prior blocks with earlier signage point, or blocks before the crossed slot boundary, are included;
|
||||
- fork validation may rebuild an MMR root by rolling back to a fork height and walking the candidate branch.
|
||||
|
||||
## Difficulty adjustment
|
||||
`BlockchainMMRManager.add_block_to_mmr()` requires sequential canonical insertion. Reorgs must roll the manager back before appending new canonical records. The current MMR implementation is in-memory and rebuilt on startup from canonical `height_map` starting at `HARD_FORK2_HEIGHT`.
|
||||
|
||||
**Location**: `consensus/difficulty_adjustment.py`
|
||||
## Block Creation Mirrors Validation
|
||||
|
||||
### Key function: `get_next_sub_slot_iters_and_difficulty()`
|
||||
Block creation constructs commitments that body/header validation later recomputes:
|
||||
|
||||
Called at epoch boundaries (every `EPOCH_BLOCKS = 4608` blocks).
|
||||
- reward claims include pool/farmer coins for the previous transaction block and intervening non-transaction blocks;
|
||||
- transaction filter includes puzzle hashes for additions/rewards and coin IDs for removals;
|
||||
- addition root groups by puzzle hash and hashes coin IDs per puzzle hash;
|
||||
- non-transaction blocks have transaction foliage, generator, and generator refs stripped at infusion;
|
||||
- `calculate_infusion_point_total_iters()` adds an extra sub-slot when SP iters exceed IP iters.
|
||||
|
||||
### Algorithm
|
||||
Creation APIs accept callbacks for signatures and fee computation. Consensus validity still comes from the deterministic commitments and later validation; callback behavior must not be trusted as validation.
|
||||
|
||||
1. Find second-to-last transaction block in previous epoch
|
||||
2. Compute elapsed time between reference points
|
||||
3. New difficulty = `old_difficulty × target_time / actual_time`
|
||||
4. Clamp to `[old / DIFFICULTY_CHANGE_MAX_FACTOR, old × DIFFICULTY_CHANGE_MAX_FACTOR]`
|
||||
(factor = 3)
|
||||
5. Truncate to `SIGNIFICANT_BITS` (8)
|
||||
## Trust Boundaries
|
||||
|
||||
### Same logic applies to sub-slot iterations (SSI)
|
||||
Inputs from peers/farmers are untrusted until they pass validation. The highest-risk external calls are:
|
||||
|
||||
### Invariant
|
||||
- Rust CLVM generator execution and `SpendBundleConditions` production;
|
||||
- Rust BLS/signature validation as part of generator execution;
|
||||
- Rust `check_time_locks()`;
|
||||
- VDF proof validation;
|
||||
- SQLite-backed `BlockStore`/`CoinStore` reads during fork reconstruction;
|
||||
- BIP158 filter construction/serialization.
|
||||
|
||||
Difficulty and SSI can change at most 3× per epoch.
|
||||
Failure mode expectations:
|
||||
|
||||
---
|
||||
- generator or VDF failure should return an invalid-block error, not partially mutate state;
|
||||
- DB lookup inconsistency often asserts because consensus code expects local stores to be internally coherent;
|
||||
- `PreValidationResult.error` must be checked before trusting `required_iters` or `conds`;
|
||||
- `conds.validated_signature` is required for blocks with generators before body validation accepts them.
|
||||
|
||||
## VDF iteration math
|
||||
## Persistence And Cache Gotchas
|
||||
|
||||
**Location**: `consensus/pot_iterations.py`
|
||||
`BlockHeightMap` is a canonical-chain cache, not a source of orphan/fork truth. It is reconciled from the DB at startup by walking backward from the stored peak and stops early only when both block hash and sub-epoch summary match. Reorg rollback truncates height-to-hash and deletes SES entries above the fork height.
|
||||
|
||||
### `calculate_iterations_quality(quality_string, size, difficulty, cc_sp_output_hash)`
|
||||
`try_block_record()` only checks the in-memory cache. Code paths that need orphan records or records outside the cache use `get_block_record_from_db()` or `block_store`. Mixing these up changes whether a valid fork appears disconnected.
|
||||
|
||||
```
|
||||
sp_quality = hash(quality_string + cc_sp_output_hash)
|
||||
iters = difficulty × DIFFICULTY_CONSTANT_FACTOR × sp_quality_int / (2^256 × expected_plot_size)
|
||||
return max(iters, 1)
|
||||
```
|
||||
`lookup_block_generators()` is fork-aware: generator refs may resolve on a fork branch first, then the canonical chain. After SF9 refs are banned, but legacy paths still matter for historical blocks and tests.
|
||||
|
||||
### `calculate_ip_iters(sub_slot_iters, signage_point_index, required_iters)`
|
||||
## Audit Focus Areas
|
||||
|
||||
```
|
||||
ip_iters = (sp_iters + NUM_SP_INTERVALS_EXTRA × sp_interval_iters + required_iters) % sub_slot_iters
|
||||
```
|
||||
- Any change that touches `ForkInfo`, `advance_fork_info()`, `include_spends()`, or `_reconsider_peak()` can corrupt reorg coin-state replay.
|
||||
- Any change that moves `_peak_height` earlier, flushes height map/MMR before DB commit, or catches commit exceptions differently can expose inconsistent peak state.
|
||||
- Any change to `pre_sp_tx_block_height()`, overflow detection, or finished-sub-slot counting can shift hard-fork gates, generator flags, challenge roots, and MMR roots.
|
||||
- Any change to `AugmentedBlockchain` height mapping, read-only snapshots, or MMR copying can make sync batch validation disagree with sequential commit validation.
|
||||
- Any change to body validation that treats DB coin records as canonical without considering `fork_info` can reject valid fork spends or accept invalid branch spends.
|
||||
- Any change to MMR bagging, pop logic, finalized-block cutoff, or fork rebuild must be checked against block creation and validation together.
|
||||
- Any change to sub-epoch summary generation must preserve the relationship between `prev_ses_block`, challenge root gating, and difficulty/SSI boundary computation.
|
||||
|
||||
### `is_overflow_block(signage_point_index)`
|
||||
## Source Pointers
|
||||
|
||||
```
|
||||
overflow = signage_point_index >= NUM_SPS_SUB_SLOT - NUM_SP_INTERVALS_EXTRA
|
||||
```
|
||||
|
||||
i.e., the last 3 signage points of a sub-slot are overflow blocks.
|
||||
|
||||
### Constraints
|
||||
|
||||
- `required_iters ∈ (0, sp_interval_iters)`
|
||||
- `signage_point_index < NUM_SPS_SUB_SLOT` (64)
|
||||
- `sub_slot_iters % NUM_SPS_SUB_SLOT == 0`
|
||||
|
||||
---
|
||||
|
||||
## Block rewards
|
||||
|
||||
**Location**: `consensus/block_rewards.py`
|
||||
|
||||
### Schedule (pool = 7/8, farmer = 1/8 + fees)
|
||||
|
||||
| Period | Per-block reward |
|
||||
| ------------------- | ---------------- |
|
||||
| Height 0 (pre-farm) | 21 000 000 XCH |
|
||||
| Years 0–3 | 2 XCH |
|
||||
| Years 3–6 | 1 XCH |
|
||||
| Years 6–9 | 0.5 XCH |
|
||||
| Years 9–12 | 0.25 XCH |
|
||||
| Year 12+ | 0.125 XCH |
|
||||
|
||||
`_blocks_per_year = 1 681 920` (32 × 6 × 24 × 365)
|
||||
|
||||
### Coinbase parent IDs
|
||||
|
||||
- Pool: `genesis_challenge[:16] + height.to_bytes(16)`
|
||||
- Farmer: `genesis_challenge[16:] + height.to_bytes(16)`
|
||||
|
||||
These are deterministic, not hashed.
|
||||
|
||||
---
|
||||
|
||||
## `AugmentedBlockchain` — Parallel validation overlay
|
||||
|
||||
**Location**: `consensus/augmented_chain.py`
|
||||
|
||||
### Purpose
|
||||
|
||||
Wraps a `BlocksProtocol` with an in-memory cache of extra blocks. Used during
|
||||
parallel batch validation: blocks in the batch aren't committed to the DB until
|
||||
all pass, but subsequent blocks in the batch need to reference earlier ones.
|
||||
|
||||
### Key invariant
|
||||
|
||||
Extra blocks must form a contiguous chain. `add_extra_block()` validates that
|
||||
each new block's `prev_hash` matches the last added block.
|
||||
|
||||
### Generator ref resolution
|
||||
|
||||
`lookup_block_generators()` first checks extra blocks (walking backward via
|
||||
`prev_header_hash`), then falls through to the underlying blockchain.
|
||||
For files in the Landmarks table above, read the source in `chia/consensus/` directly. In-module but not landmarked: `chia/consensus/block_rewards.py` (reward/fee amount math). Cross-module authority: `chia/full_node/mempool_manager.py`. For regression coverage, start with `chia/_tests/blockchain/test_blockchain.py`, `chia/_tests/blockchain/test_augmented_chain.py`, `chia/_tests/blockchain/test_block_commitments.py`, and `chia/_tests/core/consensus/test_mmr.py`.
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
# Chia Daemon Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/daemon/` is the local process-control and keychain RPC boundary. It is not
|
||||
part of the peer wire protocol: clients and node services connect to a local TLS
|
||||
websocket, exchange JSON envelopes from `chia.util.ws_message`, and either ask
|
||||
the daemon to perform privileged local work or ask it to relay messages to a
|
||||
registered service websocket.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for local daemon websocket routing, service start/stop supervision, keychain proxying, plotter process management, daemon message envelopes, and GUI/service event fanout. For service-specific RPC endpoint semantics, read `rpc.md` plus the concrete service context.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `WebSocketServer` is the daemon authority. It owns the TLS websocket listener,
|
||||
service registration table, daemon command dispatch, child-process table,
|
||||
plotting queue, keyring status notifications, and shutdown coordination.
|
||||
- `DaemonProxy` is the generic client-side request/response adapter. It owns
|
||||
request-id waiters and the listener task that turns daemon responses back into
|
||||
per-request events.
|
||||
- `KeychainServer` is the remote keychain authority behind daemon commands. It
|
||||
maps request `kc_user`/`kc_service` pairs to cached `Keychain` instances and
|
||||
normalizes keychain exceptions into daemon JSON errors.
|
||||
- `KeychainProxy` is a compatibility layer over local or remote keychain access.
|
||||
In local mode it directly calls a `Keychain`; in remote mode it reconnects to
|
||||
the daemon and reconstructs returned private keys from returned entropy.
|
||||
- Process launching and killing are local OS authority. `launch_service()`,
|
||||
`launch_plotter()`, `kill_processes()`, and `windows_signal.kill()` are the
|
||||
points where daemon commands become child processes, PID files, signals, and
|
||||
plotter log files.
|
||||
|
||||
## Why This Is Tricky
|
||||
|
||||
Public RPC docs show the daemon as a websocket route for service commands. In source, the daemon is also the local privilege concentrator: it can expose keychain operations, launch or kill services, mutate plotter queue state, and relay GUI/service messages by registered service name. That makes message envelope compatibility and registration cleanup security-relevant even though this is not P2P traffic.
|
||||
|
||||
## Wrong Assumptions To Avoid
|
||||
|
||||
- Do not apply peer-protocol sender maps, binary streamable framing, or P2P rate limits to daemon messages.
|
||||
- Do not treat service registration as harmless metadata; registered names become routing authorities for local clients.
|
||||
- Do not route arbitrary user-supplied command lines through service launch paths.
|
||||
- Do not normalize keychain errors without checking CLI, GUI, daemon proxy, and remote keychain compatibility.
|
||||
|
||||
## Wire And Routing Contracts
|
||||
|
||||
- Daemon messages are JSON dictionaries with `command`, `ack`, `data`,
|
||||
`request_id`, `destination`, and `origin`. `format_response()` flips
|
||||
origin/destination, preserves the request id, and sets `ack=True`.
|
||||
- `destination != "daemon"` is pure service forwarding. The daemon does not
|
||||
interpret the command or payload if the destination is registered; it serializes
|
||||
the original message and sends it to all websockets registered under that
|
||||
destination.
|
||||
- `register_service` adds the current websocket to `connections[service]`.
|
||||
Multiple registrations of the same websocket for the same service collapse
|
||||
through the set; one websocket may register for multiple services and
|
||||
`remove_connection()` must remove it from all of them.
|
||||
- There is no peer-style protocol enum, streamable binary framing, node-type map,
|
||||
or rate limiter here. The primary gate is mutual TLS using daemon private
|
||||
certs plus local config (`self_hostname`, `daemon_port`,
|
||||
`daemon_max_message_size`, `daemon_heartbeat`).
|
||||
- Malformed JSON or unexpected message shape is caught around `handle_message()`
|
||||
and returned as a daemon error response to the sender. Debug logging must pass
|
||||
through `redact_sensitive_data()` before recording message contents.
|
||||
|
||||
## TLS, Startup, And Shutdown
|
||||
|
||||
- The daemon server uses `ssl_context_for_server()` with client certificates
|
||||
required. The daemon has a default minimum TLS policy plus a daemon-local
|
||||
compatibility escape hatch for internal daemon connections only.
|
||||
- `async_run_daemon()` runs `chia_init()`, initializes daemon logging, acquires
|
||||
`daemon_launch_lock_path(root_path)`, creates `WebSocketServer`, installs async
|
||||
signal handlers, and waits on `shutdown_event`.
|
||||
- `stop()` cancels ping/status tasks, kills every tracked service process, clears
|
||||
`services`, and sets `shutdown_event`. `exit()` closes the `WebServer` and must
|
||||
await `await_closed()` because `WebServer.close()` only schedules cleanup.
|
||||
- `DaemonProxy.start()` creates its listener task after connecting and then
|
||||
sleeps briefly before returning. `_get()` registers the waiter before sending
|
||||
and has a source-defined response timeout.
|
||||
|
||||
## Keychain And Secret Handling
|
||||
|
||||
- Keychain commands are intercepted before normal daemon command dispatch by
|
||||
membership in `keychain_commands`. Adding a keychain RPC requires updating this
|
||||
list, `KeychainServer.handle_command()`, and usually `KeychainProxy`.
|
||||
- `KeychainServer.run_request()` uses streamable JSON conversion for newer
|
||||
typed request/response classes (`get_key`, `get_keys`, public-key and label
|
||||
operations). Older commands are hand-parsed dictionaries and have more varied
|
||||
error shapes.
|
||||
- Public-key responses intentionally override `to_json_dict()` to expose only
|
||||
`fingerprint`, `public_key`, and `label`; do not reuse private-key response
|
||||
shapes for public-only APIs.
|
||||
- Remote private-key reads return public key hex plus entropy hex. The proxy
|
||||
rebuilds the mnemonic and private key and verifies the derived G1 matches the
|
||||
returned public key before handing it to callers.
|
||||
- Passphrase operations mutate global keyring cache/state and notify
|
||||
`wallet_ui` via queued `keyring_status_changed` messages. `unlock_keyring()`
|
||||
may also run `check_keys()` once when the daemon was started with
|
||||
`--wait-for-unlock`.
|
||||
|
||||
## Service And Plotter Process Model
|
||||
|
||||
- `start_service` accepts only names validated by `validate_service()`. A service
|
||||
is considered running if the daemon has a live tracked process or a registered
|
||||
websocket for that service, which supports services started outside the daemon.
|
||||
- Child processes inherit a copied environment with `CHIA_ROOT` set to the
|
||||
daemon root. Frozen builds map service names to packaged executables; source
|
||||
runs use `shutil.which()` or the raw service name.
|
||||
- PID files live under `root_path / "run"` and are best-effort. On kill, the PID
|
||||
file is renamed to `.pid-killed` when possible; failure to write or rename PID
|
||||
files is intentionally non-fatal.
|
||||
- Plotting is a special daemon-managed pseudo-service named `chia_plotter`.
|
||||
`plots_queue` is in-memory state; plotter subscribers receive full queue state
|
||||
on registration and incremental `state_changed`/`log_changed` messages later.
|
||||
- Plotter command construction is per-plotter (`chiapos`, `bladebit`, `madmax`)
|
||||
and mutates command args before launch by appending `-D` so child plotters use
|
||||
the daemon for keychain access.
|
||||
- Serial plotting is coordinated by queue name and `PlotState`: only one
|
||||
non-parallel `RUNNING` item per queue should exist. Completion is detected by
|
||||
tailing the plotter log for plotter-specific final words, not by only waiting
|
||||
on process exit.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Broadening service forwarding or registration is security-sensitive because
|
||||
registered service names become routing authorities for all local daemon
|
||||
clients.
|
||||
- Changing response envelope fields breaks `DaemonProxy._get()` and GUI/RPC
|
||||
clients that key off `request_id`, `origin`, `destination`, and `ack`.
|
||||
- Keychain error compatibility is uneven but tested. Normalizing errors is useful
|
||||
only if CLI, GUI, daemon tests, and `KeychainProxy.handle_error()` are updated
|
||||
together.
|
||||
- Be careful with task lifetime: ping, state-change delivery, daemon-proxy
|
||||
listeners, keychain reconnect loops, and plotter tasks are deliberately
|
||||
referenced via `create_referenced_task()` or cancelled via `cancel_task_safe()`.
|
||||
- `KeychainProxy.close()` awaits the reconnect task after setting `shut_down`.
|
||||
Any change to reconnect-loop exit conditions can make shutdown hang.
|
||||
- Plotter queue state and service process state are updated from async tasks
|
||||
without an explicit lock. Keep state transitions small and preserve ordering of
|
||||
SUBMITTED -> RUNNING -> FINISHED/REMOVING notifications.
|
||||
- `launch_service()` splits `service_command` on spaces. Today daemon-controlled
|
||||
service names are allowlisted and the only appended option is the testing flag;
|
||||
do not start passing arbitrary user-supplied command lines through this path.
|
||||
- Windows process groups and signal mapping are special-cased. Changes to
|
||||
creation flags or `windows_signal.kill()` need Windows-specific validation.
|
||||
|
||||
## Test And Audit Strategy
|
||||
|
||||
- `chia/_tests/core/daemon/test_daemon.py` covers daemon command responses,
|
||||
passthrough routing to a full node, keychain RPCs, passphrase status events,
|
||||
plotting queue transitions, logging redaction, bad JSON, and plotter options.
|
||||
- `chia/_tests/core/daemon/test_daemon_register.py` covers multi-service
|
||||
registration and connection cleanup semantics.
|
||||
- `chia/_tests/core/daemon/test_keychain_proxy.py` covers local-vs-remote
|
||||
`KeychainProxy` behavior, private/public key reconstruction, and error mapping.
|
||||
- `chia/_tests/core/test_daemon_rpc.py` is the minimal daemon client smoke test.
|
||||
- `chia/_tests/cmds/test_daemon.py` anchors CLI daemon startup, keyring unlock
|
||||
flow, and daemon launcher behavior.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Daemon websocket server and process control: `chia/daemon/server.py`.
|
||||
- Client/proxy adapters: `chia/daemon/client.py`, `chia/daemon/keychain_proxy.py`.
|
||||
- Remote keychain command handling: `chia/daemon/keychain_server.py`.
|
||||
- Plotter process queue: `chia/daemon/server.py`, `chia/plotters/`.
|
||||
- Message envelope helpers: `chia/util/ws_message.py`.
|
||||
@@ -0,0 +1,98 @@
|
||||
# Chia Data Layer Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/data_layer/` implements Chia DataLayer: an off-chain key/value Merkle store whose committed roots are anchored by wallet-managed DataLayer singletons. The module is not just a local database. Correctness depends on alignment between local Merkle roots, wallet singleton history, static delta/full-tree files, mirror subscriptions, and optional downloader/uploader plugins.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for DataLayer store mutations, root publication, mirror sync, `.dat` file generation/validation, proof verification, DataLayer offers, subscriptions, and DataLayer RPC. For singleton wallet authority and offer signing, also read `wallet.md`; for RPC transport shape, read `rpc.md`.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `DataLayer` is the service coordinator. It owns lifecycle, wallet RPC access, periodic subscription management, file upload/download orchestration, and the RPC-facing mutation/read API. It does not own chain truth directly; it asks wallet RPC for DataLayer singleton state and updates.
|
||||
- `DataStore` is the local Merkle/data persistence authority. It stores root generations/status in SQLite, stores Merkle blobs under `merkle_blobs_path`, stores or references key/value blobs under `key_value_blobs_path`, and delegates tree mutation/proof mechanics to `chia_rs.datalayer.MerkleBlob`.
|
||||
- `DataLayerWallet` and `DataLayerStore` are wallet-side authorities for singleton launcher tracking, root history, mirror coins, lineage proofs, and offer solver data. The service reaches them mostly through wallet RPC request types, while wallet internals persist singleton records in the wallet DB.
|
||||
- `DataLayerRpcApi` is mostly transport normalization: it converts JSON hex/optional pagination/request shapes into typed service calls. It is not the core invariant boundary, but bugs here can silently change legacy RPC semantics.
|
||||
- `DataLayerServer` is a separate static-file HTTP process for serving `.dat` files from `server_files_location`; it does not validate store state beyond filename shape.
|
||||
- S3/plugin support is an extension boundary. Plugins are HTTP services described by `PluginRemote` values and can handle upload/download decisions, but the service still validates downloaded data by inserting it through `DataStore`.
|
||||
|
||||
## Why This Is Tricky
|
||||
|
||||
Public DataLayer docs describe proving key/value inclusion without sharing the full dataset. Source-level correctness depends on two separate truths staying aligned: wallet singleton history publishes the root sequence, while local/mirrored files provide enough data to reconstruct those roots. Mirrors and plugins can improve availability, but they are never trust anchors; every downloaded file must rebuild the wallet-advertised root before becoming local committed state.
|
||||
|
||||
## Wrong Assumptions To Avoid
|
||||
|
||||
- Do not treat a local root as current chain truth until wallet confirmation status has been reconciled.
|
||||
- Do not treat mirror URLs, plugins, or static file names as trusted data sources.
|
||||
- Do not collapse `None`, omitted root fields, and empty-root sentinels across RPC/service/wallet boundaries.
|
||||
- Do not treat offer creation as a pure read; it can stage local store mutations before proofs are generated.
|
||||
|
||||
## Store And Root Invariants
|
||||
|
||||
- A store id is the DataLayer singleton launcher id. Local roots and wallet singleton records must be interpreted against that id, not an arbitrary database namespace.
|
||||
- Empty-tree roots use two representations: `DataStore` uses `node_hash=None`; RPC/wallet-chain paths often use `bytes32.zeros` via `DataLayer.none_bytes`. Code crossing this boundary must normalize intentionally.
|
||||
- Root generations are contiguous per store. `DataStore._insert_root()` increments from the latest committed generation unless an explicit generation is supplied, and `_check_roots_are_incrementing()` treats missing generation numbers as corruption.
|
||||
- Root status drives publication state: `COMMITTED` is local confirmed/canonical, `PENDING` is ready/submitted for chain publication, and `PENDING_BATCH` is an open local batch not yet publishable. Only one pending root is expected; multiple pending roots are an error.
|
||||
- Most service reads call `_update_confirmation_status()` first. That method compares local generation with wallet `dl_latest_singleton(... only_confirmed=True)`, promotes matching pending roots to committed, shifts generations for already-confirmed roots, and clears stale pending rows. Moving reads/writes around this step changes consistency semantics.
|
||||
- Local roots can temporarily be ahead of chain roots during batch updates or unconfirmed publication. Sync code treats this as expected and avoids rolling local state backward unless explicit rollback/unsubscribe paths are used.
|
||||
|
||||
## Data And File Model
|
||||
|
||||
- Key/value data is represented as terminal leaves. Hashing is domain-separated: `key_hash(key) = sha256(0x01 || key)`, `leaf_hash(key, value) = sha256(0x02 || key_hash || value_hash)`, and internal nodes use `sha256(0x02 || left || right)`.
|
||||
- `MerkleBlob` is the operational tree format. `DataStore` keeps recent blobs in an LRU cache, writes Merkle blobs by store/root hash, and stores larger key/value blobs as zstd-compressed files while keeping smaller blobs inline in the `ids` table.
|
||||
- Delta/full-tree files are length-prefixed streams of `SerializedNode` records. Full files contain the tree for a root; delta files omit nodes already present in the previous generation by using `DeltaFileCache`.
|
||||
- File names are part of the protocol and may be grouped by store id. `download_data.is_filename_valid()` rejects names that do not round-trip through the generator.
|
||||
- Downloaded files are not trusted because they came from a mirror/plugin. `insert_from_delta_file()` feeds them into `DataStore.insert_into_data_store_from_file()`, which reconstructs missing nodes, builds a Merkle blob for the wallet-advertised root, inserts that root as committed, and reloads the blob as a correctness check.
|
||||
- `maximum_full_file_count` controls how many recent full-tree files are retained/generated. Older full files are pruned, but delta continuity still matters for peers syncing generation by generation.
|
||||
|
||||
## Mutation Lifecycle
|
||||
|
||||
- Creating a store calls wallet RPC `create_new_dl(... push=True)` and then creates the local tree at the returned launcher id. The chain singleton and local empty root are created through separate systems and can fail independently.
|
||||
- `batch_update()` mutates local `DataStore` first, then optionally publishes the pending root through wallet RPC. If `submit_on_chain=False`, the root is stored as `PENDING_BATCH` and must later be submitted.
|
||||
- `batch_insert()` verifies the store is owned by the DataLayer wallet before mutating. This ownership check is the main guard preventing arbitrary local roots from being published for non-owned singletons.
|
||||
- `submit_pending_root()` converts an open `PENDING_BATCH` root to `PENDING` and publishes it. `_get_publishable_root_hash()` rejects already-confirmed roots and still-open batches.
|
||||
- Multistore updates stage one update per store id, then call wallet RPC `dl_update_multiple()` with `LauncherRootPair`s. Duplicate store ids in one multistore request are rejected before publishing.
|
||||
- `DataStore.insert_batch()` handles duplicate-key/change-list semantics and optional autoinsert reference placement. It continues an existing `PENDING_BATCH` only when it is exactly one generation after the committed root; otherwise pending state is an internal error.
|
||||
|
||||
## Sync And Subscription Loop
|
||||
|
||||
- `periodically_manage_data()` is the background control loop. Each cycle tracks wallet subscriptions, pseudo-subscribes owned stores so files are generated even without explicit subscriptions, optionally auto-subscribes local stores, then runs bounded concurrent `update_subscription()` jobs.
|
||||
- `update_subscription()` performs four ordered steps: sync subscription URLs from wallet mirrors, fetch/validate remote data, upload local files for owned/current data, then prune old full files. Reordering these affects mirror discovery and file availability.
|
||||
- Wallet reachability failures are intentionally non-fatal. Subscription tracking stops early if the wallet RPC connection is unavailable and retries next cycle; per-subscription failures are logged without killing the loop.
|
||||
- `fetch_and_validate()` uses wallet singleton history as the target generation/root sequence, randomizes eligible server URLs, tries plugin-specific or plain HTTP delta downloads, and marks mirror/server failures with backoff in the subscriptions table.
|
||||
- Unsubscribe is queued and processed only after fetch jobs complete while holding `subscription_lock`, avoiding races between deletion of local subscription/data and active download/update work.
|
||||
- Owned stores are treated as local publication sources even when they have no external mirror URLs. Subscribed stores are treated as replication targets whose chain roots are verified through wallet state.
|
||||
|
||||
## Proofs, Offers, And Singleton Coupling
|
||||
|
||||
- Inclusion proofs are rooted in DataLayer Merkle roots, but proof verification also binds those roots to the DataLayer singleton puzzle hash. `dl_verify_proof_internal()` reconstructs the host full puzzle from `inner_puzzle_hash`, proof root, and store id, then verifies `ProofOfInclusion.valid()`.
|
||||
- `verify_proof()` checks wallet coin state for the proof coin id and reports `current_root` by testing whether that singleton coin is unspent. Historical proofs can verify successfully while not being current.
|
||||
- Offer construction mutates/uses local stores to produce the maker/taker inclusion sets and solver dependencies. `process_offered_stores()` may stage local inserts before generating proofs, so offer creation is a local data mutation path, not a pure read.
|
||||
- `make_offer()` verifies the constructed `TradingOffer` summary against local `StoreProofs` before returning. `take_offer()` verifies maker proofs, builds `proofs_of_inclusion` solver data, and then calls wallet RPC outside the data-store transaction because wallet failures may happen after chain submission.
|
||||
- Non-secure offer cancellation asks wallet RPC for the offer, derives affected store ids from the DataLayer offer summary, then clears pending roots for those stores after cancellation.
|
||||
|
||||
## RPC And Compatibility Notes
|
||||
|
||||
- RPC request fields mix legacy names (`id`, `root_hash`, `changelist`) with streamable request/response classes for newer offer/proof paths. Preserve wire shapes unless deliberately changing public RPC behavior.
|
||||
- `root_hash` omission means "latest local committed root" via `Unspecified`; explicit empty-root sentinels can mean the empty tree. `None`, omitted, and `bytes32.zeros` are not interchangeable across RPC, service, and wallet layers.
|
||||
- Pagination is byte-size based over key/value blob lengths, not item-count based. Oversized single items raise rather than splitting.
|
||||
- `DataLayerRpcClient` is a thin convenience wrapper and does not cover every server-side nuance; do not infer server behavior solely from client helper argument types.
|
||||
- Static file serving validates filenames but uses direct synchronous file reads in the request handler. It assumes files are already written under `server_files_location` by the service or plugin.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- High-risk edits move work across DB writer transactions, `_update_confirmation_status()`, pending-root status changes, or wallet RPC calls. These boundaries encode publication and rollback assumptions.
|
||||
- File-system writes for Merkle blobs, key/value blobs, and `.dat` files have TODOs around locking. Concurrent service/plugin/server access should be treated as a real consistency concern.
|
||||
- Empty-root normalization is subtle and historically uneven. Audit any change touching `bytes32.zeros`, `None`, `Root.node_hash`, or proof roots with both empty and non-empty stores.
|
||||
- Plugin and mirror URLs are external trust inputs. Downloaded data must continue to be verified against wallet-advertised roots, and file path/name validation must stay strict.
|
||||
- DataLayer wallet code depends on singleton CLVM structure, odd singleton amounts, lineage proofs, and offer solver field names. Changes in wallet puzzle drivers or offer summaries can break this module without direct edits here.
|
||||
- Tests for this module should cover lifecycle/order effects: pending batch to publish, confirmation promotion, wallet-unreachable sync retry, delta-file validation failure, grouped vs ungrouped filenames, unsubscribe retention, historical proof verification, and offer-created local mutations.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Service coordination and RPC-facing behavior: `chia/data_layer/data_layer.py`, `chia/data_layer/data_layer_rpc_api.py`.
|
||||
- Local Merkle/data persistence: `chia/data_layer/data_store.py`.
|
||||
- Static file validation/download/upload: `chia/data_layer/download_data.py`, `chia/data_layer/data_layer_server.py`, `chia/data_layer/s3_plugin_service.py`.
|
||||
- Wallet singleton authority: `chia/data_layer/data_layer_wallet.py`, `chia/data_layer/dl_wallet_store.py`.
|
||||
- Offer/proof request contracts: `chia/data_layer/data_layer_util.py`, `chia/protocols/wallet_protocol.py`.
|
||||
@@ -0,0 +1,128 @@
|
||||
# Chia Farmer Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/farmer/` is the block-production coordinator between the full node, harvesters, pool HTTP APIs, local keychain, plot-sync state, RPC clients, and optional solver services for v2 plots. It does not own chain consensus or plot lookup; its safety depends on forwarding only locally verified proof/signature material across these boundaries while keeping short-lived signage-point state coherent.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for farmer/full-node/harvester proof flow, pool partial submission, reward-target config, plot-sync receiver state, solver connection management, and farmer RPC. For local plot discovery or signatures, read `harvester.md` and `plotting.md`; for final block acceptance, read `full-node.md` and `consensus.md`.
|
||||
|
||||
## Landmarks
|
||||
|
||||
| file | owns |
|
||||
| ------------------------------- | ----------------------------------------------------- |
|
||||
| `chia/farmer/farmer.py` | mutable service state, SP/proof/pool/plot-sync caches |
|
||||
| `chia/farmer/farmer_api.py` | SP/proof/signature/solver message routing |
|
||||
| `chia/farmer/farmer_rpc_api.py` | reward target, pool state, harvester plot RPC |
|
||||
| `chia/farmer/start_farmer.py` | farmer Service construction |
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `Farmer` is the mutable service state holder: keys, reward targets, signage-point caches, proof caches, harvester plot-sync receivers, pool state, pending solver requests, and RPC/event callbacks.
|
||||
- `FarmerAPI` is the peer-protocol semantic layer. It receives full-node signage points, harvester proofs/signatures/plot-sync updates, and solver responses, then routes validated follow-up messages.
|
||||
- The full node remains the authority for chain validity and unfinished-block construction. Farmer `DeclareProofOfSpace` and `SignedValues` messages are proposals/signatures, not accepted blocks.
|
||||
- Harvesters are operator-controlled peers, but the farmer still verifies proof quality and aggregate signatures before forwarding. A harvester can affect local rewards/fees and pool partial submission, so its payloads are not blindly trusted.
|
||||
- Pool servers are external HTTP services. Pool info, farmer records, difficulty, partial acknowledgement, and errors update local `pool_state`, but pool responses do not alter consensus facts.
|
||||
- RPC is local management: reward target mutation, payout-instruction updates, harvester/plot inspection, signage-point inspection, pool login links, and solver connection control.
|
||||
|
||||
## Why This Is Tricky
|
||||
|
||||
Public farming docs describe the farmer as finding proofs and earning rewards, but the source-level role is narrower: it correlates transient proof material from harvesters, pool servers, solvers, and the full node. None of those inputs is durable truth by itself. The same proof can be used for network block proposals, pool partial accounting, UI farm-health metrics, and later signature requests, so cache keys and message routing carry more meaning than a local handler read suggests.
|
||||
|
||||
## Wrong Assumptions To Avoid
|
||||
|
||||
- Do not treat a farmer proof as an accepted block; full-node/consensus validation still decides acceptance.
|
||||
- Do not treat pool partial success as chain state; it is external accounting by a pool server.
|
||||
- Do not assume the farmer can recompute harvester-local signatures or plot metadata after the fact; it must preserve correlation to the original harvester/proof.
|
||||
- Do not treat solver responses as independent proofs; farmer pending-request state is the correlation authority.
|
||||
|
||||
## Main Runtime Flow
|
||||
|
||||
Block production is a two-stage signature workflow:
|
||||
|
||||
- Full node sends `NewSignagePoint` to the farmer.
|
||||
- Farmer caches it under `challenge_chain_sp`, builds per-pool `PoolDifficulty` hints, and sends the old or new signage-point harvester payload depending on peer protocol version.
|
||||
- Harvester returns `NewProofOfSpace`.
|
||||
- Farmer verifies the proof with `verify_and_get_quality_string()` using the SP challenge, peak height, and previous transaction block height.
|
||||
- If the proof's required iterations are below the SP interval, farmer caches the proof and asks the originating harvester to sign challenge-chain and reward-chain SP values.
|
||||
- `RespondSignatures` is processed into `DeclareProofOfSpace` and broadcast to full nodes.
|
||||
- Full node may later request foliage signatures by quality string; farmer re-contacts the original harvester and returns `SignedValues`.
|
||||
|
||||
Pool partial submission is parallel to block-winning flow. For pool contract plots, the same proof is checked against the pool's current difficulty. If good enough, farmer requests a harvester plot signature over `PostPartialPayload`, adds its farmer/taproot/authentication signatures, posts `/partial`, and updates local accounting from the pool response.
|
||||
|
||||
V2 plots add a solver hop. Harvester sends `PartialProofsData`; farmer stores pending request metadata keyed by `bytes(partial_proof)`, broadcasts `SolverInfo` to solver peers, receives `SolverResponse`, reconstructs a `ProofOfSpace`, and re-enters the normal `new_proof_of_space()` path. Solver responses for unknown or empty proofs are dropped.
|
||||
|
||||
## Mutable State Domains
|
||||
|
||||
- `sps`: maps `challenge_chain_sp` to one or more `NewSignagePoint` objects. Duplicates are ignored; entries are short-lived and drive both RPC display and proof validation.
|
||||
- `proofs_of_space`: maps SP hash to `(plot_identifier, ProofOfSpace)` pairs used later when harvester signatures arrive.
|
||||
- `quality_str_to_identifiers`: maps locally computed quality strings to `(plot_identifier, challenge_hash, sp_hash, harvester_node_id)` so full-node `RequestSignedValues` can target the same harvester/proof.
|
||||
- `number_of_responses` and `cache_add_time`: low-tech bounded-memory controls. `_periodically_clear_cache_and_refresh_task()` removes SP/proof/quality/response state after roughly three sub-slot times.
|
||||
- `pending_solver_requests`: maps partial proof bytes to original harvester data. This is a request correlation table, not a validated-proof cache.
|
||||
- `plot_sync_receivers`: one `Receiver` per harvester peer. It owns plot lists, invalid/key-missing/duplicate paths, sync identifiers, and effective plot-size summaries used by RPC/UI.
|
||||
- `pool_state`: keyed by `p2_singleton_puzzle_hash`, combining persisted `PoolingShareState`, remote pool difficulty/token data, partial counters, errors, and next update deadlines.
|
||||
- `authentication_keys`, `all_root_sks`, `_private_keys`, and `pool_sks_map`: key-derived state. These are populated after keychain access succeeds and must be refreshed when pooling config changes.
|
||||
|
||||
## Key And Signature Contracts
|
||||
|
||||
- Farmer private keys are derived as farmer and pool keys from all root keys. Harvester handshake advertises farmer public keys and old-style pool public keys so harvesters can select plots.
|
||||
- For plot signatures, the harvester provides the local-key share. Farmer finds the matching farmer private key, derives the aggregate plot public key, optionally adds the taproot share for pool-contract plots, aggregates signatures, and verifies before forwarding.
|
||||
- Original self-pooled plots with `pool_public_key` require a local pool private key and a signed `PoolTarget`; pool-contract plots set `pool_target` and `pool_signature` to `None`.
|
||||
- Pool authentication uses owner/authentication keys and current authentication tokens from `pool_protocol`. Login links, `/farmer`, and `/partial` requests are signed payloads tied to launcher/target data.
|
||||
- CHIP-22 third-party harvester reward override handling forces source signature data and logs whether the harvester's fee-quality convention appears valid. This is a reward-routing convention, not a consensus permission check.
|
||||
|
||||
## Pool State And Persistence
|
||||
|
||||
- Pooling configuration lives in `<chia root>/pooling/pooling_share_state.yaml`, guarded by `PoolingShareState.lock()`. `Farmer.__init__()` performs migration from older `config.yaml` pool entries.
|
||||
- `update_pool_state()` periodically rereads pooling config, refreshes keys, ensures per-pool state exists, fetches `/pool_info`, fetches `/farmer`, POSTs unknown farmers, PUTs payout/auth updates, and updates difficulty/points.
|
||||
- Mainnet enforces HTTPS pool URLs. Non-mainnet and self-pooling (`pool_url == ""`) follow local accounting paths without remote pool submission.
|
||||
- Pool counters are rolling time-window lists plus since-start totals. `strip_old_entries()` aging happens on signage-point handling and on stat increments; stale data can persist if the farmer stops receiving signage points.
|
||||
|
||||
## Protocol Compatibility
|
||||
|
||||
- `NewSignagePointHarvester` and `NewSignagePointHarvester2` intentionally share the same message ID. Farmer selects old/new payload shape based on the source-defined harvester protocol-version boundary.
|
||||
- `NewSignagePointHarvester2` carries `peak_height` and `last_tx_height`, while the old payload carries `filter_prefix_bits`. Any change here is a live farmer-harvester compatibility change and must move with protocol enum/sender-map/stub/rate-limit/version tests.
|
||||
- Source-signature fields in `RequestSignatures` are optional but meaningful. They carry full data corresponding to hashes when third-party harvester reward override/source verification is active.
|
||||
- Farmer broadcasts only no-reply/proposal messages to full nodes except direct request/response use through `call_api()` for harvester signatures. Do not broadcast request/response message types outside the server state-machine rules.
|
||||
|
||||
## Concurrency And Lifecycle
|
||||
|
||||
- `Farmer.manage()` starts an initialization loop that waits for key setup before launching pool-state refresh and cache-clear tasks. Harvester handshake tasks wait for `started` so public keys are available.
|
||||
- Shutdown sets `_shut_down`, awaits the two background tasks, closes keychain proxy, and clears `started`. Long-running pool HTTP calls and harvester RPC calls run on the service event loop.
|
||||
- There is no broad lock around farmer mutable dictionaries. Handlers and background tasks rely on asyncio sequencing and short-lived mutation. Multi-step edits to related caches must preserve ordering and cleanup on exceptions.
|
||||
- `on_connect()` creates plot-sync receivers for harvesters and sends handshake after key readiness. `on_disconnect()` removes the receiver and emits UI/RPC state changes.
|
||||
- `connect_to_solver` RPC closes existing solver connections before opening the requested solver peer, so solver selection is effectively single-target from the management surface even though solve requests broadcast to all current solver connections.
|
||||
|
||||
## RPC Surface
|
||||
|
||||
- Signage-point RPC reads from transient `sps`/`proofs_of_space`; missing SPs are normal after cache expiry.
|
||||
- Reward target RPC mutates `config.yaml` and in-memory encoded/decoded targets. Optional private-key search scans derived wallet addresses to report whether keys are present.
|
||||
- Pool state RPC returns shallow copies of `pool_state` with live plot counts calculated from `plot_sync_receivers`.
|
||||
- Harvester plot RPC pages/sorts/filter data from `Receiver`; sorting by optional key fields is intentionally rejected.
|
||||
- State-change fanout targets wallet UI and metrics with specific event names. Changing event shape affects UI/metrics consumers even if RPC routes stay stable.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Quality-string correlation is the bridge between initial proof handling and later full-node signature requests. Evicting or overwriting it too early causes valid full-node requests to fail.
|
||||
- `proofs_of_space[sp_hash]` is assumed present in `_process_respond_signatures()`. Handler ordering and cache expiry must ensure signature responses cannot outlive their proof entry.
|
||||
- Pool partial accounting has many early returns. Failed paths should consistently emit `failed_partial` and increment the right counter; missing one makes UI/metrics diverge from actual pool submission behavior.
|
||||
- Solver request correlation by `bytes(partial_proof)` can collide logically if the same partial proof is submitted from different harvesters/SP contexts before a response returns. Treat changes to this keying as correctness-sensitive.
|
||||
- The farmer accepts and logs harvester reward-address overrides. Any change to override validation affects operator economics and third-party harvester compatibility.
|
||||
- Network/API changes must preserve node-type sender authorization, protocol reply mapping, and old/new harvester version branching. Protocol classes alone are not the full contract.
|
||||
- Background pool updates mutate `pool_state` while signage-point/proof handlers read difficulty/token fields. Keep `None` checks and update ordering explicit to avoid submitting partials with stale or missing pool parameters.
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- Farmer-harvester tests should cover both old and new signage-point payload branches, duplicate SP handling, proof verification rejection, request-signatures routing to the original harvester, and cache-expiry behavior.
|
||||
- Pool tests should exercise missing difficulty/token, not-good-enough partials, successful `/partial` difficulty update, `TOO_LATE`, `PROOF_NOT_GOOD_ENOUGH`, network failure, and self-pooling.
|
||||
- Signature tests should assert aggregate key construction for pool-public-key plots and pool-contract/taproot plots, including reward override/source-data cases.
|
||||
- Solver tests should cover unknown response, empty proof response, successful reconstruction into `NewProofOfSpace`, and cleanup of `pending_solver_requests`.
|
||||
- RPC tests should verify event payloads, pagination/filter/sort behavior, reward-target persistence, payout-instruction update forcing the next farmer update sentinel, and solver reconnection behavior.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
In-module files are in the Landmarks table above. Cross-module authorities:
|
||||
|
||||
- Pool config mirror and external pool protocol: `chia/pools/pool_config.py`, `chia/protocols/pool_protocol.py`.
|
||||
- Harvester/solver wire contracts: `chia/protocols/harvester_protocol.py`, `chia/protocols/solver_protocol.py`.
|
||||
+61
-225
@@ -1,249 +1,85 @@
|
||||
# Full Node Orchestration — Deep Context
|
||||
# Chia Full Node Module Context
|
||||
|
||||
> Attach when touching `chia/full_node/full_node.py`, `full_node_api.py`,
|
||||
> `full_node_store.py`, `full_node_rpc_api.py`, sync logic, or block
|
||||
> processing pipeline.
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
## File map
|
||||
This module is the orchestration boundary where untrusted peer/RPC inputs become consensus, mempool, wallet-notification, and timelord/farmer side effects. The main safety property is not held in one class: it depends on preserving exact ordering between `Blockchain`, `FullNodeStore`, `MempoolManager`, SQLite stores, peer caches, and async fanout.
|
||||
|
||||
| File | Lines | Role |
|
||||
| -------------------------- | ----- | -------------------------------------------------------- |
|
||||
| `full_node.py` | ~3400 | `FullNode`: main orchestrator, sync, block/tx processing |
|
||||
| `full_node_api.py` | ~2080 | `FullNodeAPI`: all P2P message handlers |
|
||||
| `full_node_rpc_api.py` | ~1170 | `FullNodeRpcApi`: HTTP/WS RPC endpoints |
|
||||
| `full_node_rpc_client.py` | ~380 | RPC client (used by CLI and tests) |
|
||||
| `full_node_store.py` | ~1060 | `FullNodeStore`: signage points, unfinished blocks |
|
||||
| `full_node_service.py` | ~10 | Service registration |
|
||||
| `start_full_node.py` | ~120 | Service startup config |
|
||||
| `block_store.py` | ~700 | `BlockStore`: SQLite full block persistence |
|
||||
| `coin_store.py` | ~680 | `CoinStore`: UTXO database |
|
||||
| `sync_store.py` | ~140 | `SyncStore`: sync state tracking |
|
||||
| `weight_proof.py` | ~1740 | `WeightProofHandler`: weight proof creation/validation |
|
||||
| `subscriptions.py` | ~240 | `PeerSubscriptions`: wallet coin/puzzle subscriptions |
|
||||
| `hint_store.py` | ~100 | `HintStore`: hint persistence |
|
||||
| `hint_management.py` | ~60 | Hint processing from conditions |
|
||||
| `tx_processing_queue.py` | ~250 | `TransactionQueue`: async tx processing |
|
||||
| `check_fork_next_block.py` | ~40 | Fork-next-block utility |
|
||||
| `hard_fork_utils.py` | ~55 | Hard fork flag computation |
|
||||
| `full_block_utils.py` | ~370 | Block ↔ header block conversion |
|
||||
| `bundle_tools.py` | ~20 | SpendBundle utilities |
|
||||
## When To Read This
|
||||
|
||||
---
|
||||
Read this for full-node sync, block propagation, mempool coupling, peak processing, wallet/timelord/farmer notifications, full-node RPC reads, and `FullNodeStore` behavior. For pure block validity, read `consensus.md`; for wire schema or connection mechanics, read `protocols.md` or `server.md`.
|
||||
|
||||
## `FullNode` — Main orchestrator
|
||||
## Landmarks
|
||||
|
||||
**Location**: `full_node/full_node.py`
|
||||
| file | owns |
|
||||
| --------------------------------------- | ---------------------------------------------------- |
|
||||
| `chia/full_node/full_node.py` | peak processing, sync orchestration, peer/RPC fanout |
|
||||
| `chia/full_node/full_node_api.py` | peer message handlers, tx/block/SP intake |
|
||||
| `chia/full_node/mempool_manager.py` | transaction-admission authority |
|
||||
| `chia/full_node/full_node_store.py` | volatile SP/EOS/unfinished-block caches |
|
||||
| `chia/full_node/block_store.py` | full block + peak persistence, in_main_chain |
|
||||
| `chia/full_node/coin_store.py` | current-peak UTXO view, FF lineage markers |
|
||||
| `chia/full_node/sync_store.py` | peak-to-peer targets, sync flags |
|
||||
| `chia/full_node/tx_processing_queue.py` | per-peer tx backpressure/DoS queue |
|
||||
|
||||
### Key state
|
||||
## Implementation Authority
|
||||
|
||||
- `blockchain: Blockchain` — chain state + UTXO
|
||||
- `mempool_manager: MempoolManager` — transaction pool
|
||||
- `full_node_store: FullNodeStore` — signage points, unfinished blocks
|
||||
- `sync_store: SyncStore` — sync state
|
||||
- `full_node_peers: FullNodePeers` — peer discovery
|
||||
- `weight_proof_handler: WeightProofHandler` — weight proof logic
|
||||
- `subscriptions: PeerSubscriptions` — wallet subscriptions
|
||||
- `_transaction_queue: TransactionQueue` — async tx processing
|
||||
- `server: ChiaServer` — networking
|
||||
- `Blockchain.add_block()` is the validation and persistence authority for blocks and reorgs. `FullNode` may pre-validate, batch, cache, and broadcast, but it must not treat peer-provided peak, block, weight, signage point, or unfinished-block data as authoritative until consensus code accepts it.
|
||||
- `MempoolManager.add_spend_bundle()` is the transaction-admission authority. P2P `new_transaction` fee/cost values are only advertisements used for fetch priority and peer-accountability checks; the locally computed `MempoolItem` cost/fee wins.
|
||||
- `FullNodeStore` is not a consensus store. It is a bounded, volatile cache for signage points, end-of-slot bundles, unfinished blocks, future VDF-dependent objects, recent pooling data, and tx fetch bookkeeping. Do not infer chain truth from it without checking `Blockchain`/`BlockStore`.
|
||||
- RPC is semi-trusted but still crosses a boundary. Expensive endpoints run CLVM or block-generator work, read chain/mempool state, and sometimes take `priority_mutex` at low priority. Keep RPC reads consistent with the current main chain when returning spend/addition/removal state.
|
||||
|
||||
### Key dataclass: `PeakPostProcessingResult`
|
||||
## Mutation Ordering Contracts
|
||||
|
||||
After a new peak is accepted:
|
||||
- New peak processing is deliberately split into a locked phase and an unlocked fanout phase. `FullNode.peak_post_processing()` must run under `blockchain.priority_mutex`; it updates hints, `FullNodeStore`, mempool peak state, and gathers wallet/signage data. `peak_post_processing_2()` must run after releasing the lock; it sends timelord/full-node/wallet notifications and broadcasts tx changes.
|
||||
- Block additions use high-priority blockchain locking; transaction admission uses low-priority locking after expensive pre-validation. This prevents transaction work from starving block acceptance while still making mempool insertion atomic with the chain peak used for validation.
|
||||
- During batch sync, pre-validation can be pipelined, but actual `Blockchain.add_block()` calls are sequential and carry mutable `ValidationState`, `ForkInfo`, and an `AugmentedBlockchain`. The augmented view must be kept aligned with underlying MMR/cache state between batches.
|
||||
- `ForkInfo` is a rolling validation context, not just metadata. Skipped/already-known fork blocks still need `advance_fork_info()`/`run_single_block()` when they are not on the current main chain, otherwise later block-body validation sees an incomplete additions/removals view.
|
||||
|
||||
- `mempool_peak_added_tx_ids` — transactions re-added
|
||||
- `mempool_removals` — transactions removed
|
||||
- `fns_peak_result` — signage points and infusion points
|
||||
- `hints` — new hints for wallet notifications
|
||||
- `lookup_coin_ids` — coins to look up for wallet updates
|
||||
- `signage_points` — signage points to forward to farmers after new peak
|
||||
## Sync Model
|
||||
|
||||
---
|
||||
- `new_peak` announcements only populate `SyncStore` and trigger short/batch/long sync choices. Weight and height advertised by peers select candidates; weight proof validation and subsequent block validation decide whether to adopt.
|
||||
- Long sync has two separate flags: `sync_mode` means the node is actively batch-adding toward a validated target; `long_sync` prevents duplicate long-sync tasks while the node is still collecting peaks/weight proof. Transaction processing treats syncing as a hard rejection path.
|
||||
- Weight proofs are hostile input. `request_validate_wp()` checks tip height/weight against the selected peak, rejects cached-bad peaks, validates via `WeightProofHandler`, and bans peers that provide malformed or invalid proofs. The proof's recent chain must connect to sub-epoch summaries and VDF segment samples; it is not a substitute for downloading and validating blocks.
|
||||
- `SyncStore.peak_to_peer` is bounded and preserves the active target peak when evicting old entries. Code that changes eviction or target handling can strand sync with no peers even though peers previously announced the target.
|
||||
|
||||
## Block processing pipeline
|
||||
## Mempool Coupling
|
||||
|
||||
### 1. Receive block
|
||||
- `MempoolManager.peak` must always be the most recent transaction block, not merely the chain peak. Timelocks and block-generator selection depend on timestamped transaction-block context.
|
||||
- The fast `new_peak()` path is valid only for simple transaction-chain extension with `spent_coins` supplied. Reorgs, missing spent coins, or non-linear transaction block ancestry force full mempool reinitialization and revalidation.
|
||||
- `seen_bundle_hashes` and `in_flight_bundle_hashes` serve different DoS controls. `in_flight` deduplicates concurrent expensive pre-validation without churn; `seen` suppresses repeated known-invalid or recently processed bundles. On pending/conflict outcomes, `FullNode.add_transaction()` removes the seen mark so resubmission can succeed after state changes.
|
||||
- Peer-advertised tx fee/cost is checked twice: before fetching already-known txs and after local validation for fetched txs. Old nodes get a narrow cost tolerance for quote overhead; outside that tolerance, the peer can be banned.
|
||||
- Pending and conflict caches are retried on each new transaction peak. FF singleton spends may be rebased on simple extension; bundles containing only FF spends are intentionally rejected so every bundle has a normal invalidation path.
|
||||
- `TransactionQueue` is part of the DoS model. Local/trusted transactions bypass peer queues; peer queues are per-peer limited and use advertised fee/cost for priority plus round-robin deficit to prevent one peer from monopolizing validation.
|
||||
|
||||
`FullNodeAPI.respond_block()` / `FullNodeAPI.respond_blocks()` receive blocks
|
||||
from peers.
|
||||
## Signage, EOS, and Unfinished Blocks
|
||||
|
||||
### 2. Pre-validate
|
||||
- `FullNodeStore.finished_sub_slots` is rebuilt on every peak to represent the relevant SP/IP slots around the peak. Reorgs across a sub-slot-iterations change clear cached slots; same-difficulty reorgs preserve only signage points before the fork total-iterations boundary.
|
||||
- Future EOS/SP/IP caches exist because VDF-derived objects may arrive before the infused reward-chain challenge they depend on. They are bounded by key, entries-per-key, and TTL; peak processing drains the matching challenge and forwards only the recent cached signage points accepted by source policy.
|
||||
- `new_finished_sub_slot()` intentionally uses timelord locking rather than blockchain locking. It may reject a valid-looking EOS if a concurrent peak has made it obsolete; peak processing is expected to add the canonical sub-slot later.
|
||||
- Unfinished blocks are keyed by reward hash with an optional foliage hash because the old protocol cannot distinguish duplicate transaction-block variants. The v2 path prefers the deterministically "best" foliage hash and refuses to fetch worse or excessive duplicates.
|
||||
- `seen_unfinished_blocks`, `pending_tx_request`, `peers_with_tx`, and tx fetch tasks are volatile anti-duplication/backpressure state. Failure to clear timeout entries does not corrupt consensus, but it can suppress useful fetches or misattribute peer tx advertisements.
|
||||
|
||||
`pre_validate_block()` runs header validation + CLVM execution in parallel
|
||||
(thread pool). Returns `PreValidationResult` with `required_iters` and
|
||||
`conds`.
|
||||
## Persistence Semantics
|
||||
|
||||
### 3. Add to blockchain
|
||||
- `BlockStore.add_full_block()` inserts blocks with `in_main_chain=False`; `Blockchain`/store peak management later marks the canonical path. Reorg rollback clears `in_main_chain` above the fork but does not delete fork blocks.
|
||||
- `CoinStore` represents the current peak's UTXO view. Its spent-index sentinels distinguish normal unspent coins, coins spent at a height, and FF singleton lineage optimization. Rollback recalculates the FF marker based on parent lineage, so FF behavior depends on rollback correctness.
|
||||
- Coin-store writes expect exact row counts when marking spends. A mismatch is treated as invalid state rather than a soft miss, which protects the one-coin-one-record invariant but makes DB consistency assumptions visible as exceptions.
|
||||
- Hint persistence is derived from `StateChangeSummary` after block acceptance. Hint size policy lives in source; maximum-size hints also serve as puzzle-hash subscriptions for wallet notification lookup.
|
||||
|
||||
Under `blockchain.priority_mutex` (high priority):
|
||||
## RPC and Wallet Notification Gotchas
|
||||
|
||||
- `Blockchain.add_block()` validates body, updates DB, reorgs if needed
|
||||
- Returns `(AddBlockResult, Err, StateChangeSummary)`
|
||||
- `get_blockchain_state` reports `sync_mode` as `sync_mode or long_sync`; this can show syncing before block download begins. `synced` additionally requires recent transaction-block time and at least one full-node connection unless in simulator mode.
|
||||
- Recent signage point/EOS RPC responses may report `reverted=True` by searching recent chain context, because `recent_*` caches outlive reorgs for pooling/UI support.
|
||||
- `/get_additions_and_removals` explicitly checks that the requested block hash is still the main-chain hash at its height while holding the blockchain mutex; without that check fork blocks in `BlockStore` would look like canonical history.
|
||||
- `/push_tx` calls `FullNode.add_transaction()` directly and returns only `FAILED` as an `RpcError`; it does not use the peer transaction queue. Wallet P2P `send_transaction` enqueues a `TransactionQueueEntry` (trusted peers get high-priority treatment), waits up to 45 seconds on `queue_entry.done`, and returns `PENDING` on timeout.
|
||||
|
||||
### 4. Post-processing (peak_post_processing)
|
||||
## Fragility Hotspots
|
||||
|
||||
If `NEW_PEAK`:
|
||||
- The highest-risk edits are those that move work across the blockchain mutex boundary, change `sync_mode`/`long_sync` transitions, alter `ForkInfo` advancement during skipped blocks, or treat non-transaction peaks as mempool peaks.
|
||||
- Cache limits in `FullNodeStore`, `SyncStore`, and `TransactionQueue` are behavioral controls, not just memory tuning. Raising or removing them changes peer-driven resource exposure.
|
||||
- Weight-proof code mixes deterministic sampling, cached sub-epoch segments, recent-chain reconstruction, and multiprocessing VDF validation. Keep seed derivation, summary ordering, and segment count bounds stable unless the consensus proof format changes.
|
||||
- Tests around this module often need to assert timing/ordering effects rather than just return values: lock phase vs fanout phase, tx queue timeout vs eventual mempool inclusion, sync flag transitions, and reorg-driven mempool rebuilds.
|
||||
|
||||
- Update `FullNodeStore` with new signage points
|
||||
- Update `MempoolManager` with `new_peak()`
|
||||
- Process hints and subscriptions
|
||||
- Compute wallet notifications
|
||||
## Source Pointers
|
||||
|
||||
### 5. Broadcast
|
||||
|
||||
- Send `new_peak` to full node peers
|
||||
- Send `new_peak_wallet` to wallet peers
|
||||
- Send coin state updates to subscribed wallets
|
||||
- Forward new signage points to farmer
|
||||
|
||||
---
|
||||
|
||||
## `FullNodeStore` — Signage point & unfinished block tracking
|
||||
|
||||
**Location**: `full_node/full_node_store.py`
|
||||
|
||||
### Key state
|
||||
|
||||
- Signage points per challenge hash (LRU-bounded)
|
||||
- End-of-sub-slot bundles per challenge hash
|
||||
- Unfinished blocks indexed by `(reward_hash, foliage_hash)`
|
||||
- Peers that advertised each transaction (`peers_with_tx`)
|
||||
- Seen compact VDFs (dedup set)
|
||||
|
||||
### Constants
|
||||
|
||||
- `MAX_UNFINISHED_BLOCKS_PER_REWARD_HASH = 20` — eviction of worst foliage
|
||||
|
||||
### `new_peak()` returns
|
||||
|
||||
- `added_eos`: any end-of-sub-slot that becomes relevant
|
||||
- `new_signage_points`: signage points that can now be released
|
||||
- `new_infusion_points`: infusion points for timelord
|
||||
|
||||
---
|
||||
|
||||
## Sync logic
|
||||
|
||||
### Weight proof sync
|
||||
|
||||
1. Peer announces `new_peak` with higher weight
|
||||
2. Request `request_proof_of_weight` → `WeightProof`
|
||||
3. Validate weight proof (sub-epoch summaries, VDF segments)
|
||||
4. If valid: switch to batch download
|
||||
|
||||
### Batch sync
|
||||
|
||||
1. Download blocks in ranges via `request_blocks` (max 32 per request)
|
||||
2. Pre-validate batches in parallel
|
||||
3. Add blocks sequentially under blockchain lock
|
||||
4. Continue until caught up to peer's peak
|
||||
|
||||
### Long sync detection
|
||||
|
||||
If peer peak is significantly ahead, enters long sync mode. During long sync,
|
||||
transactions are not processed (mempool frozen).
|
||||
|
||||
---
|
||||
|
||||
## `FullNodeAPI` — P2P message handlers
|
||||
|
||||
**Location**: `full_node/full_node_api.py`
|
||||
|
||||
### Key handlers
|
||||
|
||||
| Handler | Trigger | Notes |
|
||||
| ---------------------------------------------------- | ------------------------ | ---------------------------------------- |
|
||||
| `new_peak()` | Peer has new peak | Triggers sync if heavier |
|
||||
| `new_transaction()` | Peer has new tx | Adds to `peers_with_tx`, schedules fetch |
|
||||
| `request_transaction()` | Peer wants a tx | Look up in mempool |
|
||||
| `respond_transaction()` | Received requested tx | Pre-validate + add to mempool |
|
||||
| `send_transaction()` | Wallet submits tx | Pre-validate + add to mempool |
|
||||
| `respond_block()` | Received single block | Add to blockchain |
|
||||
| `respond_blocks()` | Received block batch | Add batch to blockchain |
|
||||
| `new_signage_point_or_end_of_sub_slot()` | New SP/EOS | Store + broadcast |
|
||||
| `new_unfinished_block()` / `new_unfinished_block2()` | Farmer block | Validate + infuse |
|
||||
| `request_compact_vdf()` | Peer wants compact proof | Look up + respond |
|
||||
|
||||
### Transaction processing
|
||||
|
||||
`respond_transaction()` and `send_transaction()` both:
|
||||
|
||||
1. Check `seen_bundle_hashes` for dedup
|
||||
2. Run `pre_validate_spendbundle()` in thread pool
|
||||
3. Acquire blockchain lock (low priority)
|
||||
4. Call `add_spend_bundle()`
|
||||
5. On success: broadcast `new_transaction` to peers
|
||||
|
||||
---
|
||||
|
||||
## `FullNodeRpcApi` — RPC endpoints
|
||||
|
||||
**Location**: `full_node/full_node_rpc_api.py`
|
||||
|
||||
### Key endpoints
|
||||
|
||||
- `get_blockchain_state` — peak, sync status, mempool info, space estimate
|
||||
- `get_block` / `get_blocks` — fetch by height or hash
|
||||
- `get_block_record` / `get_block_records` — lightweight records
|
||||
- `get_coin_record_by_name` — single UTXO lookup
|
||||
- `get_coin_records_by_*` — batch lookups by puzzle hash, parent, hint
|
||||
- `push_tx` — submit transaction (same as `send_transaction` P2P)
|
||||
- `get_mempool_item_by_tx_id` — mempool query
|
||||
- `get_fee_estimate` — fee estimation
|
||||
- `get_network_space` — estimated network space
|
||||
|
||||
---
|
||||
|
||||
## `CoinStore` — UTXO database
|
||||
|
||||
**Location**: `full_node/coin_store.py`
|
||||
|
||||
### Schema
|
||||
|
||||
```sql
|
||||
coin_record(
|
||||
coin_name BLOB PRIMARY KEY,
|
||||
confirmed_index BIGINT,
|
||||
spent_index BIGINT, -- >0 spent at that height; 0 = normal unspent; -1 = FF lineage unspent
|
||||
coinbase INT,
|
||||
puzzle_hash BLOB,
|
||||
coin_parent BLOB,
|
||||
amount BLOB, -- 8-byte uint64
|
||||
timestamp BIGINT
|
||||
)
|
||||
```
|
||||
|
||||
### Indexes
|
||||
|
||||
- `coin_confirmed_index` — reorg rollbacks
|
||||
- `coin_spent_index` — spent coin queries
|
||||
- `coin_puzzle_hash` — address lookups
|
||||
- `coin_parent_index` — parent traversal
|
||||
- `coin_record_ph_ff_unspent_idx` (partial, new DBs only) — FF singleton optimization
|
||||
|
||||
### Key operations
|
||||
|
||||
- `new_block()` — batch insert additions, mark removals as spent
|
||||
- `rollback_to_block()` — revert coins confirmed/spent above a height
|
||||
- `get_coin_records()` — fetch by coin IDs
|
||||
- `get_coin_records_by_puzzle_hash()` — wallet queries
|
||||
- `get_unspent_lineage_info_for_puzzle_hash()` — FF singleton lineage
|
||||
|
||||
---
|
||||
|
||||
## `BlockStore` — Full block persistence
|
||||
|
||||
**Location**: `full_node/block_store.py`
|
||||
|
||||
### Schema
|
||||
|
||||
- `full_blocks` table: header_hash, height, in_main_chain flag, block_record, full block bytes
|
||||
- `sub_epoch_segments_v3` table: sub-epoch challenge segments for weight proofs
|
||||
|
||||
### Key operations
|
||||
|
||||
- `add_full_block()` — insert with block record
|
||||
- `get_full_block()` / `get_full_blocks_at()` — fetch
|
||||
- `set_in_chain()` — mark blocks as main chain
|
||||
- `set_peak()` — update peak pointer
|
||||
- `rollback()` — clear in_chain and sub-epoch data above height
|
||||
- Transaction support via `self.db_wrapper.writer_maybe_transaction()`
|
||||
For exact store schemas, volatile cache behavior, sync-target handling, transaction backpressure, mempool admission behavior, and block-processing sequence, read the source files in `chia/full_node/` listed in the Landmarks table above. For focused coverage, start with the matching tests under `chia/_tests/core/full_node/stores/`, plus `chia/_tests/core/full_node/test_tx_processing_queue.py`.
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
# Chia Harvester Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/harvester/` is the local farming edge. It connects only to farmers, learns the farmer/pool keys through the farmer-initiated harvester handshake, discovers local plot files, checks signage-point eligibility, returns V1 proofs or V2 partial proofs, signs selected farmer requests with plot-local keys, and streams plot inventory changes back to the farmer.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for farmer-harvester handshake behavior, local plot refresh, signage-point proof lookup, harvester signatures, plot-sync sender behavior, and harvester RPC/config mutation. For plot creation/cache/prover details, read `plotting.md`; for farmer-side proof handling and pool partials, read `farmer.md`.
|
||||
|
||||
## Landmarks
|
||||
|
||||
| file | owns |
|
||||
| ------------------------------------- | ------------------------------------------------ |
|
||||
| `chia/harvester/harvester.py` | plot state, refresh lifecycle, plot-sync sender |
|
||||
| `chia/harvester/harvester_api.py` | farmer-facing proof/signature/plot-sync handlers |
|
||||
| `chia/harvester/harvester_rpc_api.py` | local plot list/refresh/delete, config mutation |
|
||||
| `chia/harvester/start_harvester.py` | harvester Service construction |
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `Harvester` is the service-local authority for plot state, refresh lifecycle, decompressor configuration, RPC state notifications, and the `Sender` used for plot-sync. It does not validate farmer/full-node consensus state; it trusts the farmer to send current signage point metadata and pool difficulty hints.
|
||||
- `HarvesterAPI` is the farmer-facing peer API. It is where farmer messages become plot lookups, proof/partial-proof responses, signature responses, and plot list responses. It should stay thin around `PlotManager` and protocol types because it runs in latency-sensitive signage-point paths.
|
||||
- `PlotManager` from `chia/plotting/manager.py` owns plot discovery and the canonical in-memory map from `Path` to `PlotInfo`. The harvester accesses `plot_manager.plots` under the manager lock; background refresh mutates the same state from a thread.
|
||||
- `plot_sync.Sender` is the harvester-side inventory replication state machine. It sends ordered plot-sync messages and waits for `plot_sync_response` acknowledgements from the farmer. The farmer-side `Receiver` is the state authority for what the farmer/UI believes this harvester has.
|
||||
- RPC methods are local operator controls over the same state: list/refresh/delete plots, mutate plot directories, and persist harvester config. They are not the farmer protocol and should not be used to infer peer behavior.
|
||||
|
||||
## Why This Is Tricky
|
||||
|
||||
Public docs describe harvesters as checking plots for a farmer, but source correctness depends on when plots become usable. The farmer handshake is not just greeting metadata: it installs the keys that decide which local plots may be farmed. Plot refresh, proof lookup, signature lookup, and plot sync all read the same `PlotManager` state under different timing constraints, so moving work across the lock or handshake boundary can make the farmer's inventory diverge from the harvester's local state.
|
||||
|
||||
## Wrong Assumptions To Avoid
|
||||
|
||||
- Do not load or report plots before the farmer handshake installs acceptable keys.
|
||||
- Do not treat plot-sync inventory as proof that a plot can produce a valid proof for the current signage point.
|
||||
- Do not perform expensive prover I/O while holding the plot-manager lock.
|
||||
- Do not assume the harvester validates chain freshness; the farmer/full node supply and validate the consensus context.
|
||||
|
||||
## Service Lifecycle
|
||||
|
||||
- `start_harvester.create_harvester_service()` builds a `Service` with `NodeType.HARVESTER`, no advertised port, configured farmer peers, optional RPC, and network-specific consensus constant overrides.
|
||||
- `Harvester.manage()` creates async runtime state, then on shutdown marks `_shut_down`, shuts down the proof lookup executor, stops plot refreshing, resets the plot manager, stops plot sync, and waits for plot-sync closure.
|
||||
- A harvester starts loading plots only after receiving `harvester_handshake`. That handler installs farmer/pool public keys into `PlotManager`, binds the plot-sync sender to the farmer connection, starts plot-sync, then starts plot refreshing.
|
||||
- On farmer disconnect, the harvester stops plot-sync and plot refreshing. Reconnect must re-run the farmer handshake before plots become usable again.
|
||||
|
||||
## Plot Discovery And State
|
||||
|
||||
- Plot discovery is driven by `PlotManager.start_refreshing()`, which loads the plot cache and starts a thread that periodically scans configured directories for `*.plot` and `*.plot2` files.
|
||||
- Refresh emits `started`, `batch_processed`, and `done` callbacks. `Harvester._plot_refresh_callback()` converts those into plot-sync start, loaded batches, and final removed/invalid/no-key/duplicate lists.
|
||||
- `PlotManager` rejects or quarantines plots when keys do not match the farmer handshake, files fail to open, duplicates are found by filename, compression exceeds configured limits, compressed plots lack decompressor contexts, or uncompressed V1 files look too small to be complete.
|
||||
- `plot_manager.plots` is protected by `PlotManager`'s lock. Code that enumerates plots for RPC, signage-point lookup, or signature lookup must use `with self.harvester.plot_manager:` to avoid racing the refresh thread.
|
||||
- The plot cache is stored under `cache/plot_manager_v2.dat`; stale unused entries are evicted after refresh. Downgrade compatibility is intentionally handled by using a distinct v2 cache file.
|
||||
|
||||
## Farmer Protocol Flow
|
||||
|
||||
- The server layer enforces that farmers send `harvester_handshake`, `new_signage_point_harvester`, `request_signatures`, `request_plots`, and `plot_sync_response`; harvesters send proofs, signatures, plot inventory, plot-sync messages, and `farming_info`.
|
||||
- The signage-point message has a live compatibility split. Old farmers/harvesters use `NewSignagePointHarvester` with precomputed `filter_prefix_bits`; newer peers use `NewSignagePointHarvester2` with `peak_height` and `last_tx_height`, letting the harvester compute filter bits per plot version and enforce fork-height rules. V2 gating (skip before `HARD_FORK2_HEIGHT`) and V1 phase-out (`v1_cut_off_height()`) are keyed on `last_tx_height`, not peak height.
|
||||
- `new_signage_point_harvester()` ignores challenges until farmer/pool keys are available. This preserves the invariant that loaded plots have already been filtered against farmer-owned keys.
|
||||
- For each challenge, the harvester snapshots eligible plots under lock, applies the plot filter, skips V2 plots before `HARD_FORK2_HEIGHT`, skips V1 plots after `v1_cut_off_height()`, and performs blocking quality/proof work in the harvester executor.
|
||||
- V1 plots return full `NewProofOfSpace` messages directly. V2 plots return `PartialProofsData`; the farmer forwards each partial proof to solver services and later reconstructs a `NewProofOfSpace` for normal farmer processing.
|
||||
- The harvester sends `FarmingInfo` to the farmer after each signage-point lookup with total plots, plots passing filter, V1 proof count, and elapsed lookup time. UI/metrics consumers receive this through farmer and harvester state-change paths.
|
||||
|
||||
## Proof Lookup Invariants
|
||||
|
||||
- Plot filter and quality checks must use `calculate_pos_challenge(plot_id, challenge_hash, sp_hash)`, `calculate_iterations_quality()`, and `calculate_sp_interval_iters()` with the correct plot parameter and difficulty source.
|
||||
- Pool-specific difficulty overrides are keyed by `pool_contract_puzzle_hash`. If no matching `PoolDifficulty` exists, the global signage-point difficulty/sub-slot iters remain in force.
|
||||
- V1 full proof retrieval can fail for decompressor timeouts, line point compression (`GRResult_NoProof received`), plot I/O/prover errors, or fork phase-out checks. These failures are logged and dropped per plot rather than aborting the whole challenge.
|
||||
- V2 partial proof lookup returns only partial proofs that pass the same required-iterations threshold. It must carry plot id, plot index, `meta_group`, strength, plot public key, and pool identifiers because the farmer/solver path needs those fields to build the final `ProofOfSpace`.
|
||||
- V2 filter eligibility uses the same `calculate_prefix_bits()` → `passes_plot_filter()` path as V1, but with `NUMBER_ZERO_BITS_PLOT_FILTER_V2` and height-based adjustments (`PLOT_FILTER_V2_FIRST/SECOND/THIRD_ADJUSTMENT_HEIGHT`). Plots with `strength_v2` below `MIN_PLOT_STRENGTH` or above `MAX_PLOT_STRENGTH` are rejected by `check_plot_param()`; harvester-side and validation-side filter math must stay identical or farmers produce proofs full nodes reject. See `chia/types/blockchain_format/proof_of_space.py` for exact constants.
|
||||
- The lookup warning threshold is operationally important: late proof lookup risks missed rewards. Avoid adding synchronous work, unbounded logging, RPC calls, or long lock holds to the signage-point path.
|
||||
|
||||
## Signature Authority
|
||||
|
||||
- `request_signatures()` derives the local plot secret from the plot memo, not from a long-lived in-memory key table. It parses memo bytes into pool key-or-puzzle-hash, farmer public key, and local master secret, then derives the local secret with `master_sk_to_local_sk()`.
|
||||
- The returned signature is the harvester half of the plot signature. The farmer combines it with the farmer key, and for pooled plots also taproot signature material, before declaring proofs or submitting pool partials.
|
||||
- `plot_identifier` is overloaded: V1 proof responses combine the quality string with the resolved filename, and signature handling recovers the file path by reversing that encoding. Changing this format requires coordinated farmer changes.
|
||||
- Source signature data fields in the protocol exist for CHIP-22/third-party harvester workflows. Local harvester behavior is source-owned; do not assume those optional protocol fields are unused globally.
|
||||
|
||||
## Plot Sync Contract
|
||||
|
||||
- Plot sync is ordered by `(sync_id, message_id)`. The sender sends exactly one in-flight sync message, waits for `plot_sync_response`, then advances or resets/retries based on the receiver's expected identifier.
|
||||
- A refresh cycle is encoded as `plot_sync_start`, `plot_sync_loaded` batches, `plot_sync_removed`, `plot_sync_invalid`, `plot_sync_keys_missing`, `plot_sync_duplicates`, and `plot_sync_done`. Empty lists are still sent as final markers so the receiver can advance states.
|
||||
- `Sender._reset()` rebuilds a full initial-style inventory from the current `PlotManager` state when recovery is needed while the task is running. This makes idempotent recovery depend on the plot manager's current lock-protected snapshot.
|
||||
- Farmer `Receiver` applies deltas only at `plot_sync_done`, updates total/effective plot sizes, and emits UI updates for initial sync or non-empty deltas. Mid-sync callbacks are progress notifications, not committed inventory.
|
||||
- If changing plot sync messages, update both harvester `Sender` and farmer `Receiver`, plus protocol sender maps and request/response expectations. The message sequence is a two-sided state machine, not a batch of independent notifications.
|
||||
|
||||
## RPC And Config Mutation
|
||||
|
||||
- `HarvesterRpcApi` exposes local endpoints for plot listing, refresh triggering, file deletion, plot-directory mutation, and harvester config mutation.
|
||||
- `delete_plot()` unlinks the supplied path, triggers refresh, and emits a `"plots"` state change. There is no extra ownership model in this module; RPC authorization and local file permissions are the relevant gates.
|
||||
- `update_harvester_config()` persists config changes but does not reconfigure an already-created decompressor or restart refresh with new scan settings by itself. Treat many settings as taking effect on future refresh/service restart unless the caller explicitly triggers the relevant lifecycle.
|
||||
- RPC validates only the minimum refresh interval and basic type coercion. Validation of directory existence, plot readability, and key ownership remains in plot refresh.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Do not start plot refreshing before `harvester_handshake`; doing so loads plots without the farmer/pool key filter and can create incorrect inventory and eligibility state.
|
||||
- Do not hold the `PlotManager` lock across disk proof reads. The current path snapshots work while scheduling executor tasks and releases the lock before expensive lookups.
|
||||
- Do not alter `Plot.size` high-bit semantics without a protocol bump. The high bit distinguishes V2 strength from V1 k-size and is consumed by farmer-side plot-sync/UI calculations.
|
||||
- Be careful with `plot_identifier` parsing and path normalization. Signature lookup depends on exact V1 identifier formatting and resolved paths matching the plot manager keys.
|
||||
- GPU/compressed-plot behavior crosses into `chiapos.decompressor_context_queue`; failures may present as runtime strings from the prover/decompressor. Tests should cover timeout/drop behavior without assuming every failure raises the same Python exception.
|
||||
- Protocol edits in this area are compatibility edits. Update `harvester_protocol.py`, `protocol_message_types.py`, sender authorization, API/stub decorators, farmer receiver/handler code, and protocol version behavior together.
|
||||
|
||||
## Test And Audit Strategy
|
||||
|
||||
- For signage-point changes, test V1 and V2 paths separately: plot filter eligibility, fork-height gating, pool difficulty override, no-key handshake gating, proof/partial-proof counts, and `FarmingInfo` side effects.
|
||||
- For refresh or plot inventory changes, test `PlotManager` state plus plot-sync sender/receiver convergence. A correct local `get_plots()` result is not sufficient if the farmer receiver would apply a different inventory.
|
||||
- For signature changes, verify both self-pooled and pool-contract memo shapes, aggregate plot public key generation, missing plot behavior, and farmer-side `_process_respond_signatures()` expectations.
|
||||
- For RPC/config changes, test persisted config and runtime behavior separately. Several settings are read during service construction or refresh lifecycle rather than dynamically reconfigured.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
In-module files are in the Landmarks table above. Cross-module authorities:
|
||||
|
||||
- Plot inventory and cache authority: `chia/plotting/manager.py`, `chia/plotting/cache.py`, `chia/plotting/prover.py`.
|
||||
- Plot-sync sender/receiver contract: `chia/plot_sync/sender.py`, `chia/plot_sync/receiver.py`.
|
||||
- Config helpers used by RPC mutation: `chia/plotting/util.py`.
|
||||
@@ -0,0 +1,133 @@
|
||||
# Chia Introducer Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/introducer/` is a small bootstrap service whose job is to help wallets,
|
||||
farmers, and full nodes discover full-node peers. It is not an address-book
|
||||
authority and does not validate blockchain state. Treat every address it
|
||||
learns, vets, or returns as advisory until the receiving node's discovery and
|
||||
connection logic accepts it.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for introducer peer collection, TCP vetting, DNS fallback, introducer request/response protocol behavior, and bootstrap service startup. For receiver-side peer ingestion and address-manager policy, also read `server.md`.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `Introducer` owns service-local configuration and the background vetting loop:
|
||||
`max_peers_to_send`, `recent_peer_threshold`, `default_port`, DNS fallback
|
||||
servers, and the resolver used for DNS seeder lookups.
|
||||
- `IntroducerAPI` owns the single P2P handler,
|
||||
`request_peers_introducer`, and is responsible for filtering the server's
|
||||
introducer peer set down to vetted peers before returning
|
||||
`RespondPeersIntroducer`.
|
||||
- `ChiaServer`, not `Introducer`, owns the in-memory peer set. It creates
|
||||
`IntroducerPeers` only when the local node type is `INTRODUCER`, and it adds a
|
||||
peer only after a successful inbound handshake whose negotiated peer type is
|
||||
`FULL_NODE`.
|
||||
- `IntroducerPeers` is a volatile set, not durable storage. Entries are keyed by
|
||||
`(host, port)` through `VettedPeer.__hash__`/`__eq__`, carry local vetting
|
||||
counters, and disappear on process restart.
|
||||
- `FullNodeDiscovery` / wallet discovery own ingestion of introducer responses.
|
||||
Introducer-returned timestamps are normalized, invalid hosts are
|
||||
dropped, wrong-network default ports are filtered, and addresses enter the
|
||||
"new" table as untrusted candidates.
|
||||
|
||||
## Peer Collection And Vetting
|
||||
|
||||
- The introducer learns candidates passively from inbound full-node
|
||||
connections. A peer that never connects to the introducer as a full node is
|
||||
not eligible to be returned from the in-memory peer set.
|
||||
- Vetting is a TCP reachability check only: the service periodically samples
|
||||
recent peers, opens a raw TCP connection to `host:port` with a configured
|
||||
timeout, then closes it. It does not perform a Chia handshake or verify node
|
||||
type during vetting.
|
||||
- Positive `vetted` values mean consecutive successful reachability checks;
|
||||
negative values mean consecutive failures. Successful peers are rechecked
|
||||
after the configured interval; failed peers are not retried until the
|
||||
configured cooldown passes and are removed after repeated failures.
|
||||
- `recent_peer_threshold` gates what can be served. The API asks
|
||||
`IntroducerPeers` for recently added peers using this threshold, while the
|
||||
vetting loop samples a wider window than the DNS publication threshold so older
|
||||
candidates can be refreshed before aging out of responses.
|
||||
- API responses deliberately skip the requesting peer's own `(host, port)`.
|
||||
Keep that exclusion when changing response construction; returning a client
|
||||
to itself wastes bootstrap attempts and can amplify duplicate/self-connection
|
||||
churn.
|
||||
|
||||
## DNS Fallback Semantics
|
||||
|
||||
- DNS fallback is used only when the vetted in-memory response has fewer than
|
||||
`max_peers_to_send` peers. The introducer queries one randomly chosen DNS
|
||||
seeder for both `A` and `AAAA` records and wraps each IP with the configured
|
||||
`default_full_node_port`.
|
||||
- If `introducer.dns_servers` is omitted or empty on mainnet, startup injects
|
||||
`dns-introducer.chia.net`. Other networks depend on explicit config.
|
||||
- DNS answers are returned with normalized timestamps, the same trust level as
|
||||
introducer peers on the receiving side. Do not treat DNS fallback as stronger
|
||||
evidence than the local vetting set.
|
||||
- `get_peers_from_dns()` currently lets resolver exceptions propagate to the API
|
||||
handler. A failing DNS query can therefore fail the request instead of
|
||||
returning the partial vetted peer list unless callers add explicit handling.
|
||||
|
||||
## Wire Protocol Contracts
|
||||
|
||||
- The introducer protocol has exactly one request/reply pair:
|
||||
`request_peers_introducer` -> `respond_peers_introducer`.
|
||||
- Sender authorization is asymmetric. `WALLET`, `FARMER`, and `FULL_NODE` may
|
||||
send `request_peers_introducer`; only `INTRODUCER` may send
|
||||
`respond_peers_introducer`.
|
||||
- Runtime response validation depends on
|
||||
`protocol_state_machine.VALID_REPLY_MESSAGE_MAP`; protocol evolution must keep
|
||||
payload dataclasses, message IDs, sender map, API/stub decorators,
|
||||
rate limits, and reply map in sync.
|
||||
- Rate limits are small for requests and bounded for responses. Exact values
|
||||
belong in the rate-limit source tables.
|
||||
- Clients use introducer connections as short-lived bootstrap links. Full nodes
|
||||
send the request from `FullNodeDiscovery._introducer_client()`, ingest returned
|
||||
peers with `is_full_node=False`, and close the introducer peer after handling
|
||||
the response. Wallets follow the same close-after-response pattern.
|
||||
|
||||
## Startup And Service Shape
|
||||
|
||||
- `start_introducer.create_introducer_service()` builds a normal
|
||||
`Service[Introducer, IntroducerAPI, FullNodeRpcApi]` with
|
||||
`NodeType.INTRODUCER`. The RPC type alias is shared with full node service
|
||||
plumbing; this module does not define a dedicated introducer RPC surface.
|
||||
- The service uses public SSL paths from the full-node certificate config and
|
||||
normal ChiaServer handshake/rate-limit machinery. Introducer-specific behavior
|
||||
starts only after the connection has passed the generic server admission path.
|
||||
- `Introducer.server` is a non-null property that raises `RuntimeError` before
|
||||
`set_server()` has run. Code that needs to tolerate pre-service wiring should
|
||||
inspect `_server` or be structured to run only after service setup, rather
|
||||
than comparing `introducer.server is None`.
|
||||
- `Introducer.manage()` starts the vetting task and cancels it on shutdown. The
|
||||
task uses `_shut_down` checks plus cancellation; edits should preserve both so
|
||||
service teardown does not wait on long sleeps or stuck DNS/TCP work.
|
||||
- `on_connect()` and state-change callbacks are intentionally no-ops today. The
|
||||
meaningful connect side effect is in `ChiaServer.incoming_connection()`, which
|
||||
adds inbound full nodes to `server.introducer_peers`.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Do not broaden which peer types can populate `IntroducerPeers` without a clear
|
||||
reason. Letting wallets/farmers/self-reported peers seed the introducer would
|
||||
weaken bootstrap quality and make Sybil address injection easier.
|
||||
- Keep introducer outputs bounded by `max_peers_to_send` and receiver-side peer
|
||||
ingestion limits. Peer-list payloads are adversarial resource surfaces even
|
||||
though `RespondPeersIntroducer` is a simple streamable list.
|
||||
- Avoid treating `vetted` as proof of a valid Chia full node. It only proves a
|
||||
recent TCP accept on that address; receiving nodes must still connect,
|
||||
handshake, check network ID, and apply address-manager policy.
|
||||
- DNS failure behavior and partial-response behavior are worth testing around
|
||||
because the current API awaits DNS after collecting vetted peers. A resolver
|
||||
exception can prevent otherwise usable vetted peers from being sent.
|
||||
- Tests that exercise discovery should control DNS and async convergence. Existing
|
||||
harnesses commonly disable introducer DNS in setup and use fake resolvers for
|
||||
fallback behavior.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Introducer service state and vetting: `chia/introducer/introducer.py`.
|
||||
- Peer request/reply handling: `chia/introducer/introducer_api.py`.
|
||||
- Service startup and wiring: `chia/introducer/start_introducer.py`, `chia/introducer/introducer_service.py`.
|
||||
@@ -1,244 +0,0 @@
|
||||
# Networking & Peer Protocol — Deep Context
|
||||
|
||||
> Attach when touching `chia/server/`, `chia/protocols/`, connection handling,
|
||||
> rate limiting, or peer discovery.
|
||||
|
||||
## File map
|
||||
|
||||
### `chia/server/`
|
||||
|
||||
| File | Lines | Role |
|
||||
| -------------------------- | ----- | ---------------------------------------------------- |
|
||||
| `server.py` | ~900 | `ChiaServer`: connection management, message routing |
|
||||
| `ws_connection.py` | ~790 | `WSChiaConnection`: single peer connection |
|
||||
| `rate_limits.py` | ~160 | `RateLimiter`: per-connection rate enforcement |
|
||||
| `rate_limit_numbers.py` | ~200 | Rate limit values per message type |
|
||||
| `chia_policy.py` | ~370 | Custom asyncio event loop policy, connection limits |
|
||||
| `node_discovery.py` | ~850 | `FullNodePeers`: peer discovery and management |
|
||||
| `address_manager.py` | ~1050 | Address book for known peers |
|
||||
| `address_manager_store.py` | ~15 | Address persistence |
|
||||
| `api_protocol.py` | ~117 | `ApiProtocol`, `ApiMetadata`, `@request` decorator |
|
||||
| `capabilities.py` | ~20 | Capability detection |
|
||||
| `introducer_peers.py` | ~65 | Introducer peer handling |
|
||||
| `resolve_peer_info.py` | ~50 | DNS resolution |
|
||||
| `start_service.py` | ~350 | Service lifecycle management |
|
||||
| `signal_handlers.py` | ~100 | Graceful shutdown |
|
||||
| `ssl_context.py` | ~25 | TLS configuration |
|
||||
| `upnp.py` | ~100 | UPnP port forwarding |
|
||||
|
||||
### `chia/protocols/`
|
||||
|
||||
| File | Lines | Role |
|
||||
| --------------------------------------- | ----- | ----------------------------------------------- |
|
||||
| `protocol_message_types.py` | ~147 | `ProtocolMessageTypes` enum (109 message types) |
|
||||
| `protocol_state_machine.py` | ~88 | Valid request→response map, import-time check |
|
||||
| `protocol_message_type_to_node_type.py` | ~230 | Message type → allowed node type mapping |
|
||||
| `outbound_message.py` | ~25 | `Message`, `NodeType`, `make_msg()` |
|
||||
| `protocol_timing.py` | ~10 | Ban duration constants |
|
||||
| `shared_protocol.py` | ~80 | `Handshake`, `Capability`, `protocol_version` |
|
||||
| `full_node_protocol.py` | ~217 | Full node ↔ full node message types |
|
||||
| `wallet_protocol.py` | ~400 | Wallet ↔ full node message types |
|
||||
| `farmer_protocol.py` | ~75 | Farmer ↔ full node message types |
|
||||
| `harvester_protocol.py` | ~190 | Farmer ↔ harvester message types |
|
||||
| `timelord_protocol.py` | ~80 | Full node ↔ timelord message types |
|
||||
| `solver_protocol.py` | ~15 | Solver protocol |
|
||||
| `pool_protocol.py` | ~110 | Pool protocol messages |
|
||||
| `introducer_protocol.py` | ~15 | Introducer protocol |
|
||||
| `fee_estimate.py` | ~50 | Fee estimate messages |
|
||||
|
||||
---
|
||||
|
||||
## `WSChiaConnection` — Per-peer connection
|
||||
|
||||
**Location**: `server/ws_connection.py`
|
||||
|
||||
### Key properties
|
||||
|
||||
- WebSocket-based (aiohttp)
|
||||
- Mutual TLS authentication
|
||||
- `local_type: NodeType` — our node type
|
||||
- `peer_node_id: bytes32` — peer identity
|
||||
- `is_outbound: bool`
|
||||
- `peer_capabilities: list[Capability]`
|
||||
|
||||
### Constants
|
||||
|
||||
- `LENGTH_BYTES = 4` — message length prefix (max ~4 GiB per message)
|
||||
- `MAX_VERSION_STRING_BYTES = 128`
|
||||
- `MAX_PENDING_COMPACT_VDFS = 100`
|
||||
|
||||
### Message flow
|
||||
|
||||
1. Receive raw bytes over WebSocket
|
||||
2. Parse length prefix + `Message` (type + id + data)
|
||||
3. Look up handler in `ApiMetadata.message_type_to_request`
|
||||
4. Deserialize data via `Streamable.from_bytes()`
|
||||
5. Call handler, send reply if expected
|
||||
|
||||
### Error handling & banning
|
||||
|
||||
- `ApiError` from handlers is converted to an `error` response (not an automatic ban)
|
||||
- `ConsensusError` from handlers → close + ban for `CONSENSUS_ERROR_BAN_SECONDS`
|
||||
- Other unhandled handler exceptions → close + ban for `API_EXCEPTION_BAN_SECONDS`
|
||||
- Rate limit exceeded (for full node inbound peers) → close + ban for `RATE_LIMITER_BAN_SECONDS`
|
||||
- Protocol response mismatch (`message_response_ok` failure) → `ban_peer_bad_protocol()` using `INTERNAL_PROTOCOL_ERROR_BAN_SECONDS`
|
||||
|
||||
### Protocol state machine validation
|
||||
|
||||
On receiving a reply, `message_response_ok()` checks that the response type is
|
||||
valid for the original request type (defined in `VALID_REPLY_MESSAGE_MAP`).
|
||||
|
||||
---
|
||||
|
||||
## Rate limiting
|
||||
|
||||
**Location**: `server/rate_limits.py`, `server/rate_limit_numbers.py`
|
||||
|
||||
### Two-tier system
|
||||
|
||||
**Per-message-type limits** (`RLSettings`):
|
||||
|
||||
- `frequency`: max count per 60 seconds
|
||||
- `max_size`: max bytes per single message
|
||||
- `max_total_size`: max cumulative bytes per 60 seconds (optional)
|
||||
- `aggregate_limit`: whether to count against the global aggregate
|
||||
|
||||
**Aggregate limit** (across all non-tx message types):
|
||||
|
||||
- 1000 messages per minute
|
||||
- 100 MB per minute
|
||||
|
||||
### Transaction messages exempt from aggregate
|
||||
|
||||
`new_transaction`, `request_transaction`, `respond_transaction`,
|
||||
`send_transaction`, `transaction_ack` — these have their own per-type limits
|
||||
and do NOT count against the aggregate.
|
||||
|
||||
### Key rate limits (v1)
|
||||
|
||||
| Message | Freq/min | Size | Total/min |
|
||||
| ------------------------- | -------- | ------ | --------- |
|
||||
| `new_transaction` | 5000 | 100 B | 500 KB |
|
||||
| `respond_transaction` | 5000 | 1 MB | 20 MB |
|
||||
| `send_transaction` | 5000 | 1 MB | — |
|
||||
| `respond_blocks` | 100 | 50 MB | — |
|
||||
| `respond_proof_of_weight` | 5 | 400 MB | — |
|
||||
| `new_peak` | 200 | 512 B | — |
|
||||
| `request_block` | 200 | 100 B | — |
|
||||
| `request_blocks` | 100 | 100 B | — |
|
||||
|
||||
### V2 rate limits
|
||||
|
||||
Activated when both peers have `Capability.RATE_LIMITS_V2`. Overrides/extends
|
||||
v1 with additional message types (wallet sync, mempool updates, etc.).
|
||||
|
||||
### `Unlimited` message types
|
||||
|
||||
Some response messages use `Unlimited` instead of `RLSettings` — they have a
|
||||
per-message size limit but no frequency limit and are exempt from aggregate.
|
||||
|
||||
---
|
||||
|
||||
## Protocol state machine
|
||||
|
||||
**Location**: `protocols/protocol_state_machine.py`
|
||||
|
||||
### `VALID_REPLY_MESSAGE_MAP`
|
||||
|
||||
Maps request types to valid response types. Examples:
|
||||
|
||||
- `request_block` → `[respond_block, reject_block]`
|
||||
- `request_blocks` → `[respond_blocks, reject_blocks]`
|
||||
- `send_transaction` → `[transaction_ack]`
|
||||
- `request_puzzle_state` → `[respond_puzzle_state, reject_puzzle_state]`
|
||||
|
||||
### `NO_REPLY_EXPECTED`
|
||||
|
||||
Fire-and-forget messages: `new_peak`, `new_transaction`,
|
||||
`new_unfinished_block`, `new_signage_point_or_end_of_sub_slot`,
|
||||
`request_mempool_transactions`, `new_compact_vdf`, `coin_state_update`,
|
||||
`mempool_items_added`, `mempool_items_removed`.
|
||||
|
||||
### Import-time check
|
||||
|
||||
`static_check_sent_message_response()` verifies NO_REPLY_EXPECTED and
|
||||
VALID_REPLY_MESSAGE_MAP don't overlap. Runs at module import.
|
||||
|
||||
---
|
||||
|
||||
## `ApiProtocol` and `@request` decorator
|
||||
|
||||
**Location**: `server/api_protocol.py`
|
||||
|
||||
### Handler registration
|
||||
|
||||
Each API class (e.g., `FullNodeAPI`) has a class-level `ApiMetadata` that maps
|
||||
`ProtocolMessageTypes` → `ApiRequest`.
|
||||
|
||||
The `@metadata.request()` decorator:
|
||||
|
||||
- Registers the handler for its message type
|
||||
- Auto-deserializes `bytes` → `Streamable` subclass
|
||||
- Optionally passes raw bytes and/or peer reference
|
||||
- `execute_task=True` means handler runs as a separate asyncio task
|
||||
|
||||
### Key flags
|
||||
|
||||
- `peer_required=True` — handler receives `WSChiaConnection` as parameter
|
||||
- `bytes_required=True` — handler receives raw bytes (for forwarding)
|
||||
- `execute_task=True` — non-blocking execution
|
||||
|
||||
---
|
||||
|
||||
## Connection limits
|
||||
|
||||
**Location**: `server/chia_policy.py`
|
||||
|
||||
- Default `global_max_concurrent_connections = 250`
|
||||
- `set_chia_policy(connection_limit)` sets effective limit to `connection_limit + 100`
|
||||
- Custom event loop policy (`ChiaProactorEventLoop` on Windows,
|
||||
selector-based on Unix) enforces connection limits at the socket level
|
||||
|
||||
---
|
||||
|
||||
## Peer discovery
|
||||
|
||||
**Location**: `server/node_discovery.py`
|
||||
|
||||
### `FullNodePeers`
|
||||
|
||||
- Maintains address book of known peers
|
||||
- Periodic peer exchange via `request_peers` / `respond_peers`
|
||||
- Connects to introducer nodes for bootstrapping
|
||||
- DNS seeder support
|
||||
- Preference for outbound connections to maintain network topology
|
||||
|
||||
---
|
||||
|
||||
## Message type → node type mapping
|
||||
|
||||
**Location**: `protocols/protocol_message_type_to_node_type.py`
|
||||
|
||||
Maps each `ProtocolMessageTypes` to the set of `NodeType`s allowed to send it.
|
||||
Used to reject messages from unexpected node types (e.g., a wallet trying to
|
||||
send `new_peak` which is a full-node-only message).
|
||||
|
||||
---
|
||||
|
||||
## `Handshake` protocol
|
||||
|
||||
**Location**: `protocols/shared_protocol.py`
|
||||
|
||||
### Fields
|
||||
|
||||
- `network_id: str`
|
||||
- `protocol_version: str`
|
||||
- `software_version: str`
|
||||
- `server_port: uint16`
|
||||
- `node_type: NodeType`
|
||||
- `capabilities: list[tuple[uint16, str]]`
|
||||
|
||||
### `Capability` enum
|
||||
|
||||
Key capabilities: `BASE`, `BLOCK_HEADERS`, `RATE_LIMITS_V2`,
|
||||
`NONE_RESPONSE`, `MEMPOOL_UPDATES`. Used for feature negotiation.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Chia Plot Sync Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/plot_sync/` is the farmer-harvester inventory replication protocol. The harvester-side `Sender` converts `PlotManager` refresh callbacks into an ordered stream of plot-sync messages; the farmer-side `Receiver` validates that stream, accumulates a delta, commits it only at `plot_sync_done`, and exposes the farmer/UI view of each harvester's plots.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for harvester-to-farmer plot inventory replication, plot-sync ACK/retry behavior, farmer-side plot list state, and UI/RPC plot-count correctness. For local plot discovery/load rules, read `plotting.md`; for farmer proof handling, read `farmer.md`.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `Sender` is not the plot-state authority. It snapshots and serializes `PlotManager` state from the harvester and depends on refresh lifecycle callbacks for loaded, removed, invalid, key-missing, and duplicate plot lists.
|
||||
- `Receiver` is the farmer-local authority for what the farmer believes a connected harvester has. Farmer RPC, pool plot counts, and UI state read from `Farmer.plot_sync_receivers`, not directly from the remote harvester.
|
||||
- Protocol payloads live in `chia/protocols/harvester_protocol.py`; `chia/plot_sync/` owns sequencing, delta semantics, and ACK/error handling. Schema edits are network compatibility changes and must move with protocol message IDs, node-type sender maps, rate limits, API decorators, and tests.
|
||||
- The farmer and harvester are operator-controlled peers, but plot-sync payloads still cross a network boundary. The receiver validates ordering, duplicate additions, removals of missing plots, `last_sync_id`, and sync/message identifiers before mutating committed state.
|
||||
|
||||
## Why This Is Tricky
|
||||
|
||||
Public farmer/RPC docs expose harvester plot lists as if they are current inventory, but the implementation is a replicated state machine with recovery. Mid-sync messages are progress, not committed truth. Reset is intentionally full-state reconciliation from the harvester's current `PlotManager` snapshot, so making deltas more incremental can reduce correctness if it weakens convergence after dropped ACKs, disconnects, or refresh races.
|
||||
|
||||
## Wrong Assumptions To Avoid
|
||||
|
||||
- Do not treat plot sync as plot validation; it is farmer-local inventory replication.
|
||||
- Do not commit receiver state before `plot_sync_done`.
|
||||
- Do not treat ACKs as protocol request/reply matching; they are application-level plot-sync messages.
|
||||
- Do not assume a correct harvester `PlotManager` snapshot means the farmer/UI view has converged.
|
||||
|
||||
## Runtime Flow
|
||||
|
||||
- Farmer connection creates one `Receiver` per harvester peer in `Farmer.on_connect()`, then sends `harvester_handshake`.
|
||||
- Harvester `harvester_handshake` installs farmer/pool keys in `PlotManager`, binds `Sender` to the farmer `WSChiaConnection`, starts the sender task, and then starts plot refreshing. This ordering matters: refresh before handshake can load plots without the farmer-owned key filter.
|
||||
- A refresh cycle is encoded as `plot_sync_start`, one or more `plot_sync_loaded` batches, `plot_sync_removed`, `plot_sync_invalid`, `plot_sync_keys_missing`, `plot_sync_duplicates`, then `plot_sync_done`.
|
||||
- Each inbound sync message is acknowledged by the farmer with `plot_sync_response`. This is application-level ACK behavior from `Receiver._process()`, not a `protocol_state_machine` request/reply pair.
|
||||
- On disconnect, the farmer deletes the peer's `Receiver`; the harvester stops sender/refresh state and awaits sender closure. Reconnect creates a new receiver and must re-run handshake.
|
||||
|
||||
## Sender State Machine
|
||||
|
||||
- `Sender` keeps `_sync_id`, `_next_message_id`, `_last_sync_id`, pending `MessageGenerator`s, and one expected response. It sends exactly one in-flight plot-sync message and advances only after a matching `PlotSyncResponse`.
|
||||
- `sync_start()` waits while a sync is active, then chooses a time-based sync id and increments it if needed to avoid same-second reuse in tests. `last_sync_id` is included so the receiver can reject missed or reordered sync cycles.
|
||||
- Loaded plots are batched using the plot manager refresh batch size. Empty list phases still emit a final marker, which is how the receiver advances through every state even when a category is empty.
|
||||
- `_reset()` clears local sender progress. If the sender task is running, it rebuilds a full initial-style inventory from the current `PlotManager` snapshot so recovery converges on the harvester's present state rather than replaying stale deltas.
|
||||
- Response matching rejects unexpected, expired, wrong-sync, wrong-message, or wrong-message-type ACKs. Recoverable receiver errors with `expected_identifier` can rewind `_next_message_id`; unrecoverable errors reset the sender.
|
||||
|
||||
## Receiver State Machine
|
||||
|
||||
- `Receiver` starts idle and moves strictly through `loaded -> removed -> invalid -> keys_missing -> duplicates -> done`. Each message must match the current `sync_id` and `next_message_id`; `plot_sync_start` is allowed to establish the new sync id but still must have the expected message id.
|
||||
- `initial=True` resets all committed receiver state before processing. Non-initial starts must carry `last_sync_id == receiver.last_sync().sync_id`.
|
||||
- During a sync, changes live only in `current_sync.delta`. Valid plot additions, valid removals, invalid filenames, key-missing filenames, and duplicates are accumulated separately.
|
||||
- `plot_sync_done` computes path-list deltas for invalid/key-missing/duplicate categories, applies the valid additions/removals to `_plots`, replaces the side lists with the new synced lists, recalculates total and effective plot sizes, stores the completed sync as `_last_sync`, and returns to idle.
|
||||
- Progress callbacks with `delta=None` are emitted during batched phases. The committed update callback receives a non-`None` `Delta` only after `done`; farmer state-change fanout suppresses empty non-initial deltas.
|
||||
|
||||
## Data Contracts
|
||||
|
||||
- `Plot.filename` is the dictionary key on the farmer side. Path normalization changes affect duplicate detection, removals, RPC sorting/filtering, and signature/proof correlation elsewhere in farmer-harvester code.
|
||||
- `Plot.size` is compatibility-packed: v1 plots use k-size; v2 plots set the high bit and store strength in the low bits. `Plot.param()` reconstructs `PlotParam` from the source-owned v2 plot index/meta group policy.
|
||||
- Effective plot size is recomputed by the receiver with `_expected_plot_size(plot.param(), constants)` and `UI_ACTUAL_SPACE_CONSTANT_FACTOR`, so `Receiver` needs consensus constants even though plot sync itself is not consensus validation.
|
||||
- `Delta` is a reporting/commit artifact, not a standalone operation log. For path lists, removals are computed by comparing the previous committed list against the new synced list at `done`.
|
||||
|
||||
## Failure And Recovery
|
||||
|
||||
- Message timeout is defined in `Constants`. The sender polls for responses and sleeps for the timeout interval after send failures/timeouts before retrying.
|
||||
- Receiver exceptions are converted into `PlotSyncError` responses. `InvalidIdentifierError` includes an `expected_identifier` so the sender can resend from the receiver's expected point.
|
||||
- The sender has one special recovery case for a missed final ACK: if the receiver reports expected sync/message ids of zero after the sender already sent the final `plot_sync_done`, the sender finalizes locally.
|
||||
- Non-recoverable receiver errors, lost connection, or invalid message generators force sender reset. Because reset rebuilds from current plot manager state, recovery is eventually full-state reconciliation rather than minimal delta preservation.
|
||||
|
||||
## Concurrency And Lifecycle
|
||||
|
||||
- `Sender._run()` is an async task, but `sync_start()` may be called from the plot refresh thread and uses blocking `time.sleep()` while waiting for an active sync to finish. Avoid adding slow work or event-loop-only assumptions to refresh callback paths.
|
||||
- `Sender._messages` is appended by refresh callbacks and consumed by the async sender task without an explicit lock. The current design relies on simple append/index behavior and one active sync.
|
||||
- `Receiver` has no broad lock. Farmer handlers, callbacks, and RPC reads rely on asyncio sequencing and short mutation windows. Multi-step receiver changes must preserve callback ordering and committed-vs-in-progress separation.
|
||||
- Harvester shutdown awaits sender closure after stopping plot refreshing and resetting `PlotManager`; receiver shutdown is implicit in farmer disconnect handling.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Do not treat plot-sync ACKs as server-level replies unless the protocol state machine is deliberately changed. Today they are normal messages with node-type authorization: plot-sync data from harvesters, responses from farmers.
|
||||
- Do not commit receiver state before `plot_sync_done`. Mid-sync state is progress-only; committing early can make farmer RPC/UI observe partial refreshes and makes recovery from missing later phases inconsistent.
|
||||
- Preserve the exact phase order, final markers, and message id increments. Empty categories are still protocol states.
|
||||
- Be careful changing reset behavior. It is the main convergence mechanism after dropped ACKs, late responses, unrecoverable errors, and reconnect-like cases.
|
||||
- Edits to `Plot` encoding, especially the `size` high bit, affect farmer UI effective space, pool plot counts, and third-party harvester compatibility.
|
||||
- `set_connection()` expects a farmer connection; changes here should be checked against harvester handshake order and tests. The exception's expected-node wording is easy to regress because it is primarily tested by behavior, not user-facing text.
|
||||
|
||||
## Test And Audit Strategy
|
||||
|
||||
- Unit-test sender start/stop, connection type validation, response matching, timeout/expired ACK handling, negative duration clamping, reset, and recoverable/unrecoverable error paths.
|
||||
- Unit-test receiver phase transitions, invalid identifiers, `last_sync_id`, duplicate additions, removals of missing plots, counts-only serialization, committed totals, and effective plot-size calculation.
|
||||
- Integration-test real farmer-harvester service sync across multiple harvesters, repeated refreshes, valid/invalid/key-missing/duplicate transitions, removals, recovered invalid plots, reconnect-like starts, dropped/late/duplicate ACKs, and not-connected resets.
|
||||
- For protocol edits, include sender-map/rate-limit/API decorator coverage and farmer-harvester compatibility tests; serialization-only tests are insufficient.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Harvester sender state machine: `chia/plot_sync/sender.py`.
|
||||
- Farmer receiver state machine: `chia/plot_sync/receiver.py`.
|
||||
- Plot-sync payloads: `chia/protocols/harvester_protocol.py`.
|
||||
- Farmer receiver ownership and RPC consumers: `chia/farmer/farmer.py`, `chia/farmer/farmer_rpc_api.py`.
|
||||
- Harvester refresh callbacks that feed sync: `chia/harvester/harvester.py`, `chia/plotting/manager.py`.
|
||||
@@ -0,0 +1,111 @@
|
||||
# Chia Plotting Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/plotting/` is the shared plot lifecycle boundary. It creates plot files, opens and abstracts V1/V2 provers, scans configured harvester plot directories, persists a prover cache, parses plot memos into farming keys, and supplies the lock-protected plot inventory consumed by the harvester, plot sync, `chia plots check`, simulator block tools, and external plotter wrappers.
|
||||
|
||||
This module does not own consensus validity, farmer pooling state, or network protocol handling. It prepares local plot state and proof/prover primitives so those higher layers can make fork-, difficulty-, and protocol-aware decisions.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for plot creation, plot IDs, memo parsing, V1/V2 prover abstraction, plot cache/refresh behavior, plot directory config helpers, and diagnostics such as `chia plots check`. For live signage-point handling, read `harvester.md`; for pool singleton state, read `pools.md`.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `create_plots.py` owns in-process chiapos V1 plot creation and experimental Rust-backed V2 plot creation. It resolves farmer/pool keys from explicit CLI values or the keychain, writes plot memo bytes, computes plot IDs, and calls `DiskPlotter` or `create_v2_plot()`.
|
||||
- `PlotManager` owns local plot discovery and the canonical in-memory `Path -> PlotInfo` map. It runs a background refresh thread, batches directory scans, filters unusable plots, tracks duplicate/no-key/failed-open sets, and updates the on-disk cache.
|
||||
- `Cache` owns serialized prover metadata under `cache/plot_manager_v2.dat`. It is an optimization, not the authority for file existence or key ownership; refresh revalidates cache entries against the filesystem, keys, compression policy, and size heuristics.
|
||||
- `prover.py` is the compatibility layer over external proof libraries. It hides V1 `chiapos.DiskProver` and V2 `chia_rs.Prover` behind `ProverProtocol`, preserving common calls for plot ID, memo, parameter, compression level, serialized prover data, and quality lookup.
|
||||
- `util.py` owns config-facing plot scan helpers, refresh event/result shapes, memo parsing/streaming, duplicate filename checks, harvester config mutation helpers, and plot-size CLI validation.
|
||||
- `check_plots.py` is an operator diagnostic flow. It uses `PlotManager` loading semantics, then stress-tests quality lookup and proof validation. It should not be treated as the production harvesting path.
|
||||
|
||||
## Why This Is Tricky
|
||||
|
||||
Public plotting docs frame plots as files tied to a farmer key, pool key, or pool contract address. In source, those choices become long-lived binary contracts: plot IDs seed proof generation, memos recover local secrets for later signatures, and portable plots route rewards through singleton-controlled puzzle hashes. A plotting change can therefore break harvesting, farmer aggregate signatures, pool portability, external plotters, simulator fixtures, or diagnostics even if plot creation itself still succeeds.
|
||||
|
||||
## Wrong Assumptions To Avoid
|
||||
|
||||
- Do not treat the plot cache as authoritative; it accelerates refresh but file/key/prover checks still decide loadability.
|
||||
- Do not treat plot ID, memo, and plot public key as independent fields; they are coupled through creation, signature derivation, and proof validation.
|
||||
- Do not infer production harvesting behavior from `chia plots check`; diagnostics intentionally open and inspect cases normal harvesting may reject.
|
||||
- Do not make V2 support look like compressed V1 support; the prover contract and farmer/solver flow are different.
|
||||
|
||||
## Plot Identity And Memo Contracts
|
||||
|
||||
- Every loaded plot must have exactly one pool target form: old-style `pool_public_key` or pool-contract `pool_contract_puzzle_hash`. Creation asserts this, memo parsing returns either a `G1Element` or `bytes32`, and later farmer/harvester signature logic branches on that type.
|
||||
- V1 plot memos have two exact source-defined layouts for public-key and pool-contract plots. `parse_plot_info()` rejects any other length. This memo is the source of the local plot secret used by harvester signature responses.
|
||||
- The plot public key is derived as `local_pk + farmer_pk` for pool-public-key plots, and `local_pk + farmer_pk + taproot_pk` for pool-contract plots. This must stay aligned with `chia.types.blockchain_format.proof_of_space.generate_plot_public_key()` and farmer aggregate-signature verification.
|
||||
- V1 plot IDs are computed from pool public key or pool contract puzzle hash plus plot public key. V2 plot IDs are computed by `compute_plot_id_v2()` from strength, plot public key, pool key/hash, plot index, and meta group.
|
||||
- Current V2 creation hardcodes placeholder plot-index and meta-group values; `V2Prover.get_param()` mirrors this placeholder. Any future multi-index/group support must update creation, prover metadata, proof construction, farmer/harvester protocol payloads, and tests together.
|
||||
|
||||
## Discovery And Refresh Model
|
||||
|
||||
- Plot discovery reads configured harvester directories via `get_plot_directories()` and scans `*.plot` plus `*.plot2`. Recursive scan and symlink following are config driven; path resolution failures and unreadable directories are logged and skipped.
|
||||
- `PlotManager.start_refreshing()` loads the cache and starts a thread that periodically scans directories when `needs_refresh()` is true. `stop_refreshing()` joins that thread; `reset()` clears loaded plots and error state.
|
||||
- Refresh emits `started`, `batch_processed`, and `done` callbacks. Harvester converts those callbacks into plot-sync messages, so callback ordering and result semantics are observable by farmer/UI state.
|
||||
- `PlotManager.plots` is protected by the manager's lock. Harvester code snapshots or reads it under `with plot_manager:` and then performs expensive proof work outside the lock. Do not add disk proof reads, RPC calls, or long logging loops while holding this lock.
|
||||
- Batch refresh work opens files concurrently with a `ThreadPoolExecutor`, but result mutation is centralized through locks and the final `self.plots.update()` under the manager lock.
|
||||
|
||||
## Load Filters And Quarantine State
|
||||
|
||||
- A plot is not loaded if it fails extension-specific prover opening, has missing farmer/pool keys, duplicates an already-loaded filename, exceeds `max_compression_level_allowed`, is compressed while no parallel decompressor is configured, or appears too small under source-defined V1 plot-size policy.
|
||||
- `failed_to_open_filenames` stores retry timestamps and suppresses repeated open attempts until `retry_invalid_seconds` elapses. Removed or renamed files are dropped from this set on refresh.
|
||||
- `no_key_filenames` records plots whose farmer public key or pool public key is not currently available. `open_no_key_filenames=True` lets diagnostics open them while preserving the warning state.
|
||||
- Duplicate detection in `PlotManager` is filename-based across directories, not plot-ID validation. `find_duplicate_plot_IDs()` is a separate operator diagnostic that parses filename suffixes and is intentionally lightweight.
|
||||
- Compressed plot handling crosses the `chiapos.decompressor_context_queue` boundary. GPU initialization may fall back to CPU harvesting; runtime decompressor failures may surface as stringly `RuntimeError`s from lower libraries.
|
||||
|
||||
## Cache And Persistence
|
||||
|
||||
- Cache serialization uses `VersionedBlob(uint16(CURRENT_VERSION), bytes(CacheDataV1(...)))` with streamable entries. Field changes are persistence changes and must account for older cache files.
|
||||
- Cache entries store serialized prover data, farmer/pool identifiers, derived plot public key, and last-use time. They intentionally do not remove the need to check path existence, file size, compression policy, and key ownership during refresh.
|
||||
- Cache cleanup removes expired unused entries after refresh and bumps last-use for loaded plots. The retention lifetime belongs in source policy.
|
||||
- `Cache.load()` has a compatibility guard for suspicious oversized V1 prover data from older bladebit/chiapos behavior. Removing it can resurrect bad cache entries for users who have not manually deleted `plot_manager_v2.dat`.
|
||||
- `plot_manager_v2.dat` was introduced to avoid downgrading/upgrading cache shape collisions after compression-level metadata was added. Do not casually rename it or collapse it with older cache files.
|
||||
|
||||
## V1/V2 Prover Contract
|
||||
|
||||
- V1 uses `chiapos.DiskProver`; full proofs are retrieved locally with `get_full_proof(challenge, index, parallel_read)`, and proof validation uses `chiapos.Verifier`.
|
||||
- V2 uses `chia_rs.Prover`; quality lookup returns `PartialProof` wrappers, and full proof solving is done by `chia_rs.solve_proof()` in diagnostics or by solver services in the farmer flow.
|
||||
- `get_prover_from_file()` and `get_prover_from_bytes()` dispatch solely by filename suffix: `.plot2` means V2, `.plot` means V1. Unsupported extensions raise `ValueError`.
|
||||
- `V1Prover.get_strength()` intentionally raises because strength is V2-only. Code that needs plot parameters should use `get_param()` and branch on `size_v1` vs `strength_v2`.
|
||||
- `V2Prover.get_compression_level()` returns zero because V2 plots are not treated as compressed V1 plots. Do not reuse V1 compression policy as a V2 strength policy.
|
||||
|
||||
## Harvester And Farmer Coupling
|
||||
|
||||
- Harvester startup constructs `PlotManager`, configures decompressor state, and starts refreshing only after farmer handshake installs farmer and pool public keys. Starting refresh before handshake can load plots against empty key lists and produce wrong inventory.
|
||||
- Harvester signage-point handling reads `PlotInfo` from `PlotManager`, applies consensus plot filters and fork gates, then uses prover methods to find V1 proofs or V2 partial proofs. Plotting must preserve the metadata needed for `make_pos()` and solver requests.
|
||||
- Harvester signature responses parse the plot memo and derive the local key on demand. Memo format, plot public key generation, and resolved filename identity are therefore part of the farmer-harvester signature contract.
|
||||
- Plot sync depends on `PlotRefreshResult.loaded`, `removed`, `failed_to_open_filenames`, `no_key_filenames`, and duplicate state. A local refresh change can become a farmer/UI compatibility change through `chia.plot_sync`.
|
||||
- Farmer reconstructs and validates proofs with `verify_and_get_quality_string()`. Plotting-side changes to plot ID, parameter, strength, memo, or key derivation must be checked against consensus proof validation and farmer aggregate-signature construction.
|
||||
|
||||
## CLI, External Plotters, And Simulator Coupling
|
||||
|
||||
- `chia plots create` and `chia.plotters.chiapos` both call `resolve_plot_keys()` and `create_plots()`; external plotter wrappers for bladebit and madmax call `resolve_plot_keys()` and pass the resulting keys to subprocess CLIs.
|
||||
- Plot size validation lives in `util.validate_plot_size()` and is CLI/operator policy: mainnet min k comes from config, and `--override-k` still applies source-defined lower bounds.
|
||||
- `chia plots check` deliberately opens no-key plots, can list duplicate filename-style plot IDs, and can reveal raw memo bytes. Treat it as an admin/debug surface with more permissive loading than normal harvesting.
|
||||
- Simulator `BlockTools` uses `create_plots()`, `create_v2_plots()`, and `PlotManager` to generate deterministic local plots for tests and block generation. Test-only private keys and debug plot IDs/memos in creation should not leak into production assumptions.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Do not change memo lengths, field order, or key derivation without updating create, cache, harvester signature handling, farmer aggregate verification, external plotter arguments, and tests.
|
||||
- Do not hold the `PlotManager` lock across expensive prover I/O. The signage-point path is latency-sensitive; slow lookups risk missed rewards.
|
||||
- Be careful with V1/V2 fork semantics. Plotting exposes parameters and proof material, but consensus helpers decide V1 phase-out, V2 activation, plot filter prefix bits, and proof-size rejection.
|
||||
- Filename and path identity matter. Duplicate tracking is by filename, cache keys are `Path`s, harvester signature lookup resolves paths from `plot_identifier`, and refresh maps prover filenames back into `self.plots`.
|
||||
- Cache data can be stale, malformed, or from older external plotters. Refresh must stay robust to bad serialized prover data and treat cache failures as recoverable operator issues.
|
||||
- External libraries (`chiapos`, `chia_rs`) define much of the runtime behavior and exception surface. Tests should not assume all invalid plots or decompressor failures raise uniform Python exceptions.
|
||||
|
||||
## Test And Audit Strategy
|
||||
|
||||
- For creation/key changes, cover pool-public-key and pool-contract plots, explicit keys vs keychain-derived keys, memo bytes, plot ID calculation, and taproot inclusion.
|
||||
- For refresh changes, cover added/removed files, invalid retry timing, no-key handling, duplicate filenames, recursive scan/symlink settings, cache load/save/expiry, suspicious cache entry dropping, and compression/decompressor policy.
|
||||
- For prover changes, test suffix dispatch, V1 wrapper compatibility, V2 parameter/quality behavior, bytes round trips where supported, and unsupported extension failures.
|
||||
- For harvester-facing changes, test plot sync convergence and signage-point lookup separately. A correct `PlotManager.plots` result is not enough if farmer-side plot sync or signature routing would diverge.
|
||||
- For diagnostics, test `check_plots()` behavior with V1 and V2 paths, low challenge counts, duplicate listing, memo display, and proof validation failures without relying on production harvesting side effects.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Plot creation and key resolution: `chia/plotting/create_plots.py`, `chia/plotting/util.py`.
|
||||
- Plot discovery/cache: `chia/plotting/manager.py`, `chia/plotting/cache.py`.
|
||||
- Prover abstraction: `chia/plotting/prover.py`.
|
||||
- Operator diagnostics: `chia/plotting/check_plots.py`.
|
||||
- Consensus proof helpers: `chia/types/blockchain_format/proof_of_space.py`.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Chia Pools Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/pools/` is the plot-NFT/pooling singleton adapter. It bridges wallet-owned singleton state, CLVM pool puzzles, farmer-facing pooling configuration, consensus reward coin shape, and the external pool HTTP protocol. The module is small, but it sits on several trust boundaries: wallet sync reconstructs state from on-chain coin spends, the farmer reads a YAML mirror rather than wallet DB state, and reward claiming depends on exact consensus coin IDs and puzzle hashes.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for plot NFT / pool singleton state, pool wallet transitions, farmer-facing pool config, pool reward claims, pool protocol payloads, and portable-plot ownership. For wallet action scopes and sync, also read `wallet.md`; for farmer partial submission, read `farmer.md`; for plot IDs and memos, read `plotting.md`.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- The blockchain is the authority for pool singleton history. `PoolWallet.get_current_state()` derives the current and last state-changing `PoolState` from `WalletPoolStore` coin spends, not from the YAML config or transient target fields.
|
||||
- `PoolWallet` is the wallet-facing state machine. It creates the launcher spend, joins/leaves pools, submits the second leave transaction after the relative lock height, claims self-pooled rewards, and updates the farmer-facing config mirror.
|
||||
- `pool_puzzles.py` is the CLVM boundary. It curries precompiled pool puzzle modules from `chia-puzzles-py`, constructs singleton and p2-singleton puzzles, builds travel/absorb `CoinSpend`s, and parses pool state out of launcher/travel solutions.
|
||||
- `pool_wallet_info.py` defines streamable state contracts used by wallet RPC and by on-chain pool-state serialization. `PoolState` bytes are committed into launcher/travel solution extra data.
|
||||
- `pool_config.py` is a side-channel to the farmer. `<chia root>/pooling/pooling_share_state.yaml`, guarded by `PoolingShareState.lock()`, mirrors launcher id, pool URL, payout instructions, target puzzle hash, p2-singleton puzzle hash, owner public key, and derivation index for farmer startup/update loops.
|
||||
- The farmer and pool server are not authorities for singleton state. Farmer `pool_state` and remote pool `/pool_info`, `/farmer`, and `/partial` responses affect difficulty, payout registration, and partial accounting, but do not prove or mutate the wallet's on-chain state.
|
||||
|
||||
## Why This Is Tricky
|
||||
|
||||
Public pooling docs describe portable plots as being tied to a Plot NFT that can change pools. In source, that product goal is split across three authority layers: wallet singleton spends define the durable pool state, plot IDs bind plots to the singleton-derived puzzle hash, and the farmer YAML mirror tells the farmer where to submit partials. Pool servers smooth rewards through partial accounting, but they do not choose blocks or prove the wallet's singleton state.
|
||||
|
||||
## Wrong Assumptions To Avoid
|
||||
|
||||
- Do not treat farmer YAML as wallet truth; it is a mirror derived from wallet state transitions.
|
||||
- Do not treat a pool server response as validation of singleton lineage or payout authority.
|
||||
- Do not treat `target_state` as durable; it is pending local intent until matching chain state is observed.
|
||||
- Do not change pool puzzle tree hashes or singleton parent-info shape as a wallet-only refactor.
|
||||
|
||||
## Core State Model
|
||||
|
||||
- `PoolState.state` has the live self-pooling, leaving-pool, and farming-to-pool modes. Self-pooling uses the waiting-room inner puzzle with the immediate-lock sentinel and no pool URL; farming-to-pool uses the member inner puzzle; leaving-pool uses the waiting-room inner puzzle with the pool's lock height.
|
||||
- `target_puzzle_hash` means final payout destination, not the p2-singleton address where pool rewards are initially farmed. In self-pooling it is a local wallet puzzle hash; in farming-to-pool it is the pool's target puzzle hash.
|
||||
- `launcher_id` is the singleton identity. `launcher_id_to_p2_puzzle_hash()` derives the pool-contract puzzle hash used by plots and reward coins from the singleton mod hash, launcher id, launcher puzzle hash, delayed escape parameters, and delayed puzzle hash.
|
||||
- Pool singleton coins use the singleton amount invariant and are recognized by singleton lineage. Reward coins are separate pool reward outputs to the p2-singleton puzzle hash and are absorbed into the singleton chain when claimed.
|
||||
- `PoolWalletInfo.target` is transient local intent, not chain state. It is cleared when a matching on-chain transition is observed; rollback may also rewrite config if the derived current state changes.
|
||||
|
||||
## Lifecycle Flows
|
||||
|
||||
- Creating a pool wallet selects standard-wallet coins, creates a singleton launcher coin, asserts the launcher announcement from the standard-wallet spend, stages the launcher spend as an extra spend, and returns the p2-singleton puzzle hash plus launcher id. The actual `PoolWallet` object is created later when wallet sync sees and spends the launcher on chain.
|
||||
- Wallet sync discovers pool wallets by detecting a singleton launcher child, fetching the launcher spend, parsing a `PoolState` from its solution extra data, creating `PoolWallet`, recording the launcher spend in `WalletPoolStore`, tracking the first singleton coin, and subscribing to its coin id.
|
||||
- Subsequent singleton transitions are applied when a tracked singleton coin is spent. `apply_state_transition()` only accepts spends whose coin name matches the current tip coin, appends the spend, follows the new singleton coin, subscribes to its coin id, and updates the YAML mirror.
|
||||
- Switching from `FARMING_TO_POOL` to another final state is a two-transaction flow: first travel to `LEAVING_POOL`, then after `last_transition_height + relative_lock_height` plus a small reorg buffer, `new_peak()` submits the final travel to `SELF_POOLING` or a new pool.
|
||||
- Switching from `SELF_POOLING` or mature `LEAVING_POOL` to `FARMING_TO_POOL` is one travel transaction. Switching directly between pools charges for two transitions because it first enters `LEAVING_POOL`.
|
||||
- Claiming self-pooled rewards scans this pool wallet's unspent reward coins, filters to known farming rewards, builds repeated absorb spends that advance the singleton tip while consuming p2-singleton reward coins, optionally adds a standard-wallet fee spend tied by a coin announcement, and records an outgoing transaction paying the absorbed amount to the current target puzzle hash.
|
||||
|
||||
## CLVM And Consensus Coupling
|
||||
|
||||
- Pool puzzle construction must match the precompiled modules exactly: `POOL_MEMBER_INNERPUZ`, `POOL_WAITINGROOM_INNERPUZ`, `P2_SINGLETON_OR_DELAYED_PUZHASH`, and the wallet singleton top layer. Tree hashes are part of live plot contracts and reward addresses.
|
||||
- `pool_state_to_inner_puzzle()` maps `SELF_POOLING` and `LEAVING_POOL` to the waiting-room puzzle, and `FARMING_TO_POOL` to the member puzzle whose escape puzzle hash is the corresponding waiting-room puzzle hash.
|
||||
- `create_travel_spend()` and `create_absorb_spend()` reconstruct singleton parent info differently for eve spends versus later spends. Parent-info shape must stay aligned with singleton top-layer expectations.
|
||||
- `solution_to_pool_state()` distinguishes launcher spends, member travel spends, waiting-room travel spends, and absorb spends by positional CLVM solution shape. Parser changes are high risk because malformed or unexpected spends can make wallet state reconstruction skip or misclassify transitions.
|
||||
- Absorb spends reconstruct reward coins using `pool_parent_id(height, genesis_challenge)` and `calculate_pool_reward(height)`. This is intentionally tied to consensus coinbase rules; changing reward schedule or parent-id logic without updating absorb tests can strand claimable rewards.
|
||||
- `PoolWallet.claim_pool_rewards()` maps wallet farming reward transaction records back to block heights, then uses those heights to build reward coin spends. If reward detection or transaction-record height derivation changes in wallet code, pool reward claims can break even when pool puzzles are untouched.
|
||||
|
||||
## Persistence And Configuration
|
||||
|
||||
- `WalletPoolStore` stores ordered `(height, CoinSpend)` transitions per wallet id in SQLite. New transitions must not go backward in height and must extend the previous spend's coin name through parent linkage; duplicate identical spends are ignored.
|
||||
- Rollback deletes pool transitions above the rollback height. If the launcher spend itself rolls back, `PoolWallet.rewind()` asks `WalletStateManager` to remove the whole pool wallet; otherwise it refreshes the YAML mirror when derived state changed.
|
||||
- `PoolingShareState.acquire()` rewrites the entire YAML list on context exit unless `read_only=True`. Callers that only inspect state should use read-only acquisition to avoid unnecessary file churn and accidental persistence of mutated objects.
|
||||
- `PoolingShareState.add()` is keyed by `p2_singleton_puzzle_hash`, not launcher id. Duplicate p2-singleton entries are rejected, and migration from old `config.yaml["pool"]["pool_list"]` clears the old list after copying missing entries.
|
||||
- `PoolWallet.update_pool_config()` creates missing YAML entries and preserves existing payout instructions when present. Empty payout instructions are filled with a newly derived puzzle hash, so config updates can consume derivation state through the action scope.
|
||||
|
||||
## External Pool Protocol Boundary
|
||||
|
||||
- `POOL_PROTOCOL_VERSION` gates wallet-created states: a state with a newer version is rejected and requires upgrading the wallet.
|
||||
- Farmer pool registration signs `PostFarmerPayload` / `PutFarmerPayload` with the owner key and derives/uses an authentication key for `AuthenticationPayload` and partial submissions. The wallet stores only owner public key and payout/config data; private key lookup stays in farmer/keychain code.
|
||||
- Mainnet farmer update enforces HTTPS pool URLs. Self-pooling is represented to the farmer by `pool_url == ""`, which skips remote pool updates and partial submission while keeping local plot-NFT accounting.
|
||||
- Pool servers can redirect `/pool_info` via permanent redirects, update difficulty, request payout/auth updates, or reject partials. These responses affect farmer runtime state but should not be treated as validation of `PoolState` or singleton lineage.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Do not use `target_state` as durable truth. It is local pending intent and is reset after matching on-chain state or rollback; the spend history is the durable source.
|
||||
- Unconfirmed transaction gates are intentional. Join/self-pool/claim operations reject when this pool wallet has unconfirmed transactions to avoid overlapping singleton-tip spends.
|
||||
- The leave flow depends on wallet `new_peak()` being called and on `finished_sync_up_to()` height comparisons. Changes to wallet sync height semantics can delay or prematurely submit second-stage travel transactions.
|
||||
- Assertions are common around CLVM shape, singleton additions, and reward heights. Some indicate internal invariants rather than user input validation; converting them to recoverable paths can hide state corruption if not paired with explicit rejection semantics.
|
||||
- The farmer-facing YAML mirror can diverge from wallet DB state if updates are skipped, interrupted, or run with stale action-scope derivations. Review any change that moves `update_pool_config()` out of state-transition, create, or rollback paths.
|
||||
- Pool reward claim batching is bounded. Larger batches risk oversized spend bundles; smaller batches leave rewards claimable but unclaimed.
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- Creation tests should cover both self-pooling and farming-to-pool initial states, launcher announcement linkage, returned launcher id/p2-singleton hash, and later wallet creation from the confirmed launcher spend.
|
||||
- State transition tests should cover self-to-pool, pool-to-self two-stage leave, pool-to-pool two-stage switch, immature `LEAVING_POOL` rejection, duplicate/old transition rejection, and clearing `target_state` when the target lands on chain.
|
||||
- Reorg tests should cover rollback before launcher creation, rollback after one or more singleton transitions, YAML mirror refresh, interested coin-id resubscription, and unconfirmed transaction behavior after rollback.
|
||||
- Puzzle tests should assert tree hashes and spend solutions for member, waiting-room, p2-singleton, travel, and absorb paths against known vectors or existing CLVM behavior.
|
||||
- Reward claim tests should cover no rewards, non-farming coins ignored, fee/no-fee absorb, batching limit, correct `pool_parent_id`/`calculate_pool_reward` height use, and announcement coupling with fee spends.
|
||||
- Farmer integration tests should verify YAML migration/acquire behavior, self-pooling skip path, HTTPS enforcement on mainnet, `/pool_info` redirect handling, payout-instruction updates, missing auth key handling, and partial submission difficulty updates.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Wallet pool state machine: `chia/pools/pool_wallet.py`, `chia/pools/pool_wallet_info.py`.
|
||||
- Pool CLVM and singleton puzzle helpers: `chia/pools/pool_puzzles.py`.
|
||||
- Farmer config mirror: `chia/pools/pool_config.py`.
|
||||
- Pool HTTP protocol payloads: `chia/protocols/pool_protocol.py`.
|
||||
- Pool reward coin helpers: `chia/consensus/coinbase.py`.
|
||||
@@ -0,0 +1,84 @@
|
||||
# Chia Protocols Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/protocols/` is the wire-contract layer for Chia services. It does not usually enforce behavior itself; it defines the immutable `Streamable` payloads, numeric message IDs, node roles, request/reply state machine, and capability/version declarations that `chia/server/` and API classes enforce at runtime. Treat edits here as network compatibility changes, not local refactors.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for P2P/wallet/farmer/harvester/timelord/solver message schemas, numeric message ids, node-type sender authorization, request/reply mappings, shared handshake fields, capabilities, and protocol versioning. For API stub/concrete metadata parity, read `apis.md`; for connection lifecycle and rate-limit enforcement, read `server.md`; for message side effects, read the receiving service context.
|
||||
|
||||
## Landmarks
|
||||
|
||||
| file | owns |
|
||||
| ------------------------------------------------------ | ---------------------------------------- |
|
||||
| `chia/protocols/protocol_message_types.py` | canonical numeric wire message IDs |
|
||||
| `chia/protocols/protocol_state_machine.py` | valid reply/no-reply message maps |
|
||||
| `chia/protocols/protocol_message_type_to_node_type.py` | sender node-type authorization |
|
||||
| `chia/protocols/shared_protocol.py` | handshake fields, capabilities, version |
|
||||
| `chia/protocols/outbound_message.py` | Message framing, make_msg() id=None |
|
||||
| `chia/protocols/wallet_protocol.py` | wallet sync/subscription payload schemas |
|
||||
| `chia/protocols/full_node_protocol.py` | P2P sync/block/tx payload schemas |
|
||||
| `chia/protocols/harvester_protocol.py` | farmer-harvester schemas, Plot.param bit |
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `ProtocolMessageTypes` is the canonical numeric wire namespace. Values are serialized as `uint8`; renumbering or reusing an old value changes the network protocol even if Python names still look correct.
|
||||
- Payload dataclasses are schema contracts only. Validation of peer type, rate limits, request/reply sequencing, list-size limits, consensus validity, spend validity, and subscription limits happens in server/API/full-node/wallet code after deserialization.
|
||||
- `outbound_message.Message` is the framing payload inside the websocket protocol: message type, optional request id, raw bytes. `make_msg()` intentionally sets `id=None`; request ids are assigned later by `WSChiaConnection`.
|
||||
- `ProtocolMessageTypeToNodeType` is the sender authorization table. `WSChiaConnection` checks the negotiated peer `NodeType` against it before calling an API handler, so a schema being importable does not mean every node may send it.
|
||||
- `protocol_state_machine.VALID_REPLY_MESSAGE_MAP` is the response authority for `call_api()`. `ChiaServer.validate_broadcast_message_type()` also uses it to prevent broadcasting request/response messages as fire-and-forget announcements.
|
||||
- `ApiMetadata.request()` is the runtime binding from message type to handler and payload class. Concrete service APIs handle inbound dispatch, while `chia/apis/*_stub.py` metadata controls outbound request eligibility and response decoding. By default it derives the protocol type from the method name; unusual compatibility cases must pass `request_type=` explicitly.
|
||||
|
||||
## Evolution Lockstep
|
||||
|
||||
- Adding or changing a P2P message is a multi-file compatibility operation: update the payload schema, `ProtocolMessageTypes`, `ProtocolMessageTypeToNodeType`, API implementation/stub decorators, reply map/no-reply set if applicable, rate limits if peer-driven, tests, and usually `shared_protocol.protocol_version`.
|
||||
- Import-time checks that reply-required and no-reply classifications do not overlap. Tests separately require every `ProtocolMessageTypes` entry to have sender authorization and v1/v2 rate-limit coverage. Preserve both as static invariants.
|
||||
- Reply handling has two independent representations: decorators record `reply_types` for stubs/API metadata, while `VALID_REPLY_MESSAGE_MAP` controls runtime response validation. Updating only one gives future agents and tests contradictory protocol knowledge.
|
||||
- Stub metadata and concrete API metadata must move together. A response type may need a local stub entry even when the concrete handler lives on the remote peer, because `call_api()` looks up response metadata before deserializing.
|
||||
- The old/new harvester signage-point transition is a live compatibility pattern: `NewSignagePointHarvester2` uses the `new_signage_point_harvester` message ID (reassigned to 66 for the new protocol), with payload selection based on protocol version. Do not assume one message ID maps to exactly one Python payload shape forever. `NewSignagePointHarvester2` carries `peak_height` and `last_tx_height`; V2 gating (skip before `HARD_FORK2_HEIGHT`) and V1 phase-out are keyed on the transaction-block height (`last_tx_height`), not peak height.
|
||||
- Comments saying "also change protocol_message_types.py and protocol version" are not documentation noise. They mark schema changes that can break peers or third-party services if not coordinated.
|
||||
|
||||
## Handshake And Capabilities
|
||||
|
||||
- `shared_protocol.Handshake` establishes `network_id`, protocol version, software version, server port, node type, and capabilities. `WSChiaConnection` owns the ordering: outbound sends first, inbound replies, then normal tasks start.
|
||||
- Protocol versions are keyed by local/remote `NodeType`, not by a single global version. Farmer/harvester compatibility checks are stricter because those protocols carry third-party/operator integrations.
|
||||
- Capability parsing is permissive in `chia/server/capabilities.py`: unknown values are ignored, and duplicate/conflicting entries collapse according to parser rules. Do not treat the raw capability list as authoritative.
|
||||
- `RATE_LIMITS_V3` has an ordering contract beyond simple negotiation. If both peers advertise it, `configure_window_sizes` must be exchanged immediately after handshake, and settings must remain non-empty and bounded.
|
||||
- `ConfigureWindowSizes` uses protocol message numeric values in its settings list. It is therefore coupled to `ProtocolMessageTypes` and `chia/server/rate_limits_v3.py`; changing message IDs or adding v3-limited messages needs both sides updated.
|
||||
|
||||
## Trust Boundaries And Validation Assumptions
|
||||
|
||||
- All inbound protocol payload bytes are untrusted. `Streamable.from_bytes()` gives typed structure, not semantic trust. Hashes, heights, weights, VDFs, blocks, spend bundles, peer lists, fee estimates, and wallet subscription requests must still be validated by the receiving subsystem.
|
||||
- Full-node sync messages (`new_peak`, weight proofs, blocks, unfinished blocks, signage point/EOS messages) are advertisements or data carriers. Consensus code decides validity; protocol schemas intentionally allow hostile but well-formed objects.
|
||||
- Wallet protocol fields that look like history anchors (`height`, `header_hash`, `previous_height`, `fork_height`, `peak_hash`) are peer-supplied until the full node checks them against canonical chain state or detects reorg conditions.
|
||||
- Transaction fee/cost data in `NewTransaction` is only a fetch/priority/accountability hint. The mempool pipeline recomputes spend validity, cost, and fees from the `SpendBundle`.
|
||||
- Peer-list response messages (`RespondPeers`, `RespondPeersIntroducer`) cross an address-discovery trust boundary. Size, timestamp, address validity, and relay policy are server/discovery responsibilities, not schema guarantees.
|
||||
|
||||
## Request/Response Semantics
|
||||
|
||||
- A message requiring a reply must be sent through request paths with a nonce, not broadcast. Broadcast validation treats such messages as internal protocol errors and closes affected peers.
|
||||
- A response type may be legal only for a specific request. `respond_block`, `reject_block`, `respond_blocks`, `reject_blocks`, wallet reject variants, and subscription responses are not interchangeable just because the payload shapes look related.
|
||||
- `error` is a special response path returned from API handlers for `ApiError` on sufficiently new protocol versions. It is allowed from all node types and bypasses normal typed response decoding by returning `shared_protocol.Error`.
|
||||
- `none_response` is currently mapped to no senders and the capability is disabled/commented out. Do not build new behavior on it without re-enabling the capability path and updating reply semantics.
|
||||
- Late responses are routed by request id in `WSChiaConnection`, not by message type. Schema changes that alter which messages carry responses must preserve request-id expectations.
|
||||
|
||||
## Schema-Specific Gotchas
|
||||
|
||||
- Optional fields often encode protocol phase rather than convenience: `foliage_hash` distinguishes unfinished-block variants, wallet `previous_height` controls incremental sync/reorg behavior, and farmer/harvester source-data fields gate third-party signature workflows.
|
||||
- Some list fields are adversarial resource surfaces. Limits are frequently applied by API decorators (`list_limits`) or rate-limit tables rather than by the dataclass itself. Adding a list to a message without a receiver-side limit changes DoS exposure.
|
||||
- `wallet_protocol.CoinState` and `RespondToPhUpdates` are Rust-backed aliases kept in this module so network protocol tests still cover them. Do not replace them with local Python duplicates unless the Rust serialization contract changes too.
|
||||
- `harvester_protocol.Plot.param()` encodes a backward-compatibility hack: the high bit of `Plot.size` distinguishes v2 plot strength from v1 k-size. Consumers rely on this bit-level interpretation.
|
||||
- `fee_estimate` preserves a v1 wire type by converting `FeeRateV2` with `ceil()` into integer mojos per CLVM cost. Rounding direction is part of the compatibility contract; changing it can underquote fees.
|
||||
- `pool_protocol` is not websocket P2P despite living under `protocols/`. It defines signed HTTP payloads/responses for pool endpoints. Authentication token helpers depend on local wall-clock minutes and should be evaluated as pool API compatibility, not peer-message routing.
|
||||
|
||||
## Test And Audit Strategy
|
||||
|
||||
- For protocol edits, test more than serialization round trips. Verify sender-map coverage, state-machine reply validity, API/stub decorator registration, rate-limit table coverage, and old/new protocol-version behavior where compatibility is claimed.
|
||||
- Existing protocol tests are spread across server, wallet sync, farmer/harvester, plot sync, rate limits, mempool fee protocol, and network protocol data. A failing consumer test may be the real protocol regression signal even when `chia/protocols/` tests pass.
|
||||
- High-risk changes include numeric message ID edits, adding messages without sender mappings, changing default capabilities, changing the reply/no-reply classification, adding unbounded list fields, and broadening which node type may send wallet/full-node/farmer messages.
|
||||
- When auditing behavior, start from the message's receiving API handler and server gates. The protocol class tells what can be deserialized; it does not tell when the message is accepted, whether it is trusted, or which side effects follow.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
For exact message ids, sender maps, reply/no-reply maps, handshake fields, and capabilities, read the files in the Landmarks table above. For API stub metadata and outbound call shape, read `apis.md`, `chia/server/api_protocol.py`, and the relevant `chia/apis/*_stub.py`. For rate-limit compatibility, read `chia/server/rate_limit_numbers.py` and `chia/server/rate_limits_v3.py`.
|
||||
@@ -0,0 +1,84 @@
|
||||
# Repository Tooling Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
Scope: root-level packaging/config, service entrypoints, install wrappers, GitHub workflows, build/release scripts, developer tools, and GUI submodule integration. This is the single context file for repository tooling because these risks are tightly coupled.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for changes to repository metadata, install scripts, CI, packaged builds, root wrappers, service launcher wiring, dev tools, or GUI integration. Runtime behavior still belongs in the specific source module docs.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `pyproject.toml` is the Python packaging authority: project metadata, Python bounds, dependencies/extras, dynamic versioning, supplemental Chia index, and console scripts.
|
||||
- Console script names and targets are compatibility surfaces. Service-facing executables must stay aligned with `chia.util.service_groups`, CLI start/stop behavior, PyInstaller specs, installer scripts, and GUI expectations.
|
||||
- Setup may create local environment directories such as `.venv` and `.penv`, but wrapper commands are the repo contract for agents. Local agent/developer Python commands in this workspace must use `tools/py`; tests must use `tools/pytest`.
|
||||
- `chia-blockchain-gui` is a git submodule, not part of the Python package, but install/build scripts bridge Python versioning, Node workspace layout, packaged daemon binaries, and Electron installers.
|
||||
- `.github/workflows/` is the enforcement surface for tests, pre-commit, installers, packaging, release, dependency review, CodeQL, labels, Docker triggers, PyPI source upload, and CA updates.
|
||||
- `build_scripts/` owns platform release packaging, PyInstaller payloads, installer artifact shape, signing/notarization inputs, and dependency artifact policy.
|
||||
- `tools/` contains wrappers and operational utilities used by developers, CI, and replay/debug workflows. Treat wrapper behavior as part of the repo contract.
|
||||
|
||||
## Packaging, Install, And Service Wiring
|
||||
|
||||
- Supported Python is declared in `pyproject.toml`; install scripts and CI exercise a version matrix. Dependency changes can break wheel availability, private-index resolution, installer images, or import-time behavior even when unit tests pass.
|
||||
- Chia native dependencies (`chia_rs`, `chiapos`, `chiavdf`, `chiabip158`, `chia-puzzles-py`, `chialisp`) define Rust/FFI and puzzle-bytecode boundaries used by consensus, wallet, plotting, and CLVM paths.
|
||||
- Optional extras are meaningful: `dev` defines developer/CI quality tools, `upnp` enables private-index UPnP support, and `legacy_keyring` preserves old keyring behavior.
|
||||
- `install.sh` / `Install.ps1` choose supported Python, validate platform prerequisites, setup Poetry, sync dependencies/extras, optionally do non-editable installs, create legacy `venv` links, and optionally install plotters.
|
||||
- `activated.py` / `activated.sh` / `activated.ps1` dispatch commands inside the intended environment. Pre-commit hooks rely on this environment model.
|
||||
- `chia/__main__.py`, package startup glue, service launchers, daemon/RPC/config helpers, and service groups are coupled to root console scripts and packaged build entrypoints.
|
||||
|
||||
## Quality Gates
|
||||
|
||||
- `ruff.toml` selects broad lint coverage, enforces future annotations, bans relative imports, and bans direct `asyncio.create_task` in favor of `chia.util.task_referencer.create_referenced_task`.
|
||||
- `mypy.ini` is generated from template/exclusion files through `manage-mypy.py`; stale exclusions can fail pre-commit when configured.
|
||||
- `tach.toml` declares enforced import boundaries; dependency-cycle tools and virtual project checks cover additional architecture constraints.
|
||||
- `pytest.ini` centralizes warnings-as-errors behavior, test roots, xdist defaults, warning filters, and markers.
|
||||
- `.pre-commit-config.yaml` wires SQL checks, `__init__.py` generation, ruff format/check-fix, tach, Poetry checks, prettier/shfmt/basic hygiene, Chialisp formatting, dependency-cycle checks, mypy config generation, and mypy.
|
||||
|
||||
Because hooks may auto-fix or regenerate files, commits touching root config should be reviewed after hooks run.
|
||||
|
||||
## CI, Release, And GUI Coupling
|
||||
|
||||
- `test.yml` generates a dynamic test matrix through `chia/_tests/build-job-matrix.py`; scheduled/release/full-matrix runs broaden Python and platform coverage.
|
||||
- `pre-commit.yml` runs hooks across Linux/macOS/Windows and multiple Python versions; local single-platform checks are weaker.
|
||||
- Install-script workflows validate native install scripts, GUI install scripts, editable/non-editable behavior, and Docker install paths.
|
||||
- Installer workflows build macOS, Windows, deb, and rpm artifacts through `build_scripts/`; root metadata feeds versions, dependencies, console scripts, and GUI payload expectations.
|
||||
- `build_scripts/check_dependency_artifacts.py` protects release builders from unexpected source builds by checking artifacts from PyPI plus the Chia index.
|
||||
- GUI install/build flows initialize/update the submodule, propagate `chia version` into GUI package metadata, and package Python/PyInstaller payloads into Electron artifacts.
|
||||
|
||||
Workflow changes should be treated as product changes to supported platforms. Check path filters, branch/tag triggers, concurrency behavior, generated matrix inputs, submodule assumptions, and Chia-managed GitHub action usage.
|
||||
|
||||
## Tooling And Operational Utilities
|
||||
|
||||
- Repository wrappers (`tools/py`, `tools/pytest`) protect this workspace from global interpreter drift and must remain the default for Python commands.
|
||||
- Chain generation, full-sync replay, JSON block execution, CLVM/RPC tooling, dependency analysis, and benchmark helpers often encode external artifact formats or process behavior. Avoid cosmetic refactors that hide what contract a tool is exercising.
|
||||
- Tools that open node DBs, replay chains, or generate artifacts should preserve read-only/write intent, root path isolation, and output naming so they do not contaminate normal node state.
|
||||
- Benchmark scripts should run via `tools/py -m benchmarks.<name>` and remain consumers of production APIs, not alternate implementations or production dependencies.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Python bounds, dependencies, package metadata, console scripts, or service entrypoints.
|
||||
- Install scripts, Poetry setup, root wrappers, or environment dispatch.
|
||||
- Ruff/mypy/tach/pytest/pre-commit config that silently weakens gates.
|
||||
- CI matrix generation, workflow triggers, path filters, install/build actions, or release signing/notarization inputs.
|
||||
- GUI submodule handling, version propagation, package payload paths, or installer layout.
|
||||
- Dependency lock updates without checking private-index and wheel-artifact consequences.
|
||||
|
||||
Root/tooling code often looks like glue, but it encodes platform and release assumptions. Avoid cleanup edits unless the affected installer, hook, workflow, or package path is also validated.
|
||||
|
||||
## Verification Guidance
|
||||
|
||||
- Package metadata/dependencies: run the relevant Poetry/pre-commit path and at least an import/CLI smoke through repository wrappers.
|
||||
- Console script/service changes: check source entrypoint, service group behavior, and PyInstaller packaging surfaces.
|
||||
- Install scripts: exercise the affected native script path or CI-equivalent command when practical.
|
||||
- Static tooling config: run the affected hook directly; use broader pre-commit when behavior is cross-cutting.
|
||||
- Test matrix/config: check generated matrix behavior plus representative `tools/pytest` runs.
|
||||
- CI workflow edits: reason about event triggers and matrix dimensions; local tests do not validate GitHub Actions semantics.
|
||||
- GUI/build coupling: validate the phase that consumes the changed root contract, not just Python tests.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Packaging and tool configuration: `pyproject.toml`, `poetry.lock`, `ruff.toml`, `mypy.ini.template`, `tach.toml`, `pytest.ini`, `.pre-commit-config.yaml`.
|
||||
- Install and environment wrappers: `install.sh`, `Install.ps1`, `activated.py`, `activated.sh`, `activated.ps1`, `tools/`.
|
||||
- CI and release automation: `.github/workflows/`, `build_scripts/`.
|
||||
- GUI submodule integration: `chia-blockchain-gui`.
|
||||
@@ -0,0 +1,94 @@
|
||||
# Chia RPC Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/rpc/` is the local operator/API boundary for Chia services. It is separate from the peer protocol: service-specific RPC APIs expose HTTP POST routes and daemon websocket commands, while this module supplies the shared server wrapper, client base class, typed marshalling helper, and structured error normalization.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for shared RPC server/client transport, daemon websocket registration, common routes, typed request marshalling, structured error shape, state-change fanout, and local/admin API behavior. For endpoint side effects, read the concrete service context and RPC API class.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `RpcServer` is the shared transport authority. It owns HTTPS route registration, daemon websocket registration/reconnect, state-change fanout to the daemon/UI, common service routes, and lifecycle coordination with `WebServer`, `ClientSession`, and the daemon websocket.
|
||||
- Concrete `*RpcApi` classes own service semantics. They provide `service_name`, `service`, `get_routes()`, and `_state_changed()`; endpoint validation and side effects belong there, not in `RpcServer`.
|
||||
- `RpcServiceProtocol` is the service adapter contract. The service supplies its `ChiaServer`, connection summaries, outbound peer connection hook, state-change callback registration, and optional lifecycle management.
|
||||
- `RpcClient` is only the common client for shared routes. Service-specific clients such as full-node, wallet, and data-layer clients extend the same HTTP/JSON convention for larger endpoint surfaces.
|
||||
- `rpc_errors.py` is the compatibility bridge between legacy `"error"` strings and machine-readable `"structuredError"` payloads. Callers still depend on both shapes.
|
||||
|
||||
## Why This Is Tricky
|
||||
|
||||
Public RPC docs present HTTP and websocket access as one API surface. In source, this package is mostly transport and compatibility glue: it should preserve request/response envelopes, daemon registration, common admin routes, and error normalization while leaving business semantics in concrete service RPC APIs. A change that looks like common JSON cleanup can break CLI clients, GUI websocket consumers, service shutdown order, or language clients that depend on legacy response fields.
|
||||
|
||||
## Wrong Assumptions To Avoid
|
||||
|
||||
- Do not treat RPC as P2P. It is a local/admin surface protected by private TLS in normal service mode.
|
||||
- Do not put endpoint business validation into `RpcServer`; concrete RPC APIs own service semantics.
|
||||
- Do not assume HTTP and daemon websocket failures have identical response shape.
|
||||
- Do not assume `marshal()` covers every typed or CLVM-streamable request path.
|
||||
|
||||
## Transport And Lifecycle
|
||||
|
||||
- RPC HTTP routes are POST-only and are registered from `rpc_api.get_routes()` plus `RpcServer._routes`. Route names include their leading slash on the server; `RpcClient` method names intentionally match common route names without the slash.
|
||||
- `start_rpc_server()` creates TLS contexts from the private daemon certs/CA, installs the service state-change callback, starts the HTTPS server, and optionally starts the daemon websocket loop.
|
||||
- RPC startup is nested inside `Service.manage()` after the node service and peer server start. Shutdown closes peer connections first, then closes RPC, awaits the peer server, and finally awaits RPC cleanup. Do not make RPC shutdown depend on peer-server state still being live.
|
||||
- `RpcServer.close()` is a shutdown signal plus webserver close; `await_closed()` is responsible for closing the daemon websocket, client session, webserver, and daemon reconnect task.
|
||||
- The daemon connection loop re-creates `ClientSession` and websocket on each reconnect attempt, registers the service with the daemon, handles daemon-originated commands until the websocket closes, then sleeps for the configured retry delay unless shut down.
|
||||
- `daemon_heartbeat` and `max_message_size` apply only to the daemon websocket. HTTP request body size is controlled by `max_request_body_size`, with service startup able to override the default.
|
||||
|
||||
## Request And Response Contracts
|
||||
|
||||
- HTTP handlers call `await request.json()` before endpoint dispatch. Malformed JSON/body parsing errors are outside the endpoint `try` block in `wrap_http_handler()` and therefore can surface as HTTP-level failures rather than normalized RPC error objects.
|
||||
- Endpoint handlers may return `None` or a dict. `wrap_http_handler()` and websocket `safe_handle()` insert `"success": True` when absent; handlers that need failure semantics must return/raise explicitly.
|
||||
- HTTP endpoint exceptions produce `{success: false, error, traceback, structuredError}`. Websocket endpoint exceptions produce `{success: false, error, structuredError}` and intentionally omit traceback from the response.
|
||||
- `RpcClient.fetch()` treats transport-level HTTP errors via `raise_for_status()` and application-level failures via `ResponseFailureError`, preserving the full response dict for CLI and tests.
|
||||
- Websocket daemon commands are selected by `command`, ignored when `ack` is true, respond to `"ping"` with `pong()`, then resolve first against common `RpcServer` methods and then concrete `rpc_api` methods. Unknown commands become structured failure responses when the incoming JSON parsed successfully.
|
||||
- State-change notifications are push-only daemon websocket messages. `_state_changed()` may add service-specific payloads, and common connection changes additionally trigger a synthetic `get_connections` update for `wallet_ui`.
|
||||
|
||||
## Common Routes And Peer Coupling
|
||||
|
||||
- Common routes are operational helpers: network info, active peer connections, open/close peer connection, stop node, route discovery, version, health, and log level management.
|
||||
- `get_connections()` crosses into `ChiaServer` through the concrete service. Returned connection data includes peer node id bytes; clients convert `node_id` hex strings back to bytes.
|
||||
- `open_connection()` resolves hostnames with the configured IPv6 preference and delegates to `ChiaServer.start_client()` with the service's `on_connect` callback when present.
|
||||
- `close_connection()` closes all matching peer connections for a node id. The server connection layer remains the authority for cleanup, bans, and peer lifecycle side effects.
|
||||
- `set_log_level()` mutates global/service logging configuration at runtime. Tests rely on `reset_log_level()` restoring the configured service log level.
|
||||
|
||||
## Marshalling And Compatibility
|
||||
|
||||
- `marshal()` adapts typed `Streamable` request/response endpoints to normal RPC dict endpoints. It derives the request class from the endpoint's `request` type hint and asserts it is a `Streamable`.
|
||||
- Default marshalling uses `from_json_dict()` and `to_json_dict()`. Requests carrying the CLVM-streamable JSON CHIP use CLVM streamable JSON serialization, optionally with a named translation layer.
|
||||
- The shared translation registry maps known CHIP translation names to translation layers. Unknown translation names raise before endpoint logic runs and are normalized by the transport wrapper.
|
||||
- The helper assumes a single `request` argument and a `Streamable` response. Endpoints with custom dict semantics, multi-object responses, action-scope wrappers, or legacy shape compatibility should keep explicit dict handling in the concrete RPC API.
|
||||
- Wallet RPC also performs CLVM-streamable JSON handling in transaction wrappers. Do not assume `marshal()` is the only CLVM-streamable JSON path in the RPC system.
|
||||
|
||||
## Structured Error Semantics
|
||||
|
||||
- Prefer raising `RpcError` for intentional endpoint failures that need stable machine-readable codes or structured data. `RpcError.simple()` is the concise path when the legacy and structured messages can match.
|
||||
- Non-`RpcError` exceptions are mapped by type: validation, timestamp, consensus, protocol, util `ApiError`, and assertion errors receive specific codes; everything else becomes `UNKNOWN`.
|
||||
- `structured_error_from_exception()` imports consensus/protocol error classes lazily. Keep new imports there deliberate to avoid broad import-time coupling from the low-level RPC package into heavy subsystems.
|
||||
- The legacy `"error"` string remains compatibility surface for existing CLI/client code. Changing it can break callers even if `"structuredError"` is richer.
|
||||
- Values placed in `RpcError.data` must be JSON-serializable. The transport wrapper does not sanitize arbitrary objects before `obj_to_response()`.
|
||||
|
||||
## Test And Audit Strategy
|
||||
|
||||
- Shared RPC changes should run focused tests under `chia/_tests/rpc/` plus any service-specific RPC tests for affected semantics. Use `tools/pytest`, not bare `pytest`.
|
||||
- For common route changes, preserve the invariant that `RpcClient` method names match `RpcServer._routes` entries without the leading slash.
|
||||
- For response-shape changes, test both HTTP and daemon websocket paths; they intentionally differ on traceback exposure.
|
||||
- For marshalling changes, test normal streamable JSON and CLVM-streamable requests. Data-layer and wallet RPC are the important service-specific consumers.
|
||||
- For lifecycle changes, reason with `Service.manage()` shutdown order and daemon reconnect behavior. Leaks usually appear as unclosed aiohttp sessions, a reconnect task that never exits, or websocket state-change sends after shutdown.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Highest-risk edits: changing automatic `"success"` insertion, changing common route names, broadening `RpcClient.fetch()` failure behavior, altering daemon websocket command routing, or making RPC lifecycle assumptions about peer-server availability.
|
||||
- Error compatibility is easy to break: HTTP includes traceback in failure responses, websocket failures do not, and `ResponseFailureError` preserves the full JSON body.
|
||||
- The RPC boundary is semi-trusted local/admin surface protected by private TLS in normal service mode, not an untrusted P2P path. Do not move peer protocol rate-limiting or node-type authorization assumptions into this layer.
|
||||
- Route discovery returns every shared and service-specific route. Adding sensitive operational endpoints should be evaluated as local admin API exposure even when not reachable through the P2P protocol.
|
||||
- `marshal()` depends on runtime type hints. Missing or future-deferred annotations that cannot resolve will fail at decoration/call setup rather than inside endpoint business logic.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Shared server lifecycle and common routes: `chia/rpc/rpc_server.py`.
|
||||
- Common client behavior: `chia/rpc/rpc_client.py`.
|
||||
- Typed marshalling and CLVM streamable handling: `chia/rpc/util.py`.
|
||||
- Structured errors: `chia/rpc/rpc_errors.py`.
|
||||
- Daemon websocket envelope/routing: `chia/util/ws_message.py`, `chia/daemon/server.py`.
|
||||
@@ -0,0 +1,154 @@
|
||||
# Chia Seeder Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/seeder/` is two cooperating services over one SQLite database: a crawler
|
||||
that joins the Chia P2P network as a minimal full node to discover reachable
|
||||
peers, and an authoritative DNS server that serves the crawler's reliable-peer
|
||||
set. It is bootstrap infrastructure, not a consensus or identity authority.
|
||||
Every peer address it records or returns remains advisory until a receiving
|
||||
full node performs normal DNS/introducer ingestion, connection, handshake,
|
||||
network-id checks, and address-manager policy.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for crawler peer discovery, crawler DB scoring/persistence, DNS seed responses, seeder RPC observability, and bootstrap-peer publication behavior. For generic P2P connection policy, also read `server.md` and `protocols.md`.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `Crawler` owns P2P crawling state, the background crawl loop, peer-gossip
|
||||
ingestion, version/height/TLS observations, and the in-memory caches that are
|
||||
periodically flushed to `CrawlStore`.
|
||||
- `CrawlerAPI` is a deliberately stubby full-node API surface. It registers many
|
||||
full-node and wallet message handlers so generic `ChiaServer` protocol
|
||||
dispatch accepts expected traffic, but only `new_peak` mutates crawler state.
|
||||
The crawler actively calls `FullNodeAPI.request_peers`; it does not serve
|
||||
peers or blockchain data to remote nodes.
|
||||
- `CrawlStore` is the persistence and scoring boundary. It owns the `peer_records`,
|
||||
`peer_reliability`, and `good_peers` tables plus mirrored in-memory maps.
|
||||
`good_peers` is the crawler-to-DNS handoff table.
|
||||
- `DNSServer` owns DNS protocol behavior and does not crawl. It periodically
|
||||
reloads `good_peers`, merges configured `static_peers`, splits them into IPv4
|
||||
and IPv6 lists, and rotates answers through round-robin pointers under a lock.
|
||||
- Generic `Service` / `ChiaServer` still own TLS identity, handshakes, node-type
|
||||
checks, rate limits, request/response validation, bans, connection lifetime,
|
||||
daemon/RPC wiring, and shutdown ordering for the crawler side.
|
||||
|
||||
## Service Shape
|
||||
|
||||
- `start_crawler.create_full_node_crawler_service()` builds a normal
|
||||
`Service[Crawler, CrawlerAPI, CrawlerRpcApi]` with `NodeType.FULL_NODE`,
|
||||
`service_name="full_node"`, and the seeder service config. This is why the
|
||||
crawler can connect to full nodes and use full-node protocol messages without
|
||||
a separate crawler node type.
|
||||
- The crawler advertises `seeder.port` but connects discovered candidates on
|
||||
`seeder.other_peers_port`. The crawl loop ignores records whose stored port is
|
||||
not `other_peers_port`, so changing that config changes crawl eligibility.
|
||||
- `Crawler.manage()` lowers `server.config["peer_connect_timeout"]` from
|
||||
`seeder.peer_connect_timeout`, opens the SQLite DB, bootstraps configured
|
||||
hosts, and starts the crawl task unless `start_crawler_loop=False`.
|
||||
- The DNS service is independent of `Service` and uses low-level asyncio UDP and
|
||||
TCP protocols. It binds wildcard UDP/TCP sockets with platform-specific IPv4
|
||||
handling.
|
||||
- Both crawler and DNS default to the same `crawler_db_path`; running only one
|
||||
side is valid but DNS will wait/retry until reliable peers or static peers are
|
||||
available.
|
||||
|
||||
## Crawl Data Flow
|
||||
|
||||
- Bootstrap peers are inserted as unresolved host/IP strings with placeholder
|
||||
version/timestamp state and no reachability evidence.
|
||||
- Each crawl batch asks `CrawlStore.get_peers_to_crawl(...)` for peers whose
|
||||
reliability ban/ignore windows expired and whose last try/connect time is old
|
||||
enough. IPv6 records use a shorter retry delta than IPv4, and recently selected
|
||||
records are temporarily suppressed.
|
||||
- The crawler opens a bounded number of concurrent outbound client tasks. On connect it
|
||||
records the peer's short Chia version, requests `RespondPeers`, waits briefly
|
||||
for a qualifying `new_peak`, then closes the connection.
|
||||
- `new_peak` is the crawler's reachability signal. A peer is marked connected
|
||||
only when its peer IP parses as an IP address and the announced height is at
|
||||
least `seeder.minimum_height`; TLS version is also recorded. If no qualifying
|
||||
peak arrives, the attempt is scored as a failure.
|
||||
- `RespondPeers` entries update `best_timestamp_per_peer` and create new
|
||||
candidates only when their advertised timestamp is within the source-defined
|
||||
freshness horizon. That horizon also gates version reporting and RPC-visible
|
||||
`best_timestamp_per_peer` data.
|
||||
- After each batch the crawler writes peer records, rewrites `good_peers` from
|
||||
current reliability scores, prunes records older than `crawler.prune_peer_days`
|
||||
by `best_timestamp`, clears temporary caches, clears `server.banned_peers`, and
|
||||
emits a `crawl_batch_completed` state change.
|
||||
|
||||
## Reliability And Persistence Semantics
|
||||
|
||||
- `PeerReliability` maintains exponentially decayed success signals over several
|
||||
source-defined windows. A peer can be reliable with a small early sample or
|
||||
with progressively lower reliability thresholds over longer windows.
|
||||
- Failures update ignore/ban windows only when the peer is not currently
|
||||
reliable. The worst long-window failure cases can be ignored or banned for
|
||||
source-defined durations.
|
||||
- `PeerRecord` is streamable but mutable in practice through `object.__setattr__`
|
||||
in `update_version()`. Do not assume frozen dataclass immutability protects
|
||||
record state inside `CrawlStore`.
|
||||
- `CrawlStore.add_peer(save_db=False)` updates only memory. Durable writes happen
|
||||
through `Crawler.save_to_db()`, which calls `load_to_db()` and
|
||||
`load_reliable_peers_to_db()` with retry-on-exception behavior.
|
||||
- The schema has migration quirks: `tls_version` is added with `ALTER TABLE` and
|
||||
duplicate-column errors are ignored. Treat persisted crawler DBs as long-lived
|
||||
operational state, not throwaway cache, when changing columns or insert order.
|
||||
- `good_peers` stores only IP strings. DNS output does not include ports; clients
|
||||
receiving DNS answers use their own network default port logic.
|
||||
|
||||
## DNS Semantics
|
||||
|
||||
- `DNSServer.dns_response()` answers only the configured domain and subdomains.
|
||||
Requests outside that zone return `REFUSED`; unknown names inside the zone
|
||||
return `NXDOMAIN`; valid answers always include NS and SOA authority records.
|
||||
- `static_peers` are merged into each refresh. Literal IPs are accepted directly;
|
||||
hostnames are resolved for both `A` and `AAAA` with a source-defined lifetime
|
||||
when the async resolver is available.
|
||||
- Peer answer rotation is round-robin per address family, not random per query.
|
||||
`CrawlStore.get_good_peers()` shuffles DB results before DNS loads them, then
|
||||
DNS pointer state controls fairness between refreshes.
|
||||
- Exact response limits, EDNS0 handling, and truncation behavior live in `chia/seeder/dns_server.py`.
|
||||
|
||||
## RPC And Observability
|
||||
|
||||
- `CrawlerRpcApi` exposes `/get_peer_counts` and `/get_ips_after_timestamp`.
|
||||
Counts are derived from in-memory five-day crawler caches, not directly from
|
||||
SQLite, so they reflect the running crawler's latest processed batch.
|
||||
- `get_ips_after_timestamp` sorts IP strings lexicographically after timestamp
|
||||
filtering and supports `offset`/`limit`. It requires an `after` timestamp.
|
||||
- State-change websocket payloads are emitted only for `crawl_batch_completed`
|
||||
and `loaded_initial_peers`; missing change data is replaced with
|
||||
`/get_peer_counts` output.
|
||||
- The crawler's log summary deliberately reports both gossiped recent addresses
|
||||
and reachable recent handshakes. Keep those concepts separate: a peer can be
|
||||
seen in `RespondPeers` without being reachable or DNS-eligible.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Highest-risk edits are changes that mark peers reliable from weaker evidence,
|
||||
broaden `CrawlerAPI` into serving full-node data, remove `minimum_height` or
|
||||
IP validation from `new_peak`, or let unbounded peer lists flow into memory or
|
||||
DNS responses.
|
||||
- The crawler treats peer-gossiped timestamps as freshness hints. Keep the
|
||||
five-day filter and pruning behavior explicit; stale or future-biased gossip
|
||||
should not become DNS output without fresh reachability evidence.
|
||||
- The crawler currently clears `server.banned_peers` after each batch. Changes to
|
||||
ban handling need to account for crawler-specific connection churn without
|
||||
weakening generic `ChiaServer` behavior elsewhere.
|
||||
- DNS tests cover TCP/UDP, IPv4/IPv6 listeners, A/AAAA/ANY/NS/SOA responses,
|
||||
static IPs, static hostname resolution, DB-driven peer loading, and error
|
||||
cases. Some broad DNS query/error-condition tests are skipped as flaky, so
|
||||
changes in protocol parsing or truncation need focused fresh verification.
|
||||
- Crawler tests assert loop startup controls, handling of unknown protocol
|
||||
messages, `new_peak` reachability effects, DB-to-good-peer promotion, pruning,
|
||||
and RPC pagination. Use `tools/pytest` and async `time_out_assert` patterns
|
||||
for behavior that depends on service convergence.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Crawler service state and peer scoring: `chia/seeder/crawler.py`, `chia/seeder/crawl_store.py`, `chia/seeder/peer_record.py`.
|
||||
- Crawler peer API and RPC: `chia/seeder/crawler_api.py`, `chia/seeder/crawler_rpc_api.py`.
|
||||
- DNS server behavior: `chia/seeder/dns_server.py`.
|
||||
- Service startup: `chia/seeder/start_crawler.py`, `chia/seeder/crawler_service.py`.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Chia Server Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/server/` is the network boundary where remote TLS peers become typed protocol calls on node APIs. Its core safety property is distributed across TLS identity, handshake/capability negotiation, per-peer message routing, rate limiting, peer admission, and address discovery. Do not reason about any one of these in isolation.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for P2P websocket connections, TLS identity, peer lifecycle, handshake/capability negotiation, rate limiting, peer discovery, bans, and address-manager behavior. For message schemas, read `protocols.md`; for semantic acceptance of a message, read the receiving service context.
|
||||
|
||||
## Landmarks
|
||||
|
||||
| file | owns |
|
||||
| ----------------------------------- | ----------------------------------------------------- |
|
||||
| `chia/server/ws_connection.py` | per-peer protocol, request/response, rate-limit state |
|
||||
| `chia/server/server.py` | connection admission, bans, TLS identity, broadcast |
|
||||
| `chia/server/rate_limits.py` | v1/v2 per-message + aggregate accounting |
|
||||
| `chia/server/rate_limits_v3.py` | in-flight-window v3 limiter |
|
||||
| `chia/server/rate_limit_numbers.py` | rate-limit tables/constants |
|
||||
| `chia/server/address_manager.py` | peer address-book state machine |
|
||||
| `chia/server/node_discovery.py` | DNS/introducer discovery orchestration |
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `WSChiaConnection` is the per-peer protocol authority. It owns handshake-derived peer type/capabilities, request nonce allocation, pending-response matching, v2/v3 rate-limit state, API task lifetime, and raw websocket parsing.
|
||||
- `ChiaServer` is the process-local connection authority. It owns TLS context selection, node identity, inbound/outbound admission, duplicate/self connection rejection, bans, broadcast validation, and connection shutdown callbacks.
|
||||
- API classes own message semantics, but only after `ApiMetadata` has selected the handler, decoded the streamable payload, and `ProtocolMessageTypeToNodeType` has checked that the peer's negotiated `NodeType` is allowed to send that message.
|
||||
- Peer discovery is advisory, not trusted. `FullNodeDiscovery` bounds and sanitizes peer lists before they reach `AddressManager`; `AddressManager` decides whether and when an address becomes tried/good.
|
||||
|
||||
## Handshake And Identity Contracts
|
||||
|
||||
- Peer identity is the SHA-256 fingerprint of the TLS certificate used on the websocket. `connection_added()` replaces an existing connection with the same peer id; host/port alone are not the identity boundary.
|
||||
- Handshake ordering matters. Outbound peers send `handshake` first; inbound peers read first and then reply. If both sides negotiate `RATE_LIMITS_V3`, `configure_window_sizes` exchange must happen immediately before normal inbound/outbound/message-handler tasks start.
|
||||
- `connection_type`, `peer_capabilities`, `peer_server_port`, `version`, and `protocol_version` are not valid until `perform_handshake()` completes. Any code path that calls `call_api()`, peer-type filtering, or discovery bookkeeping before that point is using uninitialized protocol state.
|
||||
- Full nodes enforce hard-fork capability at inbound admission after handshake, using current peak height. This is an admission rule, not part of TLS authentication or generic protocol decoding.
|
||||
- Capability parsing is permissive: unknown capability ids are ignored and duplicate/conflicting entries are normalized by parser rules. Do not assume the peer's raw capability list is canonical.
|
||||
|
||||
## Message Routing And Request State
|
||||
|
||||
- Requests and responses share an id space split by direction. This split is what prevents both peers from generating the same local request id under normal operation.
|
||||
- `send_request()` registers `pending_requests` before enqueueing the message; inbound messages with matching ids are treated as responses, not API calls. Late responses to timed-out ids are discarded through `timed_out_requests` to avoid reusing an id while a stale response may still arrive.
|
||||
- `call_api()` validates the response type against `protocol_state_machine.VALID_REPLY_MESSAGE_MAP` and bans on mismatches. Fire-and-forget broadcasts must not use request/response message types; `ChiaServer.validate_broadcast_message_type()` treats that as an internal protocol error and closes relevant peers.
|
||||
- API handler dispatch has two separate gates: existence in the local API metadata and permission for the sender node type. Adding a protocol message requires updating the protocol enum, API metadata decorators, node-type sender map, and request/response state machine when replies are expected.
|
||||
- `execute_task=True` handlers are intentionally not cancelled by normal connection close through `cancel_tasks()`. They also bypass the normal API timeout path. This flag is a lifecycle contract, not just a scheduling optimization.
|
||||
|
||||
## Rate Limiting Model
|
||||
|
||||
- v1/v2 rate limiting is per-message plus aggregate non-transaction accounting over a time slot. Incoming messages always commit counters, even when they exceed limits; outgoing messages commit counters only if allowed to send.
|
||||
- `Unlimited` means "no frequency limit, size limit only"; it relies on protocol state and unsolicited-response handling elsewhere. Treating an unlimited response type as generally safe removes an important implicit coupling.
|
||||
- v3 rate limiting replaces v2 only for message types listed in `rate_limits_v3` and only when both peers advertise `RATE_LIMITS_V3`. It is in-flight-window based, not time-window based.
|
||||
- v3 request messages with a finite `window_size` increment outbound `in_flight` when sent and decrement in `send_request()`'s `finally`. Inbound finite-window messages increment `receive_window` around `_api_call()` and decrement in its `finally`. These decrements are the leak-prevention invariant.
|
||||
- v3 response/unlimited messages are allowed to carry the request nonce without being in `pending_requests`; this special case exists because responses reuse the peer's request id. Removing that distinction can misclassify legitimate responses or bypass the intended limiter path.
|
||||
- Localhost and configured exempt networks are exempt from disconnect/ban consequences, but v3-supported messages still bypass v2 accounting when v3 is negotiated. Exemption changes enforcement, not protocol shape.
|
||||
|
||||
## Connection Lifecycle
|
||||
|
||||
- `close()` is idempotent but still calls the server close callback for already-closed connections. The callback performs ban insertion and removal from `all_connections`, so "already closed" must not mean "skip server cleanup."
|
||||
- `ChiaServer.connection_closed()` refuses to ban localhost, trusted peers, and exempt networks even when a lower layer requested a ban. Audit ban behavior at the server callback, not at the call site that passed `ban_time`.
|
||||
- `incoming_connection()` waits for `connection.wait_until_closed()` before returning the websocket response. Long-lived inbound requests therefore tie request lifetime to connection lifetime by design.
|
||||
- `Service.manage()` shutdown order is intentional: release UPnP in the background, cancel reconnect loop, close peer connections, close RPC, await `ChiaServer`, then await RPC. Reordering can leave reconnect or RPC paths racing against closed peer state.
|
||||
- `chia_policy.set_chia_policy()` mutates the process event-loop policy and wraps server creation with `PausableServer`. Connection limits are enforced at accept/pause level, separately from ChiaServer's node-type inbound slot limits.
|
||||
|
||||
## Peer Discovery And Address Book
|
||||
|
||||
- `FullNodeDiscovery` is the orchestrator; `AddressManager` is the state machine. Discovery decides when to query DNS/introducers, when to make normal versus feeler connections, and when to relay addresses. AddressManager decides table placement, collision handling, retries, and selection probability.
|
||||
- Peer lists are bounded at response and ingestion boundaries before reaching address-manager state. Oversized host strings and invalid IPs are dropped before `IPAddress` construction.
|
||||
- Received peer timestamps are normalized or penalized depending on source and validity. This avoids treating peer-supplied freshness as authoritative.
|
||||
- Full-node address sharing is intentionally asymmetric for fingerprinting resistance: `request_peers()` and `neighbour_known_peers` suppress re-relaying the same address to the same neighbour. The current `request_peers()` handler does not check `peer.is_outbound`, so treat the outbound-neighbour intent as a convention, not an enforced gate.
|
||||
- During normal runtime mutation, `AddressManager` invariants depend on mutating matrices through `_set_new_matrix()` and `_set_tried_matrix()` so the sparse-position sets stay synchronized. Deserialization intentionally writes matrices directly and then reconstructs position sets; do not assume direct matrix writes are always invalid.
|
||||
- Private subnets are rejected by default in `AddressManager` unless discovery explicitly enables private networks from introducer config. Do not add bypasses in discovery without checking this address-book policy.
|
||||
|
||||
## External Inputs And Persistence
|
||||
|
||||
- Websocket frames, streamable payloads, peer lists, DNS answers, introducer responses, configured peer hostnames, and the persisted peers file are all untrusted at this boundary.
|
||||
- The peers file supports both the deprecated `PeerDataSerialization` migration path and the current binary `AddressManager.serialize_bytes()` format. Load failures fall back to a fresh address manager rather than aborting service startup.
|
||||
- Public IP discovery in `ChiaServer.get_peer_info()` depends on external HTTPS services and returns `None` on failure. Callers must handle absence of self-advertisable peer info without degrading core connectivity.
|
||||
- UPnP runs in a side thread and is best-effort. It should not be used as a readiness signal for peer networking.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Highest-risk edits: changing handshake task start order, changing request-id reuse rules, moving v3 window increments/decrements, broadening `Unlimited`, skipping response-type validation, or bypassing `ProtocolMessageTypeToNodeType`.
|
||||
- Connection cleanup bugs often show up as stale `all_connections`, stuck `pending_requests`, leaked v3 `in_flight` windows, or reconnect loops that keep targeting already-connected peers under a different hostname.
|
||||
- Peer discovery bugs often look non-deterministic because selection uses randomized bucket placement, Poisson feeler timing, per-network-group filtering, and collision testing. Tests should control randomness/time or assert invariants rather than exact peer order.
|
||||
- AddressManager counts (`new_count`, `tried_count`, `ref_count`, matrix entries, used-position sets, `random_pos`) are a single consistency domain. Any helper that updates one must update all related structures under the lock exposed by the async public methods.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
For exact websocket dispatch, peer-close behavior, rate-limit tables, and discovery/address-book rules, read the source files in `chia/server/` listed in the Landmarks table above.
|
||||
@@ -0,0 +1,147 @@
|
||||
# chia-simulator
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
Scope: `chia/simulator/`. This is distilled architectural context for future audit or implementation agents. It intentionally omits generated SSL collateral and obvious helper inventories.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for simulator block farming, reorg/revert helpers, simulator-only RPC routes, `BlockTools`, service test harnesses, deterministic plots/keys, wallet transaction helpers, and local simulator startup. For production full-node behavior, read `full-node.md`; for consensus acceptance rules, read `consensus.md`.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
`chia.simulator` is the deterministic test and local-dev control plane for Chia services. It does not implement an alternate blockchain. Instead, it fabricates blocks, plots, VDFs, keys, config roots, and RPC affordances so tests and simulator users can drive the real `FullNode`, `Blockchain`, `MempoolManager`, wallet sync, and service stack quickly.
|
||||
|
||||
The two central surfaces are:
|
||||
|
||||
- `BlockTools`: constructs consensus-shaped objects, test plots, keys, block generators, VDF proofs, signage points, sub-slots, and block sequences.
|
||||
- `FullNodeSimulator`: subclasses `FullNodeAPI` and wraps a real `FullNode` with high-level farming, reorg, revert, mempool-processing, and wallet-sync helpers.
|
||||
|
||||
The main architectural rule is: simulator conveniences may shortcut environment setup and block production, but accepted chain state must still flow through normal full-node validation/commit paths unless the function explicitly documents a destructive test-only state edit.
|
||||
|
||||
## Why This Is Tricky
|
||||
|
||||
Public simulator docs describe complete control of a private chain: farm blocks, reorg, revert, inspect state, and toggle autofarming. In source, that control is intentionally split between safe helpers that still pass through real full-node validation and destructive helpers that directly edit local stores for test convenience. Simulator code is useful because it is close to production paths; making shortcuts broader can create tests that pass against states a real node could never reach.
|
||||
|
||||
## Wrong Assumptions To Avoid
|
||||
|
||||
- Do not treat simulator block creation as consensus acceptance; generated blocks still need full-node/consensus commit unless a helper explicitly bypasses it.
|
||||
- Do not use destructive revert helpers as a model for production reorg behavior.
|
||||
- Do not assume simulator config/key/plot shortcuts are valid outside isolated test roots.
|
||||
- Do not treat simulator RPC routes as normal full-node RPC behavior; they extend the full-node API for local private-chain control.
|
||||
|
||||
## Core Simulator Contracts
|
||||
|
||||
- `BlockTools.get_consecutive_blocks()` is a block factory, not a chain authority. It can create blocks with specific timing, overflow, references, transaction generators, reward targets, and seeds; the authoritative decision happens when `FullNode.add_block()`, `FullNode.add_block_batch()`, or `Blockchain.add_block()` accepts those blocks.
|
||||
- `FullNodeSimulator.farm_new_block()` and `farm_new_transaction_block()` acquire the full node blockchain mutex, derive the next block from current persisted chain state, release the lock, then call `full_node.add_block()`. They rely on the same post-processing path as ordinary full-node operation.
|
||||
- `add_blocks_in_batches()` intentionally mirrors batch sync: it builds `ForkInfo`, `ValidationState`, and `AugmentedBlockchain`, calls `FullNode.add_block_batch()` in chunks, then runs peak post-processing and `_finish_sync()`. It is the right path for large simulator-generated chains and reorg tails.
|
||||
- `revert_block_height()` is the exception: it directly rolls `coin_store` and `block_store`, sets the block-store peak, mutates `blockchain._peak_height`, and refreshes the mempool. It does not broadcast and explicitly expects wallets to be wiped. Treat it as destructive local state surgery, not a reorg model.
|
||||
|
||||
## Block Generation Model
|
||||
|
||||
`BlockTools` lowers mainnet constants for tests (`test_constants`) and creates enough deterministic plots to find proofs quickly. The defaults reduce plot sizes, difficulty, VDF discriminant size, proof filter bits, weight-proof windows, and timing while preserving the consensus shape that validation expects.
|
||||
|
||||
`get_consecutive_blocks()` is the heart of the module. It:
|
||||
|
||||
- continues from an optional block list or creates genesis;
|
||||
- computes signage points and end-of-sub-slot bundles;
|
||||
- scans local test plots for qualified V1/V2 proofs;
|
||||
- computes required iterations, overflow status, pool/farmer targets, VDF proofs, and MMR/header commitments;
|
||||
- optionally injects a `SpendBundle` as transaction data or synthesizes spends from farmer reward coins;
|
||||
- updates local block-record, height-to-hash, difficulty, sub-slot-iterations, MMR, pending-reward, and generator-reference state as it appends blocks.
|
||||
|
||||
The factory deliberately exposes consensus edge knobs: `force_overflow`, `skip_slots`, `guarantee_transaction_block`, `keep_going_until_tx_block`, `dummy_block_references`, `block_refs`, normalized VDF proof flags, `current_time`, `genesis_timestamp`, `force_plot_id`, `skip_overflow`, `min_signage_point`, and deterministic `seed`.
|
||||
|
||||
Important coupling: block construction must stay aligned with consensus validation around `pre_sp_tx_block_height`, hard-fork gates, V1 phase-out, V2 plot activation/strength, generator reference bans, transaction-block reward maturity, and post-HF2 MMR commitments. A generated block that looks locally coherent can still be rejected by full-node validation if these gates diverge.
|
||||
|
||||
## Plot, Key, And Signature State
|
||||
|
||||
`BlockTools` owns an ephemeral root path, config, keychain, plot directory, `PlotManager`, expected plot IDs, and local key caches. In automated tests it creates config and SSL under temp roots, forces the test network, normalizes localhost to loopback, assigns free daemon/RPC/service ports, and uses pre-generated cert pools to avoid expensive cert generation.
|
||||
|
||||
Plot setup creates deterministic V1 and V2 plots with stable filenames and test private keys. `expected_plots` is the invariant checked after refresh; refresh callbacks assert batch accounting and duration. Deleted or excluded plots must update this set or refresh assertions will fail.
|
||||
|
||||
Signatures are produced from plot memo-derived local keys plus farmer/pool keys:
|
||||
|
||||
- plot signatures aggregate local, farmer, and optional taproot shares;
|
||||
- pool signatures are only returned for pool public-key plots;
|
||||
- `setup_keys()` can use simulator config fingerprint/reward address or synthesize farmer/pool keys in a local keychain.
|
||||
|
||||
Because locked-memory key material is expensive, plot local keys are cached by plot ID. Changes around `parse_plot_info()`, V1/V2 plot metadata, or keychain patching can break both block farming and harvester/farmer integration tests.
|
||||
|
||||
## Transaction And Wallet Helpers
|
||||
|
||||
There are two transaction paths:
|
||||
|
||||
- Low-level block inclusion: pass `transaction_data` to `BlockTools.get_consecutive_blocks()` or use `WalletTool` to build simple signed spends for fabricated chain tests.
|
||||
- Full-node/mempool path: use wallet action scopes or spend bundles, wait for mempool inclusion through `FullNodeSimulator`, then farm a guaranteed transaction block.
|
||||
|
||||
`FullNodeSimulator` helpers are behavior-level conveniences:
|
||||
|
||||
- `wait_transaction_records_entered_mempool()` polls the full node mempool and can fail fast if a wallet transaction is marked invalid after retries.
|
||||
- `process_transaction_records()`, `process_spend_bundles()`, and `process_coin_spends()` farm transaction blocks until expected additions exist in the coin store.
|
||||
- `farm_blocks_to_wallet()` intentionally farms extra transaction blocks because block rewards are only claimable by later transaction blocks. It verifies the exact expected coinbase coin count before waiting for wallet spendability.
|
||||
- `create_coins_with_amounts()` groups wallet-generated outputs, avoids duplicate puzzle-hash/amount pairs when necessary, farms one transaction block, filters change, and waits for wallet visibility.
|
||||
|
||||
Wallet sync checks require three states to converge: the simulator's full node reports synced, the wallet state manager reports synced, and retry-store states are exhausted. Height equality alone is not sufficient.
|
||||
|
||||
## RPC Surface
|
||||
|
||||
`SimulatorFullNodeRpcApi` extends `FullNodeRpcApi` with simulator-only endpoints:
|
||||
|
||||
- `/farm_block`
|
||||
- `/set_auto_farming`
|
||||
- `/get_auto_farming`
|
||||
- `/get_farming_ph`
|
||||
- `/get_all_blocks`
|
||||
- `/get_all_coins`
|
||||
- `/get_all_puzzle_hashes`
|
||||
- `/revert_blocks`
|
||||
- `/reorg_blocks`
|
||||
|
||||
The client wrapper converts these to typed Python calls and reuses the normal full-node RPC transport. Address inputs are decoded as testnet addresses; `farm_block()` returns a computed new peak height, while the actual block acceptance still happens through simulator/full-node methods.
|
||||
|
||||
Autofarm is implemented by installing `FullNode.simulator_transaction_callback = FullNodeSimulator.autofarm_transaction`. When enabled, accepted mempool transactions trigger a transaction-block farm to the simulator farming puzzle hash. The setting is persisted back into `config.yaml`.
|
||||
|
||||
## Service Harness
|
||||
|
||||
`setup_services.py` is the service-factory layer used by tests. It creates real `Service[...]` instances for full nodes, simulator nodes, wallets, farmer, harvester, timelord, introducer, crawler, seeder, daemon, and solver with test config mutations.
|
||||
|
||||
Key setup contracts:
|
||||
|
||||
- `setup_full_node(..., simulator=True)` returns a `SimulatorFullNodeService`; otherwise it creates a normal `FullNodeService`.
|
||||
- Most service ports are set to request ephemeral binding or assigned via `find_available_listen_port()` to avoid collisions. The helper only reserves recently returned ports inside the current process; it is not a cross-process lock.
|
||||
- In-memory DB URIs are used unless `reuse_db=True`; DB version tables may be pre-created for migration tests.
|
||||
- Test full nodes disable introducers/DNS by default, reduce peer connect churn, reserve all but one logical core, and force simulator autofarm/current-time off for deterministic tests.
|
||||
- Capability overrides preserve `BASE`, allow explicit disables, and then force important modern capabilities (`HARD_FORK_2`, `RATE_LIMITS_V3`) on in test setup.
|
||||
|
||||
`chia/_tests/util/setup_nodes.py` builds higher-level pytest environments on top of this module. `setup_simulators_and_wallets*()` creates one `BlockTools` per simulator and additional `BlockTools` if wallets outnumber simulators, then returns `FullNodeEnvironment`/`WalletEnvironment` wrappers plus the first `BlockTools`.
|
||||
|
||||
## Local Simulator Startup
|
||||
|
||||
`start_simulator.async_main()` is the user-facing simulator startup path. It loads config, extracts simulator fingerprint, farming address, and plot directory, forces `simulator.use_current_time=True`, creates `BlockTools`, sets up keys and a small plot set, initializes logging, and returns/runs a full-node service whose peer API is `FullNodeSimulator` and whose RPC API is `SimulatorFullNodeRpcApi`.
|
||||
|
||||
This path is intentionally close to a normal full-node service after `BlockTools` setup. If a change only belongs to the CLI/local simulator experience, keep it out of test fixtures unless deterministic tests also need it.
|
||||
|
||||
## Small Support Modules
|
||||
|
||||
- `WalletTool` is a minimal standard-puzzle wallet for test spend creation. It derives child keys, tracks puzzle-hash-to-key mappings, creates announcement-linked multi-coin spends, computes change, and signs all relevant aggregate-signature opcodes. It is not wallet state management.
|
||||
- `vdf_prover.get_vdf_info_and_proof()` calls `chiavdf.prove()` directly and wraps the output as `VDFInfo`/`VDFProof`. Simulator block creation depends on this being structurally identical to validation expectations, even with tiny test discriminants.
|
||||
- `TempKeyring` patches `KeyringWrapper` and `supports_os_passphrase_storage()` so tests use isolated file keyrings. Cleanup restores the old shared keyring root when needed.
|
||||
- `ssl_certs*.py` are pre-generated test collateral. `ssl_certs.py` cycles through ten CA/node-cert sets and warns if a set is reused while still marked in use.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Do not bypass full-node validation for convenience unless the API is explicitly destructive test state manipulation. Most simulator bugs become false-positive tests.
|
||||
- `get_consecutive_blocks()` has many consensus mirror points. Edits around overflow, sub-epoch summaries, MMR, `pre_sp_tx_block_height`, V1/V2 plot eligibility, transaction-generator cost, or reward maturity need consensus tests across hard-fork modes.
|
||||
- `BlockTools` caches block records and MMR state between calls when the input tip matches. Cache invalidation is by header hash; mutating a passed block list or reusing stale `BlockTools` state can produce surprising chains.
|
||||
- `WalletTool` has mutable class attributes noted by a TODO. Tests that share instances or assume pristine address/key lookup state should be treated carefully.
|
||||
- `revert_block_height()` intentionally leaves network peers and wallets uninformed. Use reorg helpers for wallet/full-node behavior and revert only for local chain reset scenarios.
|
||||
- Test setup mutates nested config dicts in place before saving. Passing shared config objects across services can leak simulator-specific settings unless callers copy or isolate roots.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Block generation and deterministic plot/key setup: `chia/simulator/block_tools.py`.
|
||||
- Simulator full-node API and RPC routes: `chia/simulator/full_node_simulator.py`, `chia/simulator/simulator_full_node_rpc_api.py`, `chia/simulator/simulator_full_node_rpc_client.py`.
|
||||
- Local simulator startup: `chia/simulator/start_simulator.py`.
|
||||
- Test service factories: `chia/simulator/setup_services.py`, `chia/_tests/util/setup_nodes.py`.
|
||||
- Minimal wallet spend helper: `chia/simulator/wallet_tools.py`.
|
||||
@@ -0,0 +1,87 @@
|
||||
# Chia Solver Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/solver/` is a small service boundary for v2 plot solving. It receives `SolverInfo` messages from farmers, turns Rust-backed `PartialProof` fragments into full proof bytes via `chia_rs.solve_proof()`, and returns `SolverResponse` messages that the farmer folds back into normal proof-of-space processing. It is not a chain, plot, key, or farmer-state authority.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for V2 partial-proof solving, solver peer admission, farmer `connect_to_solver`, solver protocol payloads, and farmer response correlation. For plot lookup and partial-proof production, read `harvester.md` and `plotting.md`; for final proof handling, read `farmer.md`.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `Solver` owns only process-local runtime state: root/config, service lifecycle flags, a `ThreadPoolExecutor`, consensus constants, and the attached `ChiaServer`.
|
||||
- `SolverAPI` is the peer API. Its only semantic endpoint is `solve`, registered for `ProtocolMessageTypes.solve` and expected to reply with `solution_response`.
|
||||
- `SolverRpcApi` is local observability only. It exposes `get_state` with `started`; it does not configure peers, submit work, or expose proof data.
|
||||
- Farmer remains the coordinator. It validates signage-point context, tracks `pending_solver_requests`, broadcasts solve requests to solver peers, reconstructs `ProofOfSpace`, and then re-enters `new_proof_of_space()`.
|
||||
- Harvester remains the plot authority. V2 harvesters supply partial proofs plus plot metadata; the solver never opens plot files or verifies plot ownership.
|
||||
- Full node and consensus remain the final acceptance authority. A solver-produced proof is still just proposed proof material until the farmer/full node/consensus path verifies and uses it.
|
||||
|
||||
## Why This Is Tricky
|
||||
|
||||
Public farmer RPC docs present the solver as turning V2 partial proofs into full proofs. The important source-level motivation is separation of concerns: harvesters keep local plot/prover ownership, farmers keep signage-point and harvester correlation, solvers perform the expensive proof expansion, and full-node/consensus still validate the final proof. That makes solver throughput and response handling a farmer hot path even though the solver service itself is intentionally small.
|
||||
|
||||
## Wrong Assumptions To Avoid
|
||||
|
||||
- Do not make solver own plot files, farmer keys, pool state, or signage-point validity.
|
||||
- Do not treat a solver response as matched by websocket request id; farmer handles it as an inbound protocol message.
|
||||
- Do not assume no response is harmless; farmer pending state and cache cleanup decide how long unresolved partial proofs remain.
|
||||
- Do not add large solver payloads without revisiting protocol and rate-limit ownership in `chia/protocols/` and `chia/server/`.
|
||||
|
||||
## Main Runtime Flow
|
||||
|
||||
- Harvester finds v2 partial proofs for a signage point and sends `PartialProofsData` to farmer.
|
||||
- Farmer rejects the data if the `sp_hash` is not in its short-lived signage-point cache.
|
||||
- For each partial proof, farmer stores pending request metadata keyed by `bytes(partial_proof)`: original `PartialProofsData` and originating harvester peer.
|
||||
- Farmer broadcasts `ProtocolMessageTypes.solve` with `SolverInfo(partial_proof, plot_id, strength, plot_size)` to all current solver connections.
|
||||
- Solver receives `solve`, checks `solver.started`, calls `solve_proof(partial_proof, plot_id, strength, size, constants.TESTNET)`, and returns `SolverResponse(partial_proof, proof)` if a proof is produced.
|
||||
- Farmer accepts `solution_response` only if the `partial_proof` matches a pending request, drops empty proofs, reconstructs a v2 `ProofOfSpace`, and calls `new_proof_of_space()` with the original harvester peer.
|
||||
|
||||
The request correlation key is the serialized partial proof only. If identical partial proofs can appear concurrently from different harvester/signage contexts, later requests overwrite earlier metadata. Treat changes to this keying as farmer/solver correctness-sensitive.
|
||||
|
||||
## Network And Protocol Contract
|
||||
|
||||
- `NodeType.SOLVER` has its own protocol version and normal shared capabilities.
|
||||
- Sender authorization is asymmetric: farmers may send `solve`; solvers may send `solution_response`.
|
||||
- `SolverAPI.solve()` advertises `solution_response` as a reply type, but the farmer sends `solve` through broadcast rather than request-id `call_api()` matching. This means the solver response is handled as an inbound message, not as request-id state-machine matching.
|
||||
- `solve` and `solution_response` are expected to remain small, rate-limited payloads. Do not add large fields to these schemas without updating rate-limit assumptions.
|
||||
- `SolverInfo` and `SolverResponse` are `Streamable` wire contracts backed by `PartialProof`, `bytes32`, `uint8`, and raw `bytes`. Schema edits require the usual protocol lockstep: message types, sender map, API/stub metadata, rate limits, compatibility tests, and protocol version considerations.
|
||||
|
||||
## Lifecycle, Config, And Peer Admission
|
||||
|
||||
- `start_solver.create_solver_service()` follows the standard Chia service composition: load selected-network constants, build `Solver`, `SolverAPI`, optional RPC, then wrap in `Service` as `NodeType.SOLVER`.
|
||||
- Solver peer and RPC ports are configured through the solver/farmer config. Farmer config defaults to a local solver peer.
|
||||
- Solver config defaults to `trusted_peers_only: True`; `on_connect()` accepts trusted peers and rejects untrusted peers unless that flag is disabled.
|
||||
- `num_threads` controls the solver executor size. The current `solve()` implementation calls `solve_proof()` synchronously from the API handler rather than scheduling work on the executor, so changing solver throughput or blocking behavior requires checking API task latency and event-loop impact.
|
||||
- Shutdown sets `_shut_down`, marks service shutdown through `manage()`, and shuts down the executor with `wait=True`.
|
||||
|
||||
## Cross-Module Coupling
|
||||
|
||||
- `chia/farmer/` is the only meaningful runtime consumer. Farmer RPC also exposes `connect_to_solver`, which closes existing solver connections before opening a requested solver peer.
|
||||
- `chia/harvester/` drives solver demand indirectly by emitting v2 `PartialProofsData`; v1 plots bypass the solver entirely and return full proofs directly.
|
||||
- `chia/protocols/` owns `SolverInfo`, `SolverResponse`, message IDs, sender authorization, and shared protocol versioning.
|
||||
- `chia/server/` owns TLS identity, handshake-derived node type, rate limits, peer filtering, broadcast validation, and API dispatch. Solver code should not bypass these gates.
|
||||
- `chia_rs` owns the actual proof-solving algorithm. Python solver behavior is mostly orchestration, error handling, and boundary enforcement around `solve_proof()`.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- `constants.TESTNET` is passed into `solve_proof()`. Any network/fork behavior change needs confirmation that this flag is the intended Rust API input for mainnet and testnet configurations.
|
||||
- The solver logs partial proof fragments and plot IDs. Avoid raising log volume or including sensitive local metadata in hot signage-point paths.
|
||||
- Returning `None` from `SolverAPI.solve()` silently produces no response to the farmer. Farmer-side pending requests are only cleaned up on send failure or matching response; absence of a response can leave entries until surrounding farmer cache cleanup handles related state.
|
||||
- Because farmer broadcasts to all solvers, multiple solvers may race to answer the same partial proof. The first matching response pops the pending request; later valid responses are logged as unknown and dropped.
|
||||
- `SolverRpcApi._state_changed()` returns no websocket events. UI/daemon expectations should treat solver RPC as polling-only unless event support is added deliberately.
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- Solver unit/service tests should cover readiness gating, successful `SolverResponse` construction, `solve_proof()` failure returning no message, and service shutdown of executor resources.
|
||||
- Farmer/solver integration tests should cover pending request insertion, unknown response drop, empty proof cleanup, successful reconstruction into `NewProofOfSpace`, solver send exceptions, and multiple solver responses for one partial proof.
|
||||
- Protocol tests should cover serialization of solver messages and static sender-map/reply metadata invariants when schemas or message IDs change.
|
||||
- Lifecycle/RPC tests should verify default/configured solver peer connection, trusted-peer admission, `get_state`, and reconnect behavior through farmer `connect_to_solver`.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Solver service/API/RPC: `chia/solver/solver.py`, `chia/solver/solver_api.py`, `chia/solver/solver_rpc_api.py`, `chia/solver/start_solver.py`.
|
||||
- Farmer-side request correlation: `chia/farmer/farmer.py`, `chia/farmer/farmer_api.py`.
|
||||
- Harvester partial-proof production: `chia/harvester/harvester_api.py`, `chia/plotting/prover.py`.
|
||||
- Wire contracts and sender authorization: `chia/protocols/solver_protocol.py`, `chia/protocols/protocol_message_types.py`, `chia/protocols/protocol_state_machine.py`.
|
||||
- Proof expansion implementation boundary: `chia_rs.solve_proof`.
|
||||
@@ -0,0 +1,62 @@
|
||||
# Chia SSL Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/ssl/` is the certificate material and permission hygiene layer for Chia's local config root. It does not build network connections itself; it provisions trust roots and node certificates that `chia/server/`, `chia/rpc/`, and `chia/daemon/` turn into mutual-TLS contexts.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for certificate generation, public/private CA material, SSL file permissions, Chia root SSL layout, daemon/RPC certificate trust, and service certificate provisioning. For connection-time TLS behavior, also read `server.md`.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `create_ssl.py` is the provisioning authority. It creates `config/ssl/`, installs the bundled Chia public CA, creates or imports the user's private CA, and generates per-service private/public cert/key pairs under the paths expected by `config.yaml`.
|
||||
- `ssl_check.py` is the filesystem-permission authority. It discovers configured cert/key paths, verifies platform-supported POSIX modes, reports unsafe files, and implements `chia init --fix-ssl-permissions`.
|
||||
- The bundled `chia_ca.crt`/`chia_ca.key` are the public peer-network CA material. The generated `private_ca.crt`/`private_ca.key` are local/user trust material used for authenticated service relationships and daemon/RPC access.
|
||||
- `chia/server/server.py`, not `chia/ssl/`, decides which CA/cert pair is used for each node type and calculates peer/node ids from the certificate fingerprint.
|
||||
|
||||
## Trust Roots And Certificate Classes
|
||||
|
||||
- There are two parallel trust domains. Public certs are signed by the bundled Chia CA and are used for ordinary public peer connections. Private certs are signed by the user's private CA and are used where the peer must be one of the user's trusted services.
|
||||
- Private certs are generated for `full_node`, `wallet`, `farmer`, `harvester`, `timelord`, `crawler`, `data_layer`, `daemon`, and `solver`. Public certs are generated for `full_node`, `wallet`, `farmer`, `introducer`, `timelord`, `data_layer`, and `solver`.
|
||||
- `ChiaServer.create()` encodes the operational matrix: harvesters use private certs as clients; harvester/farmer/wallet/data_layer servers require private-CA client certs; other peer servers use the public Chia CA. Changing generation lists or config paths without updating this matrix can silently break service startup or peer admission.
|
||||
- Daemon and RPC traffic use daemon private certs with the private CA. This is a local/admin trust boundary, distinct from public peer TLS.
|
||||
- Peer identity is the SHA-256 fingerprint of the TLS certificate selected by the server layer. Regenerating a service cert changes its node id and affects duplicate/self-connection detection.
|
||||
|
||||
## Provisioning Contracts
|
||||
|
||||
- `create_all_ssl(root_path)` assumes the default config path convention under `root_path / "config" / "ssl"`. It removes legacy `trusted.key`/`trusted.crt`, ensures `ssl/` and `ssl/ca/`, writes the bundled Chia CA, then generates private and public node certs.
|
||||
- Passing `private_ca_crt_and_key` imports an existing private CA before node cert generation. If private CA files already exist on disk, they are reused. If either private CA file is missing, a new CA is created and all private node certs are regenerated from it.
|
||||
- Public node cert generation uses `overwrite=False`; existing public cert/key pairs survive normal `create_all_ssl()` runs. Private generation follows the caller's `overwrite` argument, defaulting to replacement.
|
||||
- `node_certs_and_keys` is an injection hook for tests and packaged fixtures. It is keyed by node name then cert prefix (`"private"`/`"public"`) and bypasses generated cert creation only when both `"crt"` and `"key"` are present.
|
||||
- `write_ssl_cert_and_key()` unlinks before replacing and opens files with `O_CREAT | O_EXCL` plus explicit modes. Preserve this pattern when changing writes; SSL key material must not briefly inherit permissive umask-derived modes.
|
||||
|
||||
## Permission Model
|
||||
|
||||
- Default cert mode is `0o644`; default key mode is `0o600`. Certs may be world-readable but must not be group/other writable or executable. Keys must not grant any group/other permissions.
|
||||
- Permission checks are skipped on Windows/Cygwin because ACL support is not implemented. Do not treat a successful no-op on those platforms as proof that files are protected.
|
||||
- `get_all_ssl_file_paths()` reads configured paths from `config.yaml` with `fill_missing_services=True`, then appends the bundled Mozilla CA cert. Missing config keys are reported but do not abort the scan.
|
||||
- Missing files are ignored by permission verification because nonexistent files cannot be dangerously permissive. Startup failures for missing certs occur later when TLS contexts load cert chains.
|
||||
- `check_ssl()` warns but does not exit; `fix_ssl()` attempts `chmod` on files reported by the same verifier and reports whether anything changed or failed.
|
||||
|
||||
## Consumer Coupling
|
||||
|
||||
- `chia.server.ssl_context` is only a path resolver. It mirrors the nested config layout and returns root-relative paths for public/private service certs and CAs.
|
||||
- `ssl_context_for_server()` and `ssl_context_for_client()` live in `chia/server/server.py`. They call `verify_ssl_certs_and_keys()`, load cert chains, require peer certificates, disable hostname checking, and use CA files to validate the opposite side.
|
||||
- Server contexts require the source-defined default TLS minimum. The daemon may explicitly loosen its context via `daemon_allow_tls_1_2`; that exception belongs to daemon-local compatibility, not the general peer TLS policy.
|
||||
- `chia_init()` creates default config before `create_all_ssl()`. Migration and `chia init -c <ca-dir>` copy CA files into `config/ssl/ca`, fix copied file permissions, and rerun certificate provisioning.
|
||||
- Tests commonly generate ad hoc certs with `generate_ca_signed_cert()` and then select `private_ssl_ca_paths()` or `chia_ssl_ca_paths()` to assert which trust domain should accept the connection.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Highest-risk edits: changing node-name lists, altering public/private overwrite behavior, changing config key paths, broadening key permissions, or moving CA selection logic into `chia/ssl/` without updating `ChiaServer.create()`.
|
||||
- Regenerating the private CA invalidates every private cert issued by the old CA. This can break farmer/harvester, daemon/RPC, wallet, and data-layer trust relationships across machines.
|
||||
- Public CA material is bundled in source and written into user config. Treat changes to `chia_ca.crt`/`chia_ca.key` as network-wide compatibility changes, not local initialization details.
|
||||
- The generated end-entity cert subject/SAN is generic (`Chia`, `chia.net`) and hostname verification is disabled in Chia's peer contexts. Security depends on CA trust and certificate fingerprint identity, not DNS-name matching.
|
||||
- Permission path coverage is hand-maintained in `CERT_CONFIG_KEY_PATHS` and `KEY_CONFIG_KEY_PATHS`. Adding new SSL config entries requires adding them there or `check_ssl`/`fix_ssl` will not protect them.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Certificate and CA provisioning: `chia/ssl/create_ssl.py`.
|
||||
- SSL file permission checks and repair: `chia/ssl/ssl_check.py`.
|
||||
- Connection-time SSL context selection: `chia/server/server.py`, `chia/server/ssl_context.py`.
|
||||
@@ -0,0 +1,129 @@
|
||||
# chia-tests-root
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
Scope: `chia/_tests/`. This is distilled architectural context for future audit or implementation agents. It focuses on the shared test harness and cross-suite contracts rather than inventorying individual test files.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for shared pytest fixtures, consensus-mode parametrization, persistent chain artifacts, simulator/full-node/wallet topologies, keyring isolation, async test conventions, and CI test partitioning.
|
||||
|
||||
## Module Role
|
||||
|
||||
`chia/_tests/` is the executable specification for the Python node implementation. It spans consensus validation, full-node sync, mempool policy, wallet behavior, data-layer lifecycle, service wiring, farmer/harvester/timelord flows, command handlers, serialization, DB migrations, simulator behavior, and CI workflow generation.
|
||||
|
||||
The root module is not a passive test directory. Its shared fixtures define the meaning of many lower-level tests:
|
||||
|
||||
- consensus constants and hard-fork activation modes;
|
||||
- deterministic block and plot generation through `BlockTools`;
|
||||
- persistent block artifact lookup and regeneration;
|
||||
- in-memory blockchain/store construction;
|
||||
- simulator/full-node/wallet/service topologies;
|
||||
- keyring isolation and localhost/network assumptions;
|
||||
- async convergence, benchmark, and runtime instrumentation;
|
||||
- CI job partitioning and resource expectations.
|
||||
|
||||
When changing tests under this root, treat fixture edits as architecture changes. A small change in `chia/_tests/conftest.py`, `chia/_tests/util/setup_nodes.py`, `chia/_tests/connection_utils.py`, or wallet/data-layer conftests can alter the semantics of hundreds of tests.
|
||||
|
||||
## Consensus-Mode Contract
|
||||
|
||||
Most tests that depend on `blockchain_constants` run against five `ConsensusMode` values:
|
||||
|
||||
- plain pre-hard-fork behavior;
|
||||
- hard-fork mode with lowered activation and plot-filter reduction heights;
|
||||
- soft-fork mode with hard/soft fork heights forced to activation;
|
||||
- hard-fork mode with lowered V2-plot difficulty;
|
||||
- hard-fork mode after V1 plot phase-out.
|
||||
|
||||
This parametrization is one of the suite's strongest safeguards. It catches assumptions that only hold before or after a fork, especially around CLVM flags, canonical generator rules, plot eligibility, header commitments, transaction-block context, and cached block fixtures.
|
||||
|
||||
`@pytest.mark.limit_consensus_modes` is enforced during collection. It only works on tests that actually have the `consensus_mode` fixture; misuse fails collection rather than silently narrowing coverage. Use it to control runtime or isolate fork-specific behavior, not to hide mode-sensitive failures.
|
||||
|
||||
Persistent block fixtures are versioned by `test_chain_suffix(consensus_mode)`. `default_400_blocks`, `default_1000_blocks`, `default_10000_blocks`, long-reorg chains, compact chains, and fork-height variants are expected to match the active constants. If a new persistent chain is added, `test_build_chains.py` is part of the change so cached artifacts remain auditable.
|
||||
|
||||
## Harness Authority Boundaries
|
||||
|
||||
`BlockTools.get_consecutive_blocks()` is the authority for deterministic block shape. Use it when the test needs exact block anatomy: overflow, sub-slot boundaries, transaction-block guarantees, generator references, fork/reorg structure, VDF normalization, hard-fork commitments, or malformed variants.
|
||||
|
||||
`FullNodeSimulator` farming APIs are the authority for behavior-level lifecycle tests. Use them when exact block internals are irrelevant and the target contract is mempool admission, block confirmation, wallet sync, RPC state, or data-layer update visibility.
|
||||
|
||||
`chia/_tests/util/blockchain.py:create_blockchain()` creates a minimal in-memory production `Blockchain` with real `CoinStore`, `BlockStore`, `BlockHeightMap`, and `InlineExecutor`. It is the right base for isolated consensus/store tests where full service setup would obscure the invariant.
|
||||
|
||||
`chia/_tests/util/setup_nodes.py` is the service-topology layer. It builds real `Service` instances for full nodes, simulator nodes, wallets, farmers, harvesters, timelords, introducers, crawlers, seeders, daemons, and solvers. These are not mocks; tests using them are crossing process, RPC, websocket, DB, or peer-protocol boundaries.
|
||||
|
||||
`wallet_environments` is a higher-level wallet integration contract. It creates one simulator full node plus N wallets, connects wallets to the node, configures trusted/untrusted mode, opens wallet/full-node RPC clients, optionally prefarms rewards, and returns a `WalletTestFramework`. It also multiplies tests across trusted/untrusted full-node sync and reuse/new-puzzle-hash transaction modes unless explicitly pinned.
|
||||
|
||||
## Shared Async And Network Contracts
|
||||
|
||||
`time_out_assert()` is the standard convergence primitive. It records structured timeout metadata via `ether.record_property`, applies adjusted timeouts, and polls at short intervals. Use it for node height, peer tables, wallet sync, mempool contents, logs, ban state, and service readiness. Raw sleeps are acceptable only for narrow process/socket settle cases where no observable condition exists yet.
|
||||
|
||||
`connection_utils.py` constructs real websocket peers with generated certs and performs protocol handshakes. Dummy connections default to modern `HARD_FORK_2` and `RATE_LIMITS_V3` capabilities, which means protocol and rate-limit tests may be asserting post-fork behavior even when the peer is synthetic.
|
||||
|
||||
`self_hostname` is pinned to loopback. Tests that exercise ban logic may patch localhost exemptions; do not generalize those cases to normal local peer behavior without checking the patch.
|
||||
|
||||
Keyrings are isolated by autouse/root fixtures. Test code must not prompt for production keyring passphrases. Use `TempKeyring`-backed fixtures for keys and avoid sharing key roots between service topologies.
|
||||
|
||||
## Behavioral Test Regions
|
||||
|
||||
`blockchain/` is the consensus block-acceptance spec. It constructs real blocks and pushes them through production validation/add paths, with exact `AddBlockResult` and `Err` expectations. Shared `ForkInfo`, `AugmentedBlockchain`, generator-reference lookup, hard-fork commitments, and persistent block cache checks are central contracts.
|
||||
|
||||
`core/` is the broad integration boundary for full-node orchestration, mempool policy, server/protocol safety, RPC, stores, data layer, daemon/service startup, and custom type invariants. It often depends more on shared fixtures and topology semantics than on local helper code.
|
||||
|
||||
`wallet/` is the wallet behavior spec. Its own conftest installs autouse patches that shortcut consensus-heavy block validation and replace normal `BlockTools` with `WalletBlockTools` unless a test is marked `standard_block_tools`. This is intentional: default wallet tests focus on mempool acceptance, coin-state notifications, and wallet DB updates rather than full consensus validity.
|
||||
|
||||
`simulation/`, `harvester/`, `farmer_harvester/`, `timelord/`, `solver/`, `plot_sync/`, and `pools/` are service-flow suites. They validate real service wiring, plots, signage points, VDF/solver interactions, farming/harvesting protocols, pool wallet flows, and sync behavior. These tests are sensitive to ports, temp roots, pre-generated plots, and service cleanup.
|
||||
|
||||
`cmds/` tests usually patch RPC client factories through `get_test_cli_clients` and a temp config root. They are command-surface tests, not service integration tests, unless they explicitly use the root `chia_root` subprocess helpers.
|
||||
|
||||
`util/`, `clvm/`, `generator/`, `fee_estimation/`, `db/`, and custom-type tests are mostly fast deterministic guards for serialization, CLVM execution, cost accounting, DB wrappers, cache behavior, network protocol files, and helper APIs. Prefer local assertions here instead of service topology.
|
||||
|
||||
## Cross-Subsystem Correlation
|
||||
|
||||
These per-cluster details are not obvious from conftest or individual test files:
|
||||
|
||||
- Farmer proof flow is correlation-heavy: `sps`, `proofs_of_space`, `quality_str_to_identifiers`, `number_of_responses`, `cache_add_time`, and `pending_solver_requests` must agree across async harvester, full-node, pool, and solver messages.
|
||||
- Plotting and plot-sync tests protect the harvester-to-farmer inventory state machine: `(sync_id, message_id)` ordering, dropped/delayed/duplicated responses, quarantine behavior, and reset/retry.
|
||||
- Pool tests span config/CLI parsing, CLVM pool puzzle lifecycle, wallet pool store, `plotnft` commands, singleton identity, trusted/untrusted wallet sync, and reorg/revert.
|
||||
- Daemon tests: websocket registration, keychain proxy, and do not apply P2P message/rate-limit assumptions to daemon JSON traffic.
|
||||
- RPC structured error tests intentionally preserve both legacy `error` strings and newer `structuredError` payloads.
|
||||
- Directory config files affect runtime shape: moving tests between core subdirectories can change checked-out blocks/plots, parallelism, CI timeout behavior, and consensus-mode coverage.
|
||||
- DB wrapper behavior underpins full-node, wallet, and DataLayer stores. Reader transaction visibility, WAL mode, savepoint rollback, and foreign-key delay semantics are infrastructure contracts.
|
||||
|
||||
## CI And Test Partitioning
|
||||
|
||||
`chia/_tests/README.md` and `testconfig.py` define CI job generation. Treat `testconfig.py` as the source of truth for default settings, with the README as process guidance for regenerating workflows. Test files are discovered by `test_*.py`; each subdirectory below configured root test dirs becomes a workflow matrix job. Parent-directory jobs do not include subdirectory tests, and subdirectory jobs do not include parent tests.
|
||||
|
||||
The default CI settings are intentionally conservative: tests can run in parallel by default, block/plot checkout and timelord installation are disabled by default, and per-directory `config.py` files opt heavy suites into block artifacts, timelord install, lower parallelism, or longer job timeouts.
|
||||
|
||||
Moving or adding test files can require running workflow generation, even when no test contents changed. Changing only test contents does not require workflow regeneration.
|
||||
|
||||
Heavy suites opt into longer timeouts or artifacts for a reason. `blockchain`, `core/full_node`, `core/mempool`, wallet asset suites, `simulation`, `harvester`, `solver`, `timelord`, and persistent-chain users often rely on cached blocks/plots or long-running service flows. Do not collapse their CI config into defaults without measuring runtime and artifact needs.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Fixture-scope changes are high blast radius. `bt` is session-scoped and expensive; service fixtures are function-scoped for isolation; wallet autouse patches are function-scoped to keep consensus shortcuts local to wallet tests.
|
||||
- Consensus-mode narrowing can hide fork regressions. If a test is mode-limited, the reason should be cost or explicit fork relevance.
|
||||
- Persistent block artifacts use pickle-backed bytes and file locks under the default root's sibling `blocks` directory. In CI, missing expected artifacts fail instead of regenerating.
|
||||
- Service tests mutate config dictionaries and temp roots. Reusing mutable config objects or roots can leak ports, daemon settings, trusted peers, autofarm flags, or single-threaded settings between tests.
|
||||
- Wallet tests are not always consensus-valid by default. Use `standard_block_tools` when testing header validation, weight proofs, BIP158/additions/removals proofs, or wallet protocol behavior that relies on real block structure.
|
||||
- A final height, balance, or RPC response is usually too weak for async lifecycle tests. Assert the intermediate boundary that matters: advertised, fetched, admitted to mempool, farmed, evicted, synced, persisted, or rolled back.
|
||||
- Broad `pytest.raises(Exception)`, raw sleeps, non-empty collection checks, and topology-heavy tests for pure policy are weak signals. Tighten them when touching nearby code.
|
||||
|
||||
## Change Guidance
|
||||
|
||||
Preserve layered assertions. Good tests show the subsystem boundary being protected, not just the final observable state.
|
||||
|
||||
When adding or moving tests, check the local directory's `config.py`, root `testconfig.py` defaults, and README workflow-generation rules. Runtime, artifact checkout, timelord install, and parallelism are part of the architecture.
|
||||
|
||||
## Verification Guidance
|
||||
|
||||
For root fixture changes, include representative downstream coverage from each affected topology: a pure consensus/blockchain test, an isolated mempool or store test, a single full-node/simulator test, a wallet-environment test if wallet fixtures changed, and a service/network test if setup or connection helpers changed.
|
||||
|
||||
For workflow-layout changes, update generated workflows as described by `chia/_tests/README.md`. For persistent block fixture changes, include the relevant `test_build_chains.py` coverage and expect artifact/runtime impact.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Root fixtures and test configuration: `chia/_tests/conftest.py`, `chia/_tests/README.md`, `chia/_tests/testconfig.py`.
|
||||
- Shared topology and utility layers: `chia/_tests/util/setup_nodes.py`, `chia/_tests/util/blockchain.py`, `chia/_tests/connection_utils.py`.
|
||||
- Wallet environment harness: `chia/_tests/environments/wallet.py`, `chia/_tests/wallet/conftest.py`.
|
||||
- CI matrix generation: `chia/_tests/build-job-matrix.py`.
|
||||
@@ -0,0 +1,127 @@
|
||||
# chia-tests-blockchain
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
Scope: `chia/_tests/blockchain/`. This is distilled architectural context for future audit or implementation agents. It intentionally omits test inventory and obvious fixture descriptions.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for consensus/blockchain tests, block prevalidation/add-block helpers, fork/reorg tests, generator-reference tests, hard-fork commitment tests, and persistent block-fixture validation.
|
||||
|
||||
## Module Role
|
||||
|
||||
These tests are the executable regression spec for consensus block acceptance. They sit below full-node sync orchestration but above pure consensus helpers: most tests construct real `FullBlock` objects with `BlockTools`, pass them through production pre-validation/body-validation/add paths, and assert the exact `AddBlockResult` or `Err` that protects chain-state invariants.
|
||||
|
||||
The suite is not only checking small helpers. It validates that production sequencing remains coherent across:
|
||||
|
||||
- finished header validation and VDF/POS/slot rules;
|
||||
- block body validation, CLVM condition handling, and coin-store transitions;
|
||||
- fork choice, reorg replay, and `ForkInfo` accounting;
|
||||
- `AugmentedBlockchain` overlay behavior used by batch validation;
|
||||
- generator-ref lookup across canonical and fork branches;
|
||||
- hard-fork commitment behavior for header MMR and sub-epoch challenge roots.
|
||||
|
||||
## Primary Harness Contract
|
||||
|
||||
`blockchain_test_utils._validate_and_add_block()` is the central abstraction. Treat it as a miniature full-node block-add pipeline, not as a loose assertion helper:
|
||||
|
||||
- It wraps the target `Blockchain` in `AugmentedBlockchain` unless the caller supplies a shared overlay.
|
||||
- It computes current SSI/difficulty using `get_next_sub_slot_iters_and_difficulty(...)`.
|
||||
- It finds the previous sub-epoch summary block for validation state.
|
||||
- It runs `pre_validate_block(...)` unless `skip_prevalidation=True`.
|
||||
- It passes the prevalidated overlay `BlockRecord` into `Blockchain.add_block(...)`.
|
||||
- It checks the `BlockStore` main-chain invariant before and after each add.
|
||||
|
||||
`skip_prevalidation=True` is deliberate in malformed-body tests where the mutation would fail before the body code under test. In that mode, the helper fabricates `PreValidationResult` and signature conditions, so the test is no longer exercising header/CLVM prevalidation.
|
||||
|
||||
## State Invariants Under Test
|
||||
|
||||
The suite repeatedly defends these consensus invariants:
|
||||
|
||||
- Canonical chain has exactly one `in_main_chain` block at each height from genesis to peak.
|
||||
- Peak selection follows weight first, then lower `total_iters` on equal weight.
|
||||
- Main-chain extension and fork validation require coherent `ForkInfo`: `peak_height`, `peak_hash`, `fork_height`, `block_hashes`, additions, and removals must describe the branch being validated.
|
||||
- Coin state is fork-relative, not just DB-relative. A coin may be canonical before the fork, created on the fork, ephemeral inside the same block, or invalid because it only exists on the abandoned branch.
|
||||
- Transaction-block timestamp and previous transaction-block context drive timelocks, not necessarily the current peak height.
|
||||
- Generator references must resolve on the same branch as the block being validated and must not cross forks.
|
||||
- Post-HF2 header MMR/challenge-root commitments are gated by pre-signage-point transaction height, not naïve candidate height.
|
||||
|
||||
## Test Taxonomy
|
||||
|
||||
`test_blockchain.py` is the broad consensus regression file. Its structure mirrors the production validation pipeline: genesis/slot/header checks first, then prevalidation, body validation, reorg behavior, generator lookup, and direct `ForkInfo` accounting.
|
||||
|
||||
Header validation tests mutate otherwise valid blocks with `recursive_replace(...)`, then repair dependent hashes/signatures as needed so the intended check is reached. Common targets include bad previous hash, PoSpace, sub-slot challenge hashes, ICC/CC/RC VDFs, signage point/index constraints, deficit/SES rules, pool target signatures, foliage presence, timestamp ordering, height/weight, reward block hashes, and overflow/empty-slot cases.
|
||||
|
||||
Body validation tests build real spend bundles from `WalletTool` and assert transaction-state failures: missing or contradictory tx fields, reward claim mismatch, generator root/ref root mismatch, cost over/under-reporting, canonical generator encoding gates, Merkle/filter mismatches, duplicate outputs/removals, DB and fork double-spends, unknown fork spends, minting, invalid fees, and aggregate signature failures. Many tests bypass prevalidation only when the body path is the intended target.
|
||||
|
||||
Reorg tests are behavior-heavy. They validate short reorgs, reorgs from genesis, long reorgs over difficulty changes, heavier-lower-height branches, flip-flops back to a prior branch, stale fork-height handling, failed rollback behavior, reorg transaction replay, `get_tx_peak()` updates, and generator refs that point into the new fork. Shared `ForkInfo` and shared `AugmentedBlockchain` are reused across fork blocks to mirror full-node batch validation.
|
||||
|
||||
## Important Sibling Contracts
|
||||
|
||||
`AugmentedBlockchain` is the speculative overlay used during parallel validation. Its tests assert that extra blocks are contiguous, first-block ancestry exists in the underlying chain, fork ancestry is populated for orphan branches, generator refs search overlay before underlying chain, committed extra blocks are removed, MMR state is copied, and read-only snapshots reject mutation while preserving lookup behavior.
|
||||
|
||||
`find_fork_point.lookup_fork_chain()` and `find_fork_point_in_chain()` are tested with synthetic block graphs because boundary behavior matters at the pre-genesis/root edge, root-shared forks, same-height forks, asymmetric branch lengths, and no-common-ancestor cases. The returned fork-chain map excludes the fork point and uses the genesis challenge for the pre-genesis edge.
|
||||
|
||||
`get_block_generator()` is intentionally small but consensus-sensitive: a block with no generator must have no refs; a generator with no refs must not call lookup; refs are returned in the original ref-list order and missing refs surface as `KeyError`.
|
||||
|
||||
`test_build_chains.py` protects persistent block fixtures. It verifies cached chains still match `BlockTools` generation parameters and that additions/removals do not spend nonexistent coins. If generated-chain behavior changes intentionally, the cache regeneration path is part of the change.
|
||||
|
||||
`test_block_commitments.py` covers HF2 commitment activation by lowering fork heights, then validating both local block acceptance and two-node sync/weight-proof behavior. These tests are expensive but cover the production boundary where consensus commitments interact with full-node sync.
|
||||
|
||||
`test_blockchain_transactions.py` uses `two_nodes`, mempool submission, and block creation to bridge protocol/mempool behavior into block validation. It is useful for transaction conditions whose setup is easier through full-node APIs than through isolated consensus helpers.
|
||||
|
||||
## Review And Implementation Pitfalls
|
||||
|
||||
Do not replace shared fork `ForkInfo` with per-block fresh instances in reorg tests unless the production flow also changed. Fresh instances can hide missing additions/removals replay across fork blocks.
|
||||
|
||||
Do not treat `_validate_and_add_block_no_error()` as success-on-peak. It accepts `ALREADY_HAVE_BLOCK`, `ADDED_AS_ORPHAN`, and `NEW_PEAK`; use `_validate_and_add_block(..., expected_result=...)` when peak status matters.
|
||||
|
||||
When mutating signed/header-linked fields, update the dependent hash and plot signature or the test may fail earlier than intended. Existing tests often recalculate `transactions_info_hash`, `foliage_transaction_block_hash`, and `foliage_transaction_block_signature` after targeted body mutations.
|
||||
|
||||
Hard-fork-gated expectations often vary by `ConsensusMode`. The suite deliberately limits modes for slow or mode-sensitive cases; changing fixtures or heights can invalidate assumptions about which generated blocks are transaction blocks, overflow blocks, or generator-bearing blocks.
|
||||
|
||||
Avoid raw sleeps. Blockchain tests mostly use direct deterministic validation; full-node sync/transaction convergence uses `time_out_assert(...)`.
|
||||
|
||||
## Verification Guidance
|
||||
|
||||
For helper changes touching `_validate_and_add_block`, `AugmentedBlockchain`, generator lookup, or fork handling, include at least one focused test from the affected area plus a representative reorg/generator lookup test. For hard-fork commitment or generated-chain changes, include the relevant commitment/build-chain tests and expect longer runtime.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Consensus regression tests: `chia/_tests/blockchain/`.
|
||||
- Shared blockchain test helpers: `chia/_tests/blockchain/blockchain_test_utils.py`.
|
||||
- Chain fixture generation: `chia/_tests/blockchain/test_build_chains.py`, `chia/_tests/util/blockchain.py`.
|
||||
- Source behavior context: `.cursor/context/consensus.md`, `.cursor/context/full-node.md`.
|
||||
|
||||
## Starter Template
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from chia_rs.sized_ints import uint32
|
||||
|
||||
from chia._tests.blockchain.blockchain_test_utils import (
|
||||
_validate_and_add_block,
|
||||
_validate_and_add_block_multi_result,
|
||||
)
|
||||
from chia.consensus.augmented_chain import AugmentedBlockchain
|
||||
from chia.consensus.block_body_validation import ForkInfo
|
||||
from chia.consensus.blockchain import AddBlockResult, Blockchain
|
||||
from chia.consensus.pot_iterations import is_overflow_block
|
||||
from chia.simulator.block_tools import BlockTools
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_example(empty_blockchain: Blockchain, bt: BlockTools) -> None:
|
||||
blocks = bt.get_consecutive_blocks(5)
|
||||
for block in blocks:
|
||||
await _validate_and_add_block(empty_blockchain, block)
|
||||
|
||||
fork_blocks = bt.get_consecutive_blocks(
|
||||
3, block_list_input=blocks[:3], seed=b"fork"
|
||||
)
|
||||
# Add fork blocks and assert reorg results...
|
||||
```
|
||||
|
||||
For reorg tests, reuse a single `AugmentedBlockchain` and shared `ForkInfo` across fork blocks to mirror batch validation. Expect `ALREADY_HAVE_BLOCK` for shared prefix blocks, `ADDED_AS_ORPHAN` before the fork wins, and `NEW_PEAK` once the heavier fork overtakes. Use `is_overflow_block(constants, block.reward_chain_block.signage_point_index)` for overflow assertions.
|
||||
@@ -0,0 +1,112 @@
|
||||
# chia-tests-clvm
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
Scope: `chia/_tests/clvm/`. This is distilled architectural context for future audit or implementation agents. It intentionally omits exhaustive test inventory and obvious helper summaries.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for CLVM tests, `Program` helper behavior, puzzle compression, wallet puzzle-driver tests, SpendSim-backed smart-contract tests, singleton/custody/message-condition contracts, and CLVM verification strategy.
|
||||
|
||||
## Module Role
|
||||
|
||||
`chia/_tests/clvm/` is the executable contract for Python-facing CLVM program utilities, wallet puzzle drivers, condition builders, and lightweight smart-contract integration. It sits between pure wallet helper tests and full-node consensus tests: many cases construct real `Program`, `CoinSpend`, and `SpendBundle` objects, then either execute them directly or submit them through a small simulator backed by production mempool and coin-store code.
|
||||
|
||||
The suite protects these boundaries:
|
||||
|
||||
- `Program` convenience APIs (`at`, `replace`, `curry`, `uncurry`, run variants) and tree-hash helpers used by wallet puzzle construction.
|
||||
- CLVM serialization/deserialization and puzzle compression compatibility, including deployed dictionary versions.
|
||||
- Standard payment puzzles, taproot/hidden-puzzle signing, M-of-N delegated spends, singleton top-layer behavior, custody puzzle architecture, restrictions, and message conditions.
|
||||
- `SpendSim`'s RPC-like surface for puzzle tests that need mempool admission, block farming, hints, rollback, or puzzle/solution lookup without booting full services.
|
||||
|
||||
## Execution Harnesses
|
||||
|
||||
There are two distinct testing modes. Do not blur them.
|
||||
|
||||
Pure CLVM/unit tests run `Program` methods, compiled puzzle modules, or helper functions directly. They are appropriate for structural invariants: exact curry shapes, tree hashes, decoded puzzle driver fields, compression round trips, chialisp deserialization, and CLVM stepping.
|
||||
|
||||
Simulator-backed tests use `sim_and_client()` from `chia/_tests/util/spend_sim.py`. This is not a mock mempool. `SpendSim` wires a real full-node `CoinStore`, `HintStore`, `MempoolManager`, `Mempool`, and `simple_solution_generator()` into an in-memory DB, then represents blocks with trimmed `SimFullBlock`/`SimBlockRecord` objects. `SimClient.push_tx()` calls production `pre_validate_spendbundle()` and `add_spend_bundle()`, so failures like `GENERATOR_RUNTIME_ERROR`, timelock errors, message pairing errors, and signature/condition validation are meaningful mempool results.
|
||||
|
||||
`SpendSim.farm_block()` includes current mempool items by asking `MempoolManager.create_bundle_from_mempool()`, updates the real coin store with additions/removals and hints, records a transaction generator, advances peak state, then clears/revalidates mempool through `new_peak()`. `rewind()` rolls back the coin store and resets mempool state, which is why singleton and custody tests can reuse a pre-spend height for alternate paths.
|
||||
|
||||
`chia/_tests/clvm/coin_store.py` is a much smaller legacy-style spend harness used by older puzzle tests. It validates a bundle through `get_name_puzzle_conditions()` in mempool mode and `check_time_locks()`, then applies additions/removals to an in-memory record map. Prefer `SpendSim` for behavior that depends on real mempool admission, hints, rollback, or block inclusion.
|
||||
|
||||
## Core Behavioral Contracts
|
||||
|
||||
`Program` tests defend exact Python wrapper behavior around Rust CLVM execution:
|
||||
|
||||
- `Program.run()` and instance `run_with_cost()` default to wallet-style flags including mempool/soft-fork behavior, while module-level run helpers can use non-mempool flags. Tests that assert cost or output bytes should be explicit about which path they use.
|
||||
- `curry()` emits the canonical CLVM curry shape and `uncurry()` only recognizes that shape without trailing garbage or malformed quoted args. Wallet puzzle drivers rely on this form to inspect layered puzzles.
|
||||
- `replace()` paths use the same `f`/`r` grammar as `at()` and reject conflicting or impossible paths rather than partially rebuilding malformed trees.
|
||||
- Tree-hash helpers in `wallet.util.curry_and_treehash` must match `Program.to(...).get_tree_hash()` across atoms, ints, atom lists, and curried args, including negative and large integer encodings.
|
||||
|
||||
Puzzle compression tests are compatibility tests, not performance-only checks. `puzzle_compression.ZDICT` contains already-deployed puzzle bytes and legacy dictionaries; `lowest_best_version()` encodes which compression version is required for recognized modules. Decompression intentionally caps output, so tests around large buffers are resource-limit checks.
|
||||
|
||||
Chialisp deserialization tests run the CLVM deserializer module against serialized atoms/lists and overflow-sized atom headers. They protect canonical parsing and failure behavior at the CLVM byte-format boundary.
|
||||
|
||||
## SpendSim And RPC-Like Semantics
|
||||
|
||||
`test_spend_sim.py` is the contract for what smart-contract tests may assume from `SimClient`:
|
||||
|
||||
- farming creates realistic reward coins and heights;
|
||||
- `push_tx()` surfaces `MempoolInclusionStatus` plus `Err`;
|
||||
- hint lookup honors include-spent and height filters;
|
||||
- puzzle-hash, puzzle-hashes, parent-id, name, block-record, block, additions/removals, mempool item, and puzzle/solution lookups behave close enough to full-node RPC for puzzle-driver tests;
|
||||
- `get_puzzle_and_solution()` reconstructs coin spends from recorded block generators using production Rust lookup.
|
||||
|
||||
Because `SpendSim` stores simplified block records, it is good for puzzle and mempool behavior, not for consensus/header/weight-proof assertions.
|
||||
|
||||
## Custody Architecture Contracts
|
||||
|
||||
The custody tests exercise `wallet.puzzles.custody.*` as a small composable puzzle framework:
|
||||
|
||||
- `PuzzleWithRestrictions.memo()` is the on-chain/exported synchronization contract. `from_memo()` must reconstruct unknown members/restrictions, including recursive `MofN`, so later wallet code can fill known puzzle implementations by puzzle hash.
|
||||
- `PuzzleWithRestrictions.puzzle_reveal()` layers `INDEX_WRAPPER`, optional restriction layer, and top-level `DELEGATED_PUZZLE_FEEDER`; `puzzle_hash()` uses precalculated hashes to match the reveal without materializing every layer.
|
||||
- `solve()` must align member validator solutions, delegated-puzzle validator solutions, member solution, and optional delegated puzzle/solution in the exact order expected by the CLVM modules.
|
||||
- `MofN` rejects impossible thresholds and duplicate member nodes. Its solve format differs across threshold shapes, so tests iterate combinations to catch proof-format regressions.
|
||||
|
||||
Concrete member puzzle tests cover BLS-with-taproot, singleton-backed membership, and fixed-puzzle membership. They intentionally check both success and escape paths: invalid hidden puzzle, wrong fixed delegated puzzle, missing singleton approval message, and constructor/solve misuse errors.
|
||||
|
||||
Restriction tests cover delegated-puzzle wrapper stacks, heightlocks, fixed `CREATE_COIN` destinations, and `SEND_MESSAGE` bans. A valid restriction often wraps the delegated puzzle before submission; submitting the original delegated puzzle with only a matching solution is expected to fail.
|
||||
|
||||
## Message Conditions
|
||||
|
||||
Message-condition tests cover the paired `SEND_MESSAGE`/`RECEIVE_MESSAGE` invariant. For every nonzero sender/receiver commitment mode, a lone send or receive fails with `MESSAGE_NOT_SENT_OR_RECEIVED`, while the aggregate succeeds.
|
||||
|
||||
`MessageParticipant` is deliberately strict: no anyone-can-send/receive participant, coin-id commitments must either stand alone or match all parent/puzzle/amount fields, and manual `mode_integer` values must match supplied arguments. These are API footgun tests as much as condition tests.
|
||||
|
||||
## Singleton Contracts
|
||||
|
||||
Singleton tests cover both legacy and current top layers. They assert launcher flow, eve spend, steady-state spend, P2-singleton claims, P2-singleton-or-delayed claims, delayed escape, melting, and negative odd-amount invariants.
|
||||
|
||||
Key invariants:
|
||||
|
||||
- launcher amounts must be odd;
|
||||
- singleton spends must create exactly one odd child unless melting;
|
||||
- lineage proofs derive from the parent coin spend and include inner puzzle hash for non-launcher parents;
|
||||
- P2-singleton claims couple coin and puzzle announcements;
|
||||
- delayed escape requires elapsed seconds/blocks in simulator state;
|
||||
- legacy and current singleton layers may surface different `Err` values for the same malformed even-coin path, and tests encode that distinction.
|
||||
|
||||
## Editing And Review Guidance
|
||||
|
||||
Use `sim_and_client()` when the assertion depends on mempool admission, condition validation, hints, rollback, or inclusion in a transaction generator. Use direct `Program` execution for deterministic CLVM structure, serialization, hashing, and helper API behavior.
|
||||
|
||||
Keep negative-path assertions specific. These tests often distinguish `FAILED` from `PENDING` and specific `Err` codes such as `GENERATOR_RUNTIME_ERROR`, `ASSERT_HEIGHT_RELATIVE_FAILED`, `ASSERT_SECONDS_RELATIVE_FAILED`, `ASSERT_MY_AMOUNT_FAILED`, or `MESSAGE_NOT_SENT_OR_RECEIVED`.
|
||||
|
||||
Do not replace simulator farming/rewind flows with raw state mutation unless the production behavior being skipped is irrelevant. Many tests depend on `new_peak()`, coin-store rollback, hint persistence, or mempool clearing.
|
||||
|
||||
When modifying wallet puzzle drivers, assert both the pure representation contract (memo, puzzle hash, parse/fill behavior) and an on-chain spend path through `SpendSim`. The former catches wallet sync/recognition regressions; the latter catches CLVM or mempool acceptance regressions.
|
||||
|
||||
For compression changes, preserve deployed dictionary compatibility and the decompression output cap. Adding a dictionary version should update `LATEST_VERSION` behavior and `lowest_best_version()` expectations without making older compressed blobs undecodable.
|
||||
|
||||
## Verification Guidance
|
||||
|
||||
For changes touching `SpendSim`, include `test_spend_sim.py` plus at least one simulator-backed puzzle suite that uses rollback or mempool errors. For changes touching `Program`, include `test_program.py` and `test_curry_and_treehash.py`. For wallet puzzle-driver changes, include the relevant CLVM test file and consider the corresponding wallet module tests if the puzzle is surfaced through wallet recognition or transaction creation.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- CLVM test suite: `chia/_tests/clvm/`.
|
||||
- SpendSim harness: `chia/_tests/util/spend_sim.py`.
|
||||
- Python-facing CLVM wrappers: `chia/types/blockchain_format/program.py`, `chia/wallet/puzzles/`.
|
||||
- Source behavior context: `.cursor/context/types.md`, `.cursor/context/wallet.md`.
|
||||
@@ -0,0 +1,69 @@
|
||||
# Chia Commands Tests Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/_tests/cmds/` is the CLI contract test module. It primarily verifies that Click parsing, command helper dataclasses, config/root-path handling, RPC-client selection, wallet request construction, transaction output files, and user-facing text stay aligned with `chia/cmds/`. Most tests intentionally stop at the command/RPC boundary; they are not full wallet or node behavior tests unless they explicitly use service fixtures.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for CLI tests, command parser behavior, mocked RPC-client tests, command output expectations, transaction-output files, and command framework coverage. For CLI implementation context, also read `cmds.md`.
|
||||
|
||||
## Harness Boundaries
|
||||
|
||||
- `conftest.py` provides `get_test_cli_clients`, a module-scoped temporary Chia root with default config plus monkeypatches that replace `get_any_service_client`, `get_wallet_client`, and `cli_confirm` in both `chia.cmds.cmds_util` and the command modules that imported those functions directly. Tests using this fixture exercise the real CLI entrypoint while avoiding daemon/keychain/RPC processes.
|
||||
- `cmd_test_utils.py` is the central mock-RPC harness. `TestRpcClient` records method calls, `TestRpcClients.get_client()` maps concrete RPC client classes to mutable fake clients, and command tests assert both rendered output and exact request dataclasses/arguments sent to fake RPC methods.
|
||||
- `run_cli_command()` mutates `sys.argv` and invokes the top-level `chia` CLI with `--root-path`. It treats any non-zero `SystemExit` as failure and returns captured stdout. This means tests often fail through `AssertionError` wrapping Click output rather than directly through Click exceptions.
|
||||
- Wallet command tests share deterministic fixtures from `wallet/test_consts.py`: fixed fingerprints, wallet ids, `bytes32` helpers, `STD_TX`, and `STD_UTX`. These values are part of the expected text/request-shape contract.
|
||||
- Only a small minority of tests cross into live services: `test_cmd_framework.test_wallet_rpc_helper` uses `wallet_environments`, and `test_farm_cmd.test_farm_summary_command` uses farmer/harvester/simulator/wallet services. Treat those as integration tests with async convergence requirements, unlike the mock-RPC command tests.
|
||||
|
||||
## Command Framework Contracts
|
||||
|
||||
- `test_cmd_framework.py` is the specification for the newer class-based command system in `chia.cmds.cmd_classes` and `chia.cmds.cmd_helpers`. It verifies dataclass command wrapping, sync/async `run()` dispatch, type-hint-to-Click option generation, nested `command_helper` parsing, context injection, and rejection of unsupported/default-incompatible type shapes.
|
||||
- `TransactionEndpoint` tests enforce that transaction endpoint subclasses use `@transaction_endpoint_runner`, preserve default option parity with older decorators, write returned `TransactionRecord`s through `TransactionsOut`, and convert `--valid-at/--expires-at` into `ConditionValidTimes`.
|
||||
- `test_old_decorator_support` intentionally keeps old decorator functions (`coin_selection_args`, `tx_config_args`, `tx_out_cmd`) in sync with class-based helpers. If the legacy decorators are removed, this test is expected to be deleted rather than preserved with compatibility shims.
|
||||
- `test_tx_config_args.py`, `test_timelock_args.py`, and `test_click_types.py` pin CLI conversion behavior for amounts, fees, addresses, bytes32 values, uint64 values, coin-selection config, reuse/new-address selection, and hidden timelock options. These are user-interface contracts and should not be loosened without matching CLI migration intent.
|
||||
|
||||
## RPC And Config Boundary
|
||||
|
||||
- `test_cmds_util.py` covers `get_any_service_client`: no-SSL test server use, friendly consumed `ResponseFailureError` output, traceback formatting, `consume_errors=False`, unknown client types without explicit ports, and unexpected exception consumption. These tests protect CLI error presentation as much as networking behavior.
|
||||
- `test_peer.py` verifies service-name validation, connection table rendering, and missing config-section errors. Solver coverage matters because `solver` is configured differently from older services and may be absent from existing configs.
|
||||
- `create_service_and_wallet_client_generators()` loads config from the temp root, derives default RPC ports from `node_config_section_names`, supports DataLayer's `fill_missing_services`, and patches modules that imported service helpers by name. New command modules that import `get_any_service_client` directly may need explicit monkeypatch coverage here.
|
||||
|
||||
## Wallet CLI Coverage
|
||||
|
||||
- `wallet/test_wallet.py` is a broad command-to-wallet-RPC contract suite: get transaction(s), show balances, send XCH/CAT, get address, clawback, delete unconfirmed transactions, derivation index, sign message, add token, make/take/cancel offers, and offer summaries. It asserts exact `wallet_request_types` objects, `TXConfig`, fee/mojo conversions, timelock propagation, CAT/NFT name resolution, royalty summary calls, and transaction bundle file output.
|
||||
- `wallet/test_did.py`, `wallet/test_nft.py`, `wallet/test_vcs.py`, and `wallet/test_notifications.py` follow the same pattern for asset-specific subcommands. They do not prove DID/NFT/VC/notification wallet internals; they prove CLI parsing, ID/address conversion, RPC request construction, output text, `push` flags, fees, reuse-puzhash config, and `ConditionValidTimes`.
|
||||
- Wallet command tests commonly subclass `TestWalletRpcClient` inside a single test to expose only the RPC methods under inspection. This keeps expected call logs precise; avoid moving behavior into a global fake unless multiple commands intentionally share the same RPC surface.
|
||||
- Many assertions compare rich dataclass instances from `chia.wallet.wallet_request_types`. A failing equality usually means a CLI argument no longer maps to the same public RPC contract, not just an output formatting change.
|
||||
|
||||
## Other Command Areas
|
||||
|
||||
- `test_show.py` uses fake full-node RPC responses plus synthetic `FullBlock`/`TestBlockRecord` objects to check `chia show` output and RPC call ordering for chain state, fee estimates, height lookup, and block printing.
|
||||
- `test_farm_cmd.py` is a real-service integration check for `farm summary`. It waits for harvester-to-farmer plot sync before hitting real RPC ports, then asserts stable sections of summary output with and without pool rewards.
|
||||
- `test_daemon.py` isolates daemon startup/keyring behavior with mocks and a short-lived subprocess. It is sensitive to `sys.argv[0]`, keyring unlock messaging, and daemon process cleanup.
|
||||
- `test_dev_gh.py` validates local argument/path checks for GitHub workflow dispatch. The real dispatch test is skipped and should stay isolated from ordinary CLI test runs unless CI/auth behavior is explicitly being tested.
|
||||
- `test_sim.py` contains skipped simulator end-to-end command tests. They document intended simulator CLI flows but are not active safety coverage.
|
||||
- `wallet/test_wallet_check.py` is pure logic coverage for `chia.cmds.check_wallet_db` gap/contiguous-used-address checks and wallet type parsing from DB rows.
|
||||
|
||||
## Change Guidance
|
||||
|
||||
- Pick the smallest harness matching the boundary: direct param type/helper calls for conversion logic, `CliRunner` for Click-only parsing, `get_test_cli_clients` for command-to-RPC contracts, and live service fixtures only when the command's behavior depends on real service state.
|
||||
- For command-to-RPC tests, assert both stdout and the fake RPC call log. Output-only assertions can miss broken request payloads; call-log-only assertions can miss user-visible CLI regressions.
|
||||
- When adding a new service/client command, update the fake client mapping or monkeypatches if the command imports helper functions directly. A real command may work while tests still hit an unpatched helper path.
|
||||
- Keep fee and amount expectations explicit about units. `TransactionFeeParamType` applies a source-defined decimal-XCH fee cap, while generic `AmountParamType` produces `CliAmount` and converts later using wallet-specific mojo-per-unit.
|
||||
- Preserve root-path isolation. Tests should use temp roots, `runner.isolated_filesystem()`, or explicit temp paths for config and transaction files; touching default roots is a flake and state-leak risk.
|
||||
- Avoid raw sleeps in the few live-service tests. Use `time_out_assert` or existing service readiness checks because farm/daemon/wallet fixtures are timing-sensitive.
|
||||
|
||||
## Fragility Signals
|
||||
|
||||
- Direct imports of patch targets in command modules are easy to miss. If a command file does `from chia.cmds.cmds_util import get_wallet_client`, patching only `cmds_util` will not affect that module's local binding.
|
||||
- CLI text assertions are intentionally brittle for public UX. If wording changes are deliberate, update the specific expected strings and keep request-shape assertions intact.
|
||||
- Transaction file output uses `TransactionBundle` streamable bytes. Changes to `TransactionRecord` serialization or output-file wiring can break offline signing/push workflows even if stdout still looks correct.
|
||||
- `ChiaCliContext.expected_prefix` caches address prefix during parameter conversion. Address tests that pass without config may be depending on this cache; be careful when moving address parsing earlier or outside Click context.
|
||||
- The command framework is mid-transition between decorators and dataclass helpers. Do not add compatibility layers unless both old and new command styles are still meant to be supported.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- CLI test helpers and fixtures: `chia/_tests/cmds/cmd_test_utils.py`, `chia/_tests/cmds/conftest.py`.
|
||||
- Command framework coverage: `chia/_tests/cmds/test_cmd_framework.py`, `chia/_tests/cmds/test_cmds_util.py`, `chia/_tests/cmds/test_click_types.py`.
|
||||
- CLI implementation context: `chia/cmds/`.
|
||||
@@ -1,11 +1,7 @@
|
||||
---
|
||||
description: Data layer test patterns — DataStore logic, wallet-backed RPC flows, singleton lifecycle
|
||||
globs:
|
||||
- chia/_tests/core/data_layer/**
|
||||
---
|
||||
|
||||
# Data Layer Tests
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
## Scope
|
||||
|
||||
Use this for data layer behavior across both local store logic and chain-backed updates:
|
||||
@@ -0,0 +1,50 @@
|
||||
# Full Node Tests
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradlicts this doc, trust source and update the doc.
|
||||
|
||||
## Scope
|
||||
|
||||
Use this when testing behavior driven by full node state transitions: sync,
|
||||
block acceptance, mempool-to-block inclusion, wallet-connected flows, and reorg.
|
||||
|
||||
For harness selection and block/transaction patterns, see `patterns.md` and
|
||||
`architecture.md`. For mempool-specific tests, see `mempool.md`.
|
||||
|
||||
## Key Tips
|
||||
|
||||
- For sync tests, assert both node height and peak equality — height alone can
|
||||
lag behind peak acceptance.
|
||||
- Keep fixture scope narrow; large shared state increases intermittent failures.
|
||||
|
||||
## Starter Template
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from chia._tests.blockchain.blockchain_test_utils import _validate_and_add_block
|
||||
from chia._tests.connection_utils import add_dummy_connection, connect_and_get_peer
|
||||
from chia._tests.core.node_height import node_height_at_least
|
||||
from chia._tests.util.setup_nodes import OldSimulatorsAndWallets
|
||||
from chia._tests.util.time_out_assert import time_out_assert
|
||||
from chia.consensus.blockchain import Blockchain
|
||||
from chia.full_node.full_node_api import FullNodeAPI
|
||||
from chia.protocols import wallet_protocol
|
||||
from chia.server.server import ChiaServer
|
||||
from chia.simulator.block_tools import BlockTools
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_example(
|
||||
one_node_one_block: tuple[FullNodeAPI, ChiaServer, BlockTools],
|
||||
) -> None:
|
||||
full_node_api, server, bt = one_node_one_block
|
||||
full_node = full_node_api.full_node
|
||||
|
||||
blocks = bt.get_consecutive_blocks(3)
|
||||
for block in blocks:
|
||||
await full_node.add_block(block)
|
||||
|
||||
await time_out_assert(10, node_height_at_least, True, full_node, 3)
|
||||
```
|
||||
@@ -0,0 +1,80 @@
|
||||
# chia-tests-util
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
Scope: `chia/_tests/util/`. This is distilled architectural context for future audit or implementation agents. It intentionally omits helper-by-helper inventory and focuses on contracts that affect the wider test suite.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for shared test utilities, topology setup helpers, deterministic blockchain fixtures, SpendSim, async convergence helpers, protocol corpus generation, plot-cache monkeypatching, and cross-suite test infrastructure.
|
||||
|
||||
## Module Role
|
||||
|
||||
`chia/_tests/util/` is the shared test infrastructure layer for the repository. It is not a leaf test directory: changes here can alter the meaning, runtime, or flake profile of blockchain, full-node, mempool, wallet, server, farmer/harvester, timelord, data-layer, plotting, and protocol tests.
|
||||
|
||||
The package provides five architectural surfaces:
|
||||
|
||||
- service topology construction for real full nodes, simulators, wallets, farmers, harvesters, timelords, solvers, introducers, daemon, VDF clients, and RPC clients;
|
||||
- deterministic chain and DB construction for consensus-heavy tests and persistent fixture chains;
|
||||
- lightweight spend/mempool simulation for CLVM and wallet contract tests that need production mempool and coin-store behavior without a full node;
|
||||
- async convergence, benchmark, protocol, RPC, and process helpers used as observable test contracts;
|
||||
- protocol serialization corpora and generated tests that pin wire compatibility across node roles.
|
||||
|
||||
Treat edits here as harness changes, not local cleanup, unless the helper is demonstrably unused outside `chia/_tests/util/` itself.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
`setup_nodes.py` is the topology authority. Its context managers own keyring isolation, root/config construction, service lifetimes, port choices, capability toggles, wallet RPC client creation, and teardown order. `setup_two_nodes()` and `setup_n_nodes()` create real full-node services with separate DBs. `setup_simulators_and_wallets*()` creates simulator full nodes plus wallet services and is the fixture backbone for wallet, data-layer, RPC, and simulation tests. `setup_farmer_solver_multi_harvester()` and `setup_full_system()` cross into farmer/harvester/solver/timelord/daemon behavior and include explicit connection-settle loops.
|
||||
|
||||
`blockchain.py` is the deterministic chain authority. `create_blockchain()` constructs a production `Blockchain` over in-memory stores with an `InlineExecutor`; tests using it are still exercising real block/coin stores. `persistent_blocks()` and `new_test_db()` define the cached block-artifact contract used by long sync, weight proof, reorg, commitment, and generated-chain tests. In CI, missing persistent block files are hard failures, not cache misses.
|
||||
|
||||
`spend_sim.py` is a deliberately partial full-node substitute for Chialisp and wallet contract tests. It uses production `MempoolManager`, `Mempool`, `CoinStore`, and `HintStore`, but replaces `FullBlock` and `BlockRecord` with streamable minimal types. `SimClient` mirrors selected full-node RPC client methods so tests can later swap to a real client. This makes it useful for spend admission, mempool inclusion status, coin lookup, hints, puzzle-and-solution extraction, and block farming, but not for peer protocol, weight proofs, signage points, non-transaction blocks, or full consensus header behavior.
|
||||
|
||||
`time_out_assert.py` is the async convergence authority. Tests depend on it for retry semantics, adjusted timeouts, short polling intervals, pytest traceback hiding, and Ether/JUnit telemetry. Replacing it with sleeps or changing its comparison semantics can hide flakes or make CI diagnostics worse.
|
||||
|
||||
`build_network_protocol_files.py`, `network_protocol_data.py`, `protocol_messages_json.py`, and the generated protocol tests form the wire-compatibility corpus. Adding, removing, or reordering visited protocol messages changes the serialized byte stream and JSON fixtures that guard Streamable compatibility across farmer, full node, wallet, harvester, introducer, pool, timelord, shared, and solver protocols.
|
||||
|
||||
## Cross-Module Contracts
|
||||
|
||||
- `chia/_tests/conftest.py` imports this module heavily. Session fixtures install the plot cache, isolate keyring access, create shared `BlockTools`, construct persistent chains by consensus mode, and expose simulator/wallet/full-node fixtures through these helpers.
|
||||
- Full-node and blockchain tests rely on `create_blockchain()` and `persistent_blocks()` preserving production store semantics. A helper that looks like fixture setup can affect fork choice, generator refs, MMR commitments, coin-store rollback, or DB-version coverage.
|
||||
- Wallet and data-layer tests rely on `setup_simulators_and_wallets*()` to create wallet RPC clients against live wallet services. The returned environment couples wallet action scopes, full-node mempool admission, simulator farming, RPC reads, and wallet sync.
|
||||
- Mempool tests rely on `spend_sim.py`, `get_name_puzzle_conditions.py`, and `misc.invariant_check_mempool()` for production CLVM flags, spend-bundle validation, pool accounting, and fast-forward singleton invariants.
|
||||
- Server/protocol tests rely on generated protocol data and helpers like `time_out_messages()` and `patch_request_handler()` to observe message ordering, API metadata, request/reply state, and rate-limit coverage.
|
||||
- Farmer/harvester and plotting tests rely on `setup_farmer_solver_multi_harvester()` and `plot_cache.install()`. The plot cache monkeypatches prover methods and `chia_rs.solve_proof` globally for the test process, merging cache state on exit under a file lock.
|
||||
- `full_sync.py` is an executable sync harness, not a normal unit helper. It builds a `FullNode` from config, stubs network broadcast, streams compressed blocks from a DB, and chooses between `add_block_batch()` sync behavior and keep-up behavior. Changes here affect full-sync benchmarking and regression reproduction.
|
||||
|
||||
## Fragility And Review Signals
|
||||
|
||||
- Async topology helpers must keep `AsyncExitStack` ownership clear. Moving service creation out of the managed stack, changing keyring scope, or skipping shielded/ordered teardown can leak processes, sockets, DB handles, or keyring state into unrelated tests.
|
||||
- Default config overrides in topology helpers are part of test semantics: sync waits, coin logging, and block-creation timeout settings make tests faster and more observable. Removing them can cause slow convergence or missing coin logs.
|
||||
- Persistent block artifacts are shared by consensus mode. Changing generation parameters, suffixes, fork heights, seed values, dummy refs, transaction inclusion, or normalization flags requires updating the corresponding generated-chain tests and cached artifacts.
|
||||
- `SpendSim` has intentional simplifications: every simulated block is effectively a transaction block, reward/header hashes are synthetic, and rewinds reset the mempool. Do not generalize conclusions from it to full-node sync, consensus header validation, or non-transaction block behavior.
|
||||
- `get_name_puzzle_conditions()` chooses `run_block_generator` vs `run_block_generator2` at `HARD_FORK_HEIGHT` and always disables signature validation. It is suitable for condition extraction tests, not aggregate-signature correctness.
|
||||
- `time_out_assert_custom_interval()` records caller file/line and timeout telemetry even on failure. Changing stack distance, `adjusted_timeout()`, interval behavior, or error messages can affect both debugging and test-report processing.
|
||||
- `misc.py` mixes narrow utilities with high-blast-radius helpers. `BenchmarkRunner`, `BenchmarkData`, and `TestId` feed `process_junit`; `patch_request_handler()` mutates API metadata; `invariant_check_mempool()` reaches into private SQLite-backed mempool state by design.
|
||||
- `split_managers.py` is explicitly transitional. New code should avoid depending on split enter/exit semantics unless a test genuinely needs lifecycle control across phases.
|
||||
- `plot_cache.py` uses pickle and global monkeypatching intentionally for test speed. It should remain test-only; do not reuse its cache format or global patch behavior in production code.
|
||||
|
||||
## Change Guidance
|
||||
|
||||
Each shared helper represents a specific test layer: pure DB/store, production blockchain/store without networking, deterministic persistent chains, SpendSim, simulator-wallet lifecycle, real service wiring, or protocol corpus compatibility. When adding a helper, document that layer so future tests do not broaden runtime by accident.
|
||||
|
||||
## Verification Guidance
|
||||
|
||||
For shared harness edits, add at least one representative downstream test:
|
||||
|
||||
- `setup_nodes.py`: a wallet/simulator test plus a full-node or farmer/harvester topology user;
|
||||
- `blockchain.py`: a blockchain helper test and a persistent-chain consumer such as generated-chain or reorg coverage;
|
||||
- `spend_sim.py` or `get_name_puzzle_conditions.py`: the targeted mempool/CLVM/wallet contract tests;
|
||||
- `time_out_assert.py` or benchmark telemetry: a direct util test plus `chia/_tests/process_junit.py`-related expectations if telemetry shape changes;
|
||||
- protocol corpus files: rebuild/check generated protocol bytes and JSON tests;
|
||||
- `plot_cache.py`: its util tests plus a plotting/farmer-harvester path if monkeypatch behavior changes.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Service topology and environment setup: `chia/_tests/util/setup_nodes.py`, `chia/_tests/environments/`.
|
||||
- Deterministic chain and DB helpers: `chia/_tests/util/blockchain.py`.
|
||||
- Spend simulation and condition helpers: `chia/_tests/util/spend_sim.py`, `chia/_tests/util/get_name_puzzle_conditions.py`.
|
||||
- Async and protocol helpers: `chia/_tests/util/time_out_assert.py`, `chia/_tests/util/network_protocol_data.py`.
|
||||
- Plot and process helpers: `chia/_tests/util/plot_cache.py`, `chia/_tests/process_junit.py`.
|
||||
@@ -1,11 +1,7 @@
|
||||
---
|
||||
description: Mempool manager test patterns — acceptance/rejection, replacement, fee-per-cost, eviction, bundle selection
|
||||
globs:
|
||||
- chia/_tests/core/mempool/**
|
||||
---
|
||||
|
||||
# Mempool Tests
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
## Scope
|
||||
|
||||
Use this for mempool rule correctness:
|
||||
@@ -0,0 +1,40 @@
|
||||
# Chia Test Patterns
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
## Layered Assertions
|
||||
|
||||
Use layered assertions instead of a single final check:
|
||||
|
||||
1. **Immediate invariants** — object created, response success, expected fields present.
|
||||
2. **Eventual behavior** — `time_out_assert(...)` for async convergence.
|
||||
3. **Mempool checks** — `mempool_manager.get_spendbundle(...)`, `assert_sb_in_pool(...)`.
|
||||
4. **Wallet transitions** — `process_pending_states(...)` with `WalletStateTransition`.
|
||||
5. **Failure paths** — `pytest.raises(...)` with explicit error matching.
|
||||
6. **Log assertions** — `caplog` for protocol/service side effects.
|
||||
|
||||
## Module-by-Module Test Setup Map
|
||||
|
||||
| Module | Typical Setup | Blocks | Transaction Path | Assertion Style |
|
||||
| ------------------ | ------------------------------------------------------------------------ | -------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
|
||||
| `blockchain` | `bt`, `empty_blockchain`, `two_nodes` | `get_consecutive_blocks`, `add_block`, `add_blocks_in_batches` | `WalletTool.generate_signed_transaction`, protocol `send_transaction`, in-block `transaction_data` | direct consensus result checks, `pytest.raises`, occasional `time_out_assert` |
|
||||
| `clvm` | no network harness, or `sim_and_client` | `SpendSim.farm_block` | `sim_client.push_tx` | direct CLVM/coin-store assertions, `pytest.raises` |
|
||||
| `cmds` | `CliRunner`, `get_test_cli_clients`, temp config roots | usually none | mocked RPC client calls | output assertions, parse/validation errors |
|
||||
| `core` | mixed: `one_node_one_block`, `simulator_and_wallet`, data-layer fixtures | heavy use of `get_consecutive_blocks`, farming APIs | wallet-generated spends, protocol `send_transaction`/`respond_transaction` | heavy `time_out_assert`, mempool/state assertions, `caplog`, `pytest.raises` |
|
||||
| `db` | `DBConnection`/`PathDBConnection` fixtures | none | none | concurrency/transactionality assertions, `pytest.raises` |
|
||||
| `farmer_harvester` | `farmer_one_harvester*`, `harvester_farmer_environment` | minimal | protocol message flow | service-state `time_out_assert`, `caplog` |
|
||||
| `fee_estimation` | mostly mempool/unit harness | minimal farming | small generated spend bundles | direct estimator state assertions |
|
||||
| `generator` | pure generator/CLVM tests | none | none | deterministic program output/cost assertions |
|
||||
| `harvester` | `harvester_farmer_environment` + test plots | `default_400_blocks` for signage data | harvester protocol interactions | `time_out_assert`, mock peer assertions |
|
||||
| `pools` | pure puzzle unit tests and wallet/simulator integration | farming + reorg in integration | wallet RPC and framework tx processing | `process_pending_states`, `time_out_assert`, `pytest.raises` |
|
||||
| `simulation` | `simulator_and_wallet`, full system fixture | high-level simulator farming/reorg | wallet-generated spends | heavy `time_out_assert`, mempool/coin-store confirmations |
|
||||
| `wallet` | `wallet_environments` (primary), simulator fixtures | frequent farming/reorg | wallet action scopes, wallet RPC | `process_pending_states`, `time_out_assert`, mempool checks |
|
||||
| `weight_proof` | pre-generated block fixtures + `BlockchainMock` | `get_consecutive_blocks` for edge chains | none | proof validity/fork point assertions |
|
||||
|
||||
## Checklist
|
||||
|
||||
- Harness matches behavior under test.
|
||||
- Async convergence uses `time_out_assert`, not raw sleeps.
|
||||
- Wallet behavior uses `process_pending_states` where practical.
|
||||
- Mempool and confirmation asserted as separate steps.
|
||||
- Failure paths use `pytest.raises` with explicit error matching.
|
||||
@@ -1,11 +1,7 @@
|
||||
---
|
||||
description: Server/network test patterns — connection lifecycle, API errors, DoS/ban behavior, rate limiting
|
||||
globs:
|
||||
- chia/_tests/core/server/**
|
||||
---
|
||||
|
||||
# Server Tests
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
## Scope
|
||||
|
||||
Use this for network/server behavior rather than consensus correctness:
|
||||
@@ -0,0 +1,59 @@
|
||||
# Chia Wallet Tests Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/_tests/wallet/` is the main behavioral safety net for the wallet stack. It mixes fast unit tests for stores, puzzles, request types, and CLVM helpers with simulator-backed integration tests that exercise wallet sync, transaction construction, subwallet recognition, RPC endpoints, offers, and reorg handling. Treat it as a specification for wallet invariants, not just regression coverage.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for wallet tests, wallet fixture behavior, `wallet_environments`, wallet RPC tests, wallet store tests, asset-wallet tests, offer tests, and wallet-specific block-tool shortcuts. For source behavior, also read `wallet.md`.
|
||||
|
||||
## Test Harness Boundaries
|
||||
|
||||
- `chia/_tests/wallet/conftest.py` installs wallet-test-specific autouse patches. Unless a test is marked `standard_block_tools`, consensus-heavy block validation is shortcut and `WalletBlockTools` replaces normal `BlockTools` so wallet tests can focus on mempool acceptance, coin-state notification, and wallet DB updates.
|
||||
- `wallet_environments` is the primary integration fixture. It creates one simulator full node plus N wallet services, connects each wallet to the full node, opens wallet/full-node RPC clients, optionally trusts the full node, farms initial rewards per environment, and returns a `WalletTestFramework`.
|
||||
- `trusted_full_node` and `tx_config` are parametrized by default. Most `wallet_environments` tests run across trusted/untrusted sync and reuse/new-puzzle-hash modes unless the test explicitly pins `"trusted"` or `"reuse_puzhash"` in the indirect fixture params.
|
||||
- `new_action_scope_wrapper()` enforces reuse-puzzle-hash behavior by comparing used derivation counts around every wallet action scope. Tests that intentionally create new puzzle hashes under reuse mode must use `WalletTestFramework.new_puzzle_hashes_allowed()`.
|
||||
- `WalletBlockTools` generates structurally sufficient blocks with simplified proof/iteration fields, reward coins, transaction filters, additions/removals roots, and transaction generators. It is appropriate for wallet behavior; use `standard_block_tools` when the test depends on real consensus/header/weight-proof behavior.
|
||||
|
||||
## Wallet Test Framework Contract
|
||||
|
||||
- `WalletEnvironment` wraps a wallet service, RPC client, node API, peer server, state manager, main XCH wallet, and a local expected-balance map. Tests commonly set `wallet_aliases` such as `"xch"`, `"cat"`, `"nft"`, `"did"`, `"vc"`, or `"dl"` to make balance transitions readable.
|
||||
- `WalletStateTransition` splits expected balance effects into `pre_block_*` and `post_block_*` updates. This models the wallet distinction between pending mempool state and confirmed chain state.
|
||||
- `WalletTestFramework.process_pending_states()` is the main integration assertion path: wait for wallets to sync to the current peak, wait for pending transactions to enter and be marked in the mempool, apply/check pre-block balances, farm one transaction block, sync wallets to the new peak, apply/check post-block balances, verify previously pending transactions are confirmed, and re-check derivation reuse.
|
||||
- Balance deltas support exact fields plus comparison keys like `"<#spendable_balance"` or `">=#pending_change"` for coin-selection-dependent values. Use this when a wallet guarantee is directional but the exact selected coins/change amount is not stable.
|
||||
- New subwallet state must be initialized explicitly with `"init": True`; `"set_remainder": True` copies unspecified values from the live RPC balance. This is useful for secondary wallets but can hide unintended balance changes if overused.
|
||||
|
||||
## Behavioral Coverage Map
|
||||
|
||||
- Core XCH wallet tests exercise reward accounting, standard transaction creation, reuse-address behavior, clawback flows, fee estimation, signing, wallet DB path selection, and no-server edge cases. They usually construct spends in `WalletStateManager.new_action_scope(..., push=True)` and verify lifecycle with `process_pending_states()`.
|
||||
- `test_wallet_state_manager.py`, `test_wallet_node.py`, and sync suites cover sync-mode locking, key derivation, peer trust behavior, short/long sync, backtracking, stale or bad peer data, transaction ack/retry caches, balance computation, puzzle-hash subscriptions, coin-state validation, and reorg recovery.
|
||||
- `sync/test_wallet_sync.py` is marked `standard_block_tools` because it tests wallet/full-node protocol details that depend on real block/header/additions/removals behavior, request caps, weight proof fork points, and validation failures.
|
||||
- `simple_sync/test_simple_sync_protocol.py` and `test_new_wallet_protocol.py` focus on wallet protocol subscriptions: puzzle hash, coin id, hint, mempool item updates, request limits, reorg responses, capability gating, and sync-by-state behavior.
|
||||
- RPC tests in `rpc/` are broad endpoint integration tests, not just serialization checks. They cover send/push/create transaction paths, balance/farmed amount, coin queries and filters, CAT/DID/NFT/offer endpoints, notification RPCs, signing endpoints, resync flags, split/combine coin commands, and remote wallet RPC validation.
|
||||
- CAT, NFT, DID, VC, DataLayer, remote wallet, clawback, and offer suites encode asset-specific lifecycle invariants: launcher/eve spends, singleton lineage, hints, metadata, DID ownership, VC proofs/revocation, CR-CAT approval, mirror handling, offer cancellation/conflict states, and reorg survival.
|
||||
- Store tests (`wallet_coin_store`, `transaction_store`, `puzzle_store`, `wallet_interested_store`, `trade_store`, singleton/NFT/key-val stores) are persistence-contract tests. They matter for rollback, filtering, total-count caching, legacy migrations, unconfirmed caches, and restart behavior.
|
||||
- Puzzle/CLVM/unit suites (`singleton`, CAT/NFT/VC lifecycle fast tests, outer puzzle tests, conditions, signer protocol, taproot, bech32m, CLVM streamable/casts) are deterministic low-level guards and often run without simulator services.
|
||||
|
||||
## Editing Guidance
|
||||
|
||||
- Prefer `wallet_environments` for user-visible wallet behavior, subwallet transactions, RPC flows, and balance lifecycle assertions. Prefer smaller store/unit fixtures when the behavior under test is local persistence, parsing, puzzle construction, or pure CLVM.
|
||||
- When a test creates wallet transactions, keep the lifecycle explicit: action scope, staged transactions, mempool entry, block confirmation, wallet sync, then store/balance assertions. Avoid final-only checks that can pass while pending-state behavior regresses.
|
||||
- Do not replace `process_pending_states()` with raw sleeps or one-off balance polling unless the framework cannot model the scenario. If it cannot, document the missing lifecycle step in the test.
|
||||
- Be careful with fixture parametrization. A test that is only meaningful in trusted mode, untrusted mode, reuse mode, or fresh-derivation mode should pin that mode in the indirect fixture params; otherwise it silently multiplies runtime and may assert the wrong invariant.
|
||||
- Use `standard_block_tools` for tests that inspect consensus-valid block structure, weight proof/header behavior, BIP158 filters, additions/removals proofs, or request-header protocol details. The default wallet block patches intentionally bypass much of that.
|
||||
- Reorg tests should assert both wallet-visible state and durable stores. Many wallet bugs only appear when interested coin IDs, race caches, unconfirmed transactions, singleton records, or remote/interested stores are rolled back and then reprocessed.
|
||||
- For asset tests, assert identity through the asset's authority: CAT TAIL hash, singleton launcher id, NFT/DID/VC lineage, DataLayer launcher/root records, or offer/trade id. Wallet id/order alone is too weak.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- The autouse patches are part of the test architecture. Changing them alters the meaning and runtime of nearly every wallet integration test.
|
||||
- `reuse_puzhash` failures usually indicate a transaction path derived a fresh address outside the intended action-scope/config boundary.
|
||||
- Balance assertions are intentionally stateful. After a negative-path `process_pending_states()` raises, tests may need to restore local expected balances before continuing.
|
||||
- Trusted and untrusted sync paths intentionally diverge. If a test covers peer validation, stale proofs, rollback, or state-from-peer behavior, run or pin the mode that exercises the relevant branch.
|
||||
- RPC tests often share helpers and fixtures from CAT/store tests. Moving helpers can create import cycles or make supposedly endpoint-level tests depend on broader asset setup.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Wallet test fixtures and framework: `chia/_tests/wallet/conftest.py`, `chia/_tests/environments/wallet.py`.
|
||||
- Shared wallet source context: `chia/wallet/wallet_node.py`, `chia/wallet/wallet_state_manager.py`, `chia/wallet/wallet_rpc_api.py`.
|
||||
@@ -0,0 +1,112 @@
|
||||
# Chia Timelord Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/timelord/` is the proof-of-time production service. It does not decide chain validity: the full node sends peaks and unfinished blocks, the timelord schedules VDF work against its current local view, and the full node later validates every returned signage point, end-of-sub-slot, compact proof, and infusion point before accepting side effects.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for timelord VDF scheduling, peak/unfinished-block selection, VDF-client communication, compact proof production, timelord metrics, and full-node/timelord protocol coupling. For final block or VDF acceptance, also read `full-node.md` and `consensus.md`.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `Timelord` owns local VDF scheduling state, VDF client connections, proof collection, compact-proof work queues, and metrics events. Its state is operational state, not canonical chain state.
|
||||
- `TimelordAPI` is the full-node protocol entry point. `new_peak_timelord`, `new_unfinished_block_timelord`, and `request_compact_proof_of_time` mutate `Timelord` state under `timelord.lock`; they trust the full node enough to schedule work, but the returned products are still checked by full-node consensus paths.
|
||||
- `LastState` is the timelord's executable chain view. It can represent the first sub-slot, a peak, or an end-of-sub-slot; all challenge selection, initial classgroup form selection, deficit behavior, SES inclusion, and reward-challenge history derive from this object.
|
||||
- `iters_from_block()` is the narrow bridge from unfinished/finished reward-chain block data to local SP/IP iteration targets. It calls consensus iteration helpers and PoS validation; exceptions or failed required-iters assertions mean the candidate is not schedulable.
|
||||
- VDF clients are local subprocesses or external clients connected to the timelord TCP server. They are only accepted from configured IPs and capped while idle, but their returned proofs are still locally validated with `validate_vdf()` before broadcasting.
|
||||
- Full node remains the acceptance authority. `FullNodeAPI` ignores timelord messages while syncing, serializes IP/EOS handling through `timelord_lock`, reconstructs finished blocks from infusion VDFs, and calls normal block validation before accepting them.
|
||||
|
||||
## Runtime Model
|
||||
|
||||
Normal timelord mode runs three logical VDF chains:
|
||||
|
||||
- challenge chain
|
||||
- reward chain
|
||||
- infused challenge chain, only when `LastState.get_challenge(INFUSED_CHALLENGE_CHAIN)` is available
|
||||
|
||||
`Timelord.manage()` starts a TCP server for VDF clients, initializes `LastState`, and launches `_manage_chains()` unless bluebox mode is enabled. `_manage_chains()` repeatedly handles failures, consumes pending peaks, maps free VDF clients to unspawned chains, submits iteration targets, and checks the lowest unfinished reward-chain iteration for IP, SP, or EOS completion.
|
||||
|
||||
The key reset operation is `_reset_chains()`. It stops existing chain clients after state changes, recomputes all future work relative to the current `LastState.last_ip`, clears stale proof lists by bumping `num_resets`, requeues schedulable unfinished blocks, seeds a limited window of signage points, and always queues the end-of-sub-slot iteration. Changes here affect liveness and correctness more than performance.
|
||||
|
||||
## State And Ordering Contracts
|
||||
|
||||
- `num_resets` labels proofs by generation. Proofs whose label does not match the current reset are intentionally ignored; removing or weakening this filter can mix old-challenge proofs into current-chain messages.
|
||||
- `iters_to_submit`, `iters_submitted`, `iteration_to_proof_type`, `iters_finished`, and `proofs_finished` must stay aligned. A submitted iteration is meaningful only with its chain set and proof type.
|
||||
- SP completion requires both challenge-chain and reward-chain proofs for the same iteration and current reset. The reward-chain VDF challenge must match `LastState.get_challenge(REWARD_CHAIN)` before broadcasting `NewSignagePointVDF`.
|
||||
- IP completion requires CC/RC proofs and sometimes ICC proof. The matching unfinished block is found by recomputing IP iterations against current state; then the timelord broadcasts `NewInfusionPointVDF` and, after early-height safeguards, may locally synthesize a new `NewPeakTimelord` to continue working without waiting for the full node.
|
||||
- EOS completion requires the expected chain count for the end-of-slot iteration. It builds `EndOfSubSlotBundle`, handles optional ICC and sub-epoch-summary inclusion, broadcasts `NewEndOfSubSlotVDF`, updates `LastState`, moves viable overflow blocks into the active unfinished set, and resets chains.
|
||||
- `reward_challenge_cache` is bounded by the sub-slot block window and is used to decide whether an unfinished block's `rc_prev` is in the local chain window. It is not a general fork database.
|
||||
|
||||
## Peak And Unfinished-Block Selection
|
||||
|
||||
Peak handling intentionally does not always chase the newest heavier announcement. `new_peak_timelord()` accepts the first peak, a heavier peak, or equal weight with lower total iterations, but skips a one-height-ahead heavier peak if a cached unfinished/overflow block with lower or equal iterations would be orphaned. This lets a fast local timelord finish a lower-iteration competing block that can win fork choice.
|
||||
|
||||
Unfinished blocks are split into immediate `unfinished_blocks` and future `overflow_blocks`:
|
||||
|
||||
- Non-overflow blocks are scheduled when their IP is still ahead of the current last IP and `_can_infuse_unfinished_block()` approves them.
|
||||
- Overflow blocks whose SP belongs to the previous sub-slot are cached until the EOS window makes their IP schedulable.
|
||||
- Stale overflow blocks whose total iterations are already behind current total iterations are dropped so they cannot indefinitely block future peaks.
|
||||
- New epochs disallow overflow infusion, matching consensus expectations.
|
||||
|
||||
The helper `overflow_sp_total_iters()` captures the core overflow invariant: an overflow block's signage point may live in the previous slot even though its infusion point is in the next slot.
|
||||
|
||||
## Full-Node Coupling
|
||||
|
||||
Full node sends `NewPeakTimelord` after peak processing with reward-chain block, next difficulty/SSI, deficit, possible SES, recent reward challenges, and last challenge-block/EOS total iterations. On unfinished block creation it sends `NewUnfinishedBlockTimelord` with the unfinished reward-chain block, foliage, `rc_prev`, SES, and MMR root context.
|
||||
|
||||
Timelord responses re-enter full-node validation:
|
||||
|
||||
- `NewSignagePointVDF` is adapted into `RespondSignagePoint`, then validated and fanned out to farmers/full nodes by existing full-node signage-point logic.
|
||||
- `NewEndOfSubSlotVDF` is passed to `FullNode.add_end_of_sub_slot()`, which checks predecessor slots, updates `FullNodeStore`, drains future infusion caches, and may resend peak data to a timelord that is on the wrong state.
|
||||
- `NewInfusionPointVDF` looks up a matching unfinished block, reconstructs a `FullBlock` with MMR context, validates the pre-farm/pool signature path, and calls `add_block()`. On validation failure, the full node sends its current peak back to only the offending timelord peer.
|
||||
- `RespondCompactProofOfTime` is a compact-proof response path for bluebox mode; it updates full-node compact VDF handling rather than block production.
|
||||
|
||||
## Bluebox / Compact Proof Mode
|
||||
|
||||
Bluebox mode turns the service into compact-proof production instead of live chain production. `TimelordAPI` ignores peak and unfinished-block messages in this mode and only queues `RequestCompactProofOfTime` entries. Stale compact-proof work is discarded before appending new requests.
|
||||
|
||||
There are two compact-proof execution paths:
|
||||
|
||||
- External VDF clients receive a `BLUEBOX` job via `_manage_discriminant_queue_sanitizer()`.
|
||||
- On Windows or with `slow_bluebox`, `chiavdf.prove()` runs in a `ThreadPoolExecutor`; a tempfile path is used as the shutdown trigger.
|
||||
|
||||
Both paths intentionally choose a random target VDF field first, then fall back to the first queued item. This avoids starvation because CC SP/IP compact proofs are much more common than EOS/ICC proofs.
|
||||
|
||||
## Concurrency And Lifecycle
|
||||
|
||||
- `timelord.lock` is the module's main consistency boundary. API handlers, VDF-client mapping, iteration submission, proof appending, compact queue mutation, and reset operations rely on this lock around multi-field mutations.
|
||||
- `_do_process_communication()` performs network reads outside many lock sections but appends validated proofs and failure records under the lock. Be careful not to hold the lock across long VDF-client reads.
|
||||
- `_handle_failures()` prioritizes liveness. A current-generation VDF-client failure resets to EOS-only work; prolonged inactivity backs off the restart threshold up to a source-defined cap and resets all chains.
|
||||
- Shutdown closes the executor trigger file, cancels communication/main-loop tasks, sends stop signals to VDF clients, closes idle and assigned writers, and closes the TCP server.
|
||||
- `timelord_launcher.py` is a separate process manager for `chiavdf`'s `vdf_client` binary. It resolves the configured host, restarts clients until stopped, suppresses early stderr noise, and kills all active subprocesses on shutdown.
|
||||
|
||||
## RPC And Metrics Surface
|
||||
|
||||
`TimelordRpcApi` exposes no request routes. Its only external surface is websocket metrics payload generation for `finished_pot`, `new_compact_proof`, `skipping_peak`, and `new_peak`. Changing event names or payload keys affects daemon/UI/metrics consumers even though there is no route-level RPC API.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- `_reset_chains()` is the primary correctness/liveness hotspot. It rewrites iteration queues, active unfinished caches, reset labels, and chain process state in one operation.
|
||||
- Overflow handling is consensus-sensitive. Changes must preserve previous-slot SP math, EOS-window scheduling, stale overflow pruning, and no-overflow-in-new-epoch behavior.
|
||||
- Peak skipping is intentional fork-choice support, not stale-state behavior. Removing the orphan check can make the timelord abandon a lower-iteration unfinished block that the full node would have preferred.
|
||||
- `LastState` must mirror the full-node-provided peak/EOS fields closely enough to compute the same challenges, initial forms, deficits, SES inclusion, and transaction-block heights. Divergence typically appears as rejected VDFs or blocks, not as local exceptions.
|
||||
- `proof_label == num_resets` checks protect against stale VDF clients racing after resets. Treat these labels as a concurrency invariant.
|
||||
- The VDF client wire format is ad hoc and length-prefixed with decimal/byte encodings. Any change must be coordinated with the external `vdf_client` binary, not just Python tests.
|
||||
- Bluebox mode and normal mode are mutually exclusive in API behavior. Avoid adding code paths that partially process peaks while compact-proof workers are active.
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- Timelord state-machine tests should use `default_1000_blocks`, `create_blockchain()`, and `timelord_peak_from_block()` style helpers to exercise real block/iteration data.
|
||||
- Peak-selection tests should cover heavier peaks, equal-weight lower-total-iteration peaks, unfinished block orphan prevention, and the full-node reorg result after an IP VDF finishes a lower-iteration candidate.
|
||||
- Overflow tests should separately cover SP-total-iteration math, caching before EOS, scheduling after EOS, stale overflow pruning, and new-epoch rejection.
|
||||
- VDF-client tests can unit-test `_handle_client()` whitelist/cap behavior and `_do_process_communication()` invalid-proof handling with synthetic readers/writers and monkeypatched `validate_vdf()`.
|
||||
- Bluebox tests should cover stale queue expiry, random-field fallback behavior, slow/external proof validation failure, and response broadcasting.
|
||||
- Full-node integration tests are needed for changes that alter timelord message payloads or response ordering, because the acceptance authority and many failure paths live in `FullNode`/`FullNodeAPI`, not in this module.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
- Timelord service state and scheduling: `chia/timelord/timelord.py`, `chia/timelord/timelord_state.py`.
|
||||
- Full-node-facing API and RPC metrics: `chia/timelord/timelord_api.py`, `chia/timelord/timelord_rpc_api.py`.
|
||||
- Iteration derivation and VDF client launcher: `chia/timelord/iters_from_block.py`, `chia/timelord/timelord_launcher.py`.
|
||||
- Service startup and local types: `chia/timelord/start_timelord.py`, `chia/timelord/types.py`.
|
||||
@@ -0,0 +1,158 @@
|
||||
# chia-types
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
Scope: `chia/types/`. This is distilled architectural context for future audit or implementation agents. It intentionally omits file inventory and obvious helper summaries.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for shared blockchain-format types, CLVM `Program` helpers, `Coin`/condition contracts, proof/VDF wrappers, block generator carriers, mempool item carriers, peer info types, and streamable schemas under `chia/types/`. For ownership of behavior, also read the consuming module context: consensus, full node, wallet, server, farmer/harvester, or timelord.
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
`chia.types` is the shared type and helper boundary between consensus, full node, wallet, farmer/harvester, timelord, networking, RPC, and tests. It does not usually own orchestration or persistence. Its job is to expose stable Python-facing shapes for consensus objects, CLVM programs, block generators, mempool items, VDF/POS checks, peer addresses, fee units, condition opcodes, and lightweight protocol-adjacent records.
|
||||
|
||||
The most important design fact is that many "types" are Rust-backed `chia_rs` objects re-exported or wrapped for Python compatibility. Local Python code often adds only:
|
||||
|
||||
- compatibility imports for historical module paths;
|
||||
- CLVM convenience APIs that convert between `Program` and `SerializedProgram`;
|
||||
- validation glue around external proof libraries;
|
||||
- carrier objects that preserve metadata computed by full-node, consensus, or wallet code.
|
||||
|
||||
Treat this module as a shared contract layer. Small edits here can change consensus validation, wallet spend construction, mempool ordering, network address bucketing, or wire/persistence serialization even when the file being edited looks like a simple dataclass.
|
||||
|
||||
## Rust And Serialization Boundary
|
||||
|
||||
`chia_rs` owns the canonical implementations for core consensus types and CLVM execution-heavy objects: `Coin`, `CoinSpend`, `SpendBundle`, `SpendBundleConditions`, `ProofOfSpace`, `VDFInfo`, `VDFProof`, `BlockRecord`, `G1Element`, `G2Element`, and `SerializedProgram` (`chia_rs.Program`). Python files in this module frequently preserve old import paths while delegating behavior to Rust.
|
||||
|
||||
The boundary is compatibility-sensitive:
|
||||
|
||||
- `SerializedProgram` is an alias to `chia_rs.Program`, not a Python class. `isinstance(..., SerializedProgram)` checks are really checks against the Rust extension type.
|
||||
- `Coin`, `ClassgroupElement`, `VDFInfo`, and `VDFProof` are re-exported for legacy import paths; removing those paths can break broad call sites outside the apparent file.
|
||||
- `Streamable` dataclasses in this module are persisted or sent over APIs. Field type changes, enum numeric value changes, and list element changes should be treated like schema migrations.
|
||||
- JSON helpers such as `MempoolItem.to_json_dict()` and `MempoolSubmissionStatus.to_json_dict_convenience()` are RPC/UI compatibility surfaces, not internal debug formatting.
|
||||
|
||||
## CLVM Program Contracts
|
||||
|
||||
`blockchain_format.program.Program` is the Python CLVM s-expression wrapper used heavily by wallet puzzle construction, tests, simulator tooling, and some RPC paths. It is not the consensus-hot block validation path, but it calls Rust CLVM execution and tree hashing.
|
||||
|
||||
Key contracts:
|
||||
|
||||
- `Program.from_bytes()` intentionally parses through the Rust CLVM path. This gives a Python-compatible object while using the faster Rust parser/LazyNode path.
|
||||
- `Program.run()` and `run_with_cost()` default to wallet-style flags, while module-level run helpers preserve lower-level legacy semantics. This distinction matters for strict mempool/soft-fork behavior.
|
||||
- `_run()`, `uncurry()`, and `make_spend()` are compatibility adapters accepting both `Program` and `SerializedProgram`. They should not silently accept arbitrary objects because callers rely on type errors to catch malformed puzzle/solution construction.
|
||||
- `curry()` and `uncurry()` encode/decode the canonical CLVM curry shape. Wallet puzzle drivers and tests assume this shape when matching layered puzzles.
|
||||
- `sha256_treehash()` is deliberately iterative to avoid Python recursion limits on deeply nested CLVM. Replacing it with a recursive implementation changes a robustness property.
|
||||
- `get_tree_hash_precalc()` treats any atom matching a supplied `bytes32` as already hashed. That is a performance and semantic contract used by puzzle-hash construction.
|
||||
|
||||
Canonical serialization rules are enforced elsewhere for block/mempool validation, but this module is where many tools construct and inspect programs before they reach those validators.
|
||||
|
||||
## Coin And Condition Contracts
|
||||
|
||||
`Coin` identity and condition opcode values are consensus-level contracts even though their local Python definitions are small.
|
||||
|
||||
- `coin_as_list()` preserves the CLVM list order `[parent_coin_info, puzzle_hash, amount]`. This order feeds coin-name and puzzle semantics across wallet and consensus code.
|
||||
- `hash_coin_ids()` sorts coin IDs descending before hashing the concatenation, with a special single-coin path. Addition-root and Merkle-style commitments elsewhere must remain consistent with this ordering.
|
||||
- `ConditionOpcode` byte values are CLVM output ABI. Renaming is low risk; changing numeric bytes or deleting legacy opcodes is consensus/protocol risk.
|
||||
- `ConditionWithArgs` is a lightweight parsed-condition carrier. Semantic validation of argument counts, timelocks, announcements, and signatures happens in Rust/full-node/wallet code, not in this dataclass.
|
||||
- `SigningMode` strings are external signing domain identifiers. They must stay in lockstep with the relevant CHIP, BLS augmentation modes, and hardware-wallet/offline signing expectations.
|
||||
|
||||
## Proof Of Space And VDF Contracts
|
||||
|
||||
`blockchain_format.proof_of_space` and `blockchain_format.vdf` contain real validation gates around external proof libraries. They are consensus-critical glue, not just type helpers.
|
||||
|
||||
Proof-of-space validation in `verify_and_get_quality_string()` couples:
|
||||
|
||||
- pool key vs pool-contract puzzle-hash exclusivity;
|
||||
- plot parameter min/max checks from `ConsensusConstants`;
|
||||
- challenge derivation from plot id, original challenge hash, and signage point;
|
||||
- V1 plot-filter prefix reductions by candidate height;
|
||||
- V1 phase-out starting at `HARD_FORK2_HEIGHT` and full cutoff after configured epochs;
|
||||
- SF9 V1 proof-size rejection;
|
||||
- V2 activation gating and Rust `validate_proof_v2()`.
|
||||
|
||||
The `prev_transaction_block_height` parameter is not interchangeable with candidate height. It gates V1 phase-out and V2 activation based on the last transaction block before the current signage point, matching consensus hard-fork semantics. Candidate `height` feeds prefix-bit reductions for both V1 and V2 via `calculate_prefix_bits()`, which uses version-specific constants and height thresholds; both versions then pass through the same `passes_plot_filter()` check.
|
||||
|
||||
VDF validation in `validate_vdf()` couples `VDFProof`, `VDFInfo`, `ClassgroupElement`, and consensus constants:
|
||||
|
||||
- optional `target_vdf_info` equality check catches callers validating the wrong VDF field;
|
||||
- witness type is bounded by `MAX_VDF_WITNESS_SIZE`;
|
||||
- classgroup input must satisfy the exact serialized size expected by VDF validation;
|
||||
- discriminants and Wesolowski verification are cached, but failures return `False` rather than raising invalid-block exceptions.
|
||||
|
||||
`CompressibleVDFField` values identify which block field a compressed VDF proof belongs to. Changes must match full-node/consensus compression and decompression paths.
|
||||
|
||||
## Block Generator And Mempool Contracts
|
||||
|
||||
`BlockGenerator` is the validation-time view of a block generator: program bytes plus the raw previous generator bytes referenced by height. `NewBlockGenerator` extends it for block creation with block-ref heights, aggregate signature, additions, removals, and total cost.
|
||||
|
||||
Important boundaries:
|
||||
|
||||
- Generator reference bytes are resolved by consensus/full-node code; this module stores them but does not validate fork ancestry or ref-list limits.
|
||||
- `NewBlockGenerator.cost` is the total CLVM plus byte plus condition cost used by block creation. Do not confuse it with per-spend cost or mempool virtual cost.
|
||||
- `BlockInfo` is a structural protocol for blocks/header-like objects that expose generator fields. It supports code that works across full blocks, unfinished blocks, and similar objects without imposing inheritance.
|
||||
|
||||
`MempoolItem` is the admitted transaction carrier after expensive validation has already produced `SpendBundleConditions`. It does not independently validate spend correctness.
|
||||
|
||||
Core contracts:
|
||||
|
||||
- `fee_per_cost` controls ordering via `__lt__`; `fee_per_virtual_cost` includes `SPEND_PENALTY_COST` for alternate ordering/estimation paths.
|
||||
- `cost` and `num_spends` are derived from Rust `SpendBundleConditions`. If `conds` is absent, these properties fall back to zero for compatibility, but normal admitted items should have conditions.
|
||||
- `bundle_coin_spends` maps coin id to the original spend plus dedup/fast-forward metadata. `removals` and `to_spend_bundle()` reconstruct from this map, so missing entries produce incorrect RPC output or rebroadcast bundles.
|
||||
- `latest_singleton_lineage` records fast-forward eligibility and current unspent singleton lineage. It is interpreted by full-node mempool logic and coin-store rollback behavior; this module only stores it.
|
||||
- `assert_height`, `assert_before_height`, and `assert_before_seconds` are admission-window metadata used around new peaks. They are not replacements for Rust timelock validation.
|
||||
|
||||
## Peer And Network-Adjacent Contracts
|
||||
|
||||
`PeerInfo`, `UnresolvedPeerInfo`, and `TimestampedPeerInfo` are shared by server, discovery, RPC, tests, and introducer flows. They sit near the network trust boundary but do not decide peer admission.
|
||||
|
||||
- `PeerInfo` currently accepts `str` or `IPAddress` for compatibility and remains mutable/`unsafe_hash=True` until call sites fully migrate. Treat it as legacy-sensitive.
|
||||
- `host` returns the string form for compatibility; `ip` is the normalized address object used for policy and bucketing.
|
||||
- `get_key()` maps IPv4 into an IPv6-derived key space before appending port. `get_group()` groups addresses by network-specific byte prefixes. AddressManager peer selection and anti-sybil bucketing depend on these exact bytes.
|
||||
- `TimestampedPeerInfo` is streamable peer-list data. Freshness and address validity are normalized by server/discovery code, not trusted from the dataclass.
|
||||
|
||||
## Cross-Module Coupling
|
||||
|
||||
Primary consumers:
|
||||
|
||||
- `chia.consensus` consumes `BlockGenerator`, `ValidationState`, VDF/POS helpers, `Coin`, and generator/program aliases during block validation, difficulty/slot checks, and fork-aware generator resolution.
|
||||
- `chia.full_node` consumes `MempoolItem`, `NewBlockGenerator`, `BlockGenerator`, `PeerInfo`, VDF/POS helpers, and `WeightProof` for sync, block creation, tx admission, RPC, and peer handling.
|
||||
- `chia.wallet` consumes `Program`, `SerializedProgram`, `make_spend()`, condition opcodes, signing modes, weight-proof carriers, and peer info for puzzle construction, signing, SPV sync, and tests.
|
||||
- `chia.server` and discovery consume peer-info types for connection identity context, address-book bucketing, and RPC-facing peer data.
|
||||
- `chia.farmer`, `chia.harvester`, `chia.plotting`, and `chia.timelord` consume proof/VDF helpers and plot-key derivation functions around signage points, VDF validation, and plot creation.
|
||||
|
||||
This module generally should not import upward into those consumers. Its dependency direction is toward `chia_rs`, `clvm`, `chiapos`, `chiavdf`, and low-level `chia.util` helpers.
|
||||
|
||||
## Persistence And Compatibility Notes
|
||||
|
||||
- `WeightProof`, `RecentChainData`, `ProofBlockHeader`, `UnfinishedHeaderBlock`, `MempoolSubmissionStatus`, and `FeeRate` are `Streamable` schemas. Existing DB rows, protocol payloads, and RPC fixtures may depend on their exact field order and integer widths.
|
||||
- `MempoolInclusionStatus` values are represented as `uint8` inside some wallet records. Numeric stability matters more than enum member ordering aesthetics.
|
||||
- `FeeRate.create()` rounds up with `math.ceil()`. This prevents underquoting when converting mojos over CLVM cost and mirrors protocol fee-estimate compatibility expectations.
|
||||
- `Mojos` and `CLVMCost` are `NewType` wrappers around `uint64`; they add static clarity only. Runtime code still receives integers.
|
||||
|
||||
## Trust Boundaries
|
||||
|
||||
All serialized blocks, spend bundles, generators, conditions, peer addresses, proof bytes, VDF proofs, and weight proofs that enter these types from peers or RPC remain untrusted. `Streamable` parsing and dataclass construction only establish shape.
|
||||
|
||||
Validation authority lives elsewhere:
|
||||
|
||||
- spend correctness, conditions, signatures, and timelocks: Rust validation plus full-node mempool/consensus code;
|
||||
- block generator ancestry, ref limits, cost limits, and canonical encoding: consensus/full-node validation;
|
||||
- chain weight-proof validity: full-node and wallet weight-proof handlers;
|
||||
- peer admission, rate limits, and address freshness: server/discovery;
|
||||
- wallet ownership and puzzle-driver interpretation: wallet state and puzzle modules.
|
||||
|
||||
Do not add "convenient" validation shortcuts in this module unless they are pure helpers with the same semantics as the owning subsystem. Divergent local checks here can make callers trust a shape that the real authority later rejects.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- High-risk edits: changing `ConditionOpcode` bytes, `SigningMode` strings, streamable field order/types, Rust alias imports, `Program` default flags, proof/VDF height gates, V1 prefix filtering, or peer bucketing bytes.
|
||||
- Be careful with `prev_transaction_block_height` vs candidate `height` in proof-of-space logic. Hard-fork and phase-out gates intentionally use different notions of height in different places.
|
||||
- Do not conflate `Program` and `SerializedProgram`: wallet construction often needs Python CLVM traversal, while full-node validation and generator paths expect serialized Rust-backed bytes.
|
||||
- Do not treat `MempoolItem.to_spend_bundle()` as the original submitted bundle if `bundle_coin_spends` was transformed for dedup or fast-forward behavior; check full-node mempool semantics first.
|
||||
- Replacing Rust-backed aliases with Python duplicates can break serialization equality, hashing, JSON conversion, or consensus validation even if attributes appear identical.
|
||||
- Tests for this module should usually be consumer-driven: consensus tests for proof/generator changes, mempool tests for `MempoolItem` behavior, wallet tests for `Program`/`make_spend()`/signing changes, and server/discovery tests for `PeerInfo`.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
For exact type definitions and helper behavior, read the owning files under `chia/types/`, especially `chia/types/blockchain_format/`, `chia/types/blockchain_format/proof_of_space.py`, `chia/types/condition_opcodes.py`, `chia/types/condition_with_args.py`, `chia/types/generator_types.py`, `chia/types/mempool_item.py`, and `chia/types/peer_info.py`. For proof-of-space behavior changes, also inspect `chia/_tests/core/custom_types/test_proof_of_space.py`.
|
||||
@@ -0,0 +1,253 @@
|
||||
# Chia Util Module Context
|
||||
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
`chia/util/` is the shared infrastructure layer beneath consensus, networking,
|
||||
wallet, daemon, data layer, simulator, and tests. It is not one cohesive product
|
||||
feature; it is a collection of small primitives that become high impact because
|
||||
other modules treat them as authority for serialization, SQLite access, key
|
||||
storage, config persistence, async scheduling, cache bounds, and RPC formatting.
|
||||
|
||||
## When To Read This
|
||||
|
||||
Read this for shared serialization, SQLite transaction wrappers, keychain/keyring behavior, config/file persistence, async scheduling helpers, bounded caches, network/RPC adapters, and shared error types.
|
||||
|
||||
## Landmarks
|
||||
|
||||
| file | owns |
|
||||
| -------------------------------- | ------------------------------------------- |
|
||||
| `chia/util/streamable.py` | deterministic serialization, get_hash() |
|
||||
| `chia/util/db_wrapper.py` | SQLite writer/reader transaction semantics |
|
||||
| `chia/util/keychain.py` | mnemonic/public/private-key operations |
|
||||
| `chia/util/keyring_wrapper.py` | keyring backend selection, passphrase cache |
|
||||
| `chia/util/config.py` | YAML config load/save hygiene |
|
||||
| `chia/util/priority_mutex.py` | priority-ordered lock (block > tx) |
|
||||
| `chia/util/limited_semaphore.py` | bounded active+waiting DoS backpressure |
|
||||
| `chia/util/lru_cache.py` | LRU/FIFO bounded cache primitives |
|
||||
|
||||
## Implementation Authority
|
||||
|
||||
- `streamable.py` is the Python-side deterministic serialization authority for
|
||||
many protocol, wallet, consensus-adjacent, and RPC request/response types.
|
||||
It enforces `@streamable` + frozen dataclass + `Streamable` inheritance at
|
||||
class definition time, caches per-field parse/stream/convert functions, and
|
||||
defines `get_hash()` as `sha256(bytes(self))`. Any behavioral change here can
|
||||
alter wire format, JSON shape, object construction coercion, or consensus
|
||||
hashes for Python-defined streamables.
|
||||
- `db_wrapper.py` is the SQLite transaction authority. Stores in full node,
|
||||
wallet, data layer, simulator, and tests rely on `DBWrapper2` to serialize
|
||||
writes through one writer connection, provide pooled query-only readers, allow
|
||||
same-task nested savepoints, and expose uncommitted writes to same-task readers.
|
||||
Bypassing it loses cancellation-safe cleanup and read/write consistency rules.
|
||||
- `keychain.py`, `keyring_wrapper.py`, and `file_keyring.py` are the local key
|
||||
custody boundary. `Keychain` exposes mnemonic/public/private-key operations;
|
||||
`KeyringWrapper` selects and owns the singleton backend plus passphrase cache;
|
||||
`FileKeyring` persists encrypted key material and labels in `keyring.yaml`.
|
||||
Callers should not treat the file format or cached plaintext as a public API.
|
||||
- `config.py`, `lock.py`, `files.py`, and `path.py` are the persistence hygiene
|
||||
layer for YAML config and local files. Config writes assume an acquired
|
||||
`Lockfile` and use temp-file replacement; path handling intentionally resolves
|
||||
relative paths under the Chia root.
|
||||
- `network.py`, `ip_address.py`, `ws_message.py`, and `json_util.py` are adapters
|
||||
around `aiohttp`, DNS/IP parsing, daemon websocket payloads, and JSON
|
||||
serialization. They are low-level, but they sit on RPC/server trust boundaries.
|
||||
|
||||
## Serialization Contracts
|
||||
|
||||
- Streamable field order is dataclass field order. Reordering fields, changing
|
||||
type annotations, replacing sized ints/bytes with Python primitives, or adding
|
||||
non-default fields changes binary compatibility and possibly hash identity.
|
||||
- `Streamable.parse()` constructs objects without normal `__init__()` or
|
||||
`__post_init__()` and sets parsed fields directly. Constructor calls run
|
||||
streamable post-init coercion, but parse paths trust the field parse functions.
|
||||
Validation that must apply to untrusted bytes belongs in the parse function or
|
||||
caller, not only in a dataclass `__post_init__`.
|
||||
- Dicts serialize as lists of key/value tuples and reject duplicate keys on
|
||||
parse. Lists/bytes/strings use source-defined length prefixes. Optionals use a
|
||||
source-defined presence marker. BLS/sized-byte/Rust FFI types are delegated to their `parse`,
|
||||
`parse_rust`, `stream`, or `__bytes__` implementations.
|
||||
- `from_bytes()` rejects trailing bytes. Code that wants partial parsing must
|
||||
use `parse()` deliberately and own the remaining stream.
|
||||
- `list_limits` can truncate top-level list fields while still consuming the
|
||||
full serialized input; for Rust-backed objects `_apply_list_limits()` recurses
|
||||
through truncatable children. This is a defensive display/DoS feature, not a
|
||||
consensus validation substitute.
|
||||
- JSON conversion emits byte-like values as `0x` hex strings and large/sized ints
|
||||
as Python ints unless an object supplies a JSON override. RPC callers depend on
|
||||
this stable shape for CLI/daemon/wallet interactions.
|
||||
|
||||
## SQLite And Transaction Semantics
|
||||
|
||||
- `DBWrapper2.managed()` is preferred over `create()`/`close()` because it uses an
|
||||
async exit stack and shields connection cleanup. Most new code should use the
|
||||
managed form unless integrating with legacy lifecycle code.
|
||||
- The writer connection is the only write-capable connection. Reader connections
|
||||
are put into `pragma query_only` and pooled. `reader_no_transaction()` returns
|
||||
the writer connection when the current task already owns the writer so callers
|
||||
can read their own uncommitted changes.
|
||||
- `writer()` and `writer_maybe_transaction()` are savepoint based. Nested writers
|
||||
are allowed only within the same asyncio task. Cross-task writes are serialized
|
||||
by `_lock`; same-task nested writes become nested savepoints.
|
||||
- Savepoint cleanup is cancellation-sensitive. `_savepoint_ctx()` deliberately
|
||||
shields rollback/release and temporarily clears pending cancellation state when
|
||||
needed so orphan savepoints do not trap later writes in an
|
||||
invisible uncommitted transaction. Do not simplify this unless tests cover
|
||||
cancellation during `SAVEPOINT`, `ROLLBACK TO`, and `RELEASE`.
|
||||
- Foreign-key enforcement can be temporarily changed only for an outer writer.
|
||||
Requesting delayed enforcement inside a nested writer raises
|
||||
`NestedForeignKeyDelayedRequestError`, because nested checks would be ambiguous.
|
||||
- `reader()` starts `BEGIN DEFERRED` when needed and rolls back on exit even for
|
||||
read paths, protecting against accidental writes through a reader connection.
|
||||
- `SQLITE_MAX_VARIABLE_NUMBER`, `SQLITE_INT_MAX`, and `host_parameter_limit` are
|
||||
cross-module batching constraints for large store queries and inserts.
|
||||
|
||||
## Keychain And Secret Storage
|
||||
|
||||
- Private key entries are stored as public key bytes plus entropy bytes; public
|
||||
only entries store only public key bytes. `KeyData` reconstructs and checks
|
||||
fingerprint/public/private-key consistency when secrets are included.
|
||||
- Mnemonic handling follows BIP39-style entropy/checksum processing and accepts
|
||||
four-character word prefixes for ASCII seed phrases. Invalid ordering or
|
||||
unknown words raise before key derivation.
|
||||
- The file keyring is always encrypted, even without a user master passphrase:
|
||||
absence of a user passphrase means the stable default passphrase is used. That
|
||||
constant is compatibility-sensitive; changing it strands existing passphrase-less
|
||||
keyrings unless a migration is provided.
|
||||
- The file keyring's on-disk encryption format and the default (no-user-passphrase) passphrase are compatibility-sensitive: changing either strands existing keyrings without a migration. Exact KDF/cipher/file-layout details live in `chia/util/file_keyring.py`.
|
||||
- `FileKeyring` maintains two caches: outer encrypted file content and decrypted
|
||||
key/label data. A watchdog observer plus `Lockfile` mark/reload external file
|
||||
modifications. Writes merge staged outer properties such as passphrase hints.
|
||||
- `KeyringWrapper` caches the master passphrase and can persist it to macOS or
|
||||
Windows credential storage. On macOS headless access, `errSecInteractionNotAllowed`
|
||||
is warned and handled specially. Interactive prompting currently lives below
|
||||
some storage methods, so daemon/CLI call paths must account for possible prompts.
|
||||
- Labels are unique across fingerprints, trimmed exactly, bounded by
|
||||
`MAX_LABEL_LENGTH`, and reject leading/trailing whitespace, tabs, and newlines.
|
||||
|
||||
## Concurrency And Scheduling Primitives
|
||||
|
||||
- `PriorityMutex` gives lower enum values first and explicitly rejects nested
|
||||
acquisition by the active task. `Blockchain` uses this to let block validation
|
||||
outrank transaction processing. Starvation/fairness behavior is priority-first
|
||||
within FIFO deques; changing queue ordering changes full-node latency semantics.
|
||||
- `PriorityThreadPoolExecutor` is not a drop-in `ThreadPoolExecutor`. Work is
|
||||
ordered by `nice`, then FIFO sequence. Dedicated work is dual-posted to both a
|
||||
dedicated queue and the general queue so general threads can help while
|
||||
dedicated threads remain reserved. A shared `Future` and claim lock ensure only
|
||||
one thread runs each job.
|
||||
- `AsyncPool` keeps a target number of async workers alive, logs and consumes
|
||||
worker exceptions in the supervisor loop, and shields teardown cancellation.
|
||||
`QueuedAsyncPool` adds job/result queues plus per-job started/done state and
|
||||
cancellation flags.
|
||||
- `LimitedSemaphore` bounds both active and waiting requests. It raises
|
||||
`LimitedSemaphoreFullError` before enqueueing when no waiting slot is left;
|
||||
full-node request handlers use this as a DoS/backpressure signal.
|
||||
- `task_referencer`, `safe_cancel_task`, `log_exceptions`, `task_timing`, and
|
||||
related helpers exist to make async lifecycle explicit. Fire-and-forget tasks
|
||||
should generally be referenced or awaited through these helpers to avoid silent
|
||||
GC/cancellation surprises.
|
||||
|
||||
## Cache And Bounded-State Semantics
|
||||
|
||||
- `LRUCache` is access-order LRU, but `LRUSet` and `LRUKeyedListCache` use FIFO
|
||||
key eviction despite the `LRU` prefix. Do not assume reads promote entries in
|
||||
the latter two structures.
|
||||
- `LRUKeyedListCache` is a bounded dict-of-lists with per-key entry limits,
|
||||
total-entry accounting, optional monotonic-time TTL, and insertion-order-based
|
||||
expiry. Full-node future-object and peer-advertisement caches rely on these
|
||||
limits as resource controls, not just memory optimizations.
|
||||
- `BlockCache` implements consensus blockchain interface protocols over an
|
||||
in-memory `BlockRecord` map plus height map and delegates MMR-root computation
|
||||
to an injected `MMRManagerProtocol`. It is used as an augmented or lightweight
|
||||
chain view; it must remain consistent between header hash, height, and MMR
|
||||
manager updates.
|
||||
|
||||
## Config, Filesystem, And Process Setup
|
||||
|
||||
- Config loading retries YAML reads even though a lock should prevent partial
|
||||
reads. Missing config exits the process by default, so library-style callers
|
||||
must pass `exit_on_error=False` if they need exceptions instead.
|
||||
- `lock_and_load_config()` holds the config lock only around load. `save_config()`
|
||||
assumes the caller already acquired that lock. Code that loads, mutates, and
|
||||
saves config should keep the lock over the whole read-modify-write sequence.
|
||||
- CLI overrides are generated from flattened config keys and skip list values.
|
||||
Nested config path syntax uses dots for CLI override flattening and colons for
|
||||
`traverse_dict()`, so these are not interchangeable.
|
||||
- `process_config_start_method()` validates configured multiprocessing start
|
||||
methods and logs the selected method. Full-node and wallet process-pool code
|
||||
depends on this to avoid invalid platform choices.
|
||||
- `write_file_async()` fsyncs temp-file contents, moves into place with retries,
|
||||
chmods the final path, and cleans up temp files. It does not fsync the parent
|
||||
directory; callers needing crash-proof directory-entry durability would need a
|
||||
stronger primitive.
|
||||
|
||||
## Network, RPC, And Trust Helpers
|
||||
|
||||
- `WebServer.create()` wraps `aiohttp.web.Application`/`AppRunner`/`TCPSite` and
|
||||
records the actual ephemeral listen port after start. The current branch is
|
||||
sensitive to aiohttp site/server creation behavior; preserve the invariant that
|
||||
an ephemeral port request resolves to the selected bound IPv4/IPv6 port from runner addresses.
|
||||
- `close()` schedules cleanup in a referenced task; callers must call
|
||||
`await_closed()` to observe shutdown completion. Treating `close()` as already
|
||||
awaited can leak server resources in tests or service shutdown.
|
||||
- `is_trusted_peer()` trusts localhost unless in testing mode, explicit node-id
|
||||
entries, or configured CIDRs. This helper feeds server/RPC behavior, so changes
|
||||
affect ban/exemption and privileged-connection paths.
|
||||
- DNS resolution prefers IPv4 by default but can prefer IPv6. Host strings that
|
||||
are already IP addresses skip DNS. `parse_host_port()` uses URL parsing to
|
||||
support IPv6 bracket syntax and rejects missing host or port.
|
||||
- `ws_message.py` defines daemon-style websocket envelopes. `request_id` is a
|
||||
random `bytes32` hex string; `format_response()` swaps origin/destination and
|
||||
marks `ack=True`.
|
||||
|
||||
## Error Surface
|
||||
|
||||
- `Err` is a shared negative/positive enum across consensus, protocol, mempool,
|
||||
and API errors. The sign historically groups temporary versus permanent
|
||||
categories, but ban/disconnect policy is decided at each handling site, not by
|
||||
the sign alone: `INVALID_PROTOCOL_MESSAGE = -4` is negative yet explicitly
|
||||
bans. Renumbering or reclassifying values changes behavior and
|
||||
wire/user-visible diagnostics.
|
||||
- `ValidationError`, `ConsensusError`, `ProtocolError`, and `ApiError` all carry
|
||||
an `Err` code but are handled by different subsystems. Do not collapse them
|
||||
without auditing ban logic, RPC error formatting, and validation call sites.
|
||||
- Keychain exceptions are intentionally granular because CLI/daemon UX needs to
|
||||
distinguish locked keyring, bad passphrase, duplicate fingerprint/label,
|
||||
missing secrets, unsupported file version, and OS credential-store failures.
|
||||
|
||||
## Fragility Hotspots
|
||||
|
||||
- Changing `streamable` parse/stream/coercion behavior without checking protocol,
|
||||
wallet request, consensus-adjacent, and test fixtures. The blast radius is much
|
||||
larger than `chia/util/streamable.py`.
|
||||
- Simplifying `DBWrapper2` cancellation shields, nested savepoint behavior,
|
||||
same-task writer reads, or reader pooling. These are transaction correctness
|
||||
contracts for persistent stores.
|
||||
- Treating cache bounds and limited semaphores as arbitrary tuning constants.
|
||||
Several callers use them as peer-driven resource-exposure limits.
|
||||
- Moving keyring prompts or passphrase caching without checking daemon and CLI
|
||||
call paths. Some storage methods can currently prompt indirectly.
|
||||
- Adding inline imports to avoid cycles in keychain/keyring code without
|
||||
documenting the cycle. This area already has a few local imports caused by
|
||||
circular dependencies; new ones should be a last resort.
|
||||
- Editing config read-modify-write flows without keeping the file lock across
|
||||
the full mutation. Atomic writes protect file replacement, not logical lost
|
||||
updates between processes.
|
||||
|
||||
## Test And Audit Strategy
|
||||
|
||||
- `chia/_tests/core/util/test_streamable.py` covers decorator, binary, JSON,
|
||||
list-limit, enum, and type-conversion behavior for streamables.
|
||||
- `chia/_tests/core/test_db_validation.py`, `chia/_tests/core/test_db_conversion.py`,
|
||||
store tests under `chia/_tests/core/full_node/stores/`, wallet RPC tests, and
|
||||
data-layer tests exercise `DBWrapper2` under realistic store usage.
|
||||
- `chia/_tests/util/test_lru_cache.py` documents bounded keyed-list cache
|
||||
semantics.
|
||||
- `chia/_tests/core/full_node/test_full_node_api_rate_hardening.py` and full-node
|
||||
tests exercise `LimitedSemaphore` behavior under request pressure.
|
||||
- Keychain behavior is spread across command, daemon, simulator, and wallet
|
||||
tests because key storage is a shared local environment dependency.
|
||||
|
||||
## Source Pointers
|
||||
|
||||
For exact infrastructure behavior, read the owning utility in `chia/util/` rather than copied notes; the highest-impact primitives are listed in the Landmarks table above. Related non-landmarked files: `chia/util/file_keyring.py` (encrypted key file format) and `chia/util/errors.py` (`Err` enum).
|
||||
+63
-157
@@ -1,179 +1,85 @@
|
||||
# Wallet Layer — Deep Context
|
||||
# Chia Wallet Module Context
|
||||
|
||||
> Attach when touching `chia/wallet/`.
|
||||
Verified: 2026-07-12 against 24db9ad3901d. If source contradicts this doc, trust source and update the doc.
|
||||
|
||||
## File map (top-level)
|
||||
`chia/wallet/` is the SPV wallet, transaction-construction, subwallet, offer, and wallet-RPC boundary. Its safety properties are distributed across the wallet node's peer trust model, `WalletStateManager`'s state ownership, action-scope staging, SQLite stores, and CLVM puzzle-driver conventions. Do not treat any single wallet class as the source of truth without checking the surrounding persistence and sync path.
|
||||
|
||||
| File | Lines | Role |
|
||||
| -------------------------------- | ----- | ----------------------------------------------------- |
|
||||
| `wallet_state_manager.py` | ~3330 | `WalletStateManager`: all wallet state, coin tracking |
|
||||
| `wallet_rpc_api.py` | ~3610 | `WalletRpcApi`: full wallet RPC surface |
|
||||
| `wallet_rpc_client.py` | ~1030 | RPC client for CLI/tests |
|
||||
| `wallet_node.py` | ~1750 | `WalletNode`: SPV sync, peer communication |
|
||||
| `wallet_node_api.py` | ~210 | P2P message handlers |
|
||||
| `wallet.py` | ~670 | `Wallet`: standard XCH wallet logic |
|
||||
| `wallet_blockchain.py` | ~250 | Lightweight chain tracking for wallet |
|
||||
| `conditions.py` | ~1550 | Condition parsing and construction |
|
||||
| `coin_selection.py` | ~190 | Coin selection algorithm |
|
||||
| `trade_manager.py` | ~1060 | Offer/trade management |
|
||||
| `wallet_request_types.py` | ~2550 | RPC request type definitions |
|
||||
| `wallet_coin_store.py` | ~350 | Wallet-side coin persistence |
|
||||
| `wallet_transaction_store.py` | ~500 | Transaction record persistence |
|
||||
| `wallet_puzzle_store.py` | ~390 | Derivation/puzzle hash store |
|
||||
| `wallet_weight_proof_handler.py` | ~130 | Weight proof handling for wallet |
|
||||
| `notification_manager.py` | ~120 | Notification handling |
|
||||
| `wallet_action_scope.py` | ~170 | Action scope for atomic operations |
|
||||
| `derive_keys.py` | ~140 | Key derivation |
|
||||
| `singleton.py` | ~110 | Singleton utilities |
|
||||
| `wallet_coin_record.py` | ~80 | `WalletCoinRecord` type |
|
||||
| `transaction_record.py` | ~120 | `TransactionRecord` type |
|
||||
| `start_wallet.py` | ~120 | Service startup |
|
||||
## When To Read This
|
||||
|
||||
## Sub-wallet modules
|
||||
Read this for wallet sync, key/derivation state, transaction construction, wallet RPC transaction endpoints, asset wallets, offers, signing, and wallet-owned persistence. For consensus acceptance or full-node mempool policy, read `consensus.md` or `full-node.md`.
|
||||
|
||||
| Directory | Purpose |
|
||||
| ------------- | ----------------------------------------------- |
|
||||
| `cat_wallet/` | Chia Asset Token (CAT) wallet |
|
||||
| `did_wallet/` | Decentralized Identity wallet |
|
||||
| `nft_wallet/` | NFT wallet |
|
||||
| `vc_wallet/` | Verifiable Credentials wallet |
|
||||
| `db_wallet/` | DataLayer wallet |
|
||||
| `trading/` | Offer trading utilities |
|
||||
| `puzzles/` | CLVM puzzle definitions |
|
||||
| `util/` | Wallet utilities, tx config, puzzle compression |
|
||||
## Landmarks
|
||||
|
||||
---
|
||||
| file | owns |
|
||||
| ------------------------------------- | --------------------------------------------------- |
|
||||
| `chia/wallet/wallet_state_manager.py` | central wallet state, coin/tx records, side effects |
|
||||
| `chia/wallet/wallet_node.py` | SPV peer sync, trusted/untrusted proof validation |
|
||||
| `chia/wallet/wallet_action_scope.py` | transaction side-effect staging boundary |
|
||||
| `chia/wallet/wallet.py` | standard spend construction |
|
||||
| `chia/wallet/wallet_rpc_api.py` | tx endpoint gating, TXConfig autofill |
|
||||
| `chia/wallet/coin_selection.py` | coin selection accounting/exclusions |
|
||||
| `chia/wallet/outer_puzzles.py` | puzzle-driver registry for layered assets |
|
||||
| `chia/wallet/trade_manager.py` | offer/trade lifecycle, interested subscriptions |
|
||||
|
||||
## Coin selection
|
||||
## Implementation Authority
|
||||
|
||||
**Location**: `wallet/coin_selection.py`
|
||||
- `WalletStateManager` owns wallet identity, derivation state, coin records, transaction records, subwallet registry, interested coin/puzzle subscriptions, and side-effect application. Subwallet classes build wallet-specific spends, but state ownership and cross-wallet coordination remain centralized here.
|
||||
- `WalletNode` is the peer-facing SPV coordinator. Full-node messages are typed protocol payloads, not chain authority; trusted peers bypass much of the proof machinery, while untrusted peers require weight-proof/header/Merkle validation before wallet state can be accepted.
|
||||
- `WalletActionScope` is the transaction side-effect boundary. Spend builders stage transactions, signing responses, extra spends, singleton records, selected coins, and unused derivation records; persistence/signing/push behavior happens after the scope exits through `add_pending_transactions()`.
|
||||
- Wallet RPC transaction endpoints are not thin method calls. `tx_endpoint()` gates sync/connectivity, autofills `TXConfig`, folds in extra conditions and absolute timelocks, rejects relative timelocks, opens the action scope, and then normalizes signing/push response metadata.
|
||||
- Puzzle drivers in `outer_puzzles.py` are the registry authority for reconstructing and solving layered assets. `PuzzleInfo.type()` determines the driver; layer ordering and recursive `also` data are part of the asset contract.
|
||||
|
||||
### `select_coins(spendable_amount, config, spendable_coins, unconfirmed_removals, log, amount)`
|
||||
## Mutation Ordering Contracts
|
||||
|
||||
1. **Filter**: Remove unconfirmed removals, excluded coin IDs/amounts,
|
||||
coins outside min/max amount bounds
|
||||
2. **Max coins**: 500 per selection
|
||||
3. **Sort + exact checks**: Exact single-coin match first, then exact sum of
|
||||
all smaller coins if feasible
|
||||
4. **Selection strategy**:
|
||||
- If smaller coins are insufficient: select smallest coin over target
|
||||
- Otherwise: run randomized knapsack search, then fallback to
|
||||
`sum_largest_coins()`, then smallest over target
|
||||
- New transaction creation must preserve the action-scope lifecycle: select coins and derive puzzle hashes inside the scope, append staged `TransactionRecord`s, then let scope exit merge spends, sign, persist, subscribe to additions/removals, push if requested, and commit any unused derivation result only when `push=True`.
|
||||
- `WalletActionConfig.adjust_for_side_effects()` feeds selected coin IDs back into `TXConfig.excluded_coin_ids`. Nested or multi-step spends rely on this to avoid selecting the same coin twice before any DB mutation exists.
|
||||
- `WalletStateManager.lock` protects wallet state transitions that cross stores, network acks, and coin-state ingestion. Review changes that move DB writes, queue removal, or transaction-state updates outside this lock as ordering changes, not style edits.
|
||||
- `WalletCoinStore` derives spent state from the stored spent height on insert. Rollback deletes coins confirmed after the target height and unspends coins spent after it; code that updates only one representation creates contradictory balance views.
|
||||
- `WalletTransactionStore` has primary serialized transaction blobs plus side tables for condition-valid-times and in-memory unconfirmed caches. `rollback_to_block()` only deletes `transaction_record` rows above the target height and resets `tx_submitted`; it does not delete condition-valid-time rows or reload `unconfirmed_txs`. `WalletStateManager.reorg_rollback()` selectively re-adds certain transaction types afterward. Treat full three-representation synchronization as a review invariant, not a guarantee of the current rollback path.
|
||||
- `WalletInterestedStore` is the durable counterpart to in-memory interested caches. Unknown CATs, non-HD subscriptions, trade coin interests, and retry state depend on store/cache consistency across restart and reorg.
|
||||
|
||||
### `CoinSelectionConfig` fields
|
||||
## Sync And Peer Trust
|
||||
|
||||
- `min_coin_amount`, `max_coin_amount`
|
||||
- `excluded_coin_ids: set[bytes32]`
|
||||
- `excluded_coin_amounts: set[uint64]`
|
||||
- `NewPeakQueue` priority is semantic: coin-id subscriptions, puzzle-hash subscriptions, and coin-state updates are processed before `new_peak_wallet` because full nodes send relevant state before the corresponding peak. Reordering this queue can make otherwise valid updates look unverifiable.
|
||||
- `finished_sync_up_to` is advanced only when long sync completes or when a peer already considered synced reports a peak. Rollback may lower it via the blockchain store; using it as "current network height" conflates local wallet validation progress with peer announcements.
|
||||
- `synced_peers` is per-peer wallet-sync state, not a global proof of wallet freshness. Peer disconnect clears peer caches and synced status; trusted-peer disconnect also resets local-node synced state and may restart peer discovery.
|
||||
- Trusted sync processes coin states and reorgs directly against local assumptions. Untrusted sync validates creation/spend proofs against additions/removals roots, header inclusion, wallet chain state, and cached race data that exists because coin-state updates can arrive before the validating peak.
|
||||
- `CoinState(created_height=None)` is used for reorg/removal-from-chain semantics. Treating it as merely "unknown height" loses rollback meaning.
|
||||
- `PeerRequestCache` is per-peer and height-sensitive. Cached validation or fetched headers must be cleared past fork height; race-cache entries are applied only when the later peak/backtrack path makes their height meaningful.
|
||||
|
||||
---
|
||||
## Transaction, Signing, And RPC Boundaries
|
||||
|
||||
## Wallet sync model
|
||||
- Standard wallet spend construction balances removals against additions plus fee unless explicitly using special internal paths such as negative change. Coin selection must account for confirmed wallet coins, pending removals, trade locks, current action-scope selected coins, and user-supplied min/max/exclusion config.
|
||||
- `TransactionEndpointRequest.to_json_dict()` is intentionally banned for transport; callers must use `json_serialize_for_transport()` so tx config, timelocks, and extra conditions are not silently dropped.
|
||||
- Signing can be local, absent, or supplemented by additional signing responses. Offline/remote signing flows pass unsigned spends plus hook/path/sum hints and expect responses to bind back to the intended spend bundle.
|
||||
- `merge_spends` can rewrite transaction names and concentrate the aggregate spend bundle on one transaction record. Code that keys follow-up behavior by tx name must account for post-merge identity, not just pre-build records.
|
||||
- `parse_timelock_info()` collapses duplicated timelock conditions with min/max rules, and transaction RPCs reject relative timelocks even though condition parsing can represent them.
|
||||
- Mempool acks are accepted only when matching an in-flight send marker for that peer. Stale or unsolicited acks are ignored; failed trusted/untrusted paths differ when the peer reports syncing-related rejection.
|
||||
|
||||
**Location**: `wallet/wallet_node.py`
|
||||
## Assets, Singletons, And Puzzle Drivers
|
||||
|
||||
### Sync flow
|
||||
- CAT identity is the TAIL/limitations-program hash. CAT validity depends on ring accounting across the spend bundle and lineage proofs shaped as parent id, inner puzzle hash, and amount.
|
||||
- DID, NFT, VC, and DataLayer-style singleton assets derive identity from the launcher coin name. Their puzzle hashes bind the singleton top layer, launcher id/hash, and current inner layer; lineage proofs must match the previous coin, previous inner puzzle hash, and previous amount, with eve/launch spends using special reduced forms.
|
||||
- Ownership recognition is local. Hints and fetched parent spends can reveal candidate assets, but `puzzle_store` derivation records decide whether the inner puzzle belongs to this wallet.
|
||||
- DID/NFT/VC handlers reconstruct current state from parent spends plus local stores. DID amount oddness, NFT singleton lineage, VC amount-one expectations, and VC proof hashes are cross-module assumptions used by spend builders and recognizers.
|
||||
- Credential-restricted CAT and VC approval flows cross several authorities: CAT lineage, VC singleton lineage, DID ownership/announcements, proof hashes, and provider lists. Offer approval code has special-case assumptions around single VC/proof-checker shape.
|
||||
- Driver recursion is layer-order sensitive: CAT/CR, singleton/metadata/ownership/transfer, and VC revocation/metadata/covenant layers must be reconstructed and solved in the same order they were matched.
|
||||
|
||||
1. Receive `new_peak_wallet` from full node
|
||||
2. If behind: request weight proof → validate
|
||||
3. Subscribe to puzzle hashes via `register_for_ph_updates`
|
||||
4. Subscribe to coin IDs via `register_for_coin_updates`
|
||||
5. Receive `coin_state_update` pushes for subscribed items
|
||||
6. Process coin state changes → update local wallet DB
|
||||
## Offers And Trading State
|
||||
|
||||
### Mempool tracking
|
||||
- External offers enter as compressed/bech32 `Offer` values and are decompressed into spend bundles plus driver dictionaries. Requested payments are notarized from sorted offered coins; duplicate requested payments and missing requested-asset drivers are rejected at construction.
|
||||
- `Offer.fees()` is accounting metadata, not a validation authority. Spend validity still comes from CLVM execution, announcement assertions, and wallet/full-node admission paths.
|
||||
- `TradeStore` records are keyed by trade id and deduplicated by offer name. It also mirrors coin interests into side tables so accept/confirm/cancel tracking can react to wallet coin updates.
|
||||
- Saving a trade registers non-offer additions/removals as interested coin ids. Removing or reclassifying trade side effects without updating interested subscriptions can make future wallet state look unrelated to the trade.
|
||||
|
||||
- Wallet send path tracks acceptance via `transaction_ack`
|
||||
- Protocol supports `mempool_items_added` / `mempool_items_removed`, but the
|
||||
reference wallet node API does not currently process those message types
|
||||
## Fragility Hotspots
|
||||
|
||||
### Trust model
|
||||
- High-risk edits move work across action-scope exit, wallet-state locks, DB writer boundaries, or queue priority boundaries. These are behavioral contracts even when the code looks like plumbing.
|
||||
- Store schemas carry compatibility baggage: some values are legacy serialized blobs, some columns store bytes while names imply text/hex, and request parsing accepts both old and new JSON shapes.
|
||||
- Many asset recognizers and drivers depend on positional CLVM structure, dynamic `PuzzleInfo`/`Solver` fields, and `assert`-style invariants. Review parser changes with actual parent-spend and layer-order examples.
|
||||
- Wallet sync tests often need to assert ordering effects rather than return values: state update before peak, trusted vs untrusted rollback, race-cache application, ack matching, and action-scope commit timing.
|
||||
- Mutable defaults remain in some wallet APIs for historical reasons. Before assuming per-call isolation, check whether a list/dict is copied at the boundary or shared through the class/function default.
|
||||
|
||||
- **Trusted mode**: Connected to own full node, skip weight proof verification
|
||||
- **Untrusted mode**: Full weight proof validation required
|
||||
## Source Pointers
|
||||
|
||||
---
|
||||
|
||||
## Wallet state manager
|
||||
|
||||
**Location**: `wallet/wallet_state_manager.py`
|
||||
|
||||
### Responsibilities
|
||||
|
||||
- Key management and derivation
|
||||
- Coin record tracking (confirmed/unconfirmed/pending)
|
||||
- Transaction creation and signing
|
||||
- Sub-wallet registry and lifecycle
|
||||
- Puzzle hash generation and caching
|
||||
- Action scope management for atomic multi-step operations
|
||||
|
||||
### Key state
|
||||
|
||||
- `puzzle_store: WalletPuzzleStore` — derivation indexes and puzzle hashes
|
||||
- `coin_store: WalletCoinStore` — wallet-side coin records
|
||||
- `tx_store: WalletTransactionStore` — transaction history
|
||||
- `user_store: WalletUserStore` — sub-wallet metadata
|
||||
- `interested_store: WalletInterestedStore` — tracked coin/puzzle IDs
|
||||
|
||||
### Sub-wallet types
|
||||
|
||||
Each sub-wallet type handles its own puzzle construction, spend creation,
|
||||
and coin tracking:
|
||||
|
||||
- Standard wallet (XCH)
|
||||
- CAT wallet (fungible tokens)
|
||||
- DID wallet (identity)
|
||||
- NFT wallet (non-fungible tokens)
|
||||
- DataLayer wallet
|
||||
- VC wallet (verifiable credentials)
|
||||
|
||||
---
|
||||
|
||||
## Transaction construction
|
||||
|
||||
### General flow
|
||||
|
||||
1. Select coins via `coin_selection.py`
|
||||
2. Construct puzzle reveals and solutions
|
||||
3. Create `CoinSpend` objects
|
||||
4. Aggregate into `SpendBundle`
|
||||
5. Sign with BLS keys
|
||||
6. Submit via `send_transaction` to full node
|
||||
|
||||
### `WalletActionScope`
|
||||
|
||||
Provides atomic operation scope for multi-step wallet operations.
|
||||
Tracks additions, removals, and intermediate state. On failure,
|
||||
all changes can be rolled back.
|
||||
|
||||
---
|
||||
|
||||
## Wallet RPC surface
|
||||
|
||||
**Location**: `wallet/wallet_rpc_api.py`
|
||||
|
||||
`wallet_rpc_api.py` is one of the larger wallet modules (~3610 lines).
|
||||
Key endpoint categories:
|
||||
|
||||
- **Key management**: `log_in`, `get_public_keys`, `generate_mnemonic`
|
||||
- **Wallet info**: `get_wallets`, `get_wallet_balance`, `get_sync_status`
|
||||
- **Transactions**: `send_transaction`, `get_transactions`, `delete_unconfirmed_transactions`
|
||||
- **Coin management**: `get_spendable_coins`, `select_coins`, `get_coin_records_by_names`
|
||||
- **CAT**: `cat_spend`, `cat_get_asset_id`, `create_new_cat_wallet`
|
||||
- **NFT**: `nft_mint_nft`, `nft_transfer_nft`, `nft_get_nfts`
|
||||
- **DID**: `did_get_info`, `did_update_metadata`, `did_transfer_did`
|
||||
- **Offers**: `create_offer_for_ids`, `take_offer`, `get_all_offers`
|
||||
- **DataLayer**: `dl_*` endpoints
|
||||
- **Fee estimation**: `get_fee_estimate`
|
||||
|
||||
---
|
||||
|
||||
## Wallet blockchain
|
||||
|
||||
**Location**: `wallet/wallet_blockchain.py`
|
||||
|
||||
Lightweight chain state for the wallet. Tracks:
|
||||
|
||||
- Peak height and header hash
|
||||
- Sub-epoch summaries (for weight proof validation)
|
||||
- Does NOT store full blocks — only enough for sync verification
|
||||
For coin-selection, sync, RPC, action-scope, puzzle-driver, and trade-manager behavior, read the files in the Landmarks table above. For persistence, timelock, and offer boundaries not landmarked, read `chia/wallet/wallet_transaction_store.py`, `chia/wallet/wallet_interested_store.py`, `chia/wallet/conditions.py`, `chia/wallet/trading/offer.py`, and `chia/wallet/trading/trade_store.py`.
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for apis
|
||||
globs:
|
||||
- chia/apis/**
|
||||
---
|
||||
|
||||
# Chia APIs Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/apis.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses protocol, server, or service boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for benchmarks
|
||||
globs:
|
||||
- benchmarks/**
|
||||
---
|
||||
|
||||
# Benchmarks Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/benchmarks.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
description: Auto-attach repository tooling context for GUI submodule
|
||||
globs:
|
||||
- chia-blockchain-gui/**
|
||||
- chia-blockchain-gui
|
||||
---
|
||||
|
||||
# Repository Tooling Context Available
|
||||
|
||||
Consolidated repository tooling context exists at `.cursor/context/repo-tooling.md`.
|
||||
|
||||
Read it before broad source exploration. If runtime behavior is involved, also read the affected source module context listed in `.cursor/context/INDEX.md`.
|
||||
@@ -9,7 +9,6 @@ globs:
|
||||
- chia/consensus/condition_tools.py
|
||||
- chia/consensus/generator_tools.py
|
||||
- chia/consensus/get_block_generator.py
|
||||
- chia/consensus/cost_calculator.py
|
||||
---
|
||||
|
||||
# CLVM / Puzzle Execution Context Available
|
||||
@@ -17,5 +16,6 @@ globs:
|
||||
Deep context for this subsystem exists at `.cursor/context/clvm-execution.md`.
|
||||
Read it before exploring code — it documents the three CLVM execution paths
|
||||
(mempool, block validation, block building), resource limits (cost, atoms, pairs),
|
||||
canonical serialization, condition opcodes, AGG_SIG replay protection, and
|
||||
`MEMPOOL_MODE` flag behavior.
|
||||
canonical serialization, AGG_SIG replay protection, and `MEMPOOL_MODE` flag
|
||||
behavior. Condition opcode byte values are owned by `chia/types/condition_opcodes.py`;
|
||||
see `.cursor/context/types.md` for that ABI invariant.
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for cmds
|
||||
globs:
|
||||
- chia/cmds/**
|
||||
---
|
||||
|
||||
# Cmds Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/cmds.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for daemon
|
||||
globs:
|
||||
- chia/daemon/**
|
||||
---
|
||||
|
||||
# Daemon Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/daemon.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for data-layer
|
||||
globs:
|
||||
- chia/data_layer/**
|
||||
---
|
||||
|
||||
# Chia Data Layer Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/data-layer.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for farmer
|
||||
globs:
|
||||
- chia/farmer/**
|
||||
---
|
||||
|
||||
# Farmer Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/farmer.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for harvester
|
||||
globs:
|
||||
- chia/harvester/**
|
||||
---
|
||||
|
||||
# Harvester Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/harvester.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for introducer
|
||||
globs:
|
||||
- chia/introducer/**
|
||||
---
|
||||
|
||||
# Introducer Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/introducer.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -1,14 +0,0 @@
|
||||
---
|
||||
description: Auto-attach networking context when working in server/protocol code
|
||||
globs:
|
||||
- chia/server/**
|
||||
- chia/protocols/**
|
||||
---
|
||||
|
||||
# Networking Context Available
|
||||
|
||||
Deep context for this subsystem exists at `.cursor/context/networking.md`.
|
||||
Read it before exploring code — it documents `WSChiaConnection`, rate limiting
|
||||
(per-type and aggregate), the protocol state machine, `ApiProtocol`/`@request`
|
||||
decorator, connection limits, peer discovery, and the handshake/capability
|
||||
negotiation flow.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for plot-sync
|
||||
globs:
|
||||
- chia/plot_sync/**
|
||||
---
|
||||
|
||||
# Chia Plot Sync Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/plot-sync.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
description: Auto-attach curated context for plotting
|
||||
globs:
|
||||
- chia/plotting/**
|
||||
- chia/plotters/**
|
||||
---
|
||||
|
||||
# Plotting Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/plotting.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for pools
|
||||
globs:
|
||||
- chia/pools/**
|
||||
---
|
||||
|
||||
# Pools Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/pools.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for protocols
|
||||
globs:
|
||||
- chia/protocols/**
|
||||
---
|
||||
|
||||
# Protocols Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/protocols.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
description: Auto-attach repository tooling context for root config, CI, build scripts, and tools
|
||||
globs:
|
||||
- build_scripts/**
|
||||
- .github/**
|
||||
- tools/**
|
||||
- .coveragerc
|
||||
- .diffcover.toml
|
||||
- .git_archival.txt
|
||||
- .gitattributes
|
||||
- .gitignore
|
||||
- .gitmodules
|
||||
- .markdown-lint.yml
|
||||
- .pre-commit-config.yaml
|
||||
- .prettierrc.yml
|
||||
- .repo-content-updater.yml
|
||||
- .shellcheckrc
|
||||
- BUILD_TIMELORD.md
|
||||
- CHANGELOG.md
|
||||
- CODE_OF_CONDUCT.md
|
||||
- CONTRIBUTING.md
|
||||
- INSTALL.md
|
||||
- Install-gui.ps1
|
||||
- Install-plotter.ps1
|
||||
- Install.ps1
|
||||
- LEGACY-SUPPORT-POLICY.md
|
||||
- LICENSE
|
||||
- PRETTY_GOOD_PRACTICES.md
|
||||
- README.md
|
||||
- SECURITY.md
|
||||
- Setup-poetry.ps1
|
||||
- activated.ps1
|
||||
- activated.py
|
||||
- activated.sh
|
||||
- install-gui.sh
|
||||
- install-plotter.sh
|
||||
- install-timelord.sh
|
||||
- install.sh
|
||||
- installhelper.py
|
||||
- manage-mypy.py
|
||||
- mypy-exclusions.txt
|
||||
- mypy-strict-bytes-exclusions.txt
|
||||
- mypy.ini.template
|
||||
- poetry-check.py
|
||||
- poetry.lock
|
||||
- poetry.toml
|
||||
- pyproject.toml
|
||||
- pytest.ini
|
||||
- requirements-poetry.txt
|
||||
- ruff.toml
|
||||
- setup-poetry.sh
|
||||
- start-gui.sh
|
||||
- tach.toml
|
||||
- virtual_project.yaml
|
||||
- chia/__init__.py
|
||||
- chia/__main__.py
|
||||
- chia/py.typed
|
||||
---
|
||||
|
||||
# Repository Tooling Context Available
|
||||
|
||||
Consolidated repository tooling context exists at `.cursor/context/repo-tooling.md`.
|
||||
|
||||
Read it before broad source exploration. If runtime behavior is involved, also read the affected source module context listed in `.cursor/context/INDEX.md`.
|
||||
|
||||
For `chia/__init__.py`, `chia/__main__.py`, or `chia/py.typed`, the package-root section of `.cursor/context/architecture-overview.md` is also relevant.
|
||||
@@ -1,8 +1,3 @@
|
||||
---
|
||||
description: Route agents to pre-built context before code exploration
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# Deep Context — Read Before You Grep
|
||||
|
||||
Pre-built deep context exists in `.cursor/context/`. **Before running broad
|
||||
@@ -15,15 +10,37 @@ questions instantly.
|
||||
Match the files or intent to the right context doc:
|
||||
|
||||
| If you're touching… | Read first |
|
||||
|----------------------|------------|
|
||||
|----|---|
|
||||
| `chia/consensus/**` or block validation / reorg / difficulty | `.cursor/context/consensus.md` |
|
||||
| `chia/full_node/mempool*.py`, `eligible_coin_spends.py`, fee logic | `.cursor/context/mempool.md` |
|
||||
| `chia/full_node/full_node.py`, `full_node_api.py`, `full_node_store.py`, sync | `.cursor/context/full-node.md` |
|
||||
| `chia/server/**`, `chia/protocols/**`, connections, rate limits | `.cursor/context/networking.md` |
|
||||
| `chia/server/**`, connections, rate limits | `.cursor/context/server.md` |
|
||||
| `chia/protocols/**`, wire messages | `.cursor/context/protocols.md` |
|
||||
| `chia/apis/**`, API stub metadata | `.cursor/context/apis.md` |
|
||||
| `chia/types/**`, shared types, serialization boundary | `.cursor/context/types.md` |
|
||||
| `chia/wallet/**` | `.cursor/context/wallet.md` |
|
||||
| CLVM, puzzles, conditions, generators, `chia/types/blockchain_format/**` | `.cursor/context/clvm-execution.md` |
|
||||
| CLVM execution, generators, puzzles, conditions | `.cursor/context/clvm-execution.md` |
|
||||
| `chia/farmer/**` | `.cursor/context/farmer.md` |
|
||||
| `chia/harvester/**` | `.cursor/context/harvester.md` |
|
||||
| `chia/timelord/**` | `.cursor/context/timelord.md` |
|
||||
| `chia/plotting/**` | `.cursor/context/plotting.md` |
|
||||
| `chia/plot_sync/**` | `.cursor/context/plot-sync.md` |
|
||||
| `chia/pools/**` | `.cursor/context/pools.md` |
|
||||
| `chia/daemon/**` | `.cursor/context/daemon.md` |
|
||||
| `chia/data_layer/**` | `.cursor/context/data-layer.md` |
|
||||
| `chia/rpc/**` | `.cursor/context/rpc.md` |
|
||||
| `chia/ssl/**` | `.cursor/context/ssl.md` |
|
||||
| `chia/simulator/**` | `.cursor/context/simulator.md` |
|
||||
| `chia/solver/**` | `.cursor/context/solver.md` |
|
||||
| `chia/seeder/**` | `.cursor/context/seeder.md` |
|
||||
| `chia/introducer/**` | `.cursor/context/introducer.md` |
|
||||
| `chia/cmds/**` | `.cursor/context/cmds.md` |
|
||||
| `chia/util/**` | `.cursor/context/util.md` |
|
||||
| `benchmarks/**` | `.cursor/context/benchmarks.md` |
|
||||
| Root config, build scripts, workflows, tooling | `.cursor/context/repo-tooling.md` |
|
||||
| Cross-cutting / security review / unsure | `.cursor/context/global-invariants.md` |
|
||||
| First time / orientation / unfamiliar area | `.cursor/context/architecture-overview.md` |
|
||||
| Tests (`chia/_tests/**`) | See `.cursor/rules/testing-guide.mdc` |
|
||||
|
||||
## Protocol
|
||||
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for rpc
|
||||
globs:
|
||||
- chia/rpc/**
|
||||
---
|
||||
|
||||
# Rpc Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/rpc.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for seeder
|
||||
globs:
|
||||
- chia/seeder/**
|
||||
---
|
||||
|
||||
# Seeder Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/seeder.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for server/connection code
|
||||
globs:
|
||||
- chia/server/**
|
||||
---
|
||||
|
||||
# Server Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/server.md`.
|
||||
|
||||
Read it before broad source exploration. For wire message schemas, also read `.cursor/context/protocols.md`; for API stub metadata, read `.cursor/context/apis.md`. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for simulator
|
||||
globs:
|
||||
- chia/simulator/**
|
||||
---
|
||||
|
||||
# Simulator Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/simulator.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for solver
|
||||
globs:
|
||||
- chia/solver/**
|
||||
---
|
||||
|
||||
# Solver Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/solver.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for ssl
|
||||
globs:
|
||||
- chia/ssl/**
|
||||
---
|
||||
|
||||
# Ssl Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/ssl.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for timelord
|
||||
globs:
|
||||
- chia/timelord/**
|
||||
---
|
||||
|
||||
# Timelord Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/timelord.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
description: Auto-attach curated context for shared types and serialization boundary
|
||||
globs:
|
||||
- chia/types/**
|
||||
---
|
||||
|
||||
# Types Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/types.md`.
|
||||
|
||||
Read it before broad source exploration. For CLVM execution paths and resource limits, also read `.cursor/context/clvm-execution.md`. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
description: Auto-attach curated context for util
|
||||
globs:
|
||||
- chia/util/**
|
||||
- chia/legacy/**
|
||||
---
|
||||
|
||||
# Util Context Available
|
||||
|
||||
Curated context for this subsystem exists at `.cursor/context/util.md`.
|
||||
|
||||
Read it before broad source exploration. If the change crosses subsystem boundaries, also read `.cursor/context/INDEX.md` and the coupled modules listed there.
|
||||
@@ -1,90 +0,0 @@
|
||||
---
|
||||
description: Blockchain consensus test patterns — block validation, reorgs, overflow slots, fork behavior
|
||||
globs:
|
||||
- chia/_tests/blockchain/**
|
||||
---
|
||||
|
||||
# Blockchain Tests
|
||||
|
||||
## Scope
|
||||
|
||||
Use this for consensus-level chain behavior in `chia/_tests/blockchain/**`:
|
||||
|
||||
- block validation outcomes (`NEW_PEAK`, `ADDED_AS_ORPHAN`, `INVALID_BLOCK`)
|
||||
- reorgs and fork preference behavior
|
||||
- overflow/slot edge cases
|
||||
- block-record / fork-info invariants
|
||||
|
||||
Primary references:
|
||||
|
||||
- `chia/_tests/blockchain/test_blockchain.py`
|
||||
- `chia/_tests/blockchain/blockchain_test_utils.py`
|
||||
|
||||
## Preferred Harness
|
||||
|
||||
- `empty_blockchain` fixture for most unit/integration-style blockchain tests.
|
||||
- `bt` (`BlockTools`) for deterministic block construction.
|
||||
- `create_blockchain(...)` when two-chain comparisons are needed.
|
||||
|
||||
## Common Build/Add Pattern
|
||||
|
||||
1. Build deterministic blocks with `bt.get_consecutive_blocks(...)`.
|
||||
2. Add with `_validate_and_add_block(...)` (or multi-result helper variants).
|
||||
3. Assert expected add result/error at each step.
|
||||
|
||||
Useful options in `get_consecutive_blocks(...)`:
|
||||
|
||||
- `block_list_input` to continue a chain
|
||||
- `seed` for alternate fork chain
|
||||
- `force_overflow` for overflow peak/slot scenarios
|
||||
- `skip_slots` for sub-slot boundary behavior
|
||||
|
||||
## Reorg Pattern
|
||||
|
||||
For fork validation flows:
|
||||
|
||||
- Initialize `fork_info = ForkInfo(...)`.
|
||||
- Reuse a single `AugmentedBlockchain(b)` while validating fork blocks.
|
||||
- Expect:
|
||||
- `ALREADY_HAVE_BLOCK` for shared prefix blocks
|
||||
- `ADDED_AS_ORPHAN` before fork wins
|
||||
- default success (`NEW_PEAK`) once heavier fork overtakes
|
||||
|
||||
## Overflow-Specific Guidance
|
||||
|
||||
- Use `is_overflow_block(constants, block.reward_chain_block.signage_point_index)` for assertions.
|
||||
- If a test requires overflow peak precondition, enforce it explicitly with `force_overflow=True`.
|
||||
- Keep reorg assertions layered: precondition peak state -> fork progression results -> final peak state.
|
||||
|
||||
## Starter Template
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from chia_rs.sized_ints import uint32
|
||||
|
||||
from chia._tests.blockchain.blockchain_test_utils import (
|
||||
_validate_and_add_block,
|
||||
_validate_and_add_block_multi_result,
|
||||
)
|
||||
from chia.consensus.augmented_chain import AugmentedBlockchain
|
||||
from chia.consensus.block_body_validation import ForkInfo
|
||||
from chia.consensus.blockchain import AddBlockResult, Blockchain
|
||||
from chia.consensus.pot_iterations import is_overflow_block
|
||||
from chia.simulator.block_tools import BlockTools
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_example(empty_blockchain: Blockchain, bt: BlockTools) -> None:
|
||||
# Build a base chain
|
||||
blocks = bt.get_consecutive_blocks(5)
|
||||
for block in blocks:
|
||||
await _validate_and_add_block(empty_blockchain, block)
|
||||
|
||||
# Fork from earlier point with different seed
|
||||
fork_blocks = bt.get_consecutive_blocks(
|
||||
3, block_list_input=blocks[:3], seed=b"fork"
|
||||
)
|
||||
# Add fork blocks and assert results...
|
||||
```
|
||||
@@ -1,133 +0,0 @@
|
||||
---
|
||||
description: Full node integration test patterns — sync, propagation, mempool-to-block, reorg, wallet-connected flows
|
||||
globs:
|
||||
- chia/_tests/core/full_node/**
|
||||
---
|
||||
|
||||
# Full Node Tests
|
||||
|
||||
## Scope
|
||||
|
||||
Use this when testing behavior driven by full node state transitions:
|
||||
|
||||
- node sync / backtrack / batch sync
|
||||
- block acceptance and propagation
|
||||
- mempool to block inclusion
|
||||
- wallet-connected full node flows
|
||||
- reorg and chain preference behavior
|
||||
|
||||
Primary references:
|
||||
|
||||
- `chia/_tests/core/full_node/test_full_node.py`
|
||||
- `chia/_tests/core/full_node/test_transactions.py`
|
||||
- `chia/_tests/core/full_node/full_sync/test_full_sync.py`
|
||||
|
||||
## Go-To Fixtures and Harness
|
||||
|
||||
Pick the smallest fixture that covers your case:
|
||||
|
||||
- single-node state checks: `one_node_one_block`, `one_node`
|
||||
- two/three/five node sync tests: `two_nodes`, `three_nodes`, `five_nodes`
|
||||
- wallet + full node integration: `simulator_and_wallet`, `setup_two_nodes_and_wallet`, `three_nodes_two_wallets`
|
||||
- custom service setup: `setup_simulators_and_wallets(...)` from `chia/_tests/util/setup_nodes.py`
|
||||
|
||||
Typical test setup pattern:
|
||||
|
||||
1. Connect peers with `start_client(...)` or `connect_and_get_peer(...)`.
|
||||
2. Seed chain with deterministic blocks or farmed tx blocks.
|
||||
3. Submit spend(s) and wait for mempool visibility.
|
||||
4. Farm include block and assert final chain/wallet state.
|
||||
|
||||
## Block Creation Patterns
|
||||
|
||||
Use deterministic block construction when sequence matters:
|
||||
|
||||
- `bt.get_consecutive_blocks(...)`
|
||||
- Common options: `block_list_input`, `seed`, `guarantee_transaction_block`, `skip_slots`, `force_overflow`.
|
||||
|
||||
Add blocks with:
|
||||
|
||||
- `await add_blocks_in_batches(blocks, full_node)` for bulk chain setup.
|
||||
- `await full_node.add_block(block)` when each step needs validation.
|
||||
|
||||
Use simulator helpers when internals are less important:
|
||||
|
||||
- `farm_new_transaction_block(FarmNewBlockProtocol(...))`
|
||||
- `farm_blocks_to_puzzlehash(...)`
|
||||
|
||||
## Transaction Submission Patterns
|
||||
|
||||
Wallet-first path (common):
|
||||
|
||||
1. Build tx in `new_action_scope(..., push=True)`.
|
||||
2. `await wallet.generate_signed_transaction(...)`.
|
||||
3. Read created records from `action_scope.side_effects.transactions`.
|
||||
|
||||
Protocol path (for peer/mempool behavior):
|
||||
|
||||
1. Wrap spend bundle in `wallet_protocol.SendTransaction(...)`.
|
||||
2. Submit via `full_node_api.send_transaction(...)` using a dummy or connected peer.
|
||||
|
||||
## How To Assert Steps Happened
|
||||
|
||||
Use layered checks:
|
||||
|
||||
1. **Mempool entered**
|
||||
- `time_out_assert(..., mempool_manager.get_spendbundle, expected_bundle, tx_name)`
|
||||
2. **Block inclusion**
|
||||
- farm tx block, then assert mempool no longer contains bundle
|
||||
3. **Heights/sync convergence**
|
||||
- `time_out_assert(..., node_height_at_least|node_height_exactly, ...)`
|
||||
4. **Wallet convergence**
|
||||
- `wait_for_wallet_synced(...)` + balance assertions
|
||||
5. **Failure path**
|
||||
- `pytest.raises(...)` and explicit `Err`/status checks where applicable
|
||||
|
||||
## Anti-Flake Guidance
|
||||
|
||||
- Prefer `time_out_assert` and sync wait helpers over raw sleeps.
|
||||
- Use deterministic seeds in `get_consecutive_blocks` for fork/reorg scenarios.
|
||||
- For sync tests, assert both node height and peak equality when possible.
|
||||
- Keep fixture scope narrow; large shared state increases intermittent failures.
|
||||
|
||||
## Quick Checklist
|
||||
|
||||
- Fixture matches topology (single node vs multi-node vs wallet-connected).
|
||||
- Block sequence is deterministic if order-sensitive.
|
||||
- Mempool, inclusion, and sync are asserted as separate steps.
|
||||
- Negative path validates expected `Err` or status, not just generic failure.
|
||||
|
||||
## Starter Template
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from chia._tests.blockchain.blockchain_test_utils import _validate_and_add_block
|
||||
from chia._tests.connection_utils import add_dummy_connection, connect_and_get_peer
|
||||
from chia._tests.core.node_height import node_height_at_least
|
||||
from chia._tests.util.setup_nodes import OldSimulatorsAndWallets
|
||||
from chia._tests.util.time_out_assert import time_out_assert
|
||||
from chia.consensus.blockchain import Blockchain
|
||||
from chia.full_node.full_node_api import FullNodeAPI
|
||||
from chia.protocols import wallet_protocol
|
||||
from chia.server.server import ChiaServer
|
||||
from chia.simulator.block_tools import BlockTools
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_example(
|
||||
one_node_one_block: tuple[FullNodeAPI, ChiaServer, BlockTools],
|
||||
) -> None:
|
||||
full_node_api, server, bt = one_node_one_block
|
||||
full_node = full_node_api.full_node
|
||||
|
||||
# Build and add blocks
|
||||
blocks = bt.get_consecutive_blocks(3)
|
||||
for block in blocks:
|
||||
await full_node.add_block(block)
|
||||
|
||||
# Assert height convergence
|
||||
await time_out_assert(10, node_height_at_least, True, full_node, 3)
|
||||
```
|
||||
@@ -1,119 +0,0 @@
|
||||
---
|
||||
description: Detailed block creation, transaction submission, and assertion patterns for Chia tests
|
||||
globs:
|
||||
- chia/_tests/**
|
||||
---
|
||||
|
||||
# Chia Test Patterns
|
||||
|
||||
## How We Make Blocks
|
||||
|
||||
### 1) Deterministic block lists (`BlockTools`)
|
||||
|
||||
Use `bt.get_consecutive_blocks(...)` when you need exact block structure, specific spend inclusion, or malformed block variants.
|
||||
|
||||
Common options:
|
||||
- `block_list_input=` continue from existing chain
|
||||
- `transaction_data=` include a spend bundle
|
||||
- `guarantee_transaction_block=True` force tx block
|
||||
- `force_overflow`, `skip_slots`, `seed`, `time_per_block`
|
||||
|
||||
Then add blocks with:
|
||||
- `await full_node.add_block(block)` for single steps, or
|
||||
- `await add_blocks_in_batches(blocks, full_node)` for larger sets
|
||||
|
||||
### 2) High-level farming APIs (`FullNodeSimulator`)
|
||||
|
||||
Use these for behavior tests where exact block internals do not matter:
|
||||
- `farm_blocks_to_puzzlehash()`
|
||||
- `farm_blocks_to_wallet()`
|
||||
- `farm_rewards_to_wallet()`
|
||||
- `farm_new_transaction_block()`
|
||||
- `reorg_from_index_to_new_index()`
|
||||
- `revert_block_height()`
|
||||
|
||||
### 3) Pre-generated persistent chains
|
||||
|
||||
Use fixtures like:
|
||||
- `default_400_blocks`, `default_1000_blocks`, `default_10000_blocks`
|
||||
- reorg variants and compact variants
|
||||
|
||||
These come from `persistent_blocks(...)` and are used heavily in consensus/timelord/weight proof tests.
|
||||
|
||||
## How We Submit Transactions
|
||||
|
||||
### A) Wallet-internal (most common)
|
||||
|
||||
1. `async with wallet.wallet_state_manager.new_action_scope(..., push=True) as action_scope:`
|
||||
2. `await wallet.generate_signed_transaction(...)`
|
||||
3. Records from `action_scope.side_effects.transactions`
|
||||
4. Wait via `wait_transaction_records_entered_mempool` or `process_pending_states`.
|
||||
|
||||
### B) Wallet RPC
|
||||
|
||||
`WalletRpcClient`: `send_transaction(...)`, `create_signed_transactions(...)`, `push_transactions(...)`, `push_tx(...)`.
|
||||
|
||||
### C) Full node protocol-level
|
||||
|
||||
Build `wallet_protocol.SendTransaction(spend_bundle)`, send via `full_node_api.send_transaction(...)` with dummy peers from `chia/_tests/connection_utils.py`.
|
||||
|
||||
### D) CLVM simulator
|
||||
|
||||
`status, err = await sim_client.push_tx(spend_bundle)`
|
||||
|
||||
## How We Assert Steps Happened
|
||||
|
||||
Use layered assertions instead of a single final check:
|
||||
|
||||
1. **Immediate invariants** — object created, response success, expected fields present.
|
||||
2. **Eventual behavior** — `time_out_assert(...)` for async convergence.
|
||||
3. **Mempool checks** — `mempool_manager.get_spendbundle(...)`, `assert_sb_in_pool(...)`.
|
||||
4. **Wallet transitions** — `process_pending_states(...)` with `WalletStateTransition`.
|
||||
5. **Failure paths** — `pytest.raises(...)` with explicit error matching.
|
||||
6. **Log assertions** — `caplog` for protocol/service side effects.
|
||||
|
||||
## Module-by-Module Test Setup Map
|
||||
|
||||
| Module | Typical Setup | Blocks | Transaction Path | Assertion Style |
|
||||
|---|---|---|---|---|
|
||||
| `blockchain` | `bt`, `empty_blockchain`, `two_nodes` | `get_consecutive_blocks`, `add_block`, `add_blocks_in_batches` | `WalletTool.generate_signed_transaction`, protocol `send_transaction`, in-block `transaction_data` | direct consensus result checks, `pytest.raises`, occasional `time_out_assert` |
|
||||
| `clvm` | no network harness, or `sim_and_client` | `SpendSim.farm_block` | `sim_client.push_tx` | direct CLVM/coin-store assertions, `pytest.raises` |
|
||||
| `cmds` | `CliRunner`, `get_test_cli_clients`, temp config roots | usually none | mocked RPC client calls | output assertions, parse/validation errors |
|
||||
| `core` | mixed: `one_node_one_block`, `simulator_and_wallet`, data-layer fixtures | heavy use of `get_consecutive_blocks`, farming APIs | wallet-generated spends, protocol `send_transaction`/`respond_transaction` | heavy `time_out_assert`, mempool/state assertions, `caplog`, `pytest.raises` |
|
||||
| `db` | `DBConnection`/`PathDBConnection` fixtures | none | none | concurrency/transactionality assertions, `pytest.raises` |
|
||||
| `farmer_harvester` | `farmer_one_harvester*`, `harvester_farmer_environment` | minimal | protocol message flow | service-state `time_out_assert`, `caplog` |
|
||||
| `fee_estimation` | mostly mempool/unit harness | minimal farming | small generated spend bundles | direct estimator state assertions |
|
||||
| `generator` | pure generator/CLVM tests | none | none | deterministic program output/cost assertions |
|
||||
| `harvester` | `harvester_farmer_environment` + test plots | `default_400_blocks` for signage data | harvester protocol interactions | `time_out_assert`, mock peer assertions |
|
||||
| `pools` | pure puzzle unit tests and wallet/simulator integration | farming + reorg in integration | wallet RPC and framework tx processing | `process_pending_states`, `time_out_assert`, `pytest.raises` |
|
||||
| `simulation` | `simulator_and_wallet`, full system fixture | high-level simulator farming/reorg | wallet-generated spends | heavy `time_out_assert`, mempool/coin-store confirmations |
|
||||
| `wallet` | `wallet_environments` (primary), simulator fixtures | frequent farming/reorg | wallet action scopes, wallet RPC | `process_pending_states`, `time_out_assert`, mempool checks |
|
||||
| `weight_proof` | pre-generated block fixtures + `BlockchainMock` | `get_consecutive_blocks` for edge chains | none | proof validity/fork point assertions |
|
||||
|
||||
## Agent Templates
|
||||
|
||||
### Wallet transfer with robust checks
|
||||
|
||||
1. Use `wallet_environments` with needed prefarm.
|
||||
2. Build tx inside `new_action_scope(..., push=True)`.
|
||||
3. Call `wallet_environments.process_pending_states([...])` with pre-block and post-block expected deltas.
|
||||
|
||||
### Protocol/mempool acceptance test
|
||||
|
||||
1. Create spend bundle (`WalletTool` or wallet action scope).
|
||||
2. Submit with `full_node_api.send_transaction(...)` via dummy peer.
|
||||
3. Assert mempool inclusion, farm tx block, assert eviction + coin-store updates.
|
||||
|
||||
### Consensus block validation test
|
||||
|
||||
1. Build base chain using `bt.get_consecutive_blocks`.
|
||||
2. Mutate crafted block (or use transaction conditions).
|
||||
3. Validate with `_validate_and_add_block(...)` expecting specific `Err`.
|
||||
|
||||
## Checklist
|
||||
|
||||
- Harness matches behavior under test.
|
||||
- Async convergence uses `time_out_assert`, not raw sleeps.
|
||||
- Wallet behavior uses `process_pending_states` where practical.
|
||||
- Mempool and confirmation asserted as separate steps.
|
||||
- Failure paths use `pytest.raises` with explicit error matching.
|
||||
@@ -1,34 +1,44 @@
|
||||
---
|
||||
description: Chia test harness selection and routing to domain-specific guides
|
||||
alwaysApply: true
|
||||
description: Chia test harness routing — pick the right guide and harness before writing tests
|
||||
globs:
|
||||
- chia/_tests/**
|
||||
---
|
||||
|
||||
# Chia Test Guide
|
||||
|
||||
When writing or modifying tests under `chia/_tests/`, follow this sequence:
|
||||
When writing or modifying tests under `chia/_tests/`, read the matching on-demand
|
||||
guide before any code search.
|
||||
|
||||
## 1. Route to the right guide
|
||||
## Route to the right guide
|
||||
|
||||
Identify the area from the user-provided path or intent, then read the matching guide **before any code search**:
|
||||
|
||||
- `chia/_tests/blockchain/**` or consensus/reorg/overflow intent -> `testing-guide-blockchain.mdc`
|
||||
- `chia/_tests/core/full_node/**` or sync/full-node intent -> `testing-guide-full-node.mdc`
|
||||
- `chia/_tests/core/mempool/**` or mempool intent -> `testing-guide-mempool.mdc`
|
||||
- `chia/_tests/core/data_layer/**` or data-layer intent -> `testing-guide-data-layer.mdc`
|
||||
- `chia/_tests/core/server/**` or server/network intent -> `testing-guide-server.mdc`
|
||||
| Path or intent | Read first |
|
||||
|----------------|------------|
|
||||
| `chia/_tests/blockchain/**` or consensus/reorg/overflow | `.cursor/context/testing/blockchain.md` |
|
||||
| `chia/_tests/core/full_node/**` or sync/full-node | `.cursor/context/testing/full-node.md` |
|
||||
| `chia/_tests/core/mempool/**` or mempool | `.cursor/context/testing/mempool.md` |
|
||||
| `chia/_tests/core/data_layer/**` or data-layer | `.cursor/context/testing/data-layer.md` |
|
||||
| `chia/_tests/core/server/**` or server/network | `.cursor/context/testing/server.md` |
|
||||
| `chia/_tests/core/**` cross-subsystem or service wiring | `.cursor/context/testing/architecture.md` |
|
||||
| `chia/_tests/wallet/**` | `.cursor/context/testing/wallet.md` |
|
||||
| `chia/_tests/clvm/**` or SpendSim | `.cursor/context/testing/clvm.md` |
|
||||
| `chia/_tests/cmds/**` or CLI | `.cursor/context/testing/cmds.md` |
|
||||
| `chia/_tests/util/**` or shared harness | `.cursor/context/testing/infrastructure.md` |
|
||||
| Any test area or CI partitioning | `.cursor/context/testing/architecture.md` |
|
||||
| Block/tx/assertion patterns for all tests | `.cursor/context/testing/patterns.md` |
|
||||
|
||||
After reading, state: `Guide loaded: <guide-file> (area: <area>)`.
|
||||
If area is ambiguous, ask one clarifying question instead of broad repository search.
|
||||
|
||||
## 2. Pick the right harness
|
||||
## Pick the right harness
|
||||
|
||||
1. **Pure logic / parsing / utility** — no simulator, standard pytest.
|
||||
2. **Chain state / farming / wallets / mempool / reorg** — simulator fixtures from `chia/_tests/conftest.py`:
|
||||
- `simulator_and_wallet`, `one_node`, `two_nodes`, `one_node_one_block`
|
||||
- `wallet_environments` for wallet-heavy tests
|
||||
3. **Service wiring (farmer/harvester/solver/timelord)** — `setup_*` in `chia/_tests/util/setup_nodes.py`.
|
||||
4. **Lightweight CLVM spend sim** — `sim_and_client()` from `chia/_tests/util/spend_sim.py`.
|
||||
2. **Chain state / farming / wallets / mempool / reorg** — simulator fixtures from
|
||||
`chia/_tests/conftest.py`: `simulator_and_wallet`, `one_node`, `two_nodes`,
|
||||
`one_node_one_block`; `wallet_environments` for wallet-heavy tests.
|
||||
3. **Service wiring (farmer/harvester/solver/timelord)** — `setup_*` in
|
||||
`chia/_tests/util/setup_nodes.py`.
|
||||
4. **Lightweight CLVM spend sim** — `sim_and_client()` from
|
||||
`chia/_tests/util/spend_sim.py`.
|
||||
|
||||
Key shared utilities: `chia/_tests/conftest.py`, `chia/simulator/block_tools.py`, `chia/_tests/util/time_out_assert.py`, `chia/_tests/environments/wallet.py`.
|
||||
|
||||
For detailed block/tx/assertion patterns, see `testing-guide-patterns.mdc`.
|
||||
Key shared utilities: `chia/_tests/conftest.py`,
|
||||
`chia/simulator/block_tools.py`, `chia/_tests/util/time_out_assert.py`,
|
||||
`chia/_tests/environments/wallet.py`.
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# Chia Context
|
||||
|
||||
This repo ships curated architectural context for coding agents.
|
||||
|
||||
Start with `.cursor/context/INDEX.md` for routing, critical invariants,
|
||||
cross-module coupling, architecture overview, and the module table.
|
||||
|
||||
For focused work, read the matching context doc under `.cursor/context/` before
|
||||
broad source exploration. For review or audit work, also check
|
||||
`.cursor/context/global-invariants.md`.
|
||||
|
||||
Cursor behavior is guided by `.cursor/rules/`, including context routing,
|
||||
test harness selection, and repository tooling guardrails. Bugbot review
|
||||
guidance lives in `.cursor/BUGBOT.md`.
|
||||
Reference in New Issue
Block a user