* update the fast FullBlock parser (that just skips fields) to support the new serialization of RewardChainBlock
* add test and shard the test, to enable it running in parallel
* disable expensive tests on MacOS Intel
* document some aspects of the full_blocks_utils test
* context manager for BlockTools
* Missed one usage and fixed up Self import
* Ignore cleanup errors in tempdir cleanup for BlockTools
* Remove debug prints
* Fixed up sync blocktools and use @contextmanager
* Address review comments
* Add return type
* Empty commit for github
* Strange github things
* adjust order of finding ports
* Add context handler features to MempoolManager
* No need to manually shut_down mempoolmanager
* Fix up issues with ordering in full node
* Fix dumb programmer issues
* Fix up some bugbot issues
* Move timing to inside context manager
* fix wrong indent
* migrate casts to wallet util
* reset mempool_check_conditions
* ruff to re-order imports
* reset full_node_rpc_api
* add tests for casts
* ruff and type
* move to util to avoid consensus depending on wallet
* ruff again
* add a deterministic seed for random
<!-- Merging Requirements:
- Please give your PR a title that is release-note friendly
- In order to be merged, you must add the most appropriate category
Label (Added, Changed, Fixed) to your PR
-->
<!-- Explain why this is an improvement (Does this add missing
functionality, improve performance, or reduce complexity?) -->
### Purpose:
This PR is an optimisation for the way that AddressManager and
ExtendedPeerInfo are serialized, which happens periodically in a task
called from node_discovery.py
As well as the format of the serialization, the serialization algorithm
itself has had a multitude of speedups, including removing redundant
loops, minimising allocations, and so on.
<!-- Does this PR introduce a breaking change? -->
### Current Behavior:
Previously we were serialising ExtendedPeerInfo as a string and
AddressManager as an `@streamable` `.to_bytes()`.
Copying all the data to this class was taking a deal of time.
The serialising function also had a lot of complicated pieces of
redundant code, looping over data multiple times when 1 pass might do.
### New Behavior:
This PR instead serialises AddressManager directly using a bytearray,
and refactors the code to reduce the complexity and number of loops.
More comments have been added in this area of the codebase as well.
### Testing Notes:
This PR includes a new benchmark file for the AddressManager's
serialisation.
Here are my personal results comparing this PR with `main`:
On main: `Average serialize time: 1.544302 seconds`
Sum of my optimisations: `Average serialize time: 0.468669 seconds`
Feel free to run `benchmarks/address_manager_store.py` to see for
yourself.
I have also added in a migration test in
`chia/_tests/core/full_node/test_address_manager.py` which creates an
old serialisation, loads it using the current loading code, then
serialises it using the new serialization, and then loads that.
It is also checked at each step along the way to ensure that the
serialisation has remained good.
<!-- Attach any visual examples, or supporting evidence (attach any
.gif/video/console output below) -->
* introduce create_block_generator2() to mempool, which picks transactions and compresses them incrementally, allowing us to fill blocks even when compressing
* improve test, comment and pragma no cover
* fix mempool test
* simplify item selection logic in create_block_generator2()
* add test coverage of timeout in create_block_generator2()
* address review comments
* fixup
* no need to pass get_unspent_lineage_info_from_puzzle_hash function into create_bundle_from_mempool(), we already have it as a member now
* cache results from get_lineage_info_for_puzzle_hash()
Validate fast forward spends before adding their spend bundle to the mempool.
Make sure that all fast forward spends of a spend bundle would still have unspent coins.
* make item_inclusion_filter optional and move it to the end with default None, to simplify calls
* move block generator creation into the Mempool class
* update benchmark
* add test
* put otherwise unheld tasks into the pit
* and the rest too
* cull less often
* task referencer
* private
* report unexpectedly unreferenced tasks
* also warn for automated catching in tests
* reporting task
* undo some
* fixup for 3.9
* ban asyncio.create_task
* oof
* simplify including using `Task.add_done_callback()` for culling
* refactor get_block_generator() to prepare for easier composing of a layered blockchain class
* make get_block_generator() part of the interface to the blockchain rather than an independent, free function passed in the validation. This keeps the view of the blockchain more focused to a single class.
* add test for get_block_generator
* add unit test for Blockchain.lookup_block_generators()
* address review comments
* address review comments
make MempoolItem and InternalMempoolItem contain the SpendBundleConditions directly, rather than the NPCResult. A mempool item is guaranteed to be valid, so the error state of NPCResult will never be engaged anyway.
* Initial draft of mempool updates [skip ci]
* More implementation work, but capability issue [skip ci]
* Temp (will probably revert)
* Get first test passing
* Setup spent coin tests
* Add unrelated test data to make tests better
* Refactor test
* Finish testing
* Update tests
* Include removal reason and block inclusion tests
* Remove unnecessary dict
* Add missing message to test
* Fix after rebase
* Broadcast support from node for mempool updates
* Remove redundant code
* Use new_block
* Filter and early return suggestions
* Reword comments on PeakPostProcessingResult
* Improve mempool fn name and doc comment
* Bump wallet protocol version
* Add ratelimits
* Add tests for querying mempool items
* Asserts and more tests
* Add peers for spend bundle tests
Arvid NorbergandGitHub
Earle LoweandGitHub
Kyle AltendorfandGitHub
Richard KissandGitHub
Amine KhaldiandGitHub
pengbanbanandGitHub
RigidityandGitHub