<!-- Merging Requirements:
- Please give your PR a title that is release-note friendly
- In order to be merged, you must add the most appropriate category
Label (Added, Changed, Fixed) to your PR
-->
<!-- Explain why this is an improvement (Does this add missing
functionality, improve performance, or reduce complexity?) -->
### Purpose:
This PR is an optimisation for the way that AddressManager and
ExtendedPeerInfo are serialized, which happens periodically in a task
called from node_discovery.py
As well as the format of the serialization, the serialization algorithm
itself has had a multitude of speedups, including removing redundant
loops, minimising allocations, and so on.
<!-- Does this PR introduce a breaking change? -->
### Current Behavior:
Previously we were serialising ExtendedPeerInfo as a string and
AddressManager as an `@streamable` `.to_bytes()`.
Copying all the data to this class was taking a deal of time.
The serialising function also had a lot of complicated pieces of
redundant code, looping over data multiple times when 1 pass might do.
### New Behavior:
This PR instead serialises AddressManager directly using a bytearray,
and refactors the code to reduce the complexity and number of loops.
More comments have been added in this area of the codebase as well.
### Testing Notes:
This PR includes a new benchmark file for the AddressManager's
serialisation.
Here are my personal results comparing this PR with `main`:
On main: `Average serialize time: 1.544302 seconds`
Sum of my optimisations: `Average serialize time: 0.468669 seconds`
Feel free to run `benchmarks/address_manager_store.py` to see for
yourself.
I have also added in a migration test in
`chia/_tests/core/full_node/test_address_manager.py` which creates an
old serialisation, loads it using the current loading code, then
serialises it using the new serialization, and then loads that.
It is also checked at each step along the way to ensure that the
serialisation has remained good.
<!-- Attach any visual examples, or supporting evidence (attach any
.gif/video/console output below) -->
* Add some checks when trying to join the same pool already joined
* Add test to test the specific cli case
* Remove line added during debugging
* Add test
* misc updates
* remove cli test
* final tests for joining same pool again
* Update check from code review
* update test framework
* Use click.ClickException for error
* Add check for wallet synced up front as the first thing checked for joining a pool
* Add test for unsynced RPC call
* log how long it took to call peak_post_processing_2() if block validation as a whole took a surprisingly long time
* log timing for Mempool.new_peak()
* Update chia/full_node/mempool_manager.py
Co-authored-by: Kyle Altendorf <sda@fstab.net>
---------
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* store merkle, key, and value blobs in files
* update expected columns
* fixup tests
* more
* cleanup disk
* better wrapped path identification
* todos for potential filesystem locking
* suppress not precheck
* handle not-present new path configuration entries
* log exception
* replace hard coded 32
* three layers of directories for bytes paths
<!-- Merging Requirements:
- Please give your PR a title that is release-note friendly
- In order to be merged, you must add the most appropriate category
Label (Added, Changed, Fixed) to your PR
-->
<!-- Explain why this is an improvement (Does this add missing
functionality, improve performance, or reduce complexity?) -->
### Purpose:
<!-- Does this PR introduce a breaking change? -->
### Current Behavior:
### New Behavior:
<!-- As we aim for complete code coverage, please include details
regarding unit, and regression tests -->
### Testing Notes:
<!-- Attach any visual examples, or supporting evidence (attach any
.gif/video/console output below) -->
* Somewhat hacky workaounds for DIDs from other wallets
* cannot use set as not hashable
* Add a couple tests
* Change DIDCoinData to use bytes instead of bytes32
* Remove some unused code
* Remove some more unused code
* Make the alternate recovery did spendable I think
* Use Optional[bytes32] for DIDCoinData to show the list should be 32 bytes or nothing (where nothing is the NIL program)
* Replace specific test with parameterization of other tests to get more testing of the Alternate DIDs
* Empty commit for github
* minor formatting change
* No need to cast to bytes since as_atom returns bytes
<!-- Merging Requirements:
- Please give your PR a title that is release-note friendly
- In order to be merged, you must add the most appropriate category
Label (Added, Changed, Fixed) to your PR
-->
<!-- Explain why this is an improvement (Does this add missing
functionality, improve performance, or reduce complexity?) -->
### Purpose:
<!-- Does this PR introduce a breaking change? -->
### Current Behavior:
### New Behavior:
<!-- As we aim for complete code coverage, please include details
regarding unit, and regression tests -->
### Testing Notes:
<!-- Attach any visual examples, or supporting evidence (attach any
.gif/video/console output below) -->
* add `chia dev data sync-time` for doing datalayer sync time tests
* add chia stop -d all at the end
* less subprocessing
* remove dead log file code
* add subcommand help messages
* correct for 3.9 support
* allow for days of timing
* format done message
* more info
* fixup
* fixup
* stuff
* more try finally
* add minutes
* add `--working-path`
Amine Khaldi
Chris MarslenderandGitHub
matt-o-howandGitHub
Arvid NorbergandGitHub
wjblankeandGitHub
Earle LoweandGitHub
