* prover protocol and v2Prover
* format name
* format
* refactor filename
* tests/raise unimplemented
* add get_filename_str to mock
* rename methods
* rename
* refactor
* improve coverage
* test from bytes
* replace blspy imports with chia_rs imports for BLS types
* remove blspy-stubs, since we're dropping the blspy dependency. chia_rs has type stubs already
* Timeout for retrieving a decompressor.
* Pre-commit fix
* Use new chiapos
* Use chiapos 2.0.1
---------
Co-authored-by: wallentx <william.allentx@gmail.com>
* Renaming instances of 'decompresser' to 'decompressor'
* Making 'parallel_decompressors_count' singular
* Using chiapos version that has matching spelling
* Putting the clsp file back where I found it
* fix ver
---------
Co-authored-by: Jack Nelson <j.nelson@chia.net>
* Make `list_to_batches` work with collections instead of lists only
* Move `to_batches` into `chia.util.misc`
* Only support `set` and `list`
* Drop `tests.generator.test_to_batches` exclusion
* Improve type restrictions and be more coverage friendly in tests
* configure isort to add the future annotations import
* apply the new isort setting
* remove type ignores for new mypy (#13539)
https://pypi.org/project/mypy/0.981/
* another
* Log the number of cache entries and the time it took to parse them
* Log `started` and `done` as `INFO` and `batch_processed` as `DEBUG`
* Log the plot found message with `DEBUG` level
* Some additional debug logs in `new_signage_point_harvester`
After changing `plot_paths` to be a set in #11204 the plots are loaded
out of order. Sorting the list before splitting it to batches recovers
the load order.
Fixes#11499
* plotting|tests: Extend the lifetime of plot data cache entries
* plotting|tests: Add `prover` and `farmer_public_key` to the cache
* plotting: Missing type hints in `Cache` and `CacheEntry`
* setup: Bump `chiapos` to 1.0.10
* plotting: Move cache classes into `chia.plotting.cache`
* plotting: Introduce `CacheEntry.from_disk_prover`
* Make `Cache` a dataclass
* `staticmethod` -> `classmethod`
* Store cache data as `bytes` in `DiskCache`
* streamable: Merge `strictdataclass` into `Streamable` class
* tests: Test not supported streamable types
* streamable: Reorder decorators
* streamable: Simplify streamable decorator and force correct usage/syntax
* streamable: Just move some stuff around in the file
* streamable: Improve syntax error messages
* mypy: Drop `type_checking.py` and `test_type_checking.py` from exclusion
* streamable: Use cached fields instead of `__annotations__`
This is now possible after merging `__post_init__` into `Streamable`
* Introduce `DefinitionError` as `StreamableError`
* `/t` -> ` `
* protocols|server: Define new harvester plot refreshing protocol messages
* protocols: Bump `protocol_version` to `0.0.34`
* tests: Introduce `setup_farmer_multi_harvester`
Allows to run a test setup with 1 farmer and mutiple harvesters.
* plotting: Add an initial plot loading indication to `PlotManager`
* plotting|tests: Don't add removed duplicates to `total_result.removed`
`PlotRefreshResult.removed` should only contain plots that were loaded
properly before they were removed. It shouldn't contain e.g. removed
duplicates or invalid plots since those are synced in an extra sync step
and not as diff but as whole list every time.
* harvester: Reset `PlotManager` on shutdown
* plot_sync: Implement plot sync protocol
* farmer|harvester: Integrate and enable plot sync
* tests: Implement tests for the plot sync protocol
* farmer|tests: Drop obsolete harvester caching code
* setup: Add `chia.plot_sync` to packages
* plot_sync: Type hints in `DeltaType`
* plot_sync: Drop parameters in `super()` calls
* plot_sync: Introduce `send_response` helper in `Receiver._process`
* plot_sync: Add some parentheses
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* plot_sync: Additional hint for a `Receiver.process_path_list` parameter
* plot_sync: Force named parameters in `Receiver.process_path_list`
* test: Fix fixtures after rebase
* tests: Fix sorting after rebase
* tests: Return type hint for `plot_sync_setup`
* tests: Rename `WSChiaConnection` and move it in the outer scope
* tests|plot_sync: More type hints
* tests: Rework some delta tests
* tests: Drop a `range` and iterate over the list directly
* tests: Use the proper flags to overwrite
* test: More missing duplicates tests
* tests: Drop `ExpectedResult.reset`
* tests: Reduce some asserts
* tests: Add messages to some `assert False` statements
* tests: Introduce `ErrorSimulation` enum in `test_sync_simulated.py`
* tests: Use `secrects` instead of `Crypto.Random`
* Fixes after rebase
* Import from `typing_extensions` to support python 3.7
* Drop task name to support python 3.7
* Introduce `Sender.syncing`, `Sender.connected` and a log about the task
* Add `tests/plot_sync/config.py`
* Align the multi harvester fixture with what we do in other places
* Update the workflows
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* tests: Allow to use custom keys in `BlockTools.new_plot`
* tests: Create three plots with `G1Element()` as farmer/pool key
But not add them as default plots. Just to have plots in the test cache with different keys from
within and outside the keychain for plot loading tests.
create
* test|github: Bump `test-cache` version to `0.28.0`
* plotting: Drop plots from `no_key_filenames` after the keys appeared
* plotting: Drop plots from `no_key_filenames` if the plot was removed
* tests: Test loading of plots where we miss the keys
* plotting: Use `x.copy()` instead of `list(x)`
* plotting: Add a comment about the missing keys plot removal
* tests: Fix after #9595
* plotting: Fix try/except in `PlotManager._refresh_task`
* tests: Move the start refreshing call into `PlotRefreshTester` ctor
It being in `run()` before where it was called every test iteration lead
to not detecting an issue with the try/except introduced in #9409. See
ee1dd1912f4678d7b960b670473bd4b3ffbd890b for the fix.
Without this the plot manager thread silently dies on exceptions and
refreshing just stops. This change leads to refreshing reset and a error
log on exceptions.
* chia|tests: Pass the loaded/removed plots with `PlotRefreshResult`
Instead of just the counts.
* plotting: Default to `False` for `expected_result_matched`
* tests: Validate the loaded/removed plots match the expectations
* plotting|util: Introduce `chia.util.generator_tools.list_to_batches`
* tests: Test `list_to_batches` in `test_list_to_batches.py`
* util|tests: Return an empty iterator for empty input lists
* tests: Test list sizes from 1-10 in `test_valid`
* plotting: Move plot removals out of `PlotManager.refresh_batch`
Moving this checks up in the trace so that it only runs once for each
refresh cycle reduces `plot_removed` calls per cycle from `batch_count *
plot_count` to `plot_count` only. This checks can be quite expensive
depending on how/where the plots are stored so this should noticeably
improve refresh times in slow setups and large farms even for the
initial refresh cycle. I don't see any downside here considered that
even if a plot gets removed during a refresh cycle (after the checks
run) it will catch it in the next cycle.
**Note**: This has become an issue now (e.g. in #8972) because prior to
way, that it only really processed in batches for the very first run
after a restart. For each repeated refresh cycle it just walked through
all files in one `refresh_batch` call. Now after #8385 fixed this
behavior the checks are called `batch_count * plot_count` for all
refresh cycles, even for each repeated cycle where it was `plot_count`
before.
* plotting: Move `file_path.exists` call down
There is no point in testing it before here, it only leads to redundant
calls for repeated refresh cycles. The important point here is just that
it runs after
```
if file_path in self.plots:
return self.plots[file_path]
```
so it could still be moved around differently if there are other
suggestions, tell me. I even think it might be possible to just drop it
with no bad impact but lets just move it for now.
* plotting|harvester|tests: Improve batch processing
Let it only process `batch_size` files per `refresh_batch` call.
* plotting: Improve thread shutdown
* plotting|harvester|tests: Drop `loaded_size` from `PlotRefreshResult`
* rename: Drop `_plots` and `_files` suffixes in `PlotRefreshResult`
* plotting: Drop `PlotRefreshResult.__add__`
I introduced it but i don't really like it because its not a "real" add
any only works in one direction because of `remaining`.
* plotting: Refactor file exists check in `process_file`
* plotting: Refactor processing of the `ThreadPoolExecutor.map()` result
I might miss something but imo the use of `reduce` here just reduces
readability and has no value.
* plotting: Let `process_file` return `Optional[PlotInfo]`
Matt HauffandGitHub
Arvid NorbergandGitHub
Almog De PazandGitHub
Izumi HoshinoandGitHub
dustinfaceandGitHub
