After changing `plot_paths` to be a set in #11204 the plots are loaded
out of order. Sorting the list before splitting it to batches recovers
the load order.
Fixes#11499
* plotting|tests: Extend the lifetime of plot data cache entries
* plotting|tests: Add `prover` and `farmer_public_key` to the cache
* plotting: Missing type hints in `Cache` and `CacheEntry`
* setup: Bump `chiapos` to 1.0.10
* plotting: Move cache classes into `chia.plotting.cache`
* plotting: Introduce `CacheEntry.from_disk_prover`
* Make `Cache` a dataclass
* `staticmethod` -> `classmethod`
* Store cache data as `bytes` in `DiskCache`
* streamable: Merge `strictdataclass` into `Streamable` class
* tests: Test not supported streamable types
* streamable: Reorder decorators
* streamable: Simplify streamable decorator and force correct usage/syntax
* streamable: Just move some stuff around in the file
* streamable: Improve syntax error messages
* mypy: Drop `type_checking.py` and `test_type_checking.py` from exclusion
* streamable: Use cached fields instead of `__annotations__`
This is now possible after merging `__post_init__` into `Streamable`
* Introduce `DefinitionError` as `StreamableError`
* `/t` -> ` `
* protocols|server: Define new harvester plot refreshing protocol messages
* protocols: Bump `protocol_version` to `0.0.34`
* tests: Introduce `setup_farmer_multi_harvester`
Allows to run a test setup with 1 farmer and mutiple harvesters.
* plotting: Add an initial plot loading indication to `PlotManager`
* plotting|tests: Don't add removed duplicates to `total_result.removed`
`PlotRefreshResult.removed` should only contain plots that were loaded
properly before they were removed. It shouldn't contain e.g. removed
duplicates or invalid plots since those are synced in an extra sync step
and not as diff but as whole list every time.
* harvester: Reset `PlotManager` on shutdown
* plot_sync: Implement plot sync protocol
* farmer|harvester: Integrate and enable plot sync
* tests: Implement tests for the plot sync protocol
* farmer|tests: Drop obsolete harvester caching code
* setup: Add `chia.plot_sync` to packages
* plot_sync: Type hints in `DeltaType`
* plot_sync: Drop parameters in `super()` calls
* plot_sync: Introduce `send_response` helper in `Receiver._process`
* plot_sync: Add some parentheses
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* plot_sync: Additional hint for a `Receiver.process_path_list` parameter
* plot_sync: Force named parameters in `Receiver.process_path_list`
* test: Fix fixtures after rebase
* tests: Fix sorting after rebase
* tests: Return type hint for `plot_sync_setup`
* tests: Rename `WSChiaConnection` and move it in the outer scope
* tests|plot_sync: More type hints
* tests: Rework some delta tests
* tests: Drop a `range` and iterate over the list directly
* tests: Use the proper flags to overwrite
* test: More missing duplicates tests
* tests: Drop `ExpectedResult.reset`
* tests: Reduce some asserts
* tests: Add messages to some `assert False` statements
* tests: Introduce `ErrorSimulation` enum in `test_sync_simulated.py`
* tests: Use `secrects` instead of `Crypto.Random`
* Fixes after rebase
* Import from `typing_extensions` to support python 3.7
* Drop task name to support python 3.7
* Introduce `Sender.syncing`, `Sender.connected` and a log about the task
* Add `tests/plot_sync/config.py`
* Align the multi harvester fixture with what we do in other places
* Update the workflows
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* tests: Allow to use custom keys in `BlockTools.new_plot`
* tests: Create three plots with `G1Element()` as farmer/pool key
But not add them as default plots. Just to have plots in the test cache with different keys from
within and outside the keychain for plot loading tests.
create
* test|github: Bump `test-cache` version to `0.28.0`
* plotting: Drop plots from `no_key_filenames` after the keys appeared
* plotting: Drop plots from `no_key_filenames` if the plot was removed
* tests: Test loading of plots where we miss the keys
* plotting: Use `x.copy()` instead of `list(x)`
* plotting: Add a comment about the missing keys plot removal
* tests: Fix after #9595
* plotting: Fix try/except in `PlotManager._refresh_task`
* tests: Move the start refreshing call into `PlotRefreshTester` ctor
It being in `run()` before where it was called every test iteration lead
to not detecting an issue with the try/except introduced in #9409. See
ee1dd1912f4678d7b960b670473bd4b3ffbd890b for the fix.
Without this the plot manager thread silently dies on exceptions and
refreshing just stops. This change leads to refreshing reset and a error
log on exceptions.
* chia|tests: Pass the loaded/removed plots with `PlotRefreshResult`
Instead of just the counts.
* plotting: Default to `False` for `expected_result_matched`
* tests: Validate the loaded/removed plots match the expectations
* plotting|util: Introduce `chia.util.generator_tools.list_to_batches`
* tests: Test `list_to_batches` in `test_list_to_batches.py`
* util|tests: Return an empty iterator for empty input lists
* tests: Test list sizes from 1-10 in `test_valid`
* plotting: Move plot removals out of `PlotManager.refresh_batch`
Moving this checks up in the trace so that it only runs once for each
refresh cycle reduces `plot_removed` calls per cycle from `batch_count *
plot_count` to `plot_count` only. This checks can be quite expensive
depending on how/where the plots are stored so this should noticeably
improve refresh times in slow setups and large farms even for the
initial refresh cycle. I don't see any downside here considered that
even if a plot gets removed during a refresh cycle (after the checks
run) it will catch it in the next cycle.
**Note**: This has become an issue now (e.g. in #8972) because prior to
way, that it only really processed in batches for the very first run
after a restart. For each repeated refresh cycle it just walked through
all files in one `refresh_batch` call. Now after #8385 fixed this
behavior the checks are called `batch_count * plot_count` for all
refresh cycles, even for each repeated cycle where it was `plot_count`
before.
* plotting: Move `file_path.exists` call down
There is no point in testing it before here, it only leads to redundant
calls for repeated refresh cycles. The important point here is just that
it runs after
```
if file_path in self.plots:
return self.plots[file_path]
```
so it could still be moved around differently if there are other
suggestions, tell me. I even think it might be possible to just drop it
with no bad impact but lets just move it for now.
* plotting|harvester|tests: Improve batch processing
Let it only process `batch_size` files per `refresh_batch` call.
* plotting: Improve thread shutdown
* plotting|harvester|tests: Drop `loaded_size` from `PlotRefreshResult`
* rename: Drop `_plots` and `_files` suffixes in `PlotRefreshResult`
* plotting: Drop `PlotRefreshResult.__add__`
I introduced it but i don't really like it because its not a "real" add
any only works in one direction because of `remaining`.
* plotting: Refactor file exists check in `process_file`
* plotting: Refactor processing of the `ThreadPoolExecutor.map()` result
I might miss something but imo the use of `reduce` here just reduces
readability and has no value.
* plotting: Let `process_file` return `Optional[PlotInfo]`
* Revert "plotting: Fix failed_to_open_filenames re-try interval"
This reverts 8a8abc41a8 which is part of
#7848. Seems like i confused myself there.
* plotting|util: Introduce `retry_invalid_seconds` in plot refresh params
* plotting: Cleanup `failed_to_open_filenames`
* tests: Test re-try of plots which failed to load
* harvester|plotting|tests: Introduce `PlotManager` class
This moves all plots related and plot-directory related stuff from the
harvester into the class `PlotManager`, adjusts all related code
accordingly and adds some extra wrappers there.
* harvester|plotting|tests: Return how many new plots were loaded
* plotting: Fix `failed_to_open_filenames` re-try interval
With `< 1200` it just tries it on the next refresh.
* plotting: Fix and improve duplicates handling
* harvester|plotting: Thread locks for `PlotManager.plots`
* chia|tests: Load plots in batches
* chia|tests: Move plot refreshing into a separate thread
* plotting: Properly handle removed plots
And fix tests accordingly. It seems like this fix https://github.com/Chia-Network/chia-blockchain/pull/3350 wasn't really a fix, rather adjusting to allow for a bug?
* plotting|harvester|tests: Introduce `PlotRefreshResult`
* tests: Expand `test_farmer_harvester_rpc.py`
* chia|tests: Move some stuff from `plot_tools.py` into new file `util.py`
* refactor: Rename `plot_tools.py` to `manager.py`
* chia|tests: Use pure dataclass for `PlotsRefreshParameter`
With `uint16` as type saving to config doesn't work, this is a
preparation for the next commit.
* harvester: Adjust deprecation message, use `info` instead of `warning`
* plotting: Fix typo
* refactor: Rename `filename` to `file_path`
Fits better and does avoid shadowing with filename from outer scope.
* chia|tests: Move some methods from `plotting.manager` to `plotting.util`
* plotting: Make `refresh_callback` mandatory
Kyle AltendorfandGitHub
dustinfaceandGitHub
Earle LoweandGitHub