Commit Graph
411 Commits
Author SHA1 Message Date
dustinfaceandGitHub 20f5bb22fb streamable: Improve dataclass_from_dict validation, add more tests (#10652)
* Improve and test `dataclass_from_dict` tuple validation

* Improve and test `dataclass_from_dict` list validation

* Introduce `convert_hex_string`

* Improve and test `dataclass_from_dict` byte type validation

* Improve and test `dataclass_from_dict` unhashable validation

* Improve and test `dataclass_from_dict` primitive validation

* Improve `dataclass_from_dict` validation

* `raise TypeError(...) from e`

* Just `raise`, don't `raise e`

* Use `repr` of `item` not `str`

* `type(item) == f_type` -> `isinstance(item, f_type)`
2022-06-14 12:40:33 -05:00
Arvid NorbergandGitHub 5ae87189c4 Rust coin (#11807)
* use rust implementation of Coin class, from chia_rs

* bump chia_rs dependency
2022-06-13 07:21:04 -05:00
ab53d48995 Request header blocks, and new rate limits (#11636)
* new blob block api method integrated into wallet

* direct msg streaming of headers, rename, tests

* perform_handshake call fix

* updated trusted sync with new block header calls

* add max blocks limit to fetch

* added tests for rejected block header msgs

* avoid parsing transactions info if not required

* avoid looking up capabilities setting

* move block tests out of a class

* test fix

* Merge changes

* added docs and increased rate limits

* increased block header request interval from 32 to 128

* remove fetching hashes and use height range

* fetching by height in db v2

* update capabilities, other fixes

* fixed range block header call

* Add type hints

* Start work on optimizing fetch_last_tx_from_peer

* Huge speedup in trusted wallet sync

* Revert unintentional changes

* Fix trade issue

* Improve the code

* Str format

* Optimize handling of farming rewards

* Fix bug

* Performance fixes

* Optimizations to wallet syncing

* Don't return all coins in respond_additions

* Revert concurrency numbers

* More optimization of the caches

* Small optimization in coin_added

* Optimize request_additions significantly by using a cache

* fixes from feedback

* capabilities check fixes

* Increase rate limits to allow 250tps in verification requests

* Start work on rate limits

* New rate limit versioning support

* Revert unrelated changes

* revert return False

* Lint

* Revert cbi

* try tests with trusted peer

* Revert unrelated wallet changes

* Revert more debug changes

* Add test and throw on an error if not found

* Reject invalid requests

* Revert bad change with uint32, and change warning to info

* Parametrize wallet sync test

* Merge and LGTM

* More clean way to choose peers

* Fix lint

* add the new RejectBlockHeaders, RequestBlockHeaders and RespondBlockHeaders to the network protocol regression test and regenerate test files

* Rate limit diffs only

* Improve performance

* Simpler

* Lint

Co-authored-by: Sebastjan <trepca@gmail.com>
Co-authored-by: arvidn <arvid@libtorrent.org>
2022-06-11 01:35:41 -05:00
Kyle AltendorfandGitHub 90516263aa remove chia.util.path.mkdir() (#10842)
* remove chia.util.path.mkdir()

* remove submodule change

* less tmpdir, more tmp_path
2022-06-08 10:24:40 -05:00
dustinfaceandGitHub 4230af1a59 tests: Refactor and improve streamable __post_init__ tests (#11726) 2022-06-03 18:42:05 -05:00
dustinfaceandGitHub a375e3cb8f farmer: Make Plot{Info|Path}RequestData streamable + use from_json_dict (#11762)
* Make `PlotInfoRequestData` and `PlotPathRequestData` streamable

* Use `from_json_dict` instead of `dataclass_from_dict`

* Recover `PaginatedRequestData(Protocol)`
2022-06-03 12:37:03 -05:00
1aeeffedbe update unknown condition test (#11746)
* since condition opcode 1 is about to become well defined, use some other opcode to test unknown conditions

* Update tests/core/full_node/test_mempool.py

Co-authored-by: Richard Kiss <him@richardkiss.com>

Co-authored-by: Richard Kiss <him@richardkiss.com>
2022-06-03 12:36:17 -05:00
dustinfaceandGitHub d3baa4c236 Use Plot.from_json_dict instead of dataclass_from_dict (#11760) 2022-06-03 12:35:17 -05:00
dustinfaceandGitHub 5c5ab111ad streamable: Fix default value assignments for dataclass_from_dict (#11732)
* streamable: Use constructor in `dataclass_from_dict`

This fixes default value assignments after #10561 but also leads to less perfomance due to `__post_init__` being called which at least gets mitigated by #11730.

* tests: Test default values with `from_json_dict`

* Convert to `str`, then compare.
2022-06-03 12:14:57 -05:00
Arvid NorbergandGitHub 3c2fc1947d upgrade chia-rs to streamable support (#11503)
* upgrade chia-rs to streamable support

* bump chia_rs dependency
2022-06-01 12:56:11 -05:00
8b2f7ec24c Add Windows CI (#10681)
* Add Windows CI

* rebuild workflows

* rebuild workflows

* try different activation

* catch up windows workflow template

* debug dir

* hard code dev extra on windows

* it is pytest

* .exe

* -m pytest for windows

* --module pytest

* Revert "debug dir"

This reverts commit 74b5bbfd92.

* debug pkg_resources import on windows

* debug tests.conftest import on windows

* rebuild workflows

* debug

* debug oops

* debug more

* debug more

* Update pytest.ini

* Update pytest.ini

* fix it

* Revert "debug more"

This reverts commit 367ed61dfa.

* Revert "debug more"

This reverts commit 772b6707c6.

* Revert "debug oops"

This reverts commit 5065c1e172.

* Revert "debug"

This reverts commit ecf721e453.

* Revert "debug tests.conftest import on windows"

This reverts commit ea71a51625.

* Revert "debug pkg_resources import on windows"

This reverts commit eb6afb1dfb.

* more warning ignore

* they're regexes...

* --numprocesses 0 for windows ci

* specify each test file individually since pytest on windows does not seem to handle the wildcards

* relative...

* select most recent supported python on windows, support INSTALL_PYTHON_VERSION

* sort test files when generating workflows

* report full python version on windows, actually use the selected version for the env

* oops

* debug dir

* skip timelord-requiring jobs on windows

* oops

* handle wildcard expansion for coverage on windows

* rebuild workflows

* download blocks and plots on windows

* correct and and or in yaml

* -d for Install.ps1

* git config --global core.autocrlf false

* add missing windows plot_sync workflow

* no check resource usage on windows

* delete outdated windows workflows that need the timelord

* catch up windows workflow template

* try again without xdist

* rebuild workflows

* Revert "try again without xdist"

This reverts commit 4bedf5f047.

* maybe now no xdist on windows

* maybe now no xdist on windows

* rebuild workflows

* debug chiapos==1.0.9

* Revert "debug chiapos==1.0.9"

This reverts commit 4b374e128a.

* debug blspy==1.0.9

* catch up windows workflow template

* rebuild workflows

* Various windows CI fixes

* Windows CI fixes and workflow updates

* ignore another warning for the keyring

* pre-commit fixup

* rebuild workflows

* catchup windows test workflows with long_lived/** and release/** setup

* Update test_full_sync.py

* Try without parallelization

* Some other tweaks for windows

* some more timeout tweaks

* more timeout tweaks for windows

* rebuild workflows

* Adjusting more timeouts

* Skip ipv6 test on windows

* More timeout tweaks

* More timeout adjustments

* More time out adjustments

* more windows tweaks

* Try running some things with -n 0

* Fix up windows test template and re-gen workflows

* Try this again

* Try all windows tests with -n 0

* misc update

* Undo some timeout changes

* Adjust timeout

* Increase timeout

* Coverage combine should combine the files in the current directory

* Trying retry test on windows

* timeout adjustment

* use tmp_path

* Few minor updates based on review

* lint fix

* Update tests/build-workflows.py

* Update tests/core/server/test_dos.py

* Try windows with pytest xdist n=2

* Check windows with xdist n=3

* Trying n=2 again

* Adjust timeout

* Update testconfig.py

* Update build-workflows.py

* Update testconfig.py

* Update .pre-commit-config.yaml

* Update testconfig.py

* Update build-workflows.py

* Update build-test-macos

* Update build-test-ubuntu

* Update build-test-windows

* skip

* Apply suggestions from code review

* tidy

* Update test_pool_config.py

Co-authored-by: Earle Lowe <e.lowe@chia.net>
Co-authored-by: Earle Lowe <30607889+emlowe@users.noreply.github.com>
2022-05-31 14:04:45 -05:00
Mariano SorgenteandGitHub 8dc1d66cd2 Increase from 180 to 300 (#11707) 2022-05-31 13:40:08 -05:00
William AllenandGitHub 855fe055fa Merge pull request #11645 from AmineKhaldi/main_checkpoint_1.4.0_46ebed3
Merge release/1.4.0's 46ebed3 into main
2022-05-27 12:26:50 -05:00
dustinfaceandGitHub 5844c6cda2 tests: Avoid using match of pytest.raises together with paths (#11638) 2022-05-26 15:15:30 -07:00
Amine Khaldi 83b49e73a1 Merge branch 'release/1.4.0' into main_checkpoint_1.4.0_46ebed3 2022-05-26 08:51:50 +01:00
dustinfaceandGitHub 7142ebcae7 tests: Consolidate farmer/harvester fixtures (#11513)
* tests: Consolidate farmer/harvester fixtures

* Fixup after #11578
2022-05-25 15:15:49 -07:00
arvidn a328fdcabf fix jsonify bool 2022-05-25 09:05:47 +02:00
dda517ee1a streamable: Cache convert functions from dataclass_from_dict (#10561)
* streamable: Cache convert functions for dict -> dataclass conversion

* tests: Test `dataclass_from_dict` with non-streamable classes

* `Any` -> `object`

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Move comment into `dataclass_from_dict`

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-05-24 13:22:22 -07:00
dustinfaceandGitHub c400d816af harvester: Validate the path in add_plot_directory (#11625)
Make sure it exists, is a directory and hasn't been added already.
2022-05-24 13:21:44 -07:00
Kyle AltendorfandGitHub 60c1867a44 async sleep and require connection reset error for DoS test (#11612) 2022-05-23 16:14:04 -07:00
Kyle AltendorfandGitHub 12ec1a2b3e add tests.util.misc.assert_maximum_duration for benchmark assertions (#11589)
* add tests.util.misc.assert_maximum_duration for benchmark assertions

* fix some imports

* more

* lint

* correct duration

* hint fixup

* rename to caller_file_and_line()

* set default timer to thread_time()

* extract gc manager and default to disabling

* self calibrate overhead

* wrap none-ness behind .results() method

* add messages

* from __future__ import annotations

* yield out self

* final from typing_extensions

* split it

* just use Future

* tweak

* tweak

* correction in loosely hint checked file

* all future indexing in hints

* union for the hints

* rename

* assert_runtime()

* another rename...

* message -> label

* label all results with test names

* oops
2022-05-23 12:50:48 -07:00
Arvid NorbergandGitHub 00a2a15844 Coin simplification (#11567)
* factor out as_list() from Coin into free function. Remove unused name_str() from Coin. Minor optimization of hash_coin_ids() for common case.

* extend Coin unit test
2022-05-23 10:43:17 -07:00
Arvid NorbergandGitHub 3d8081cc0d restore missing hints being stored as None (instead of 0-length bytes) (#11568) 2022-05-23 10:42:40 -07:00
Mariano SorgenteandGitHub 817baa3096 Bind port 0 to fix race condition when grabbing available ports (#11578)
* port 0 to fix flakiness

* Try fixing setup_full_system

* Try fixing setup_full_system, and lint

* More attempts to fix

* No more calls to get random ports in setup_nodes

* Revert accidental changes

* Timelord extra arg

* Try with port 0

* Fix daemon test, and lint

* Try without 0.0.0.0

* Back to 0.0.0.0

* Try a few timelord changes to get test running

* Increase timeout again

* Use the correct interface to get the port

* INFO logging to debug issue

* Revert "INFO logging to debug issue"

This reverts commit 7c379e5cca.

* Fix advertised port log

* Add extra log

* Logging back

* Rollback the timelord changes

* Try port 0 timelord

* Revert "Try port 0 timelord"

This reverts commit 4997faf3b2.

* Try full green, change ordering

* Remove unused var

* speed up simulation and cleanup

* Now try without the port config

* Fix a flaky call to get_event_loop

* Try getting the port dynamically

* No dynamic port

* Try changing the ordering

* Try adding a sleep

* Back to what works

* Timelord before vdf clients

* Dynamic port for 1st timelord

* Revert "Dynamic port for 1st timelord"

This reverts commit 0f322a15b7.

* Revert "Timelord before vdf clients"

This reverts commit 3286c34696.

* Revert "Back to what works"

This reverts commit 30380dffb7.

* Revert "Try adding a sleep"

This reverts commit 9212b665a6.

* Revert "Try changing the ordering"

This reverts commit a62597d70d.

* Revert "No dynamic port"

This reverts commit 5d2e15749b.

* Revert "Try getting the port dynamically"

This reverts commit ef9cd75679.

* Revert "Fix a flaky call to get_event_loop"

This reverts commit 01a000fdfb.

* Try one to 0

* Just not 0

* Don't get port dynamically

* Cleanup a bit

* Fix

* Some cleanup work

* Some cleanup work

* Fix daemon test

* Cleanup

* Remove arguments
2022-05-23 08:13:49 -07:00
b644f37277 Fix a few additional flaky tests (#11597)
* Fix a few additional flaky tests

* Lint

* Update tests/core/full_node/stores/test_full_node_store.py

Co-authored-by: Jeff <jeff@chia.net>

* Fix merge conflict

* Remove unnecessary function

Co-authored-by: Jeff <jeff@chia.net>
2022-05-23 07:55:05 -07:00
Mariano SorgenteandGitHub 766e1c21f5 Fix several flaky tests (#11576)
* Increase some of the timeouts to reduce flakiness

* Experiment with CI for flakiness

* Force more runs

* Add argument name

* Fix flaky mempool test

* Attempt to fix another flaky test

* Fix wallet retry test

* Missing arg

* Lint

* Increase benchmark test time

* Debug

* Debug

* Simplify retry test

* Lint

* More lint

* Return false instead of assert

* No need to check disconnect twice (since we don't break, this was flaky)

* Remove useless changes

* Accidental changes revert

* Accidental changes revert 2

* Accidental changes revert 3

* Revert fixture
2022-05-23 07:54:38 -07:00
JeffandGitHub 99275e6e37 Support searching derived addresses on testnet. (#11449)
* Support searching derived addresses on testnet.

* Added tests
2022-05-16 13:29:33 -07:00
Arvid NorbergandGitHub 51fc5f6e88 remove the cache from CoinStore. It appears the cost of maintaining the cache outweighs the gains. (#11540) 2022-05-16 13:28:42 -07:00
Mariano SorgenteandGitHub 56e6e68eb3 Uses the new from_bytes_unchecked method in blspy, to improve perfo… (#11463)
* Uses the new `from_bytes_unchecked` method in blspy, to improve performance

* Update test

* Fix merge conflict

* Use from_bytes_unchecked in post_init and from_json

* More uniform code
2022-05-16 09:49:50 -07:00
Arvid NorbergandGitHub 04b543e26a add test for streamable -> json conversion (#11527)
* add test for streamable -> json conversion

* fixup test
2022-05-15 15:47:17 -07:00
7c91f470f4 simplify SizedBytes and StructStream (#11429)
* simplify SizedBytes and StructStream

* lint

* super()  !!!

* do not pass parameter up to super().__init__()

* Update chia/util/struct_stream.py

Co-authored-by: Arvid Norberg <arvid@libtorrent.org>

* parse fixed-width int data from class name

* add int512 and uint128 .from_bytes(), test .parse() failures

* test serialization against struct.pack()

* use typing_extensions for final

* override ignore

* stop using struct for StructStream

oh the irony

* fixup .to_bytes() to accept parameters again for where we use that

* bring back signed parameter

* format

* adjust tests for new exception

* eliminate custom coding for uint128 and int512

* tidy

* remove unused StructStream.PACK attribute

* add direct tests for parse_metadata_from_name()

* stricter hinting

* remove no-longer-needed typeshed work-around

* apply strict type checking to all touched files

* remove StructStream override of .to_bytes()

* tidy

* types touchup

* add unused parameter comments

Co-authored-by: Arvid Norberg <arvid@libtorrent.org>
Co-authored-by: wjblanke <wjb98672@gmail.com>
2022-05-14 02:05:27 -07:00
Kyle AltendorfandGitHub d1e4a38462 less except -> false (#10219)
* less except -> false

* establish_connection() and .perform_handshake() can just return None

* remove more pointless, and no wrong, assertions
2022-05-14 02:03:56 -07:00
Arvid NorbergandGitHub 3795a52152 fix some LGTM issues (#11453) 2022-05-12 15:17:24 -07:00
Arvid NorbergandGitHub a4ec7e7cb8 Replace MerkleSet with the rust implementation of computing the merkle root. This saves about 20% of the CPU work in the main thread when validating blocks. (#11432) 2022-05-10 04:42:45 -07:00
Mariano SorgenteandGitHub e0df18f85d Ms.hint refactor (#11222)
* Some work on hints

* More cleanup and test_generator_tools

* Change relevant calls

* More refactor and cleanup

* More refactor

* Small fix

* Create wrapper object PeakPostProcessingResult

* Lint and small fix

* Fix a hint bug

* Fix hint update_wallets

* Fix test_full_sync

* pre-commit

* Start with PR review comments

* More efficient iteration

* Remove tx_removals_additions_and_hints

* Revert mozilla ca

* Fix issue with startup
2022-05-05 08:19:57 -07:00
Arvid NorbergandGitHub 1b2d119cc3 optimize hash_coin_list() by sorting pure coin_ids, rather than Coins that need to compute its name multiple times during sorting, and also by sorting the actual binary coin names, not hexadecimal strings (#11421) 2022-05-04 07:24:55 -07:00
Arvid NorbergandGitHub bb3550f43b more merkle tree tests (#11424) 2022-05-04 07:24:10 -07:00
d00d045d9c farmer|rpc|tests: Implement paginated harvester plot endpoints (#11365)
* farmer|rpc|tests: Implement paginated harvester plot endpoints

* Simplify filtering

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Let the API handle the exceptions

* Simplify the other filtering too

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Simplify count assertions

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Refactor `is_filter_match` to `plot_matches_filter`

And just convert to `Plot` in tests.

* Move `chia.util.misc.KeyValue` to `chia.rpc.farmer_rpc_api.FilterItem`

* Rename `peer_id` to `node_id` to be match `get_harvesters_{summary}`

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-05-03 09:17:05 -07:00
Arvid NorbergandGitHub 340e26e154 simplify merkle set (#11416)
* _make_middle doesn't need to be a member function

* add some unit tests
2022-05-03 07:49:29 -07:00
dustinfaceandGitHub 7ed610bc84 test: Give setup_farmer and setup_harvester a separate chia root (#11408)
* test: Give `setup_farmer` and `setup_harvester` a separate chia root

* test: Wait for `last_sync_time` in `get_harvesters_{summary}`

To make sure the first sync from the harvester to the farmer is done 
before we check plot counts.
2022-05-02 19:32:37 -07:00
dustinfaceandGitHub e9e536925c farmer|rpc: Add plot_count to get_pool_state RPC endpoint (#11364)
* farmer|rpc: Add `plot_count` to `get_pool_state` RPC endpoint

* Test `plot_count` of `get_pool_state` farmer RPC endpoint
2022-05-02 13:02:00 -07:00
dustinfaceandGitHub b693aeb407 farmer|rpc: Introduce get_harvesters_summary RPC endpoint (#11245) 2022-04-28 19:36:21 -07:00
Arvid NorbergandGitHub 3b4cbd42ce lookup coins in large batches (#11328)
* lookup coins in large batches, rather than one at a time, in block_body_validation

* make get_coin_records use the cache. and make the blockchain change tracking also use the new batch lookup

* make building the sql query a bit more elegant

* fix assert

* fix test_blockchain assert tests

* add type annotation

* await aiosqlite.Cursor.execute() immediately
2022-04-28 19:34:43 -07:00
Mariano SorgenteandGitHub 7630d220f5 Optimize BLS verification when public key is repeated (#11318)
* Optimize BLS verification when public key is repeated

* Fix accidental commit

* Lint

* Fix isort

* Address comments
2022-04-27 11:38:48 -07:00
Earle LoweandGitHub cb98258a8e Check observer addresses (#11259) 2022-04-21 16:59:11 -07:00
Kyle AltendorfandGitHub 1088801e73 Remove code related to no-longer-used websockets library (#11123)
* Remove logging tweak for no-longer-used websockets library

Follow up after https://github.com/Chia-Network/chia-blockchain/pull/10611.

* Update pylintrc

* Update test_daemon.py
2022-04-20 11:18:42 -07:00
ca4881ff3e Fix incorrect return in "rollback_to_block" (#10954)
* Add comments and fix a coin subscription bug

* Updated rollback test

* simplify test

* expand test

Co-authored-by: Mariano Sorgente <sorgente711@gmail.com>
2022-04-20 11:12:00 -07:00
Arvid NorbergandGitHub 743758c788 apply stricter mempool conditions (no garbage at the end of condition arguments). Fixup test (#10737) 2022-04-20 11:10:30 -07:00
79cbadf987 streamable: Enable isort + more mypy (#10539)
* isort: Fix `streamable.py` and `test_streamable.py`

* mypy: Drop `streamable.py` and `test_streamable.py` form exclusion

And fix all the mypy issues.

* Fix `pylint`

* Introduce `ParseFunctionType` and `StreamFunctionType`

* Use `object` instead of `Type[Any]` for `is_type_*` functions

* Some `Any` -> `object`

* Use `typing.overload` for `recurse_jsonify`

* Move some comments

* Drop `Union`, use `Literal` properly

* Explicitly ignore the return of `f_type.parse`

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Merge two `recurse_jsonify` overloads

* Typing for the base definition of `recurse_jsonify`

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-04-20 11:09:41 -07:00
Kyle AltendorfandGitHub 0aaa343691 replace some asyncio.get_event_loop().run_until_complete() with asyncio.run() (#11131) 2022-04-20 11:03:31 -07:00