Commit Graph
122 Commits
Author SHA1 Message Date
ab53d48995 Request header blocks, and new rate limits (#11636)
* new blob block api method integrated into wallet

* direct msg streaming of headers, rename, tests

* perform_handshake call fix

* updated trusted sync with new block header calls

* add max blocks limit to fetch

* added tests for rejected block header msgs

* avoid parsing transactions info if not required

* avoid looking up capabilities setting

* move block tests out of a class

* test fix

* Merge changes

* added docs and increased rate limits

* increased block header request interval from 32 to 128

* remove fetching hashes and use height range

* fetching by height in db v2

* update capabilities, other fixes

* fixed range block header call

* Add type hints

* Start work on optimizing fetch_last_tx_from_peer

* Huge speedup in trusted wallet sync

* Revert unintentional changes

* Fix trade issue

* Improve the code

* Str format

* Optimize handling of farming rewards

* Fix bug

* Performance fixes

* Optimizations to wallet syncing

* Don't return all coins in respond_additions

* Revert concurrency numbers

* More optimization of the caches

* Small optimization in coin_added

* Optimize request_additions significantly by using a cache

* fixes from feedback

* capabilities check fixes

* Increase rate limits to allow 250tps in verification requests

* Start work on rate limits

* New rate limit versioning support

* Revert unrelated changes

* revert return False

* Lint

* Revert cbi

* try tests with trusted peer

* Revert unrelated wallet changes

* Revert more debug changes

* Add test and throw on an error if not found

* Reject invalid requests

* Revert bad change with uint32, and change warning to info

* Parametrize wallet sync test

* Merge and LGTM

* More clean way to choose peers

* Fix lint

* add the new RejectBlockHeaders, RequestBlockHeaders and RespondBlockHeaders to the network protocol regression test and regenerate test files

* Rate limit diffs only

* Improve performance

* Simpler

* Lint

Co-authored-by: Sebastjan <trepca@gmail.com>
Co-authored-by: arvidn <arvid@libtorrent.org>
2022-06-11 01:35:41 -05:00
Kyle AltendorfandGitHub 90516263aa remove chia.util.path.mkdir() (#10842)
* remove chia.util.path.mkdir()

* remove submodule change

* less tmpdir, more tmp_path
2022-06-08 10:24:40 -05:00
a87e8e34ac black the generated protocol messages code (#11786)
* black the generated protocol messages code

* Update tests/util/build_network_protocol_files.py

Co-authored-by: Arvid Norberg <arvid@libtorrent.org>

* more focused noqa

Co-authored-by: Arvid Norberg <arvid@libtorrent.org>
2022-06-08 10:21:03 -05:00
Arvid NorbergandGitHub 3c203e335c add regression test for json formatting of protocol messages (#11770) 2022-06-03 12:34:03 -05:00
Arvid NorbergandGitHub db1cc4d32a generalize script to generate protocol message test data (#11751) 2022-06-02 13:11:24 -05:00
Arvid NorbergandGitHub d08d1b0a11 make the protocol regression test also ensure all messages are covered (#11749) 2022-06-02 13:10:54 -05:00
8b2f7ec24c Add Windows CI (#10681)
* Add Windows CI

* rebuild workflows

* rebuild workflows

* try different activation

* catch up windows workflow template

* debug dir

* hard code dev extra on windows

* it is pytest

* .exe

* -m pytest for windows

* --module pytest

* Revert "debug dir"

This reverts commit 74b5bbfd92.

* debug pkg_resources import on windows

* debug tests.conftest import on windows

* rebuild workflows

* debug

* debug oops

* debug more

* debug more

* Update pytest.ini

* Update pytest.ini

* fix it

* Revert "debug more"

This reverts commit 367ed61dfa.

* Revert "debug more"

This reverts commit 772b6707c6.

* Revert "debug oops"

This reverts commit 5065c1e172.

* Revert "debug"

This reverts commit ecf721e453.

* Revert "debug tests.conftest import on windows"

This reverts commit ea71a51625.

* Revert "debug pkg_resources import on windows"

This reverts commit eb6afb1dfb.

* more warning ignore

* they're regexes...

* --numprocesses 0 for windows ci

* specify each test file individually since pytest on windows does not seem to handle the wildcards

* relative...

* select most recent supported python on windows, support INSTALL_PYTHON_VERSION

* sort test files when generating workflows

* report full python version on windows, actually use the selected version for the env

* oops

* debug dir

* skip timelord-requiring jobs on windows

* oops

* handle wildcard expansion for coverage on windows

* rebuild workflows

* download blocks and plots on windows

* correct and and or in yaml

* -d for Install.ps1

* git config --global core.autocrlf false

* add missing windows plot_sync workflow

* no check resource usage on windows

* delete outdated windows workflows that need the timelord

* catch up windows workflow template

* try again without xdist

* rebuild workflows

* Revert "try again without xdist"

This reverts commit 4bedf5f047.

* maybe now no xdist on windows

* maybe now no xdist on windows

* rebuild workflows

* debug chiapos==1.0.9

* Revert "debug chiapos==1.0.9"

This reverts commit 4b374e128a.

* debug blspy==1.0.9

* catch up windows workflow template

* rebuild workflows

* Various windows CI fixes

* Windows CI fixes and workflow updates

* ignore another warning for the keyring

* pre-commit fixup

* rebuild workflows

* catchup windows test workflows with long_lived/** and release/** setup

* Update test_full_sync.py

* Try without parallelization

* Some other tweaks for windows

* some more timeout tweaks

* more timeout tweaks for windows

* rebuild workflows

* Adjusting more timeouts

* Skip ipv6 test on windows

* More timeout tweaks

* More timeout adjustments

* More time out adjustments

* more windows tweaks

* Try running some things with -n 0

* Fix up windows test template and re-gen workflows

* Try this again

* Try all windows tests with -n 0

* misc update

* Undo some timeout changes

* Adjust timeout

* Increase timeout

* Coverage combine should combine the files in the current directory

* Trying retry test on windows

* timeout adjustment

* use tmp_path

* Few minor updates based on review

* lint fix

* Update tests/build-workflows.py

* Update tests/core/server/test_dos.py

* Try windows with pytest xdist n=2

* Check windows with xdist n=3

* Trying n=2 again

* Adjust timeout

* Update testconfig.py

* Update build-workflows.py

* Update testconfig.py

* Update .pre-commit-config.yaml

* Update testconfig.py

* Update build-workflows.py

* Update build-test-macos

* Update build-test-ubuntu

* Update build-test-windows

* skip

* Apply suggestions from code review

* tidy

* Update test_pool_config.py

Co-authored-by: Earle Lowe <e.lowe@chia.net>
Co-authored-by: Earle Lowe <30607889+emlowe@users.noreply.github.com>
2022-05-31 14:04:45 -05:00
dustinfaceandGitHub 5844c6cda2 tests: Avoid using match of pytest.raises together with paths (#11638) 2022-05-26 15:15:30 -07:00
Kyle AltendorfandGitHub 12ec1a2b3e add tests.util.misc.assert_maximum_duration for benchmark assertions (#11589)
* add tests.util.misc.assert_maximum_duration for benchmark assertions

* fix some imports

* more

* lint

* correct duration

* hint fixup

* rename to caller_file_and_line()

* set default timer to thread_time()

* extract gc manager and default to disabling

* self calibrate overhead

* wrap none-ness behind .results() method

* add messages

* from __future__ import annotations

* yield out self

* final from typing_extensions

* split it

* just use Future

* tweak

* tweak

* correction in loosely hint checked file

* all future indexing in hints

* union for the hints

* rename

* assert_runtime()

* another rename...

* message -> label

* label all results with test names

* oops
2022-05-23 12:50:48 -07:00
Arvid NorbergandGitHub afae2ad5fe make recurse_jsonify() work directly on types (#11537)
* make recurse_jsonify() work directly on types, circumventing the dataclasses.asdict() step. This enables simpler integration of non dataclasses into the Streamable and JSON protocols

* add benchmark for Streamable.to_json_dict()
2022-05-18 09:15:03 -07:00
Kyle AltendorfandGitHub f9226fb60d rename test files that are missing test_ prefix (#10712)
* rename test files that are missing test_ prefix

* update mypy and isort excludes

* skip test_get_host_addr6() in GitHub Actions under macOS

* rebuild workflows
2022-05-16 09:51:00 -07:00
7c91f470f4 simplify SizedBytes and StructStream (#11429)
* simplify SizedBytes and StructStream

* lint

* super()  !!!

* do not pass parameter up to super().__init__()

* Update chia/util/struct_stream.py

Co-authored-by: Arvid Norberg <arvid@libtorrent.org>

* parse fixed-width int data from class name

* add int512 and uint128 .from_bytes(), test .parse() failures

* test serialization against struct.pack()

* use typing_extensions for final

* override ignore

* stop using struct for StructStream

oh the irony

* fixup .to_bytes() to accept parameters again for where we use that

* bring back signed parameter

* format

* adjust tests for new exception

* eliminate custom coding for uint128 and int512

* tidy

* remove unused StructStream.PACK attribute

* add direct tests for parse_metadata_from_name()

* stricter hinting

* remove no-longer-needed typeshed work-around

* apply strict type checking to all touched files

* remove StructStream override of .to_bytes()

* tidy

* types touchup

* add unused parameter comments

Co-authored-by: Arvid Norberg <arvid@libtorrent.org>
Co-authored-by: wjblanke <wjb98672@gmail.com>
2022-05-14 02:05:27 -07:00
Arvid NorbergandGitHub 3795a52152 fix some LGTM issues (#11453) 2022-05-12 15:17:24 -07:00
dustinfaceandGitHub 984c442b5e util: Implement Paginator class as interface to access a list by pages (#11247)
* util: Implement `Paginator` class as interface to access a list by pages

* Be less restrictive about page sizes and refactor tests

* Make the pages based of 0 instead of 1 and some more test refactoring

* More tests

* Adjust workflows after rebase

* Introduce `Paginator.create`

* `<=` instead of `- 1`
2022-04-29 18:35:47 -07:00
Kyle AltendorfandGitHub 0aaa343691 replace some asyncio.get_event_loop().run_until_complete() with asyncio.run() (#11131) 2022-04-20 11:03:31 -07:00
Arvid NorbergandGitHub 6259643c36 make listen port colissions in CI less likely (#11164) 2022-04-13 14:47:14 -07:00
2f9e718073 Ms.fast test blockchain (#11051)
* more work on test blockchain

* Optimize test_blockchain.py

* Fix weight proof bug

* Rename variable

* first rc_sub_slot hash bug

* New plots

* try with a new ID

* Run without cache

* Address test blocks and plots preparation.

* Update constant in test_compact_protocol().

* Update this constant too.

* Revert accidental altering of the gui submodule in ae7e3295f280a591e76c4dffdea75fb74ea5de6f.

* Fix benchmark test

* Revert mozilla-ca change

* Rebase on main

Co-authored-by: almog <almogdepaz@gmail.com>
Co-authored-by: Amine Khaldi <amine.khaldi@reactos.org>
2022-04-08 09:57:59 -07:00
Arvid NorbergandGitHub fe77c69018 run_generator2 rust call and compact conditions data structure (#8862)
* use run_generator2 rust call and compact spend bundle conditions data structure pervasively.

* address review comments
2022-04-04 11:53:13 -07:00
Mariano SorgenteandGitHub 890c7d3754 Fix remaining linting issues (#10962)
* FIx remaining linting issues

* Revert type:ignore

* Revert token_bytes change
2022-04-02 13:22:55 -07:00
Kyle AltendorfandGitHub 219e815fc4 set CHIA_ROOT in tests instead of symlinking (#10682)
* attempt to checkout test-cache directly to desired location

* rebuild workflows

* maybe we can use CHIA_ROOT

* use CHIA_ROOT to find blocks and plots for tests

* oops

* more informational printing

* oops

* --capture no for debugging

* flake8

* import os

* undo some unrelated changes now covered elsewhere

* undo some debug changes

* rebuild workflows
2022-03-28 13:15:24 -07:00
Arvid NorbergandGitHub 7a82f0d299 use rust clvm in Program.run() (#10878)
* remove Program.from_serialized_program

* run the rust clvm implementation (instead of python) even for wallet programs
2022-03-28 11:58:59 -07:00
Arvid NorbergandGitHub a100dda37e new DBWrapper supporting concurrent readers (#10166)
* new DBWrapper supporting concurrent readers

* adress review comments

* fixup default database version, when file doesn't exist

* remove unused argument
2022-03-28 11:58:00 -07:00
JeffandGitHub 292c26d3d8 Hardcoded SSL test certs/keys (#10828)
* Hardcoded SSL test certs/keys

* Added a second set of certs/keys.
Cert/key sets are infinitely cycled-through using get_next_private_ca_cert_and_key() and get_next_nodes_certs_and_keys()

* More cert/key sets and a tool to generate them

* Updated SSL generator to sign with the appropriate root CA.
Fixed linter issues.

* Linter fixes

* Updated generate_ssl_for_nodes() based on feedback
2022-03-25 09:24:59 -07:00
Kyle AltendorfandGitHub 4ea82fdb09 use DEFAULT_ROOT_PATH in tests (#10801) 2022-03-24 09:29:05 -07:00
Arvid NorbergandGitHub b5d51c38f8 run tests in parallel in CI (#10499) 2022-03-22 15:04:07 -07:00
Kyle AltendorfandGitHub 25917d56a5 stop helping mkdir() do what it already does (#10802)
* stop helping mkdir() do what it already does

* flake8
2022-03-21 16:22:05 -07:00
Kyle AltendorfandGitHub f5d692f5c3 require test-cache repo is found in CI (#10711) 2022-03-17 09:02:41 -07:00
Arvid NorbergandGitHub 7bab39161f more entropy in random listen ports (#10743) 2022-03-16 12:03:56 -07:00
Kyle AltendorfandGitHub c63324abe9 remove some event_loop() fixtures (#10420)
* remove event_loop() fixtures

* flake8

* flake8

* remove sys.exit() from daemon shutdown

* bump full node test timeout.  a lot...  to see.

* fixup some tests

* back to module scope event loop fixture for test_full_node.py

* Update test_full_node.py

* Iterator...

* for the whole directory

* some fixtures back to module scope for reduced runtime

* back to 40 minute workflow timeouts

* these are being addressed separately
2022-03-14 06:08:35 -07:00
JeffandGitHub 184920a55c Better management of KeyringWrapper's keys_root_path when using TempKeyring for tests (#10636)
* Better management of KeyringWrapper's keys_root_path when using TempKeyring for tests.

* Move keys_root_path restoration code into `cleanup()`
Added an assert to detect if an unexpected shared KeyringWrapper is injected during a test.

* Conditionally restore keys_root_path for testing
2022-03-11 18:48:45 -08:00
Arvid NorbergandGitHub 5b0fb70d97 a problem with using random.randint() to pick a port is that all processes (running in parallel) are seeded the same, and so pick the same ports at the same time, causing conflicts. This uses proper entropy instead. (#10621) 2022-03-10 11:08:30 -08:00
Adam KellyandGitHub ff324095cc Remove block tools and keychain globals (#10358)
* Use bt fixture

* rebase

* Use local_hostname

* Fix test_json (inheritance from unittest.TestCase)

* Use correct BlockTools fixture for test_simulation

* Pass bt fixture into cost calculation tests

* flake8

* Add missing parameters to test functions

* Fix from rebase issues

* Remove set_shared_instance

* Update comment

* Remove unneeded comments

* Remove unused code

* Remove unused code, run `multiprocessing.set_start_method("spawn")` at correct time.

* Revert unrelated change

* Set daemon_port. Teardown services in correct order.
    BIG thanks to Mariano Sorgente for debugging help.

* Add back type signature - rebase issue

* Apply review fixes from Jeff

* Document why we need a later pytest-asyncio version

* Correct type for _configure_legacy_backend

* See what's going on during CI mypy run

* github workflows

* mypy typing

* Remove legacy Keyring create method

* Start daemon first

* Shutdown daemon coroutine properly

* Remove un-needed daemon_port argument

* Set chia-blockchain-gui to hash in main

* Remove connect_to_daemon_port

* Remove code that set "daemon_port" before calling `setup_daemon`

* Remove self_hostname fixture and extra self_hostname global

* Fix two test files that were not importing self_hostname

* self_hostname fixture

* Remove more unused test code

* Simplify  fixture
2022-03-10 11:06:49 -08:00
Kyle AltendorfandGitHub a0897d8d31 context manager for socket in find_available_listen_port(), catch OSError (#10567) 2022-03-09 03:40:52 -08:00
Arvid NorbergandGitHub 018b67f0ad improve the picking of free ports for tests (#10491) 2022-03-02 20:47:50 -08:00
Arvid NorbergandGitHub 874cc23c71 add db validate function to check consistency of blockchain database (#10398) 2022-02-25 08:00:16 -08:00
Arvid NorbergandGitHub 97ef7b275a use free listen ports when running tests (#10307) 2022-02-22 16:22:11 -08:00
Arvid NorbergandGitHub aa33d7e94b use a proper temporary file for the blockchain database in tests, rather than files in the current directory with potentially colliding names (#10369) 2022-02-22 09:39:38 -08:00
Arvid NorbergandGitHub 58019afd5b chunk coin_store request into smaller sql queries (#10359)
* chunk coin_store request into smaller sql queries, to not exceed the limit of 999 on old versions of sqlite

* extend tests for chunks
2022-02-21 20:07:05 -08:00
Kyle AltendorfandGitHub 47a0a9101c apply isort (#10280) 2022-02-17 19:46:51 -08:00
Kyle AltendorfandGitHub f6e479cc07 Improve StructStream hints to support decimal and other parameters like int itself (#9949)
* Improve StructStream hints to support decimal and other parameters like int itself

* get SupportsIndex from typing_extensions
2022-02-17 07:35:13 -08:00
Arvid NorbergandGitHub 34159d3529 optimize get_block_generator() (#10104)
* optimize get_block_generator()

* add a v1 compatible get_generator() to speed up get_block_generator() with v1 databases. Add test. Add error log in case generator_from_block() fails.

* speed up test_full_block_utils
2022-02-17 07:32:29 -08:00
JeffandGitHub 4920a464c1 Remove the "Cleaning up temp keychain in dir" message to reduce noise. (#10264)
* Remove the "Cleaning up temp keychain in dir" message to reduce noise.

* Remove unnecessary temp variable
2022-02-16 18:51:56 -05:00
Arvid NorbergandGitHub 3d063fea8f remove the GeneratorArgs type, and instead include the heights and generator programs as separate lists in BlockGenerator. The heights are not necessary when validating blocks, so this makes it easier to omit them in that case. The heights are only used when generating/farming a block. (#10006) 2022-01-31 18:49:39 -08:00
b8ada1ceb7 introduce restrictions on generators at a specific height (#9957)
* introduce restrictions on generators at a specific height. disallow division on negative numbers and disallow redundant leading zeros on integer condition arguments (produced by a generator)

* use SOFT_FORK_HEIGHT constant

* there is no need to specify height when validating block in block_creation

* Update tests/core/full_node/test_mempool.py

Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>

Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
2022-01-28 12:29:11 -08:00
Arvid NorbergandGitHub 93d967525b revert allow-upgrades configuration option. It makes it significantly more complicated to do proper database migrations (#9768) 2022-01-13 14:53:27 -08:00
Kyle AltendorfandGitHub e498a406a0 Make the hard coded 2 reserved cores configurable (#9709)
* Make the hard coded 2 reserved cores configurable

* add missing reserved_cores parameters

* add missing reserved_cores parameters
2022-01-10 20:58:29 -08:00
Arvid NorbergandGitHub 2801d96477 unify the name we use for strict/safe mode (#9545)
* unify the name we use for the special mode we run CLVM in, when accepting transactions into the mempool. We currently call this, strict mode, safe mode, safe and unsafe. Neither of these names are very descriptive. This patch renames this mode to mempool-mode

* import as
2021-12-13 08:38:35 -08:00
Arvid NorbergandGitHub 70d31a8c68 V2 database framework (#9442)
* fix block_store benchmark linting issues

* basic support for database versioning

* update benchmark to run v1 and v2
2021-12-03 13:52:10 -08:00
Kyle AltendorfandGitHub 8291f0221a Make the sized bytes types hint compatible (#9369)
* Rework sized bytes for type hinting compatibility

* add a bunch of type: ignores

* this will be handled elsewhere

* noqa E501 instead of changing code

* normalize comment plurality

* @classmethod

* Revert "@classmethod"

This reverts commit 95db80e339.

* add ignore in benchmarks

* just E501 again...

* add some new type: ignores
2021-12-02 09:43:39 -08:00
Arvid NorbergandGitHub a9d0346fd0 cache heigh-to-hash and height-to-ses maps between runs (#8771)
* factor out map of block height to block hash and sub epoch summary from blockchain into its own class, BlockHeightMap

* save and restore the height-to-hash map and height-to-sub epoch summary map to flat files, to imrove startup time. Add unit tests

* add back assert
2021-12-01 09:43:16 -08:00