Commit Graph
230 Commits
Author SHA1 Message Date
dustinfaceandGitHub 20f5bb22fb streamable: Improve dataclass_from_dict validation, add more tests (#10652)
* Improve and test `dataclass_from_dict` tuple validation

* Improve and test `dataclass_from_dict` list validation

* Introduce `convert_hex_string`

* Improve and test `dataclass_from_dict` byte type validation

* Improve and test `dataclass_from_dict` unhashable validation

* Improve and test `dataclass_from_dict` primitive validation

* Improve `dataclass_from_dict` validation

* `raise TypeError(...) from e`

* Just `raise`, don't `raise e`

* Use `repr` of `item` not `str`

* `type(item) == f_type` -> `isinstance(item, f_type)`
2022-06-14 12:40:33 -05:00
Arvid NorbergandGitHub 5ae87189c4 Rust coin (#11807)
* use rust implementation of Coin class, from chia_rs

* bump chia_rs dependency
2022-06-13 07:21:04 -05:00
ab53d48995 Request header blocks, and new rate limits (#11636)
* new blob block api method integrated into wallet

* direct msg streaming of headers, rename, tests

* perform_handshake call fix

* updated trusted sync with new block header calls

* add max blocks limit to fetch

* added tests for rejected block header msgs

* avoid parsing transactions info if not required

* avoid looking up capabilities setting

* move block tests out of a class

* test fix

* Merge changes

* added docs and increased rate limits

* increased block header request interval from 32 to 128

* remove fetching hashes and use height range

* fetching by height in db v2

* update capabilities, other fixes

* fixed range block header call

* Add type hints

* Start work on optimizing fetch_last_tx_from_peer

* Huge speedup in trusted wallet sync

* Revert unintentional changes

* Fix trade issue

* Improve the code

* Str format

* Optimize handling of farming rewards

* Fix bug

* Performance fixes

* Optimizations to wallet syncing

* Don't return all coins in respond_additions

* Revert concurrency numbers

* More optimization of the caches

* Small optimization in coin_added

* Optimize request_additions significantly by using a cache

* fixes from feedback

* capabilities check fixes

* Increase rate limits to allow 250tps in verification requests

* Start work on rate limits

* New rate limit versioning support

* Revert unrelated changes

* revert return False

* Lint

* Revert cbi

* try tests with trusted peer

* Revert unrelated wallet changes

* Revert more debug changes

* Add test and throw on an error if not found

* Reject invalid requests

* Revert bad change with uint32, and change warning to info

* Parametrize wallet sync test

* Merge and LGTM

* More clean way to choose peers

* Fix lint

* add the new RejectBlockHeaders, RequestBlockHeaders and RespondBlockHeaders to the network protocol regression test and regenerate test files

* Rate limit diffs only

* Improve performance

* Simpler

* Lint

Co-authored-by: Sebastjan <trepca@gmail.com>
Co-authored-by: arvidn <arvid@libtorrent.org>
2022-06-11 01:35:41 -05:00
dustinfaceandGitHub 504b67ee2c streamable: Move some class methods out of Streamable (#11853)
* Make `function_to_parse_one_item` a free function

* Make `function_to_stream_one_item` a free function
2022-06-09 15:53:45 -05:00
Amine Khaldi 11b68de354 Merge release/1.4.0's 6171703 into main 2022-06-08 21:16:10 +01:00
Earle Lowe 463e303c91 Code readability 2022-06-08 08:50:23 -07:00
Kyle AltendorfandGitHub 90516263aa remove chia.util.path.mkdir() (#10842)
* remove chia.util.path.mkdir()

* remove submodule change

* less tmpdir, more tmp_path
2022-06-08 10:24:40 -05:00
Kyle AltendorfandGitHub 61fa0edc02 stop and join watchdog observer (#11706)
This is in accordance with the example.  For inotify at least the `.stop()`
call ends up calling the removed unschedule call anyways.

https://github.com/gorakhargosh/watchdog/tree/72f2eb7203659c11c4f0703c818dcb88fe527e99#example-api-usage
2022-06-08 10:20:08 -05:00
dustinfaceandGitHub 5c5ab111ad streamable: Fix default value assignments for dataclass_from_dict (#11732)
* streamable: Use constructor in `dataclass_from_dict`

This fixes default value assignments after #10561 but also leads to less perfomance due to `__post_init__` being called which at least gets mitigated by #11730.

* tests: Test default values with `from_json_dict`

* Convert to `str`, then compare.
2022-06-03 12:14:57 -05:00
Earle Lowe f14e743cfc Fix up syntax 2022-06-03 09:28:53 -07:00
Earle Lowe f9fe04767e Added utility function and code cleanup 2022-06-03 09:09:18 -07:00
dustinfaceandGitHub 2495e8dfd3 streamable: Introduce streamable.Field (#11745)
* streamable: Cache `dataclasses.Field` objects

* Introduce `streamable.Field`
2022-06-02 13:10:29 -05:00
Mariano SorgenteandGitHub 718b830e61 Optimize get_hash by not double converting (#11668) 2022-06-01 16:10:42 -05:00
Arvid NorbergandGitHub 3c2fc1947d upgrade chia-rs to streamable support (#11503)
* upgrade chia-rs to streamable support

* bump chia_rs dependency
2022-06-01 12:56:11 -05:00
Kyle AltendorfandGitHub 1e7d86e884 remove big_ints list (#11630) 2022-05-31 15:36:49 -05:00
Amine Khaldi 83b49e73a1 Merge branch 'release/1.4.0' into main_checkpoint_1.4.0_46ebed3 2022-05-26 08:51:50 +01:00
arvidn a328fdcabf fix jsonify bool 2022-05-25 09:05:47 +02:00
Mariano SorgenteandGitHub ab1ac66baa Optimize std_hash in coin.py (#11549) 2022-05-24 19:01:15 -07:00
dustinfaceandGitHub 15c51434f3 plotting|util: plotting.cache.DiskCache -> util.misc.VersionedBlob (#11587) 2022-05-24 16:20:38 -07:00
neurosis69andGitHub 569d610238 allow bigger chunks of bind variables per SQL statement (#11596)
* changes for testscenario

* revert change

* implement sqlite version dependent variable

* use parameter at module level, not as instance var

* linefix

* isort, black
2022-05-24 16:18:28 -07:00
dda517ee1a streamable: Cache convert functions from dataclass_from_dict (#10561)
* streamable: Cache convert functions for dict -> dataclass conversion

* tests: Test `dataclass_from_dict` with non-streamable classes

* `Any` -> `object`

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Move comment into `dataclass_from_dict`

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-05-24 13:22:22 -07:00
Arvid NorbergandGitHub 0efd323a12 fix built-in profiler to work for mac (#11590) 2022-05-23 16:10:52 -07:00
Kyle AltendorfandGitHub 71a8ac2e69 Get get_args() and get_origin() from typing_extensions (#11571)
* Get get_args() and get_origin() from typing_extensions

* Update streamable.py

* cleanup
2022-05-19 10:59:21 -07:00
Arvid NorbergandGitHub afae2ad5fe make recurse_jsonify() work directly on types (#11537)
* make recurse_jsonify() work directly on types, circumventing the dataclasses.asdict() step. This enables simpler integration of non dataclasses into the Streamable and JSON protocols

* add benchmark for Streamable.to_json_dict()
2022-05-18 09:15:03 -07:00
Mariano SorgenteandGitHub 31ed32628f Optimize code to not perform useless subgroup checks (#11546)
* Optimize code to not perform useless subgroup checks

* Revert less important optimizations
2022-05-18 09:09:56 -07:00
Mariano SorgenteandGitHub 56e6e68eb3 Uses the new from_bytes_unchecked method in blspy, to improve perfo… (#11463)
* Uses the new `from_bytes_unchecked` method in blspy, to improve performance

* Update test

* Fix merge conflict

* Use from_bytes_unchecked in post_init and from_json

* More uniform code
2022-05-16 09:49:50 -07:00
7c91f470f4 simplify SizedBytes and StructStream (#11429)
* simplify SizedBytes and StructStream

* lint

* super()  !!!

* do not pass parameter up to super().__init__()

* Update chia/util/struct_stream.py

Co-authored-by: Arvid Norberg <arvid@libtorrent.org>

* parse fixed-width int data from class name

* add int512 and uint128 .from_bytes(), test .parse() failures

* test serialization against struct.pack()

* use typing_extensions for final

* override ignore

* stop using struct for StructStream

oh the irony

* fixup .to_bytes() to accept parameters again for where we use that

* bring back signed parameter

* format

* adjust tests for new exception

* eliminate custom coding for uint128 and int512

* tidy

* remove unused StructStream.PACK attribute

* add direct tests for parse_metadata_from_name()

* stricter hinting

* remove no-longer-needed typeshed work-around

* apply strict type checking to all touched files

* remove StructStream override of .to_bytes()

* tidy

* types touchup

* add unused parameter comments

Co-authored-by: Arvid Norberg <arvid@libtorrent.org>
Co-authored-by: wjblanke <wjb98672@gmail.com>
2022-05-14 02:05:27 -07:00
298ea9ea20 Resend transactions (#11167)
* Resend transactions

* Don't recheck transactions more frequently than timeout

* Add wallet resend parameter to config, move timeout code out of tx store, but close to call site

* Add a test for wallet transaction resend

* Add test for wallet retry

* isort new files for precommit and update workflows

* Use correct fixture name

* LGTM - remove unused import

Co-authored-by: Earle Lowe <e.lowe@chia.net>
2022-05-14 02:03:10 -07:00
Kyle AltendorfandGitHub 6788c0a5ef Use context manager for locking clvm files (#11467) 2022-05-10 04:41:57 -07:00
dustinfaceandGitHub 541b99e311 harvester: Introduce recursive_plot_scan (#11468)
* harvester: Introduce `recursive_plot_scan`

* Change log and merge boths lookups

* Drop redundant assignments
2022-05-09 16:43:53 -07:00
d00d045d9c farmer|rpc|tests: Implement paginated harvester plot endpoints (#11365)
* farmer|rpc|tests: Implement paginated harvester plot endpoints

* Simplify filtering

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Let the API handle the exceptions

* Simplify the other filtering too

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Simplify count assertions

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Refactor `is_filter_match` to `plot_matches_filter`

And just convert to `Plot` in tests.

* Move `chia.util.misc.KeyValue` to `chia.rpc.farmer_rpc_api.FilterItem`

* Rename `peer_id` to `node_id` to be match `get_harvesters_{summary}`

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-05-03 09:17:05 -07:00
Arvid NorbergandGitHub 340e26e154 simplify merkle set (#11416)
* _make_middle doesn't need to be a member function

* add some unit tests
2022-05-03 07:49:29 -07:00
dustinfaceandGitHub 984c442b5e util: Implement Paginator class as interface to access a list by pages (#11247)
* util: Implement `Paginator` class as interface to access a list by pages

* Be less restrictive about page sizes and refactor tests

* Make the pages based of 0 instead of 1 and some more test refactoring

* More tests

* Adjust workflows after rebase

* Introduce `Paginator.create`

* `<=` instead of `- 1`
2022-04-29 18:35:47 -07:00
Arvid NorbergandGitHub d3f2ae367d remove unused functions on MerkleSet (#11362) 2022-04-29 08:17:37 -07:00
dustinfaceandGitHub b693aeb407 farmer|rpc: Introduce get_harvesters_summary RPC endpoint (#11245) 2022-04-28 19:36:21 -07:00
Mariano SorgenteandGitHub 7630d220f5 Optimize BLS verification when public key is repeated (#11318)
* Optimize BLS verification when public key is repeated

* Fix accidental commit

* Lint

* Fix isort

* Address comments
2022-04-27 11:38:48 -07:00
dustinfaceandGitHub c0f3d4d231 daemon|util: Don't remove from list while iterating (#11208)
* daemon: Iterate over a copy of `websockets`

Currently when it fails to send the response to one websocket it skips 
the next websocket in list becaue we remove from the list while 
iterating.

* util: Iterate over a copy of `peers_with_peak`

Currently when a peer is closed it skips the check for the next peer in 
the list becaue we remove from the list while iterating.
2022-04-22 10:55:57 -07:00
Kyle AltendorfandGitHub 1088801e73 Remove code related to no-longer-used websockets library (#11123)
* Remove logging tweak for no-longer-used websockets library

Follow up after https://github.com/Chia-Network/chia-blockchain/pull/10611.

* Update pylintrc

* Update test_daemon.py
2022-04-20 11:18:42 -07:00
79cbadf987 streamable: Enable isort + more mypy (#10539)
* isort: Fix `streamable.py` and `test_streamable.py`

* mypy: Drop `streamable.py` and `test_streamable.py` form exclusion

And fix all the mypy issues.

* Fix `pylint`

* Introduce `ParseFunctionType` and `StreamFunctionType`

* Use `object` instead of `Type[Any]` for `is_type_*` functions

* Some `Any` -> `object`

* Use `typing.overload` for `recurse_jsonify`

* Move some comments

* Drop `Union`, use `Literal` properly

* Explicitly ignore the return of `f_type.parse`

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Merge two `recurse_jsonify` overloads

* Typing for the base definition of `recurse_jsonify`

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-04-20 11:09:41 -07:00
dustinfaceandGitHub a48fd43100 streamable: Simplify and force correct usage (#10509)
* streamable: Merge `strictdataclass` into `Streamable` class

* tests: Test not supported streamable types

* streamable: Reorder decorators

* streamable: Simplify streamable decorator and force correct usage/syntax

* streamable: Just move some stuff around in the file

* streamable: Improve syntax error messages

* mypy: Drop `type_checking.py` and `test_type_checking.py` from exclusion

* streamable: Use cached fields instead of `__annotations__`

This is now possible after merging `__post_init__` into `Streamable`

* Introduce `DefinitionError` as `StreamableError`

* `/t` -> `    `
2022-04-08 18:29:32 -07:00
Arvid NorbergandGitHub 21fb6f260e transition to using chia_rs module (#11094) 2022-04-08 09:37:10 -07:00
Arvid NorbergandGitHub fe77c69018 run_generator2 rust call and compact conditions data structure (#8862)
* use run_generator2 rust call and compact spend bundle conditions data structure pervasively.

* address review comments
2022-04-04 11:53:13 -07:00
dustinfaceandGitHub 62de4a883c streamable|pools: Fix Optional parsing in dataclass_from_dict (#10573)
* Test more `Optional` parsing in `dataclass_from_dict`

* Fix optional parsing in `dataclass_from_dict`

* Fix pool wallet / tests
2022-04-04 11:50:59 -07:00
Mariano SorgenteandGitHub 890c7d3754 Fix remaining linting issues (#10962)
* FIx remaining linting issues

* Revert type:ignore

* Revert token_bytes change
2022-04-02 13:22:55 -07:00
Kyle AltendorfandGitHub a691d3c4b2 asyncio.get_event_loop() is deprecated in 3.10, stop using it (mostly) (#10418)
* asyncio.get_event_loop() is deprecated in 3.10, stop using it

https://docs.python.org/3.10/library/asyncio-eventloop.html#asyncio.get_event_loop
> Deprecated since version 3.10: Deprecation warning is emitted if there is no running event loop. In future Python releases, this function will be an alias of get_running_loop().

* black
2022-03-28 13:20:50 -07:00
Kyle AltendorfandGitHub e0a1fc4eaf Switch to integrated lock_and_load_config() context manager (#10698)
* minor lock scope reduction

* use the lock in tests

* Use the passed root_path in configure CLI command

* switch to lock_and_load_config()

* oops

* cleanup

* make _load_config_maybe_locked() private

* black

* Remove future improvement opportunity TODO comment
2022-03-28 12:52:51 -07:00
Adam KellyandGitHub efcf0cf0ad await the db commit in the async version of set_db_version (#10906) 2022-03-28 12:48:14 -07:00
Arvid NorbergandGitHub 1bcf409ee1 single thread executor (#10919)
* add inline executor and an option to run single-threaded

* add option to run test_full_sync in single-thread mode, to include block validation in profiles. Also attempt to speed it up by disabling db_sync
2022-03-28 12:47:46 -07:00
Arvid NorbergandGitHub a100dda37e new DBWrapper supporting concurrent readers (#10166)
* new DBWrapper supporting concurrent readers

* adress review comments

* fixup default database version, when file doesn't exist

* remove unused argument
2022-03-28 11:58:00 -07:00
Arvid NorbergandGitHub ba6eb6af3d when creating a new blockchain database implicitly, make it v2 (#10498)
* when creating a new blockchain database implicitly, make it v2

* fix config deadlock
2022-03-17 09:09:26 -07:00