* Improve and test `dataclass_from_dict` tuple validation
* Improve and test `dataclass_from_dict` list validation
* Introduce `convert_hex_string`
* Improve and test `dataclass_from_dict` byte type validation
* Improve and test `dataclass_from_dict` unhashable validation
* Improve and test `dataclass_from_dict` primitive validation
* Improve `dataclass_from_dict` validation
* `raise TypeError(...) from e`
* Just `raise`, don't `raise e`
* Use `repr` of `item` not `str`
* `type(item) == f_type` -> `isinstance(item, f_type)`
* streamable: Use constructor in `dataclass_from_dict`
This fixes default value assignments after #10561 but also leads to less perfomance due to `__post_init__` being called which at least gets mitigated by #11730.
* tests: Test default values with `from_json_dict`
* Convert to `str`, then compare.
* Uses the new `from_bytes_unchecked` method in blspy, to improve performance
* Update test
* Fix merge conflict
* Use from_bytes_unchecked in post_init and from_json
* More uniform code
* simplify SizedBytes and StructStream
* lint
* super() !!!
* do not pass parameter up to super().__init__()
* Update chia/util/struct_stream.py
Co-authored-by: Arvid Norberg <arvid@libtorrent.org>
* parse fixed-width int data from class name
* add int512 and uint128 .from_bytes(), test .parse() failures
* test serialization against struct.pack()
* use typing_extensions for final
* override ignore
* stop using struct for StructStream
oh the irony
* fixup .to_bytes() to accept parameters again for where we use that
* bring back signed parameter
* format
* adjust tests for new exception
* eliminate custom coding for uint128 and int512
* tidy
* remove unused StructStream.PACK attribute
* add direct tests for parse_metadata_from_name()
* stricter hinting
* remove no-longer-needed typeshed work-around
* apply strict type checking to all touched files
* remove StructStream override of .to_bytes()
* tidy
* types touchup
* add unused parameter comments
Co-authored-by: Arvid Norberg <arvid@libtorrent.org>
Co-authored-by: wjblanke <wjb98672@gmail.com>
* isort: Fix `streamable.py` and `test_streamable.py`
* mypy: Drop `streamable.py` and `test_streamable.py` form exclusion
And fix all the mypy issues.
* Fix `pylint`
* Introduce `ParseFunctionType` and `StreamFunctionType`
* Use `object` instead of `Type[Any]` for `is_type_*` functions
* Some `Any` -> `object`
* Use `typing.overload` for `recurse_jsonify`
* Move some comments
* Drop `Union`, use `Literal` properly
* Explicitly ignore the return of `f_type.parse`
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* Merge two `recurse_jsonify` overloads
* Typing for the base definition of `recurse_jsonify`
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* streamable: Merge `strictdataclass` into `Streamable` class
* tests: Test not supported streamable types
* streamable: Reorder decorators
* streamable: Simplify streamable decorator and force correct usage/syntax
* streamable: Just move some stuff around in the file
* streamable: Improve syntax error messages
* mypy: Drop `type_checking.py` and `test_type_checking.py` from exclusion
* streamable: Use cached fields instead of `__annotations__`
This is now possible after merging `__post_init__` into `Streamable`
* Introduce `DefinitionError` as `StreamableError`
* `/t` -> ` `
* minor lock scope reduction
* use the lock in tests
* Use the passed root_path in configure CLI command
* switch to lock_and_load_config()
* oops
* cleanup
* make _load_config_maybe_locked() private
* black
* Remove future improvement opportunity TODO comment
* Add configuration locking
Extracted from https://github.com/Chia-Network/chia-blockchain/pull/10631
* note that fasteners will likely be replaced by filelock
* Fix test_multiple_writers on macOS
* create_all_ssl() doesn't need to be inside the config access lock
* add warnings about not using async within get_config_lock() get lock contexts
* no need to pre-touch the lock file
* .yaml.lock instead of just .lock
* test_multiple_writers() is sync
* Revert "add warnings about not using async within get_config_lock() get lock contexts"
This reverts commit 681af3835b.
* reduce lock context size in chia_init()
* use an exit stack in load_config()
* avoid config existence precheck
* only lock around the read in load_config()
* do not raise e, just raise
* tidy new imports
* fix queue empty check in test_config.py
* remove commented out code in test_config.py
* remove unused import
Co-authored-by: Jeff Cruikshank <jeff@chia.net>
* Use bt fixture
* rebase
* Use local_hostname
* flake8
* Remove set_shared_instance
* Remove unneeded comments
* Revert unrelated change
* Add back type signature - rebase issue
* Correct type for _configure_legacy_backend
* See what's going on during CI mypy run
* github workflows
* mypy typing
* Remove legacy Keyring create method
* Start daemon first
* Set chia-blockchain-gui to hash in main
* Fix two test files that were not importing self_hostname
* self_hostname fixture
* Convert all class fixtures to top level functions
* Use bt fixture
* rebase
* Use local_hostname
* Fix test_json (inheritance from unittest.TestCase)
* Use correct BlockTools fixture for test_simulation
* Pass bt fixture into cost calculation tests
* flake8
* Add missing parameters to test functions
* Fix from rebase issues
* Remove set_shared_instance
* Update comment
* Remove unneeded comments
* Remove unused code
* Remove unused code, run `multiprocessing.set_start_method("spawn")` at correct time.
* Revert unrelated change
* Set daemon_port. Teardown services in correct order.
BIG thanks to Mariano Sorgente for debugging help.
* Add back type signature - rebase issue
* Apply review fixes from Jeff
* Document why we need a later pytest-asyncio version
* Correct type for _configure_legacy_backend
* See what's going on during CI mypy run
* github workflows
* mypy typing
* Remove legacy Keyring create method
* Start daemon first
* Shutdown daemon coroutine properly
* Remove un-needed daemon_port argument
* Set chia-blockchain-gui to hash in main
* Remove connect_to_daemon_port
* Remove code that set "daemon_port" before calling `setup_daemon`
* Remove self_hostname fixture and extra self_hostname global
* Fix two test files that were not importing self_hostname
* self_hostname fixture
* Remove more unused test code
* Simplify fixture
* Serialize/deserialize peer data alongside existing sqlite implementation (to be removed)
* Simplified AddressManagerStore. No longer uses sqlite and is no longer async.
* Removed aiosqlite usage from AddressManagerStore.
Added PeerStoreResolver class to determine the appropriate location for "peers.dat"
Updated initial-config.yaml to include "peers_file_path" default, replacing "peer_db_path" (similar change for "wallet_peers_path")
* Minor comment changes/additions
* Added migration from sqlite peer db.
Made AddressManagerStore's serialization async as it was right at the edge of blocking for too long.
* Minor tweaks to checking for migration
* Removed AddressManagerSQLiteStore class scaffolding
* makePeerDataSerialization now returns bytes instead of a PeerDataSerialization object
* Async file I/O for write_file_async using aiofiles
Added more tests
* Separate out the synchronous part of move_file
* Renamed write_file to files since we're opening up the capabilities a bit
* Update references to write_file
* Renamed test_write_file to test_files
* Tests covering move_file and move_file_async
* Minor refinements to behavior and tests
* Use aiofiles for reading peers.dat
* Added missing mypy typing info for aiofiles. Also added types-PyYAML to dev_dependencies so that `mypy chia tests` doesn't require running with --install-types
* Add types-aiofiles to the linting workflow
* Directory perms can now be passed into write_file_async.
Added an explicit f.flush() followed by os.fsync() after writing the temp file contents.
* tests: Move `default_replacements` into `generate_replacements`
* tests: Directly use `root_test_dirs`
Instead of passing it into `subdirs`
* tests: Make `build-workflow.py` callable from everywhere
* tests: Adjust the output of `build-workflows.py`
* tests: Add `-f` to `build-workflows.py` to allow failing on CI
* github: Run `build-workflows.py`
* pre-commit: Run `build-workflows.py` as first hook
* tests: Skip DID wallet tests until they are reliable and fixed
* tests: Skip `test_using_legacy_cryptfilekeyring`
* tests|github: Drop `root_test_dirs` and just search in all directories
This removes the requirement to add new test directories to the
`testconfig.py`. This change uncovered two more tests which were not
enabled on CI because their parent dir's were not added to
`root_test_dirs`.
- `tests/weight_proof/test_weight_proof.py`
- `tests/util/test_struct_stream.py`
* tests|github: Don't list test files, use `tests/dir/tests_*.py` instead
* test: Improve workflow update checks by comparing old vs new
Instead of doing a `git diff` which can lead to false positives if there
are other local changes unlrelated to test workflows. This also fixes
the issue that newly added files were not tracked by `git diff` which
means the script / the CI would trigger an alert.
* tests: Adjust `build-workflows.py` to work as expected on windows
* tests: Read/Write in bytes to avoid and detect unexpected `\r\n`
* tests: Skip `test_pool_cmdline.py`
* Prority locking to consensus
* Remove pstats
* Linting
* Do some stuff outside of lock
* Fix startup
* Add log timings
* Try some different locking
* Add limit
* catch excp
* CLVM inside lock
* Try using a semaphore instead
* use events for lock queue
* test
* Add logging for message types
* type
* remove seed
* check new peak waiters
* correct FullNodeAPI self.full_node.new_peak._waiters typo
* correct logging string typos
* only warn about new_peak Waiters if there is at least 1
* remove no-longer-accepted parameter to FullNode.peak_post_processing()
* only warn about respond_transaction Waiters if there is at least 1
* lint
* Change some constants
* Small fix and logging changes
* Put message types outside
* Change some log levels so we can test with info
* More logging
* Increase rate limits but decrease paralelism
* tweaks
* Log dropped tx
* Fix pool rpc test
* Test fixes
* Mempool optimization
* Remove from seen if fails
* Increase queue sizes
* Message types info
* More test and logging
* Small changes to networking just in case
* Decrease logging
* Decrease logging even further
* Decrease logging even further even further
* Decrease logging 3
* Transaction queue
* Don't cancel tasks or close connection
* Cancel tasks on disconnect (for shutdown purposed)
* Fix typo
* Catch cancelled
* Do multiple at a time
* More accurate farmer response time
* More efficiently create tasks
* Increase queue size and priority by fee
* Revert priority
* Don't re-request too many times for dropped TX
* Handle cancelled error so we don't go into a bad state
* Catch cancelled in syncing tasks
* Reduce new_peak_sem to improve performance
* Less bytes conversion
* Missing file, and 2 workers for CLVM
* Validate BLS in a new thread
* tests
* Change semaphore constants
* correct a cancellation triggered exception and assertion
* Fix send_transaction, dont use BaseException, fix tests
* Fix more tests
* only log transaction handler cancellation in debug
* typing in log
* move unfinished validation to diff proc
* it is asyncio.CancelledError
* Add a test for bad signature
* Fix more tests, reduce logging, lint
* One more lint
* blockchain tests, pass bytes directly, single call
* Try to fix rl_wallet failures
* Fix mempool test
* catch everything
* Don't test RL wallet
* Fix more tests and return error code
* Improve error handling in multiprocess
* Add pre-validation time
* Add pre-validation time in logs, and revert pytest.ini changes
* Add log correctly
* Ms.bls cache experiment (#9115)
* Logging for cache
* Less logging
* Return to original plan
* Clean up
* Remove coment
* Remove log
* formalize LockQueue shutdown
* Comments
* Fix blockchain test
* Improve cache
* Remove logs
* Fix sign_coin_spends
* Fix pool wallet
Co-authored-by: Kyle Altendorf <sda@fstab.net>
Co-authored-by: Yostra <straya@chia.net>
* Expose GUI option to save passphrase (where available)
Prevent GUI from showing the option to remove migrated keys
* Backend support for getting/setting the passphrase hint. The hint data is stored in keyring.yaml in the clear.
* CLI support for showing/setting/removing the passphrase hint
* Tests for setting/retrieving passphrase hints
* Squelch LGTM false positive
* Fixed issue when running without passphrase support enabled
* CLI updates based on feedback
* Initialize passphrase_hint earlier.
* Minor restructuring to address linter complaint.
* Fixes cherry-picked from fix_keyring_tests.
* Logging for the failing test_using_legacy_cryptfilekeyring test
* See if reordering the tests makes a difference.
* Revert "See if reordering the tests makes a difference."
This reverts commit b53807880b.
* Log changes to the _configure_legacy_backend patch
* Fixed patching
* Checking if something isn't cleaned up properly
* Revert "Checking if something isn't cleaned up properly"
This reverts commit ce995ba773.
* Revert "Fixed patching"
This reverts commit 66a70a16ff.
* Revert "Log changes to the _configure_legacy_backend patch"
This reverts commit 26791a40b7.
* Revert "Logging for the failing test_using_legacy_cryptfilekeyring test"
This reverts commit 4fd4873565.
* Move test_keyring_wrapper.py into a standalone test
* Small refactorings/changes per feedback
* Fix to allow migration of keys on macOS
* Fixed file_keyring synchronization test failures on macOS.
Fixed sporadic test failures on macOS when fsevents are delivered for the keyring after deletion.
TempKeyring-based tests now patch supports_os_passphrase_storage() to return False.
* Fixed LGTM issues
* Re-added the cleaning up temp keychain statement. This is being removed in a separate PR.
* Windows keyring passphrase support
* Test tweaks for Windows (mostly to support my slow VM)
* Linter fixes
* Restored prompt_for_passphrase() function which was accidentally removed during a rebase.
* Added CHIA_PASSPHRASE_SUPPORT env variable to allow more testing
* Added 'service' as a Keychain ctor param. Removed 'testing'
* Detect existing keys in the Mac Keychain
* Fix to allow migration of keys on macOS
* Added dump_keyring.py tool to show decrypted contents of keyring.yaml
* Prompt to save passphrase to macOS keychain
* Master passphrase retrieval/removal from the macOS Keychain.
Fixed typos.
* Warn if errSecInteractionNotAllowed is detected when accessing the macOS Keychain
* Fixed file_keyring synchronization test failures on macOS.
Fixed sporadic test failures on macOS when fsevents are delivered for the keyring after deletion.
TempKeyring-based tests now patch supports_os_passphrase_storage() to return False.
* TempKeyring mocks-out legacy_keyring setup to allow tests to succeed on macOS (which could find existing keys in the Keychain)
* Fixed pylint error
* Use with_name instead of with_stem (which is new to Python 3.9)
* Fixed keychain tests that started prompting for the keyring passphrase.
* Fixed LGTM issues
* Re-added the cleaning up temp keychain statement. This is being removed in a separate PR.
* Linter fixes
* Fixed keyring assignment on macOS when passphrase support is disabled.
* Include 'can_save_passphrase' flag in keyring_status response
* More linter fixes
* Fixed determination of the user_passphrase_is_set flag. This was returning true for a newly created keyring without any keys (or passphrase set)
* Removed the tidy_passphrase function per feedback
* Added some comments based on feedback
* Update chia/cmds/passphrase_funcs.py
Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com>
Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com>
* refactor: Introduce `parse_uint32` in `streamable.py`
* refactor: Introduce `write_uint32` in `streamable.py`
* tests: Add `test_uint32` in `test_streamable.py`
Adds tests for `write_uint32` and `parse_uint32`
* Moved keyring handling into a KeyringWrapper class
* Update click to 8.0.x for prompt_required support
* Renamed KeyringWrapper to _KeyringWrapper
* Expose password management options on Linux
* CLI support for setting/removing a password
* Global option for specifying the master password
* Cache the password instead of setting on the context
* Password bootstrapping during chia init
* Tidying up _KeyringWraper's interface
* Initial pass migrating the legacy keyring contents
* Encryption/decryption of keyring.yaml contents
* FileKeyring backend encrypts with ChaCha20Poly1305
* Tightened up keyring migration and initialization
* Fixed issues identified by linters
* Remove root_path from Keychain
* Prevent double-migration if setting master passwd
* KeyringWrapper tests are mostly complete
* FileKeyring will now honor the service param
* Tests for get/set/delete password
* Formatting/commenting updates
* Writer lock support with tests - WIP
* keyring.yaml is now watched for modifications
* Reader/Writer lock for get/delete password
* Fixed linter issues
* Reader lock tests
* Formatting update
* Hook up CHIA_ROOT support for KeychainWrapper
* Quick fix to address test failures
* Fixed failures when existing legacy keyring exists
* Fixed test failures caused by reusing the same temp dir
* keyring.yaml now lives in ~/.chia_keys by default. Can be overridden with CHIA_KEYS_ROOT or --keys-root-path
* Fixed migration failure when setting a password (not using the default)
* KeyringWrapper now uses supports_keyring_password to determine if a FileKeyring should be used. Patched tests to work regardless of whether supports_keyring_password return False
* The daemon now takes a --have-gui option that will prevent calling check_keys() during startup. If the keyring is locked, we want the GUI to prompt for the password.
* Added is_keyring_locked RPC call
* Added 'unlock_keyring' RPC command
* Added KeychainProxy and KeychainServer to handle RPC messages related to keyring operations. WalletNode no longer directly accesses the Keychain class.
* Turn on macOS support for testing keyring passwords
* Fixed get_key_for_fingerprint to use the ocal keychain if the platform doesn't need to remotely access the daemon's keychain.
Fixed key reconstruction when sent over RPC.
* Farmer now accesses the keychain over RPC
* Fixes for linter issues and some restructuring to support tests that use setup_nodes.py
* Couple of fixes to unblock the GUI from launching when a keyring password is set
* Added a keychain RPC call for add_private_key()
* Added remaining keychain proxy RPC calls for delete_key_by_fingerprint and delete_all_keys
* Check for None when inspecting request arguments
* Run check_keys after unlocking the keyring when the daemon is launched via GUI
* Added check_keys RPC method.
Fixed deserialization of key entropy in get_all_private_keys. This was preventing the GUI from being able to show key details.
* Added get_first_private_key to keychain_server/proxy.
create_plots now uses the keychain proxy when launched from the daemon.
* Added a comment about KeychainProxy in chia plots check
* Workaround import conflict when importing from 'tests.*' due to fasteners name conflict
* Simulator now uses KeychainProxy if launched by the daemon.
KeychainServer/Proxy now takes keychain user/testing params for testing scenarios.
* Added "set_keyring_passphrase" RPC message
* Reworking KeychainProxy usage to handle local keychain tests and RPC keychain tests.
* Replace my prior usage of asyncio.run() with asyncio.get_event_loop().run_until_complete()
* Silencing file_keyring logging for the moment.
* Updated tests to use test keychains and appropriate BlockTools construction
BlockTools should now be created with create_block_tools(_async) to handle async scenarios.
Updated block_tools to be async compatible
Updated fasteners to fix installation of top-level 'tests' in site-packages
* Added 'remove_keyring_passphrase' RPC message to the daemon
Minor tweak to TempKeyring to default to some test params
* Fixed linter issues
* Remove flake8 ignore statement now that the fasteners module has been updated
* Some initial renaming changes: password -> passphrase
* Fixed wallet RPC issue where get_key_for_fingerprint wasn't awaited-upon.
Fixed legacy keyring initialization (for migration scenarios)
* Fixed improperly merged file
* Fixed linter issues.
More renaming.
* Updated spots that were still using an incorrect keychain call
* Renamed use_password_cache, obtain_current_password
* Renamed supports_keyring_password
* Renamed has_master_password
* Renamed has_cached_password, get_cached_master_password
* Linter fixes
* Renamed master_password_is_valid
* Renamed set_cached_master_password
* Renamed set_master_password
* Renamed remove_master_password
* Renamed has_cached_master_password
* Renaming in file_keyring and keyring_wrapper
Updated default keyring payload used for tests
* Renamed get_password
Other renaming updates
* Renamed set_password
Other renaming updates
* Renamed remaining password occurrences (where appropriate)
* password -> passphrase
* Added tests for setting an emoji and Japanese master passphrase
* Attempt to notify the daemon when a keyring passphrase is set/updated/removed
* Missed one password -> passphrase replacement.
* Fixed some file synchronization issues found when running tests on macOS
* Adjusted timeout values for test_writer_lock_reacquisition_failure for macOS.
* Removed logging statements previously added for debugging
* Prompt for keyring passphrase up-front when launching a service.
Changed --have-gui flag to --wait-for-unlock
* Updated set_keyring_passphrase RPC message to fix optional current_passphrase param when the keyring is using the default passphrase.
* Minor test cleanup to deduplicate some code.
* Fixed regression when setting a new master passphrase
* Minor refactoring and docs/commenting updates
* Renaming password -> passphrase went too far. Keyring backends use password terminology for compatibility with third party backends.
* Disabling macOS support (previously added for testing only)
* Disabling passphrase support in preparation for sending out the PR
* Fixed improper merge (vscode didn't save changes during rebase)
* Update chia/cmds/init_funcs.py
Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com>
* skip_check_keys -> should_check_keys
* Shuffling some imports around to break cycles reported by LGTM
* Handle unlocking the daemon if it's already launched and waiting for unlock.
* Replaced uses_keychain_proxy decorator in farmer.py. Fixed async usage of get_reward_targets.
Linter/reformatting fixes
* Replaced uses_keychain_proxy decorator with a clearer method.
* Cleanup the temp keyring dir using shutil.rmtree()
* Restored self._root_path (had been changed to self.root_path)
* Minor cleanup
* ensure_keychain_proxy() now throws if connect_to_keychain_and_validate() fails
* Plot key resolution now yields a PlotKeys object which can be passed into create_plots.
De-indented test_invalid_icc_sub_slot_vdf to keep git blame tidy.
* Added 'keyring_status' daemon RPC message to support the GUI
* Minor changes relating to PR feedback
* Addressed more PR feedback (mostly type annotations)
* Commented-out macOS file keyring usage. This can be re-enabled for testing purposes.
* Addressed test failures that require multiple keyrings in the same process. Each TempKeyring will now set a custom KeyringWrapper instance.
* Fixed logic for communicating user_passphrase_is_set in the keyring_status RPC response.
* Updated type annotations and method signature for set_passphrase to expect a string instead of bytes.
* Fixed Wallet RPC tests
* Fixed full_node_store tests. BlockTools should be created using the create_block_tools(_async) function(s)
* Fixed test failures in test_pool_rpc
* Fixed test_daemon. After BlockTools.setup_plots is run, the config file needs to be re-read to refresh stale plot_directories.
* Suppressing LGTM false positives regarding passphrase leakage in CLI error output. Seems that LGTM sees MIN_PASSPHRASE_LEN as sensitive data.
* Second attempt at suppressing LGTM false positives
* Third attempt at addressing LGTM false positives
* Removed test_keyring_wrapper param from Keychain ctor. Test setup now sets the keyring_wrapper property directly.
* Reformatting
* More targeted update of the test config to refresh just the "plot_directories" value
* More LGTM suppressions
Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com>
Co-authored-by: wjblanke <wjb98672@gmail.com>
* Add an explicit error message when mnemonic words are not in the dictionary; should help users self-service issues like #3425 faster.
* fix lint
* fix lint x2
* Avoid importing `test_constants` as it takes a long time.
* Factor out `parse_*` functions.
* First crack at refactoring `Streamable.parse`.
* Don't add `_parse_functions` attribute to `Streamable`.
This no longer requires an extra `_parse_functions` attribute on a
`Streamable`, as it may be confusing serializers or other functions
that use `__annotations__`.
* Fix lint problems with `black`.
* Fix `parse_tuple`.
* Defer some parsing failures to parse time rather than class-creation time.
* Tidy up & remove some obsolete stuff.
* Decorate `RequestBlocks` as `streamable`.
* Fix wrong uses of Streamable class
Revert an earlier commit and error out on class creation in case a
Streamable subclass is instantiated incorrectly, e.g. containing a
non-serializable member.
Fix cases where Streamable parent class was forgotten.
* Fix wrong types when creating DerivationRecord and WalletCoinRecord
* additional unit tests for streamable parsers
* add type annotations (#3222)
Co-authored-by: Rostislav <rostislav@users.noreply.github.com>
Co-authored-by: arvidn <arvid@libtorrent.org>
* Improve performance of streamable
* __annotations__ instead of get_type_hints
* Fix overly strict type checker
* Lints and fix test
* Lint
* Adds support for empty streamables
* Use different strategy to fix lint and support empty Streamables
* Test for empty streamable
* Changelog
dustinfaceandGitHub
Kyle AltendorfandGitHub
Amine Khaldi
arvidn
Mariano SorgenteandGitHub
Adam KellyandGitHub
JeffandGitHub
OliverandGitHub
elliottbackandGitHub
