Commit Graph
55 Commits
Author SHA1 Message Date
8b2f7ec24c Add Windows CI (#10681)
* Add Windows CI

* rebuild workflows

* rebuild workflows

* try different activation

* catch up windows workflow template

* debug dir

* hard code dev extra on windows

* it is pytest

* .exe

* -m pytest for windows

* --module pytest

* Revert "debug dir"

This reverts commit 74b5bbfd92.

* debug pkg_resources import on windows

* debug tests.conftest import on windows

* rebuild workflows

* debug

* debug oops

* debug more

* debug more

* Update pytest.ini

* Update pytest.ini

* fix it

* Revert "debug more"

This reverts commit 367ed61dfa.

* Revert "debug more"

This reverts commit 772b6707c6.

* Revert "debug oops"

This reverts commit 5065c1e172.

* Revert "debug"

This reverts commit ecf721e453.

* Revert "debug tests.conftest import on windows"

This reverts commit ea71a51625.

* Revert "debug pkg_resources import on windows"

This reverts commit eb6afb1dfb.

* more warning ignore

* they're regexes...

* --numprocesses 0 for windows ci

* specify each test file individually since pytest on windows does not seem to handle the wildcards

* relative...

* select most recent supported python on windows, support INSTALL_PYTHON_VERSION

* sort test files when generating workflows

* report full python version on windows, actually use the selected version for the env

* oops

* debug dir

* skip timelord-requiring jobs on windows

* oops

* handle wildcard expansion for coverage on windows

* rebuild workflows

* download blocks and plots on windows

* correct and and or in yaml

* -d for Install.ps1

* git config --global core.autocrlf false

* add missing windows plot_sync workflow

* no check resource usage on windows

* delete outdated windows workflows that need the timelord

* catch up windows workflow template

* try again without xdist

* rebuild workflows

* Revert "try again without xdist"

This reverts commit 4bedf5f047.

* maybe now no xdist on windows

* maybe now no xdist on windows

* rebuild workflows

* debug chiapos==1.0.9

* Revert "debug chiapos==1.0.9"

This reverts commit 4b374e128a.

* debug blspy==1.0.9

* catch up windows workflow template

* rebuild workflows

* Various windows CI fixes

* Windows CI fixes and workflow updates

* ignore another warning for the keyring

* pre-commit fixup

* rebuild workflows

* catchup windows test workflows with long_lived/** and release/** setup

* Update test_full_sync.py

* Try without parallelization

* Some other tweaks for windows

* some more timeout tweaks

* more timeout tweaks for windows

* rebuild workflows

* Adjusting more timeouts

* Skip ipv6 test on windows

* More timeout tweaks

* More timeout adjustments

* More time out adjustments

* more windows tweaks

* Try running some things with -n 0

* Fix up windows test template and re-gen workflows

* Try this again

* Try all windows tests with -n 0

* misc update

* Undo some timeout changes

* Adjust timeout

* Increase timeout

* Coverage combine should combine the files in the current directory

* Trying retry test on windows

* timeout adjustment

* use tmp_path

* Few minor updates based on review

* lint fix

* Update tests/build-workflows.py

* Update tests/core/server/test_dos.py

* Try windows with pytest xdist n=2

* Check windows with xdist n=3

* Trying n=2 again

* Adjust timeout

* Update testconfig.py

* Update build-workflows.py

* Update testconfig.py

* Update .pre-commit-config.yaml

* Update testconfig.py

* Update build-workflows.py

* Update build-test-macos

* Update build-test-ubuntu

* Update build-test-windows

* skip

* Apply suggestions from code review

* tidy

* Update test_pool_config.py

Co-authored-by: Earle Lowe <e.lowe@chia.net>
Co-authored-by: Earle Lowe <30607889+emlowe@users.noreply.github.com>
2022-05-31 14:04:45 -05:00
Mariano SorgenteandGitHub 817baa3096 Bind port 0 to fix race condition when grabbing available ports (#11578)
* port 0 to fix flakiness

* Try fixing setup_full_system

* Try fixing setup_full_system, and lint

* More attempts to fix

* No more calls to get random ports in setup_nodes

* Revert accidental changes

* Timelord extra arg

* Try with port 0

* Fix daemon test, and lint

* Try without 0.0.0.0

* Back to 0.0.0.0

* Try a few timelord changes to get test running

* Increase timeout again

* Use the correct interface to get the port

* INFO logging to debug issue

* Revert "INFO logging to debug issue"

This reverts commit 7c379e5cca.

* Fix advertised port log

* Add extra log

* Logging back

* Rollback the timelord changes

* Try port 0 timelord

* Revert "Try port 0 timelord"

This reverts commit 4997faf3b2.

* Try full green, change ordering

* Remove unused var

* speed up simulation and cleanup

* Now try without the port config

* Fix a flaky call to get_event_loop

* Try getting the port dynamically

* No dynamic port

* Try changing the ordering

* Try adding a sleep

* Back to what works

* Timelord before vdf clients

* Dynamic port for 1st timelord

* Revert "Dynamic port for 1st timelord"

This reverts commit 0f322a15b7.

* Revert "Timelord before vdf clients"

This reverts commit 3286c34696.

* Revert "Back to what works"

This reverts commit 30380dffb7.

* Revert "Try adding a sleep"

This reverts commit 9212b665a6.

* Revert "Try changing the ordering"

This reverts commit a62597d70d.

* Revert "No dynamic port"

This reverts commit 5d2e15749b.

* Revert "Try getting the port dynamically"

This reverts commit ef9cd75679.

* Revert "Fix a flaky call to get_event_loop"

This reverts commit 01a000fdfb.

* Try one to 0

* Just not 0

* Don't get port dynamically

* Cleanup a bit

* Fix

* Some cleanup work

* Some cleanup work

* Fix daemon test

* Cleanup

* Remove arguments
2022-05-23 08:13:49 -07:00
Mariano SorgenteandGitHub 766e1c21f5 Fix several flaky tests (#11576)
* Increase some of the timeouts to reduce flakiness

* Experiment with CI for flakiness

* Force more runs

* Add argument name

* Fix flaky mempool test

* Attempt to fix another flaky test

* Fix wallet retry test

* Missing arg

* Lint

* Increase benchmark test time

* Debug

* Debug

* Simplify retry test

* Lint

* More lint

* Return false instead of assert

* No need to check disconnect twice (since we don't break, this was flaky)

* Remove useless changes

* Accidental changes revert

* Accidental changes revert 2

* Accidental changes revert 3

* Revert fixture
2022-05-23 07:54:38 -07:00
ecada58d36 Coin Selection Refactor With CAT Coin Selection Refactor (#9975)
* add exact match and best exact match algorithms

* optimize algorithm further

this might be good.

* lint

* fix bad logic

* add final algorithms

* delete lint

* oops

* Update coin_selection.py

* simplify and fix knapsack algoritm

* simplify code and correct logic

* make it way better.

* clarify comments and check for edge cases.

* add comments and stuff

* improve coin selection

addressed comments
Thanks!

* add coin_selection rpc tests.

* clean up and add new unit tests

* undo test changes

* add extra test cases

* move coin_selection to its own function and switch to it for cat and main wallet.

* add cat tests

* lint

* make function align with standards

also removed test

* make test better

* add proper types

* Improve code clarity

* wallet: fix coin selection bugs

* wallet: add an assert just in case

* tests: add some sleeps to reduce flakiness

* Isort

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* fix bad merge

* lint

* fix tests

* address aforementioned changes.

* remove wallet test

* isort

* more tests and fixes

* lint

* rename to amount for coin selection rpc

* fix incase we have no smaller coins

* fix tests + lint

* re add asserts

* oops

missed me.

* lint

* fix test

* Squashed commit of the following:

commit 34a2235de5
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Apr 13 10:09:42 2022 -0400

    clarify comment

commit adbf7f4f94
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Tue Apr 12 20:27:05 2022 -0400

    linty lint

commit 5ebc1ac9fd
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Tue Apr 12 20:17:19 2022 -0400

    add failure test and final changes

commit 7e5a21b4c2
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Tue Apr 12 19:35:18 2022 -0400

    add descriptions and slim down code

commit 31c95b916d
Merge: d7b91295b d9b0ef5f3
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Mon Apr 11 10:12:05 2022 -0400

    Merge branch 'jack-cat-coinselection' into jn_coinselection_dust

commit d7b91295b5
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Sun Apr 10 20:31:09 2022 -0400

    lint

commit 30dc7c0ab4
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Sun Apr 10 20:25:52 2022 -0400

    fix tests

commit 6c8c2e4874
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Thu Mar 31 15:06:00 2022 -0400

    remove duplicate code.

commit 9f79b6f304
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Thu Mar 31 15:01:10 2022 -0400

    address more concerns

commit 67c1b3929f
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Thu Mar 31 12:59:05 2022 -0400

    fix logic error

commit 2d19a53245
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Thu Mar 31 11:47:52 2022 -0400

    simplify and de duplicate code

commit 6ab1cc79bb
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Mar 30 21:34:50 2022 -0400

    add function and select individual coin

commit 582c17aa8d
Merge: ce2165942 618fbaeba
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Mar 30 21:14:37 2022 -0400

    Merge branch 'jack-cat-coinselection' into jn_coinselection_dust

commit ce21659429
Merge: 16aabb3fd 6daba28db
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Mar 30 20:53:21 2022 -0400

    Merge branch 'jack-cat-coinselection' into jn_coinselection_dust

commit 16aabb3fd5
Merge: 0b9fc2845 2286fe426
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Mar 30 20:49:02 2022 -0400

    Merge branch 'jack-cat-coinselection' into jn_coinselection_dust

commit 0b9fc28455
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Mar 30 20:38:12 2022 -0400

    lint

commit 62e74c72f4
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Mar 30 20:34:22 2022 -0400

    fix logic and tests

commit e738f44320
Author: Jack Nelson <jack@jacknelson.xyz>
Date:   Wed Mar 30 18:52:05 2022 -0400

    deal with dust and add tests

* make sure that we do not use any dust

* minor change

* address concerns

* adjust comments

* adjust comment

Co-authored-by: Mariano Sorgente <sorgente711@gmail.com>
Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-05-16 10:42:46 -07:00
298ea9ea20 Resend transactions (#11167)
* Resend transactions

* Don't recheck transactions more frequently than timeout

* Add wallet resend parameter to config, move timeout code out of tx store, but close to call site

* Add a test for wallet transaction resend

* Add test for wallet retry

* isort new files for precommit and update workflows

* Use correct fixture name

* LGTM - remove unused import

Co-authored-by: Earle Lowe <e.lowe@chia.net>
2022-05-14 02:03:10 -07:00
9ff3fc993f Ms.plot load perf2 (#10978)
* 2.7 seconds -> 0.45 seconds

* Merge

* Work on create_plots refactor

* Try to fix tests

* Try to fix tests

* Use new functions

* Fix block_tools by adding dir

* Extra argument

* Try to fix cyclic import

* isort

* Drop warning

* Some cleanups around `exclude_final_dir` and directory adding

* Cleanup `min_mainnet_k_size` checks

* Drop unrelated changes

* Fixes after rebase

* Fix cyclic import

* Update tests/block_tools.py

Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>

* Update tests/block_tools.py

Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>

Co-authored-by: xdustinface <xdustinfacex@gmail.com>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
2022-04-11 09:16:01 -07:00
Adam KellyandGitHub 7cd23b0077 Fix the case of claiming a large number of coins (#11038)
* Fix the case of claiming a large number of coins with a fee from a pool wallet

* Revert change to unrelated test

* Set PoolWallet.DEFAULT_MAX_CLAIM_SPENDS to 300

* A few review improvements
2022-04-08 09:57:08 -07:00
Amine KhaldiandGitHub a2fa8dda01 Prepare test blocks and plots only for tests that need them. This saves us a couple more hours of CI running time. (#10975) 2022-04-05 10:34:57 -07:00
dustinfaceandGitHub 62de4a883c streamable|pools: Fix Optional parsing in dataclass_from_dict (#10573)
* Test more `Optional` parsing in `dataclass_from_dict`

* Fix optional parsing in `dataclass_from_dict`

* Fix pool wallet / tests
2022-04-04 11:50:59 -07:00
Mariano SorgenteandGitHub 890c7d3754 Fix remaining linting issues (#10962)
* FIx remaining linting issues

* Revert type:ignore

* Revert token_bytes change
2022-04-02 13:22:55 -07:00
Kyle AltendorfandGitHub e5ab4cd842 back to a single option for workflow parallel config (#10979) 2022-03-31 23:01:20 -07:00
Mariano SorgenteandGitHub 02bf8489d2 Ms.parallel pool t (#10966)
* Try parallel pool tests

* Also change workflow files

* Run less combinations

* Todo for bad test

* Try lower n
2022-03-31 08:23:34 -07:00
Adam KellyandGitHub c5426068aa Consolidate test fixtures (#10778)
* Rename confusing fixtures, especially ones with the same name but different implementation

* revert premature fixture rename: two_wallet_nodes_start_height_1

* Consolidate test fixtures
2022-03-29 13:25:30 -07:00
Amine KhaldiandGitHub 94a4cb2e99 Contextualize some store test db names. (#10942) 2022-03-29 10:15:02 -07:00
Kyle AltendorfandGitHub e0a1fc4eaf Switch to integrated lock_and_load_config() context manager (#10698)
* minor lock scope reduction

* use the lock in tests

* Use the passed root_path in configure CLI command

* switch to lock_and_load_config()

* oops

* cleanup

* make _load_config_maybe_locked() private

* black

* Remove future improvement opportunity TODO comment
2022-03-28 12:52:51 -07:00
Kyle AltendorfandGitHub 48cd97cf47 minor followup to config locking (#10696)
* minor lock scope reduction

* use the lock in tests
2022-03-17 09:03:58 -07:00
dustinfaceandGitHub 66c7427622 wallet: Optional wallet type parameter for get_wallets and wallet show (#10739)
* wallet: Add optional `type` parameter to `get_wallets` and `wallet show`

* tests: Use the `type` parameter for `get_wallets` in pool rpc tests

* cmds: Ask for the name of the wallet type in CLI
2022-03-17 08:42:48 -07:00
Kyle AltendorfandGitHub c63324abe9 remove some event_loop() fixtures (#10420)
* remove event_loop() fixtures

* flake8

* flake8

* remove sys.exit() from daemon shutdown

* bump full node test timeout.  a lot...  to see.

* fixup some tests

* back to module scope event loop fixture for test_full_node.py

* Update test_full_node.py

* Iterator...

* for the whole directory

* some fixtures back to module scope for reduced runtime

* back to 40 minute workflow timeouts

* these are being addressed separately
2022-03-14 06:08:35 -07:00
Adam KellyandGitHub 70639be906 Ak.convert fixtures (#10612)
* Use bt fixture

* rebase

* Use local_hostname

* flake8

* Remove set_shared_instance

* Remove unneeded comments

* Revert unrelated change

* Add back type signature - rebase issue

* Correct type for _configure_legacy_backend

* See what's going on during CI mypy run

* github workflows

* mypy typing

* Remove legacy Keyring create method

* Start daemon first

* Set chia-blockchain-gui to hash in main

* Fix two test files that were not importing self_hostname

* self_hostname fixture

* Convert all class fixtures to top level functions
2022-03-11 16:26:54 -08:00
Mariano SorgenteandGitHub fe964a48bd Fix invalid DB commit (#10594)
* Fix invalid DB commit

* More fixes

* Add raise

* Fix test
2022-03-10 11:11:04 -08:00
dustinfaceandGitHub 7a8cae2936 pools: Fix plotnft claim command's output (#10609)
If you currently claim rewards `claim_cmd` fails to print the txhash 
with the lookup hint in `submit_tx_with_confirmation`

```
Error performing operation on Plot NFT -f 172057028 wallet id: 12: 
'dict' object has no attribute 'name'
```

Because `submit_tx_with_confirmation` expects a `TransactionRecord` as 
result from its callable parameter `func` but `pw_absorb_rewards` 
returns a dict which includes the `TransactionRecord` as value for the 
key `transaction`. This PR makes sure all other methods used as `func` 
callable have the same return behaviour as `pw_absorb_rewards`. We could 
have adjusted it the other way around (only return `TransactionRecord` 
in `pw_absorb_rewards`) but then we would drop information in the RPC 
client.  


With this PR you get:

```
Do chia wallet get_transaction -f 172057028 -tx 
0x34f74a1ffd9da9a493b78463e635996fd03d4f805ade583acb9764df73355f9c to 
get status
```
2022-03-10 11:10:03 -08:00
Adam KellyandGitHub ff324095cc Remove block tools and keychain globals (#10358)
* Use bt fixture

* rebase

* Use local_hostname

* Fix test_json (inheritance from unittest.TestCase)

* Use correct BlockTools fixture for test_simulation

* Pass bt fixture into cost calculation tests

* flake8

* Add missing parameters to test functions

* Fix from rebase issues

* Remove set_shared_instance

* Update comment

* Remove unneeded comments

* Remove unused code

* Remove unused code, run `multiprocessing.set_start_method("spawn")` at correct time.

* Revert unrelated change

* Set daemon_port. Teardown services in correct order.
    BIG thanks to Mariano Sorgente for debugging help.

* Add back type signature - rebase issue

* Apply review fixes from Jeff

* Document why we need a later pytest-asyncio version

* Correct type for _configure_legacy_backend

* See what's going on during CI mypy run

* github workflows

* mypy typing

* Remove legacy Keyring create method

* Start daemon first

* Shutdown daemon coroutine properly

* Remove un-needed daemon_port argument

* Set chia-blockchain-gui to hash in main

* Remove connect_to_daemon_port

* Remove code that set "daemon_port" before calling `setup_daemon`

* Remove self_hostname fixture and extra self_hostname global

* Fix two test files that were not importing self_hostname

* self_hostname fixture

* Remove more unused test code

* Simplify  fixture
2022-03-10 11:06:49 -08:00
cf54aae2b7 Wallet consistancy (#10532)
* use db transaction, -1 in synced up to height, delete unused funcitons

* use transaction info in key-val-store/pool-store

* cat stores

* db lock

* remove unused lock, set synced not always in transaction

* fix store tests

Co-authored-by: wjblanke <wjb98672@gmail.com>
2022-03-04 09:48:36 -08:00
JeffandGitHub 51dffd8501 Preserve existing pool payout_instructions when creating a PoolWallet (#10507)
* Preserve existing pool payout_instructions when creating a PoolWallet

* Updated the logged string when payout_instructions needs to be generated.

* Tests for update_pool_config

* isort

* Logging change, linter fixes, and more comments.
2022-03-02 10:25:13 -08:00
Arvid NorbergandGitHub bf4a632e6b simplify self_hostname. It doesn't need to depend on initial-config.yaml -> create_block_tools() -> global variable bt (#10371) 2022-02-25 08:09:46 -08:00
Arvid NorbergandGitHub d80293e390 more test fixes (#10395)
* WIP: fix more hard coded listen sockets in tests

* correct comment

* remove seemingly redundant event_loop fixtures (that sometimes seem to cause problems)

* when a test fails, the error is the most important thing to print

* fix mempool tests to me truly independent, by giving each test case a fresh block, and fresh reward coins to use for the test
2022-02-23 07:51:18 -08:00
Arvid NorbergandGitHub 97ef7b275a use free listen ports when running tests (#10307) 2022-02-22 16:22:11 -08:00
Matt HauffandGitHub e61c0f477b Accommodate multiple coin fees (#10251) 2022-02-17 11:20:19 -08:00
Kyle AltendorfandGitHub 5d2770cbdd extend pools test workflow timeout to 60 minutes (#10271) 2022-02-16 19:10:37 -08:00
Arvid NorbergandGitHub ce533879e2 run pytest in strict asyncio mode. add missing @pytest.mark.asyncio (#10230) 2022-02-16 11:01:04 -05:00
41717337e6 This patch enables fees for the plotnft commands - create, claim, joi… (#9968)
* This patch enables fees for the plotnft commands - create, claim, join, and leave.

It also corrects a mismatch in the wallet code that appeared to expect Announcement objects, but actually required bytes.

* Update to using Announcement class. Publish both claim tx and fee tx for correct wallet accounting.

* Update keysearch for new wallet

* Update test for fee, and update wallet_id invariant

* Rename variable tracking coin in absorb SpendBundle

* Update RPC failure cmdline diagnostic

* Remove fee parameter in sign method. Add publish transaction method. Add  entry in RPC API replies.

* Fix fee tx in absorb. Update absorb tests to test for fees

* conflict

* lint

Co-authored-by: William Blanke <wjb98672@gmail.com>
2022-02-14 20:07:06 -08:00
Mariano SorgenteandGitHub 7fa1861def Ms.wallet refactor (#10146)
* wallet fixes

* Don't show false positive synched

* Code cleanup and lint

* Fixes

* Revert issue

* Fix reorg issue

* WIP wallet node

* More wallet refactor

* More wallet changes

* More

* Fix subscription bugs

* Fix some tests

* Fix pool tests

* More tweaks

* Lint and small issues

* call update_ui at the correct points

* Small changes

* new peak queue

* Fix peer height issue

* Rollback more for safety, and tweak logging

* Small WSM style fixes

* Change fork point in long_sync

* More fixes with real world testing

* Fix reversed filter

* Fix function name

* Fix coin store bug properly

* Raise CancelledError

* Fix rollback issue

* Lint

* Small fix

* Fix CAT issue

* Fix test trades race condition

* Fix test trades race condition

* Try to reduce flakiness

* Test coin store and fix additional method

* Improve flakiness
2022-02-14 11:28:36 -08:00
Mariano SorgenteandGitHub 3b03872146 Ms.flaky2 (#10080)
* tests: add missing await to fix flaky test

* tests: More flaky pool tests

* WIP

* wallet: apply multiple states per block for pool wallet

* lint
2022-02-02 16:58:15 -05:00
dustinfaceandGitHub c893c44ab6 tests: Make sure pool plots getting removed from BlockTools at the end (#10052)
Prior to this PR the pool plots created by the pool tests were not 
removed from the global `BlockTools.expected_plots` if the test failed 
which happens here and there because they are flaky. Since the global 
`BlockTools` obviously outlives the parameterization of a test (i.e. in 
`test_absorb_pooling`) there were many CI runs stucking at:

```
2022-01-30T23:25:54.0633324Z 23:25:54 chia.plotting.manager: ERROR 
_refresh_callback raised:  with the traceback: Traceback (most recent 
call last):
2022-01-30T23:25:54.0634415Z 2022-01-30T23:25:54.063 full_node 
chia.plotting.manager   : ERROR    _refresh_callback raised:  
with the traceback: Traceback (most recent call last):
2022-01-30T23:25:54.0635521Z   File 
"/home/runner/work/chia-blockchain/chia-blockchain/chia/plotting/manager.py", 
line 281, in _refresh_task
2022-01-30T23:25:54.0636160Z     
self._refresh_callback(PlotRefreshEvents.batch_processed, batch_result)
2022-01-30T23:25:54.0636916Z   File 
"/home/runner/work/chia-blockchain/chia-blockchain/tests/block_tools.py", 
line 178, in test_callback
2022-01-30T23:25:54.0682983Z     assert update_result.remaining == 
len(self.expected_plots) - self.total_result.processed
2022-01-30T23:25:54.0683833Z   File 
"/home/runner/work/chia-blockchain/chia-blockchain/chia/plotting/manager.py", 
line 281, in _refresh_task
2022-01-30T23:25:54.0684293Z AssertionError
```

And getting canceled after a long time like here after ~40min 
https://github.com/Chia-Network/chia-blockchain/runs/5007793256?check_suite_focus=true.

This PR introduces the class `TemporaryPoolPlot` which provides an async 
context manager which makes sure plots are created on entry and removed 
on exit.
2022-02-01 09:32:15 -05:00
Mariano SorgenteandGitHub 34edf6e35e pool_wallet: Deterministic derivation of auth key, and only count poo… (#9922)
* pool_wallet: Deterministic derivation of auth key, and only count poolWallets for indeces

* cli: Use the same limit for cli creation

* farmer: no more auth keys in config

* farmer: handle case of no authentication sk

* farmer: remove async, and fix test

* lint

* test: Add a test for many pool wallets, and user store

* test: Wait for wallet to be synced in test

* wallet: fix startup (no dns servers) and change to server checking of plotnfts

* cli: change to server checking

* tests: fix user store test

* Improve caching

* PR comments

* Remove useless class

* Try to fix test again
2022-01-30 12:31:55 -05:00
Kyle AltendorfandGitHub 48cfe11641 Set pools CI test timeout to 45 minutes (#9817)
* Set pools CI test timeout to 45 minutes

There have been several timeout failures recently with successful runs right on the edge at 28 minutes and such.

* pre-commit for workflow rebuild
2022-01-15 16:31:54 -08:00
89f15f591c Merge standalone wallet into main (#9793)
* wallet changes from pac

* cat changes

* pool tests

* pooling tests passing

* offers

* lint

* mempool_mode

* black

* linting

* workflow files

* flake8

* more cleanup

* renamed

* remove obsolete test, don't cast announcement

* memos are not only bytes32

* trade renames

* fix rpcs, block_record

* wallet rpc, recompile settlement clvm

* key derivation

* clvm tests

* lgtm issues and wallet peers

* stash

* rename

* mypy linting

* flake8

* bad initializer

* flaky tests

* Make CAT wallets only create on verified hints (#9651)

* fix clvm tests

* return to log lvl warn

* check puzzle unhardened

* public key, not bytes. api caching change

* precommit changes

* remove unused import

* mypy ci file, tests

* ensure balance before creating a tx

* Remove CAT logic from full node test (#9741)

* Add confirmations and sleeps for wallet (#9742)

* use pool executor

* rever merge mistakes/cleanup

* Fix trade test flakiness (#9751)

* remove precommit

* older version of black

* lint only in super linter

* Make announcements in RPC be objects instead of bytes (#9752)

* Make announcements in RPC be objects instead of bytes

* Lint

* misc hint'ish cleanup (#9753)

* misc hint'ish cleanup

* unremove some ci bits

* Use main cached_bls.py

* Fix bad merge in main_pac (#9774)

* Fix bad merge at 71da0487b9

* Remove unused ignores

* more unused ignores

* Fix bad merge at 3b143e7050

* One more byte32.from_hexstr

* Remove obsolete test

* remove commented out

* remove duplicate payment object

* remove long sync

* remove unused test, noise

* memos type

* bytes32

* make it clear it's a single state at a time

* copy over asset ids from pacr

* file endl linter

* Update chia/server/ws_connection.py

Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>

Co-authored-by: Matt Hauff <quexington@gmail.com>
Co-authored-by: Kyle Altendorf <sda@fstab.net>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
2022-01-13 12:08:32 -08:00
Kyle AltendorfandGitHub 8291f0221a Make the sized bytes types hint compatible (#9369)
* Rework sized bytes for type hinting compatibility

* add a bunch of type: ignores

* this will be handled elsewhere

* noqa E501 instead of changing code

* normalize comment plurality

* @classmethod

* Revert "@classmethod"

This reverts commit 95db80e339.

* add ignore in benchmarks

* just E501 again...

* add some new type: ignores
2021-12-02 09:43:39 -08:00
dustinfaceandGitHub f3526a909d tests|github|pre-commit: Improve build-workflows.py and make it a pre-commit hook (#8728)
* tests: Move `default_replacements` into `generate_replacements`

* tests: Directly use `root_test_dirs`

Instead of passing it into `subdirs`

* tests: Make `build-workflow.py` callable from everywhere

* tests: Adjust the output of `build-workflows.py`

* tests: Add `-f` to `build-workflows.py` to allow failing on CI

* github: Run `build-workflows.py`

* pre-commit: Run `build-workflows.py` as first hook

* tests: Skip DID wallet tests until they are reliable and fixed

* tests: Skip `test_using_legacy_cryptfilekeyring`

* tests|github: Drop `root_test_dirs` and just search in all directories

This removes the requirement to add new test directories to the 
`testconfig.py`. This change uncovered two more tests which were not 
enabled on CI because their parent dir's were not added to 
`root_test_dirs`.
- `tests/weight_proof/test_weight_proof.py`
- `tests/util/test_struct_stream.py`

* tests|github: Don't list test files, use `tests/dir/tests_*.py` instead

* test: Improve workflow update checks by comparing old vs new

Instead of doing a `git diff` which can lead to false positives if there
are other local changes unlrelated to test workflows. This also fixes
the issue that newly added files were not tracked by `git diff` which
means the script / the CI would trigger an alert.

* tests: Adjust `build-workflows.py` to work as expected on windows

* tests: Read/Write in bytes to avoid and detect unexpected `\r\n`

* tests: Skip `test_pool_cmdline.py`
2021-11-09 10:57:40 -08:00
Adam KellyandGitHub 869f39893c Plotnft fees cmdline fix (#9129)
* Add fees to plotnft commandline and RPCs

* Return calculated fee based on number of transactions

* Extend tests to support fees. Default fee to 0 if not present in RPC call.

* Make plotnft cmdline fees use XCH, like

* fix validator

* cmdline tests

* Remove duplicate option
2021-11-04 15:29:11 -07:00
Adam KellyandGitHub 996f518810 Plotnft fees (#9116)
* Add fees to plotnft commandline and RPCs

* Return calculated fee based on number of transactions

* Extend tests to support fees. Default fee to 0 if not present in RPC call.
2021-11-04 10:44:36 -07:00
8a028c3594 Ms.mempool locking (#9050)
* Prority locking to consensus

* Remove pstats

* Linting

* Do some stuff outside of lock

* Fix startup

* Add log timings

* Try some different locking

* Add limit

* catch excp

* CLVM inside lock

* Try using a semaphore instead

* use events for lock queue

* test

* Add logging for message types

* type

* remove seed

* check new peak waiters

* correct FullNodeAPI self.full_node.new_peak._waiters typo

* correct logging string typos

* only warn about new_peak Waiters if there is at least 1

* remove no-longer-accepted parameter to FullNode.peak_post_processing()

* only warn about respond_transaction Waiters if there is at least 1

* lint

* Change some constants

* Small fix and logging changes

* Put message types outside

* Change some log levels so we can test with info

* More logging

* Increase rate limits but decrease paralelism

* tweaks

* Log dropped tx

* Fix pool rpc test

* Test fixes

* Mempool optimization

* Remove from seen if fails

* Increase queue sizes

* Message types info

* More test and logging

* Small changes to networking just in case

* Decrease logging

* Decrease logging even further

* Decrease logging even further even further

* Decrease logging 3

* Transaction queue

* Don't cancel tasks or close connection

* Cancel tasks on disconnect (for shutdown purposed)

* Fix typo

* Catch cancelled

* Do multiple at a time

* More accurate farmer response time

* More efficiently create tasks

* Increase queue size and priority by fee

* Revert priority

* Don't re-request too many times for dropped TX

* Handle cancelled error so we don't go into a bad state

* Catch cancelled in syncing tasks

* Reduce new_peak_sem to improve performance

* Less bytes conversion

* Missing file, and 2 workers for CLVM

* Validate BLS in a new thread

* tests

* Change semaphore constants

* correct a cancellation triggered exception and assertion

* Fix send_transaction, dont use BaseException, fix tests

* Fix more tests

* only log transaction handler cancellation in debug

* typing in log

* move unfinished validation to diff proc

* it is asyncio.CancelledError

* Add a test for bad signature

* Fix more tests, reduce logging, lint

* One more lint

* blockchain tests, pass bytes directly, single call

* Try to fix rl_wallet failures

* Fix mempool test

* catch everything

* Don't test RL wallet

* Fix more tests and return error code

* Improve error handling in multiprocess

* Add pre-validation time

* Add pre-validation time in logs, and revert pytest.ini changes

* Add log correctly

* Ms.bls cache experiment (#9115)

* Logging for cache

* Less logging

* Return to original plan

* Clean up

* Remove coment

* Remove log

* formalize LockQueue shutdown

* Comments

* Fix blockchain test

* Improve cache

* Remove logs

* Fix sign_coin_spends

* Fix pool wallet

Co-authored-by: Kyle Altendorf <sda@fstab.net>
Co-authored-by: Yostra <straya@chia.net>
2021-11-04 09:29:05 -07:00
dustinfaceandGitHub 849b5811de tests|plotting: Some refactoring/improvements around BlockTools (#8483)
* tests: Use `time_out_assert` to wait for plot refreshing

* tests: Make `plot_manager` member of `BlockTools`

This also:

- Moves some code 
related to refreshing into `BlockTools.refresh_plots`.
- Moves setting the public keys of `PlotManager` into `setup_keys.

* tests: Directly use the plot info, don't create a sorted list upfront

* tests: Refactor a loop to use `values` instead of `items`

* plotting: Return paths of created and existed plots in `create_plots`

* tests: Introduce `new_plot` in `BlockTools`

* tests: Use `BlockTools.new_plot` in `TestPoolWalletRpc`

* tests: Allow and fix `flake8` validation in `test_pool_rpc.py`

* tests: Introduce `BlockTools.delete_plot` and use it in pool tests

* tests: Cleanup plots potentially left from a failed run in pool tests
2021-10-04 10:42:36 -07:00
Arvid NorbergandGitHub 31dab2ae5f Rust only conditions (#8467)
* no need to check announcements in python, it's all done in rust

* remove python condition parsing, use rust unconditionally

* no need to assert_my_amount, it's done in rust

* assert my puzzlehash is done in rust, no need to check it in python

* parent id is checked in rust, no need to do it in python

* my coin ID is checked in rust, no need to do it in python

* reorder condition checks to have to ones we won't see at the end

* no need for the RUST_CONDITION_CHECKER constant anymore

* update pool tests to use get_name_puzzle_condition
2021-09-18 10:25:19 -07:00
dustinfaceandGitHub ada12b3f42 tests: Fix rmtree call in create_pool_plot (#8463) 2021-09-13 13:20:36 -07:00
0096f755a1 DID Update Merge (#8129)
* fixed the bugs related to DID puzzles

* change test so that wallets recover into other wallets

* fix parent_info fetching when recovering

* fix did_test

* delete did tests related to singleton behaviours as that is tested elsewhere

* linting fixes

* update hash commit for did_innerpuz.clvm

* update DID wallet RPC calls

* more lint fixes

* delete further references to deprecated did_spend function

* fix bug in wallet state manager related to falsely detecting pool create

* added did_rpc test

* make sure amounts are uint64

* lint fixes

* Fix deadlock in DID wallet creation, and introduce create_new_did_wallet API call

* rename solution_to_pool_state

* Remove tests dir from packages

* added potential fix

* Allow getting unconfirmed balance from wallet_state_manager when under lock

* Remove a deadlock from create_new_did_wallet

* Update DID wallet test to use async check for farmed chia in wallet setup

* Fix unsigned arithmetic error

* Make DID wallet tests compatible with WalletStateManager lock checking

* check if removal belongs to the wallet

* unconfirmed

* did test cleanup

* fix temppuzhash to be an innerpuz

* clean up DID Wallet, add test for previously broken case

* added state_change call when coin added
added did_info for transaction sent

* update delete wallet parameters

* add comments to DID innerpuz

* fix duplicating bug with tx store

* re-enable did tests

* fix cc wallet bare raise

* remove unused assignation

* fix bare spend in did_wallet

* fix small bug

* messages are a cons box where the first value denotes type of message

* cc_wallet uses new parameter for get_confired_balance

* updates to the puzzle based upon suggestion by Richard

* update git submodule chia-blockchain-gui to did_branch

* updated gui to did_branch 76f9e6cea9f58a30984580b594631f3ae2679752

* updated gui to 041ac79be4af360d856a6527a4b8f4a074415a9a

Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com>
Co-authored-by: Yostra <straya@chia.net>
Co-authored-by: William Blanke <wjb98672@gmail.com>
2021-09-07 12:48:41 -07:00
325d7f8245 Keyring passphrase protection (#7249)
* Moved keyring handling into a KeyringWrapper class

* Update click to 8.0.x for prompt_required support

* Renamed KeyringWrapper to _KeyringWrapper

* Expose password management options on Linux

* CLI support for setting/removing a password

* Global option for specifying the master password

* Cache the password instead of setting on the context

* Password bootstrapping during chia init

* Tidying up _KeyringWraper's interface

* Initial pass migrating the legacy keyring contents

* Encryption/decryption of keyring.yaml contents

* FileKeyring backend encrypts with ChaCha20Poly1305

* Tightened up keyring migration and initialization

* Fixed issues identified by linters

* Remove root_path from Keychain

* Prevent double-migration if setting master passwd

* KeyringWrapper tests are mostly complete

* FileKeyring will now honor the service param

* Tests for get/set/delete password

* Formatting/commenting updates

* Writer lock support with tests - WIP

* keyring.yaml is now watched for modifications

* Reader/Writer lock for get/delete password

* Fixed linter issues

* Reader lock tests

* Formatting update

* Hook up CHIA_ROOT support for KeychainWrapper

* Quick fix to address test failures

* Fixed failures when existing legacy keyring exists

* Fixed test failures caused by reusing the same temp dir

* keyring.yaml now lives in ~/.chia_keys by default. Can be overridden with CHIA_KEYS_ROOT or --keys-root-path

* Fixed migration failure when setting a password (not using the default)

* KeyringWrapper now uses supports_keyring_password to determine if a FileKeyring should be used. Patched tests to work regardless of whether supports_keyring_password return False

* The daemon now takes a --have-gui option that will prevent calling check_keys() during startup. If the keyring is locked, we want the GUI to prompt for the password.

* Added is_keyring_locked RPC call

* Added 'unlock_keyring' RPC command

* Added KeychainProxy and KeychainServer to handle RPC messages related to keyring operations. WalletNode no longer directly accesses the Keychain class.

* Turn on macOS support for testing keyring passwords

* Fixed get_key_for_fingerprint to use the ocal keychain if the platform doesn't need to remotely access the daemon's keychain.

Fixed key reconstruction when sent over RPC.

* Farmer now accesses the keychain over RPC

* Fixes for linter issues and some restructuring to support tests that use setup_nodes.py

* Couple of fixes to unblock the GUI from launching when a keyring password is set

* Added a keychain RPC call for add_private_key()

* Added remaining keychain proxy RPC calls for delete_key_by_fingerprint and delete_all_keys

* Check for None when inspecting request arguments

* Run check_keys after unlocking the keyring when the daemon is launched via GUI

* Added check_keys RPC method.
Fixed deserialization of key entropy in get_all_private_keys. This was preventing the GUI from being able to show key details.

* Added get_first_private_key to keychain_server/proxy.
create_plots now uses the keychain proxy when launched from the daemon.

* Added a comment about KeychainProxy in chia plots check

* Workaround import conflict  when importing from 'tests.*' due to fasteners name conflict

* Simulator now uses KeychainProxy if launched by the daemon.
KeychainServer/Proxy now takes keychain user/testing params for testing scenarios.

* Added "set_keyring_passphrase" RPC message

* Reworking KeychainProxy usage to handle local keychain tests and RPC keychain tests.

* Replace my prior usage of asyncio.run() with asyncio.get_event_loop().run_until_complete()

* Silencing file_keyring logging for the moment.

* Updated tests to use test keychains and appropriate BlockTools construction

BlockTools should now be created with create_block_tools(_async) to handle async scenarios.

Updated block_tools to be async compatible

Updated fasteners to fix installation of top-level 'tests' in site-packages

* Added 'remove_keyring_passphrase' RPC message to the daemon

Minor tweak to TempKeyring to default to some test params

* Fixed linter issues

* Remove flake8 ignore statement now that the fasteners module has been updated

* Some initial renaming changes: password -> passphrase

* Fixed wallet RPC issue where get_key_for_fingerprint wasn't awaited-upon.

Fixed legacy keyring initialization (for migration scenarios)

* Fixed improperly merged file

* Fixed linter issues.
More renaming.

* Updated spots that were still using an incorrect keychain call

* Renamed use_password_cache, obtain_current_password

* Renamed supports_keyring_password

* Renamed has_master_password

* Renamed has_cached_password, get_cached_master_password

* Linter fixes

* Renamed master_password_is_valid

* Renamed set_cached_master_password

* Renamed set_master_password

* Renamed remove_master_password

* Renamed has_cached_master_password

* Renaming in file_keyring and keyring_wrapper

Updated default keyring payload used for tests

* Renamed get_password
Other renaming updates

* Renamed set_password
Other renaming updates

* Renamed remaining password occurrences (where appropriate)

* password -> passphrase

* Added tests for setting an emoji and Japanese master passphrase

* Attempt to notify the daemon when a keyring passphrase is set/updated/removed

* Missed one password -> passphrase replacement.

* Fixed some file synchronization issues found when running tests on macOS

* Adjusted timeout values for test_writer_lock_reacquisition_failure for macOS.

* Removed logging statements previously added for debugging

* Prompt for keyring passphrase up-front when launching a service.

Changed --have-gui flag to --wait-for-unlock

* Updated set_keyring_passphrase RPC message to fix optional current_passphrase param when the keyring is using the default passphrase.

* Minor test cleanup to deduplicate some code.

* Fixed regression when setting a new master passphrase

* Minor refactoring and docs/commenting updates

* Renaming password -> passphrase went too far. Keyring backends use password terminology for compatibility with third party backends.

* Disabling macOS support (previously added for testing only)

* Disabling passphrase support in preparation for sending out the PR

* Fixed improper merge (vscode didn't save changes during rebase)

* Update chia/cmds/init_funcs.py

Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com>

* skip_check_keys -> should_check_keys

* Shuffling some imports around to break cycles reported by LGTM

* Handle unlocking the daemon if it's already launched and waiting for unlock.

* Replaced uses_keychain_proxy decorator in farmer.py. Fixed async usage of get_reward_targets.

Linter/reformatting fixes

* Replaced uses_keychain_proxy decorator with a clearer method.

* Cleanup the temp keyring dir using shutil.rmtree()

* Restored self._root_path (had been changed to self.root_path)

* Minor cleanup

* ensure_keychain_proxy() now throws if connect_to_keychain_and_validate() fails

* Plot key resolution now yields a PlotKeys object which can be passed into create_plots.

De-indented test_invalid_icc_sub_slot_vdf to keep git blame tidy.

* Added 'keyring_status' daemon RPC message to support the GUI

* Minor changes relating to PR feedback

* Addressed more PR feedback (mostly type annotations)

* Commented-out macOS file keyring usage. This can be re-enabled for testing purposes.

* Addressed test failures that require multiple keyrings in the same process. Each TempKeyring will now set a custom KeyringWrapper instance.

* Fixed logic for communicating user_passphrase_is_set in the keyring_status RPC response.

* Updated type annotations and method signature for set_passphrase to expect a string instead of bytes.

* Fixed Wallet RPC tests

* Fixed full_node_store tests. BlockTools should be created using the create_block_tools(_async) function(s)

* Fixed test failures in test_pool_rpc

* Fixed test_daemon. After BlockTools.setup_plots is run, the config file needs to be re-read to refresh stale plot_directories.

* Suppressing LGTM false positives regarding passphrase leakage in CLI error output. Seems that LGTM sees MIN_PASSPHRASE_LEN as sensitive data.

* Second attempt at suppressing LGTM false positives

* Third attempt at addressing LGTM false positives

* Removed test_keyring_wrapper param from Keychain ctor. Test setup now sets the keyring_wrapper property directly.

* Reformatting

* More targeted update of the test config to refresh just the "plot_directories" value

* More LGTM suppressions

Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com>
Co-authored-by: wjblanke <wjb98672@gmail.com>
2021-08-04 12:46:55 -07:00
wjblankeandGitHub d3599da0a0 reverting 7645 (#7708) 2021-07-22 09:00:36 -07:00
dustinfaceandGitHub 5b690066c1 cmds|util|tests: Drop 0x prefix from JSON conversions (#7645)
Not sure if that has been done for a specific reason but i feel like its 
better to not have them in general. Let me know if there are any 
objections, im happy to close this PR if its for a reason i missed.
2021-07-20 13:01:58 -07:00
Arvid NorbergandGitHub 98e330e41e remove left-over debug logging from test_wallet_pool_store (#7533) 2021-07-15 10:24:41 -07:00