Commit Graph
1095 Commits
Author SHA1 Message Date
Earle Lowe 70112169d8 Removed "failed to write files" log spam from data layer (#15082)
remove log spam for datalayer
2023-04-19 12:01:58 -05:00
Almog De Paz a34a7dbb2a DL plugin Add missing files (#15062)
* add_missing_files

* lint

* dont error on fail to write

* use resource

* remove prints

* use dl folder and file names

* rearrange loop

* remove redundant str conversion

* dont break on failed upload
2023-04-19 10:48:06 -05:00
Earle LoweandKyle Altendorf 433a414496 Add plugin_info and healthz endpoints to s3 plugin service, add chia data plugins check cli (#15028)
* Add plugin_id and healthz endpoints to s3 plugin service

* Add `chia data plugins check` cli

* plugin_id -> plugin_info

* updates based on reviews

* Use gather to run the lookups

* Update chia/data_layer/data_layer.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Fix typo

* short_help -> help

---------

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2023-04-18 18:45:35 -05:00
dustinface d9be2004c7 wallet: Some cleanup of WalletNode.fetch_puzzle_solution (#14896)
* wallet: Introduce `WalletNode.get_coin_spend_for_coin_state`

* Rename `fetch_puzzle_solution` and `get_coin_spend_for_coin_state`

* Free `fetch_coin_spend` and `fetch_coin_spend_for_coin_state`
2023-04-18 11:26:52 -05:00
Earle LoweandKyle Altendorf 7c719c35c7 Added some sanity checks to upload and download for S3 plugin (#15057)
* Added some sanity checks to upload and download

* Add server files path to config

* small fix

* send filenames and not paths now

* minor tweaks

* Update chia/data_layer/s3_plugin_service.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* use bytes32.fromhex and other fixes

* set host to "localhost"

---------

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2023-04-17 22:18:56 -05:00
Earle Lowe 2faa72a50a Add very basic logging support to S3 plugin (#15031)
* Add very basic logging support to plugin

* Address review comments
2023-04-17 11:16:23 -05:00
28a5d75506 S3 plugin config and endpoint updates (#15037)
* Addition logging for datalayer wrt plugins

* some plugin modifications to config and endpoints

* Add entry point and some minor config checking

* Improved config file reading for store_ids

* Changed to upload_bucket and download_url

* Apply suggestions from code review

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Code review suggestions

* uploaders and downloaders are now URLs and this was missed earlier

* Update pyinstaller.spec to include plugin binary

* updates based on code review

* Update chia/data_layer/s3_plugin_service.py

Co-authored-by: Almog De Paz <almogdepaz@gmail.com>

* minor updates

* Add boto3 to setup for s3 plugin

* Add None check

---------

Co-authored-by: Kyle Altendorf <sda@fstab.net>
Co-authored-by: Almog De Paz <almogdepaz@gmail.com>
2023-04-14 18:07:00 -06:00
5b39e71842 Dl plugin service (#14883)
* generic downloader

* add s3 Downloader

* tests

* pre commit

* create uploader protocol, factor out upload code

* fix pre commit

* add s3 uploader

* add add helper for testing with real bucket

* typing

* lint

* logs

* start service separation

* working aiohttp s3 service

* lint

* multiple instances config

* break if failed to write

* better error handling

* update from conf before checking

* lint

* redundant config

* also update bucket

* improve exception handling

* remove old tests

* uae multiple uploaders

* Update chia/data_layer/data_layer.py

pr comments

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* pr comments

* Apply typos fixes from code review

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Update chia/data_layer/s3_plugin_service.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* scheme in conf

* use byte32 for store_ids list

* use dataclass instead of tuple for return value

* initial config handling for uploaders and downloaders

---------

Co-authored-by: Kyle Altendorf <sda@fstab.net>
Co-authored-by: Earle Lowe <30607889+emlowe@users.noreply.github.com>
Co-authored-by: Earle Lowe <e.lowe@chia.net>
2023-04-12 11:33:41 -06:00
dustinface a0659e923e data_layer: Simplify wallet get and create code (#14878)
* Apply `final` decorator to `DataLayerWallet`

* Simplify `DataLayerWallet.create_new_dl_wallet`

* Simplify `DataLayerWallet.create`

* Improve `WalletStateManager.get_dl_wallet`

* More use of `WalletStateManager.get_dl_wallet`

* Only try `get_dl_wallet`
2023-04-05 12:56:39 -05:00
dustinface 5e5395305d wallet: Drop some parameters from WalletStateManager.add_new_wallet (#14919) 2023-03-27 18:07:19 -05:00
dustinface 170bd986fe wallet: Some WalletType cleanup (#14897)
* Return `WalletTye` in `WalletProtocol.type`

* Compare to `WalletType` in `WalletStateManager.create`
2023-03-27 10:18:36 -05:00
Kyle Altendorf e1d232883e hint most chia.types (#14741)
* hint various chia.types

* hmm

* ints

* test todo
2023-03-06 16:34:22 -06:00
Kyle Altendorf e5edbd9b3d fix DataStore.get_tree_generation() detection of no generations (#14721)
* fix DataStore.get_tree_generation() detection of no generations

* revert unwanted change
2023-03-06 12:17:30 -06:00
Kyle Altendorf ff5ef6e073 remove leading newlines from indented code blocks (#14653)
* remove leading newlines from indented code blocks

* catch another one
2023-02-23 21:53:48 -06:00
Kyle Altendorf c678f00acb cleanup adjacent string literals (#14650) 2023-02-22 17:16:07 -06:00
Arvid Norberg 91923ef8fa optimize offers (#14531)
cache addition coins in offers, rather than executing the clvm puzzles repeatedly
2023-02-10 15:14:23 -06:00
Amine Khaldi 5169f10e50 Merge commit '9cc25cc820accad778bf626500aedd299f62a825' into checkpoint/main_from_release_1.7.0_9cc25cc820accad778bf626500aedd299f62a825 2023-02-09 14:28:39 +01:00
Matt Hauff 9cc25cc820 Quex.offer mod tests (#14478)
* add test

* do some forwards compat stuff

* Add a forwards compat example

* fix test

* Add more forwards compat tests

* Add more forwads compat tests

* Add forwards compatibility for nft1s

* add forward compatibility for nft offers

* Add forwards compatibility tests for DL offers

* Update DL test offers

* lint

* isort

* offer mod bytes

* Fix compression test

* isort again

* Add special offers for <=3.7 (CATs)

* Add special offers for <=3.7 (NFT1s)

* Add special offers for <=3.7 (NFT0s)

* Add special offers for <=3.7 (DLs)

* Check for conflicting items during aggregation

* isort

* Return created old offers properly marked
2023-02-06 22:04:39 -06:00
Adam Kelly 58c669164c Typecheck _set_state_changed_callback (#14464) 2023-02-01 15:31:46 -06:00
Arvid Norberg 68c9aafc0a move SerializedProgram to its own file (#14391)
* move SerializedProgram into its own file (to fixup type annotations)

* fix type annotations for SerializedProgram

* fixup import statements for new SerializedProgram module
2023-01-27 17:20:23 -06:00
neurosis69 f836034649 Dl wallet store index fix (#13316)
* index fixes

Signed-off-by: Thomas Hein <thomashein@gmx.at>

* restructure

Signed-off-by: Thomas Hein <thomashein@gmx.at>

* black

Signed-off-by: Thomas Hein <thomashein@gmx.at>

* fix typo

Signed-off-by: Thomas Hein <thomashein@gmx.at>

Signed-off-by: Thomas Hein <thomashein@gmx.at>
2023-01-26 11:19:09 -06:00
Arvid Norberg fda2163b6b remove additions() and reserved_fee() from CoinSpend (#14394) 2023-01-26 11:16:14 -06:00
Earle Lowe 71d9e981c3 Make sure to close the DL WalletRpcClient on DL shutdown (#14337)
close walletrpcclient on shutdown
2023-01-19 17:08:12 -06:00
Chida82 f7d6084dad Add gzip support to Data Layer download client (#14320)
add gzip
2023-01-17 12:15:49 -06:00
Chida82 5fd468df15 Add proxy support to Data Layer download client (#14281)
* add proxy

* fix format document

* format document

* fix whitespace

* fix end of file
2023-01-06 14:18:36 -06:00
Earle Lowe 3140c223bd TechDebt: add get_name to WalletProtocol (#13995)
* add get_name to WalletProtocol

* remove async from did_wallet::get_name

* Return "Standard Wallet" for get_name
2022-12-20 13:26:06 -06:00
Chida82 bc6371aa83 add strem like procedure and log progress (#13828) 2022-11-05 17:51:17 -05:00
Florin Chirica cd0c5c0f24 Fix pending root when changelist is empty. (#13119)
* Fix pending root when changelist is empty.

* Check explicitly.

* Fix test.

* Make more explicit error.

* Fix test.
2022-11-01 20:28:23 -05:00
Amine Khaldi d1e1a5d33d Merge commit 'bcb4c1ed48a8be9032f686d69d2eda7c23807194' into checkpoint/main_from_release_1.6.1_bcb4c1ed48a8be9032f686d69d2eda7c23807194 2022-10-29 11:37:15 +01:00
Florin ChiricaandKyle Altendorf ed55423db6 Add atari get sync status RPC. (#13440)
* Add atari get sync status RPC.

* Try to fix test.

* Try to fix test.

* Some fixes.

* Update tests/core/data_layer/test_data_rpc.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-10-28 15:05:03 -05:00
Matt Hauff bcb4c1ed48 Minor change to DL mirror sync (#13697)
* Syncing change for DL mirrors

* Make function call a constant

* bad whitespace merge

* typo
2022-10-24 18:58:08 -05:00
Jack Nelson e8ff7ea9a4 Coin Selection CLI / RPC Expansion (#13325)
* add max coin amount everywhere

* add excluded amounts to bare minimum

offers really does not need it passed.

* add excluded coin ids

* finish rpc stuff

* finalize commands

add coins command group & expand various coin selection methods

* fix argument processing & clean up code

* fix grammer

* small miss

* Update wallet_rpc_api.py

* fix test

* add combination command

* fix error

* move command group

* add new built in funcs for direct wallet coin searches.

* add get coin rpc

* add last command

* switch to pageable function

* fix args

* expand multi rpc call and tests

+ fix other things

* da lint

* add reverse coin sorting and ability to select coins

* Update wallet_state_manager.py

* fix types

* align with protocol

* fix asserts

* fix imports

* isort

* Update wallet_coin_store.py

* Update wallet_rpc_api.py
2022-10-24 14:03:53 -05:00
Matt Hauff c41d5a9eda Force a 'resync' if we start to track a singleton that we have already synced (#13591) 2022-10-03 21:04:39 -05:00
Kyle Altendorf 5b39550f73 convert datalayer to DBWrapper2 (#13582)
* convert datalayer to DBWrapper2 (all write)

* more read, less write

* remove unneeded connection managers

* and...  close it

* data store now creates its own wrapper

* Drop unused hint DataLayer.batch_update_db_wrapper

* require named arguments for most of `DBWrapper2.create()`
2022-10-03 17:50:12 -05:00
Kyle Altendorf 3e9aebe3ce loop.run_until_complete() -> asyncio.run() (#13581) 2022-10-03 14:44:53 -05:00
dustinface 8cdb431064 data_layer: Use WebServer in DataLayerServer (#13546)
* Don't wait on shutdown inside `start`

* `stop` -> `close`

* Use `WebServer`
2022-10-03 14:43:30 -05:00
Matt HauffandKyle Altendorf 28213628ae Better proof to singleton mapping (#13458)
* Better proof to singleton mapping

* mypy

* black

* add one for one same values test cases

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-09-30 17:41:09 -05:00
Kyle Altendorf 3b084a165b configure isort to add the future annotations import (#13327)
* configure isort to add the future annotations import

* apply the new isort setting

* remove type ignores for new mypy (#13539)

https://pypi.org/project/mypy/0.981/

* another
2022-09-30 03:40:22 -05:00
Kyle Altendorf 81837a9e1c Refactor away from post init for UPnP (#11837)
* Refactor away from post init for UPnP

* refactor UPnP to not be reusable

* update upnp in datalayer too

* add if_needed parameter to UPnP.shutdown()

* simpler

* be more lenient about startup failures, like before

* too many ways for UPnP to fail right now, revert to a smaller refactor approach instead of fixing them all up
2022-09-26 11:19:33 -05:00
Florin Chirica 89dc1ae11e Add root hash to CLI. (#13306)
* Add root hash to CLI.

* Explicitly throw exception if we don't find root hash.

* Throw inside RPC only.
2022-09-23 19:43:04 -05:00
Kyle Altendorf 1dbf575a3e RpcApiProtocol and RpcServiceProtocol (#11522)
* provide a skeleton hint protocol for rpc apis

* more

* more self.custom_get_connections = None

* fix RpcApiProtocol.service hinting

* catch up with RpcApiProtocol._state_changed()

* FullNode.server is now non-optional.  'ish

* drop another todo

* custom_get_connections -> get_connections

* just make get_connections() a mandatory method with a helper provided

* lots

* try again

* ._start_with_fingerprint()

* fill out the Service generic parameter in start scripts

* ... instead of pass

* add docstrings to the protocols

* remove no longer needed hasattr/not None checks

* remove outdated comment

* refactor Harvester._is_shutdown to ._shut_down
2022-09-21 22:49:01 -05:00
Arvid Norberg e4df3f9bf4 make a WalletProtocol, defining the shared API of all wallets (#13437)
* make a WalletProtocol, defining the shared API of all wallets and fix type checking

* add protocol check for DataLayerWallet. remove outdated comment
2022-09-15 20:16:01 -05:00
Amine Khaldi a822c21469 Merge commit '96dfdfdd044f0c5c72252d6a23884b9871239703' into checkpoint/main_from_release_1.6.0_96dfdfdd044f0c5c72252d6a23884b9871239703 2022-09-14 21:53:45 +01:00
Kyle Altendorf 96dfdfdd04 fix unique error after cancelling offer (#13454)
* fix unique error after cancelling offer

* remove unwanted commented test

* add back the original

* tidy tests
2022-09-14 15:52:27 -05:00
Amine Khaldi 18ecb326a1 Merge commit '36d44bc2f5b8ef10b0743541478ced85554318f2' into checkpoint/main_from_release_1.6.0_36d44bc2f5b8ef10b0743541478ced85554318f2 2022-09-14 13:07:51 +01:00
Matt HauffandKyle Altendorf 36d44bc2f5 Hint to the ephemeral coin's innerpuzhash (#13448)
* Hint to the ephemeral coin's innerpuzhash

* update datalayer rpc test reference offer blobs and trade ids

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-09-13 20:14:01 -05:00
Amine Khaldi 0c1e0bcf1a Merge commit 'e73f6ddb56feefc740d7c0dbc46f3379366c7798' into checkpoint/main_from_release_1.6.0_e73f6ddb56feefc740d7c0dbc46f3379366c7798 2022-09-09 20:21:29 +01:00
Amine Khaldi c03a516a76 Merge commit '1adc4d36b08c21548af6effec73ec57f929b61f2' into checkpoint/main_from_release_1.6.0_1adc4d36b08c21548af6effec73ec57f929b61f2 2022-09-09 16:20:38 +01:00
Amine Khaldi 24bcf16bed Merge commit '484f232c39fa5a21765d6b8721e9613f43782935' into checkpoint/main_from_release_1.6.0_484f232c39fa5a21765d6b8721e9613f43782935 2022-09-09 11:15:38 +01:00
Amine Khaldi 5b3627a2b7 Merge commit '3e25de2553c128048a8fa24dc441fd76b6b3a915' into checkpoint/main_from_release_1.6.0_3e25de2553c128048a8fa24dc441fd76b6b3a915 2022-09-08 21:18:59 +01:00