Commit Graph
78 Commits
Author SHA1 Message Date
Matt HauffandGitHub 40db4635a8 [LABS-245] Enable PEP604 Ruff rules (#20269)
* Enable PEP604 Ruff rules

* Fix harcoded signature in test

* Hack CLVMStreamable test with note to fast follow
2025-11-18 12:34:00 -08:00
Florin ChiricaandGitHub 7d077cdc46 DL Code Cleanup (#19782)
* DL code cleanup

* Lint.
2025-07-14 22:25:57 -04:00
Kyle Altendorf fb8de7a1b3 Merge commit 'dac837e79961fa5ce9611d5531d789211031c465' into catchup/long_lived_datalayer_merkle_blob_from_main_dac837e79961fa5ce9611d5531d789211031c465 2025-07-11 22:35:57 -04:00
Matt HauffandGitHub f782fa8341 [CHIA-1315] Port (wallet) DataLayer endpoints to @marshal (#19689)
* Port `create_new_dl`

* Forgot default_raise

* Port `dl_track_new`

* Port `dl_stop_tracking`

* Port `dl_latest_singleton`

* Port `dl_singletons_by_root`

* fix test

* Port `dl_update_root`

* Port `dl_update_multiple`

* fix tests

* Port `dl_history`

* Port `dl_owned_singletons`

* Port `dl_get_mirrors`

* Make `Mirror` streamable (and fix test)

* Port `dl_new_mirror`

* push=True

* Port `dl_delete_mirror`

* Port `dl_verify_proof`

* Fix test

* fix import
2025-06-27 11:29:24 -07:00
Kyle Altendorf 668548cd33 Merge commit 'da20e4145fb6279d6880fd998b62456237e2f700' into catchup/long_lived_datalayer_merkle_blob_from_main_da20e4145fb6279d6880fd998b62456237e2f700 2025-06-23 22:03:50 -04:00
6050235bdb [CHIA-3043] Add (hopefully) non-controversial ruff rules (#19684)
* Add (hopefully) non-controversial ruff rules

* mypy-exclusions.txt

* PIE

* Fix pre-commit?

* PYI

* RSE

* S + ignores

* Whitespace fix

* Re-add build-init-files.py

* Add comment about security ignores

* Update mypy-exclusions.txt

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Fix the executable status of build-init-files.py

* Set executable bit for tools/run_block.py and readd shebang

* Fix new errors

---------

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2025-06-11 08:36:08 -07:00
+1
Kyle AltendorfGitHubwjblankeAmine KhaldiArvid Norbergdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Earle LoweEarle Lowegithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>ChiaAutomationZachary Brown
85c47c6649 catchup: into long_lived/datalayer_merkle_blob from main @ 329ce24c89 (#19351)
* update pin to chia-blockchain-gui (#19292)

* Update the anchor against the checkpoint.

* mypy annotation for `test_dos.py` (#19036)

* remove unused function, get_block_path()

* add type annotation to test_dos.py

* use chia_rs directly for foliage (#19308)

* use chia_rs directly for weight proof (#19315)

* build(deps): bump ruff from 0.8.5 to 0.9.6 (#19260)

* build(deps): bump ruff from 0.8.5 to 0.9.6

Bumps [ruff](https://github.com/astral-sh/ruff) from 0.8.5 to 0.9.6.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/0.8.5...0.9.6)

---
updated-dependencies:
- dependency-name: ruff
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update source to ruff 0.9.6

* Apply suggestions from code review

Co-authored-by: Kyle Altendorf <sda@fstab.net>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Earle Lowe <e.lowe@chia.net>
Co-authored-by: Earle Lowe <30607889+emlowe@users.noreply.github.com>
Co-authored-by: Kyle Altendorf <sda@fstab.net>

* build(deps): bump mypy from 1.14.1 to 1.15.0 (#19319)

* build(deps): bump mypy from 1.14.1 to 1.15.0

Bumps [mypy](https://github.com/python/mypy) from 1.14.1 to 1.15.0.
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/python/mypy/compare/v1.14.1...v1.15.0)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* adjust accordingly

* remove another

* and windows

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kyle Altendorf <sda@fstab.net>

* build(deps): bump pyinstaller from 6.11.1 to 6.12.0 (#19318)

Bumps [pyinstaller](https://github.com/pyinstaller/pyinstaller) from 6.11.1 to 6.12.0.
- [Release notes](https://github.com/pyinstaller/pyinstaller/releases)
- [Changelog](https://github.com/pyinstaller/pyinstaller/blob/develop/doc/CHANGES.rst)
- [Commits](https://github.com/pyinstaller/pyinstaller/compare/v6.11.1...v6.12.0)

---
updated-dependencies:
- dependency-name: pyinstaller
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* use chia_rs directly for sized bytes (#19326)

* build(deps): bump keyring from 25.5.0 to 25.6.0 (#19150)

Bumps [keyring](https://github.com/jaraco/keyring) from 25.5.0 to 25.6.0.
- [Release notes](https://github.com/jaraco/keyring/releases)
- [Changelog](https://github.com/jaraco/keyring/blob/main/NEWS.rst)
- [Commits](https://github.com/jaraco/keyring/compare/v25.5.0...v25.6.0)

---
updated-dependencies:
- dependency-name: keyring
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump filelock from 3.16.1 to 3.17.0 (#19164)

Bumps [filelock](https://github.com/tox-dev/py-filelock) from 3.16.1 to 3.17.0.
- [Release notes](https://github.com/tox-dev/py-filelock/releases)
- [Changelog](https://github.com/tox-dev/filelock/blob/main/docs/changelog.rst)
- [Commits](https://github.com/tox-dev/py-filelock/compare/3.16.1...3.17.0)

---
updated-dependencies:
- dependency-name: filelock
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump pre-commit from 4.0.1 to 4.1.0 (#19166)

Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 4.0.1 to 4.1.0.
- [Release notes](https://github.com/pre-commit/pre-commit/releases)
- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pre-commit/pre-commit/compare/v4.0.1...v4.1.0)

---
updated-dependencies:
- dependency-name: pre-commit
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump types-setuptools from 75.6.0.20241126 to 75.8.0.20250225 (#19320)

build(deps): bump types-setuptools

Bumps [types-setuptools](https://github.com/python/typeshed) from 75.6.0.20241126 to 75.8.0.20250225.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-setuptools
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump boto3 from 1.35.90 to 1.37.1 (#19322)

Bumps [boto3](https://github.com/boto/boto3) from 1.35.90 to 1.37.1.
- [Release notes](https://github.com/boto/boto3/releases)
- [Commits](https://github.com/boto/boto3/compare/1.35.90...1.37.1)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* use chia_rs directly for pool target (#19332)

* address issue introduced in #19207 (#19328)

address issue introduced in #19207 where empty transaction blocks would not have None transactions_block_generator

* CA Cert updates (#19312)

adding ca updates

Co-authored-by: ChiaAutomation <automation@chia.net>

* bump chia_rs to 0.21.0 (#19337)

* bump chia_rs to 0.21.0

* update mempool tests with the new, stricter, rules

* use chia_rs directly for reward chain block (#19336)

* cache `get_unspent_lineage_info()` (#19335)

* no need to pass get_unspent_lineage_info_from_puzzle_hash function into create_bundle_from_mempool(), we already have it as a member now

* cache results from get_lineage_info_for_puzzle_hash()

* fix

* fixup

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: wjblanke <wjb98672@gmail.com>
Co-authored-by: Amine Khaldi <amine.khaldi@reactos.org>
Co-authored-by: Arvid Norberg <arvid@libtorrent.org>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Earle Lowe <e.lowe@chia.net>
Co-authored-by: Earle Lowe <30607889+emlowe@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: ChiaAutomation <automation@chia.net>
Co-authored-by: Zachary Brown <z.brown@chia.net>
2025-03-05 11:52:01 -05:00
Kyle AltendorfandGitHub 2a53889d9c correct datalayer side value types (#19330) 2025-03-05 09:30:51 -05:00
Kyle Altendorf 7a6e9b3d12 Merge commit '329ce24c89e29f8c7f0d4a8bb4d04fc2b0438d95' into catchup/long_lived_datalayer_merkle_blob_from_main_329ce24c89e29f8c7f0d4a8bb4d04fc2b0438d95 2025-03-04 16:43:20 -05:00
Kyle AltendorfandGitHub d50b6e0883 use chia_rs directly for sized bytes (#19326) 2025-02-28 09:35:25 -08:00
Kyle AltendorfandGitHub 1567fa3df6 use chia_rs proof of inclusion for datalayer (#19327) 2025-02-27 14:19:38 -05:00
Kyle AltendorfandGitHub c09d63ccb0 extract sibling hash and side methods as functions (#19324) 2025-02-26 10:52:21 -05:00
Kyle AltendorfandGitHub 5c68281a81 more datalayer rust transition prep (#19306)
* more datalayer rust transition prep

* tidy
2025-02-26 07:57:50 -05:00
Kyle Altendorf b54c22faf0 Merge commit 'e446d6b726a03f6d917083b4d039abfec503e54e' into catchup/long_lived_datalayer_merkle_blob_from_main_e446d6b726a03f6d917083b4d039abfec503e54e 2025-02-25 11:30:14 -05:00
Kyle AltendorfandGitHub e446d6b726 use chia_rs directly for sized ints (#19303)
* use chia_rs directly for sized ints

* Update data_layer_rpc_api.py
2025-02-24 14:33:09 -08:00
Kyle AltendorfandGitHub 849eef1b0a remove ProofOfInclusion.as_program() (#19305) 2025-02-22 11:03:44 -05:00
Kyle AltendorfandGitHub 63d35aa560 Change delta format (#19175)
Change the delta file format to diff only against the previous
generation, as opposed to all existing generations. This maintains
backwards compatibility for now, but adds to the DB only the minimum
amount of hashes so we can sync. Once all clients update to this new
delta format, it will no longer be needed to store all the hashes into
the DB.
2025-01-31 09:46:05 -05:00
Florin Chirica d07eefce8e Add comments. 2025-01-30 19:59:18 +02:00
Kyle Altendorf 4d5f168ef5 make ProofOfInclusion.root_hash a method, from a property 2025-01-24 09:50:18 -05:00
Florin Chirica 11e1c623bb Lint. 2024-10-30 02:28:44 +02:00
Florin Chirica 7765bbd504 Lint. 2024-10-29 06:01:22 +02:00
Florin ChiricaandGitHub 81083b2176 Merge branch 'long_lived/datalayer_merkle_blob' into fc.dl_integration 2024-10-29 05:20:52 +02:00
Florin Chirica de52d3899e Migrate old DB. 2024-10-25 17:57:41 +03:00
Kyle AltendorfandGitHub 9c8557a1f8 Remove Python 3.8 support and update source to 3.9 standards (#18687)
* manual python 3.8 removals

* poetry

* pyupgrade

* black

* isort

* manually tidy imports

* work around shadowed type

* catch up build network protocol files for pyupgrade changes

* placate pylint

* re-enable 20.04

* placate flake8
2024-10-16 14:21:15 -07:00
Kyle AltendorfandGitHub a6a27bfe8e optimize datalayer hashing (#18510) 2024-09-11 09:41:54 -07:00
a4da91a86d Changed: Enhance DataLayer Plugin Registration System for Improved Third-Party Integration (#17711)
* feat: load plugins from file

* feat: add tests

* feat: add file based datalayr plugin support

* fix: isort

* match mozilla-ca repo with main

* chore: fix linting

* add additional test coverage

* Update chia/util/misc.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* move utility to datalayer folder

* fix lint

* use PluginRemote schema

* remove check

* remove check

* Update chia/data_layer/util/plugin.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* load plugin using yaml.safe_load instead of json.load

* add default value to headers if doesnt exist

* fix: syntax error

* add log exception

* Update chia/data_layer/util/plugin.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

---------

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2024-07-16 11:05:11 -07:00
Kyle AltendorfandGitHub 7de41fc9b4 tidy import aiosqlite as aiosqlite (#18278) 2024-07-15 13:36:01 -07:00
Kyle AltendorfandGitHub cf7b54966e root hash, does None mean empty or unspecified? (#18244)
* mark some cases where a none root hash means unspecified, not empty

* propagate, hint for help tracking, etc

* cp

* enum trick

* tidy

* tidy

* fixup
2024-07-11 13:44:36 -07:00
a58cc334a4 clvm mypie (#16898)
* clvm mypie

* update exclusions

* wip

* some

* undo for now

* more

* more

* drop the program usage in datalayer tests

* ignore tree hash errors

* ignore enough in _sexp_replace...

* again

* almost

* Fix test_singleton.py

* dl fixup

* did fixup

* touchup

* nft fixup

* fixup tests

* Revert "fixup tests"

This reverts commit 5df1a705ec.

* or this i guess

* blah

* Fix get_owner_did

* tidy

* blah

* `.as_atom()`

* `.as_int()`

* Update chia/wallet/nft_wallet/nft_puzzles.py

* `rrrff`

* stop using `CastableType`

* undo stuff around using `CastableType`

* `announcement_assertions: List[Program]`

* Update chia/types/blockchain_format/program.py

* more

* clvm==0.9.10

---------

Co-authored-by: Amine Khaldi <amine.khaldi@reactos.org>
Co-authored-by: Matt <quexington@gmail.com>
2024-05-22 12:28:07 -05:00
Kyle AltendorfandGitHub 98df4ce222 change datalayer python and logging from tree id to store id (#17864)
* change python and logging from tree id to store id

* always more

* oops

* again

* and...

* comment

* more
2024-05-10 13:45:45 -07:00
Amine KhaldiandGitHub 5350428eb1 Speedup leaf_hash in data_layer_util.py (#17759)
Speedup leaf_hash in data_layer_util.py.
2024-03-25 08:45:52 -07:00
Amine KhaldiandGitHub cd6f4f56de Speedup key_hash in data_layer_util.py (#17758)
Speedup key_hash in data_layer_util.py.
2024-03-25 08:45:37 -07:00
368fecc17e DL multiple batch updates. (#17414)
* DL multiple batch updates.

* Lint.

* Lint.

* Add RPCs.

* Lint

* Lint

* Lint

* Add test.

* Lint.

* Add publish pending root option.

* Improve test.

* Fix cmd.

* Lint.

* Fix test

* Revert

* Add test.

* More tests.

* Improve test.

* Apply suggestions from code review

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Lint.

* Test.

* Latest local root.

* Change publish to submit.

* More submit instead of publish.

* Coverage.

* Lint.

* Improve coverage.

* Lint.

* Lint.

---------

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2024-02-23 09:56:00 -08:00
fa220629ec DL RPC pagination. (#17347)
* DL RPC pagination.

* Lint.

* Lint.

* Add RPCs.

* Lint.

* Lint.

* Add test.

* Lint.

* Add root_hash.

* Coverage.

* Fix bug.

* Update chia/cmds/data.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Dedupe.

* Lint.

* Add reference dict to test.

* Fix test.

* Fix test.

* Fix test.

* Fix test.

* Lint.

* Fix test

* Lint

* Remove short option.

* Add test.

* Lint.

* Update tests/core/data_layer/test_data_rpc.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

---------

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2024-02-14 11:08:20 -08:00
63417e765a Add support for generating and verifying DataLayer Proofs of Inclusions (#16845)
* initial implementaion of get_proof and verify_proof for datalayer

* Add wallet RPC and stop using offer blobs

* Don't require DL wallet to validate DL proof

* Convert to dl_verify_proof to free function

* More updates to DL proofs get/verify

* More updates to DL proofs get/verify

* More updates to DL proofs get/verify

* remove some unused verification functions

* oops deleted some important things

* fix type

* Add tests for DL proofs

* WIP using streamable classes for RPC

* Tests pass with new serializable classes

* use clvm tree hash calls instead of magic numbers and std_hash

* expanded tests to include more ways to call the same thing

* minor tests updates from feedback

* add in CLI tests

* Add separate create_multiple_keys_option

* Add helper and fixup key option

* Update tests/core/data_layer/test_data_rpc.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Add layer of indirection for the proof response

---------

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2024-02-05 08:35:28 -08:00
Florin ChiricaandGitHub 41c69f5144 DL Compress get_keys_values output by hash. (#17243)
* DL Compress get_keys_values output by hash.

* Lint.

* Lint.

* Lint.

* Remove get_node from delete.
2024-01-12 09:41:00 -06:00
Florin ChiricaandGitHub ebf0c00a21 DL: Use unsubscribe queue to relax subscriptions lock. (#17068) 2023-12-19 10:59:27 -06:00
Kyle AltendorfandGitHub 90c1441af0 apply pyupgrade --py38-plus --keep-runtime-typing (#14646) 2023-10-24 12:01:23 -05:00
Kyle AltendorfandGitHub 1bbfcfaf89 add new plugins: config allowing for custom headers (#16194)
* add new `plugins:` config allowing for custom headers

* add a test
2023-09-06 17:42:25 -05:00
a4471bfa8a pass DataLayer root and generation instead of requerying (#15613)
* Cache root changes in insert_batch loop

* Reduce selects for root and generation

* remove sql logging, extracted to https://github.com/Chia-Network/chia-blockchain/pull/15690

* remove optimization extracted to https://github.com/Chia-Network/chia-blockchain/pull/15691

* remove optimization extracted to https://github.com/Chia-Network/chia-blockchain/pull/15728

* Delete test_batch_speed.py

benchmark provided in another already merged pr

* add `Root.to_row()`

* shift optimization to `DataStore.get_keys_values_dict()`

* rename to `intermediate_root`

* less tuple returning

* Revert "shift optimization to `DataStore.get_keys_values_dict()`"

This reverts commit b12ae08e3b.

* add `test_data_store.test_insert_batch_reference_and_side()`

---------

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2023-07-19 13:47:17 -05:00
Kyle AltendorfandGitHub f66b521067 Add DataLayer clear pending roots interfaces (#15516)
* add clear pending roots cli and rpc

* add tests

* drop extracted changes

* remove extracted change

* tidy

* add --yes

* handle todo

* more todo
2023-06-29 20:49:54 -05:00
433a414496 Add plugin_info and healthz endpoints to s3 plugin service, add chia data plugins check cli (#15028)
* Add plugin_id and healthz endpoints to s3 plugin service

* Add `chia data plugins check` cli

* plugin_id -> plugin_info

* updates based on reviews

* Use gather to run the lookups

* Update chia/data_layer/data_layer.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Fix typo

* short_help -> help

---------

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2023-04-18 18:45:35 -05:00
Kyle AltendorfandGitHub c678f00acb cleanup adjacent string literals (#14650) 2023-02-22 17:16:07 -06:00
ed55423db6 Add atari get sync status RPC. (#13440)
* Add atari get sync status RPC.

* Try to fix test.

* Try to fix test.

* Some fixes.

* Update tests/core/data_layer/test_data_rpc.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-10-28 15:05:03 -05:00
Kyle AltendorfandGitHub 5b39550f73 convert datalayer to DBWrapper2 (#13582)
* convert datalayer to DBWrapper2 (all write)

* more read, less write

* remove unneeded connection managers

* and...  close it

* data store now creates its own wrapper

* Drop unused hint DataLayer.batch_update_db_wrapper

* require named arguments for most of `DBWrapper2.create()`
2022-10-03 17:50:12 -05:00
Amine Khaldi 98237e6227 Reflect the changes from main on atari related code. 2022-08-30 22:35:57 +01:00
Amine Khaldi fd471e457f Reflect the changes from main on atari related code. 2022-08-30 20:31:51 +01:00
Kyle Altendorf 2fbfe49605 Merge branch 'long_lived/atari' into atari-cancel_offer_rpc 2022-08-22 12:11:45 -04:00
Kyle AltendorfandGitHub 782f7cc41c Merge pull request #13080 from Chia-Network/atari-fixup_fee_optionality
atari - fixup fee optionality
2022-08-20 09:20:27 -04:00
Kyle Altendorf ebd4dfeaaa atari - fixup fee optionality 2022-08-19 17:09:15 -04:00