Commit Graph
425 Commits
Author SHA1 Message Date
dustinface 8e2e51a8c8 Add optional outbound parameter to get_connections (#13879) 2022-11-15 17:40:16 -06:00
Earle LoweandKyle Altendorf f30f0b3512 Call close callback for already closed connections (#13727)
* always call close callback

* Some further explorations

* some more explorations

* Add delay while connections are closed in test

* Add a delay to make sure connections drain

* Update chia/server/server.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Update tests/connection_utils.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* use log_exceptions context manager

* set close flag correctly

* Added callback type and other changes

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-11-15 16:23:06 -06:00
dustinfaceandKyle Altendorf 2706f5995b server: Enable and fix mypy in ws_connection.py (#13878)
* server: Enable and fix `mypy` in `ws_connection.py`

* Apply suggestions from code review

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Tweak error message

* Tweak formatting

* Make `WSChiaConnection.close_callback` optional

* Tweak assert message

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Don't provide a default for `close_callback`

* Adjust assertion

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-11-12 14:18:11 -06:00
Kyle Altendorf 5cf8a291f7 adjust approach to optionality around ApiMetadata (#13888) 2022-11-10 19:58:12 -06:00
dustinface 0befdec071 server: Introduce WSChiaConnection._get_extra_info (#13876) 2022-11-09 17:41:50 -06:00
dustinface 50b630aee9 server: Don't pass the queue into WSChiaConnection._wait_and_retry (#13877) 2022-11-09 14:57:59 -06:00
dustinface 8394857ca2 server: Return earlier in WSChiaConnection.__getattr__ (#13875)
* server: Return earlier in `WSChiaConnection.__getattr__`

* Tweak message names
2022-11-09 14:19:43 -06:00
dustinface 24961dd02c server: Drop some unused code (#13867)
* Drop unused `Delivery` enum

* Drop unused `get_outgoing_connections`

* Drop `connection_utils.py`
2022-11-08 18:04:56 -06:00
dustinface 50d148eadd Cleanup WSChiaConnection imports/usage (#13868) 2022-11-08 10:10:59 -06:00
hugepants 1220fcfd69 Revert rate limiting messages from warning back to debug (#13758)
In 1.5.1 this was changed from `debug` to `warning` causing a lot of excess noise in default logging as reported in https://github.com/Chia-Network/chia-blockchain/issues/13132
2022-11-05 17:48:07 -05:00
Kyle Altendorf cb0fbe3208 switch to context manager for task timing (#13473) 2022-11-05 17:46:40 -05:00
dustinfaceandKyle Altendorf cf77f58bf2 server: Make ChiaServer a dataclass (#13574)
* server: Make `ChiaServer` a `dataclass`

* Use existing logger

Co-authored-by: Kyle Altendorf <sda@fstab.net>

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-11-03 11:28:23 -05:00
dustinface 7a5d9579ef server: More use of ChiaServer.get_connections (#13573)
* Drop `get_full_node_connections` -> use `get_connections`

* Drop `connection_by_type`

I think the housekeeping required to have it isn't worth it since we
don't handle huge numbers of connections and `get_connections` should be
less enough overhead. Im open for complains though.
2022-10-05 11:21:38 -05:00
Kyle Altendorf 305875d0d4 use Logger.warning() instead of the deprecated Logger.warn() (#13588)
https://github.com/Chia-Network/chia-blockchain/actions/runs/3173913911/jobs/5170075702
```python-traceback
Traceback (most recent call last):
  File "/Users/runner/work/chia-blockchain/chia-blockchain/chia/server/node_discovery.py", line 299, in _connect_to_peers
    await self._query_dns(dns_address)
  File "/Users/runner/work/chia-blockchain/chia-blockchain/chia/server/node_discovery.py", line 237, in _query_dns
    self.log.warn(f"querying DNS introducer failed: {e}")
  File "/Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/logging/__init__.py", line 1492, in warn
    warnings.warn("The 'warn' method is deprecated, "
DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
```
2022-10-03 14:45:18 -05:00
dustinface 31f31296a7 server: Use WebServer in ChiaServer (#13545)
* server: Use `WebServer`

* Fix `yaml.representer.RepresenterError: ('cannot represent an object', 49405)`
2022-10-03 14:43:04 -05:00
dustinface 35acbabf80 server: Use get_extra_info to get the ssl_object (#13578) 2022-09-30 17:50:46 -05:00
Kyle Altendorf 2fc65e1178 more explicit and complete handling of api decorator data (#13542)
* more explicit and complete handling of api decorator data

* fix

* .message_class

* actually, those are different types...

* tweak

* simplify

* learn that functools.wraps copies random attributes
2022-09-30 17:47:56 -05:00
dustinface c3ebc62246 server: Drop some ChiaServer attributes (#13513)
* Drop unused `ChiaServer.tasks`

* Drop unused `ChiaServer._tasks`
2022-09-30 17:41:51 -05:00
Kyle Altendorf 3b084a165b configure isort to add the future annotations import (#13327)
* configure isort to add the future annotations import

* apply the new isort setting

* remove type ignores for new mypy (#13539)

https://pypi.org/project/mypy/0.981/

* another
2022-09-30 03:40:22 -05:00
dustinface 345698aebd server: Create ssl contexts earlier + store them instead of all paths (#13512) 2022-09-29 12:14:20 -05:00
Kyle Altendorf bbd032e1c9 embrace click 8 (#13559)
* click==8.1.3, black==22.8.0

* black

* click option creator hinting

* ignores for to-be-resolved issue

* remove unused import
2022-09-29 11:24:36 -05:00
Gene Hoffman 1224ef924f Use codespell to catch some typos (#13557) 2022-09-28 04:27:34 -05:00
Kyle Altendorf 81837a9e1c Refactor away from post init for UPnP (#11837)
* Refactor away from post init for UPnP

* refactor UPnP to not be reusable

* update upnp in datalayer too

* add if_needed parameter to UPnP.shutdown()

* simpler

* be more lenient about startup failures, like before

* too many ways for UPnP to fail right now, revert to a smaller refactor approach instead of fixing them all up
2022-09-26 11:19:33 -05:00
Kyle Altendorf 23ff56579a parse api method hints earlier, and less, using get_type_hints() (#13509)
* parse api method hints earlier, and less, using get_type_hints()

* exploring (last)

* cleanup

* tidy comment

* be descriptive
2022-09-26 11:17:44 -05:00
Kyle Altendorf 3342a2a6a2 less _ for used variables (#13536) 2022-09-26 11:16:57 -05:00
dustinface d78c122505 Drop redundant RPC server instances in DataLayer related RPC tests (#13518) 2022-09-26 11:16:25 -05:00
dustinface 77932fcc11 util: Refactor select_port to not load the config (#13514)
* util: Refactor `select_port` to not load the config

* Fix `prefer_ipv6` config lookup

* Use `str2bool`
2022-09-23 17:47:26 -05:00
Kyle Altendorf 1dbf575a3e RpcApiProtocol and RpcServiceProtocol (#11522)
* provide a skeleton hint protocol for rpc apis

* more

* more self.custom_get_connections = None

* fix RpcApiProtocol.service hinting

* catch up with RpcApiProtocol._state_changed()

* FullNode.server is now non-optional.  'ish

* drop another todo

* custom_get_connections -> get_connections

* just make get_connections() a mandatory method with a helper provided

* lots

* try again

* ._start_with_fingerprint()

* fill out the Service generic parameter in start scripts

* ... instead of pass

* add docstrings to the protocols

* remove no longer needed hasattr/not None checks

* remove outdated comment

* refactor Harvester._is_shutdown to ._shut_down
2022-09-21 22:49:01 -05:00
Kyle Altendorf b9e6fd577e Remove kwargs from Service.start() (#12294) 2022-09-16 16:22:53 -05:00
dustinface 70fe981fb3 cmds: Implement chia beta (#12389)
* cmds: Implement `chia beta` for the beta test program

* Unhide and document all `beta` subcommands

* Refactor all subcommands

* Introduce `chia beta configure`

* Introduce `chia beta status`

* Test all `chia beta` commands

* Use a separate file logger for beta logs

* Write the plotting call args to the log file

* Sort potential submissions

* Some refactoring around log file log handler creation

* JSON dump the plotting args
2022-09-12 16:57:31 -05:00
Mariano SorgenteandKyle Altendorf 87a3e93cc4 Ms.mypy enable (#13320)
* Remove mempool.additions

* Don't re run the program, and remove program from mempool item

* Removals only stores item ids, and stores a list

* Move pending cache down to prevent cache dos

* Separate validation from adding to pool, and remove mypy exceptions

* Fix bug with replacing

* Add to mypy

* Revert cbgui

* precommit fail

* Properly update the seen dict

* lint error

* Fix mempool bug

* Enable mypy for a bunch of files

* Update after merge with main

* 99/260 remaining

* Address comments

* Enable more mypy

* Merge conflict mypy file

* More fixes

* pytest.ini and cb-gui

* One more flake8 fix

* Flake8 and tests

* More test fixes

* isort

* Make LGTM happy

* Update chia/full_node/block_store.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Update chia/full_node/full_node_store.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Update chia/wallet/util/peer_request_cache.py

Co-authored-by: Kyle Altendorf <sda@fstab.net>

* Apply all suggestions from PR comments

* Revert cb-gui

* 100 back to 1000, minor tweaks

Co-authored-by: Kyle Altendorf <sda@fstab.net>
2022-09-08 12:57:15 -05:00
Amine Khaldi a615dee862 Merge commit '27df8e28e3c2082f8959607c07188e56395c6715' into checkpoint/main_from_release_1.6.0_27df8e28e3c2082f8959607c07188e56395c6715 2022-09-01 21:25:38 +01:00
Amine Khaldi 08d9e9b768 Merge commit '95ca86cba85bc58ff45ad5c7a87b008bc3e48af0' into checkpoint/main_from_release_1.6.0_95ca86cba85bc58ff45ad5c7a87b008bc3e48af0 2022-09-01 16:06:45 +01:00
Kyle Altendorf 27df8e28e3 fill in data layer certs at startup (#13225)
* fill in data layer certs at startup

* tidy

* don't overwrite

* add overwrite parameter to write_ssl_cert_and_key()
2022-08-30 02:57:36 -05:00
Kyle Altendorf 95ca86cba8 Fill missing services in config for data layer (#13220)
* fill missing services in config for data layer

* adjust

* here too
2022-08-29 15:59:18 -05:00
Amine Khaldi eeb2277955 Merge commit '12fcaf0e982590efcf26a954c67cd760035d9e6e' into checkpoint/main_from_release_1.6.0_12fcaf0e982590efcf26a954c67cd760035d9e6e 2022-08-29 17:45:49 +01:00
Arvid Norberg 49ad81b5a0 instrumentation of asyncio tasks (#13046)
* Add support for instrumentation of asyncio tasks, to measure wall-clock time (as opposed to CPU time)

* fine tune output
2022-08-22 22:20:55 -05:00
Amine Khaldi 2836dd9574 Merge commit 'e5297f411d716b384e0514fd57aeb8536eda0527' into checkpoint/long_lived_atari_from_main_e5297f411d716b384e0514fd57aeb8536eda0527 2022-08-19 19:51:55 +01:00
Kyle Altendorf bd7ed405eb No traceback for OSError 113 in WSChiaConnection.outbound_handler() (#13016)
* No traceback for OSError 113 in WSChiaConnection.outbound_handler()

```python-traceback
Aug 16 21:24:08 fullnode chia_full_node[58416]: 2022-08-16T21:24:08.489 full_node full_node_server        : ERROR    Exception: [Errno 113] No route to host with 121.236.62.254
Aug 16 21:24:08 fullnode chia_full_node[58416]: 2022-08-16T21:24:08.490 full_node full_node_server        : ERROR    Exception Stack: Traceback (most recent call last):
Aug 16 21:24:08 fullnode chia_full_node[58416]:   File "/farm/chia-blockchain/chia/server/ws_connection.py", line 262, in outbound_handler
Aug 16 21:24:08 fullnode chia_full_node[58416]:     await self._send_message(msg)
Aug 16 21:24:08 fullnode chia_full_node[58416]:   File "/farm/chia-blockchain/chia/server/ws_connection.py", line 416, in _send_message
Aug 16 21:24:08 fullnode chia_full_node[58416]:     await self.ws.send_bytes(encoded)
Aug 16 21:24:08 fullnode chia_full_node[58416]:   File "/farm/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/web_ws.py", line 315, in send_bytes
Aug 16 21:24:08 fullnode chia_full_node[58416]:     await self._writer.send(data, binary=True, compress=compress)
Aug 16 21:24:08 fullnode chia_full_node[58416]:   File "/farm/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 688, in send
Aug 16 21:24:08 fullnode chia_full_node[58416]:     await self._send_frame(message, WSMsgType.BINARY, compress)
Aug 16 21:24:08 fullnode chia_full_node[58416]:   File "/farm/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 659, in _send_frame
Aug 16 21:24:08 fullnode chia_full_node[58416]:     await self.protocol._drain_helper()
Aug 16 21:24:08 fullnode chia_full_node[58416]:   File "/farm/chia-blockchain/venv/lib/python3.8/site-packages/aiohttp/base_protocol.py", line 87, in _drain_helper
Aug 16 21:24:08 fullnode chia_full_node[58416]:     await asyncio.shield(waiter)
Aug 16 21:24:08 fullnode chia_full_node[58416]:   File "/usr/lib/python3.8/asyncio/selector_events.py", line 848, in _read_ready__data_received
Aug 16 21:24:08 fullnode chia_full_node[58416]:     data = self._sock.recv(self.max_size)
Aug 16 21:24:08 fullnode chia_full_node[58416]: OSError: [Errno 113] No route to host
Aug 16 21:24:08 fullnode chia_full_node[58416]:
```

* Update ws_connection.py

* refactor
2022-08-17 01:03:10 -05:00
Kyle Altendorf 5f1e483511 make web socket timeout a warning with a traceback (#12824) 2022-08-16 13:45:40 -05:00
Amine Khaldi 34d6197890 Merge commit '0bc54ae08b6655ec5242a8b21460d34b5e4a38c6' into checkpoint/long_lived_atari_from_main_0bc54ae08b6655ec5242a8b21460d34b5e4a38c6 2022-08-15 13:55:39 +01:00
Amine Khaldi afd2725997 Merge commit 'f0a0f071169ef5b892261d07bf15ca701ca6e5b0' into checkpoint/long_lived_atari_from_main_f0a0f071169ef5b892261d07bf15ca701ca6e5b0 2022-08-11 13:12:19 +01:00
Amine Khaldi 87d89693b1 Merge commit '11b210809bd3c7e3e0ec199df5548f49abd6c377' into checkpoint/long_lived_atari_from_main_11b210809bd3c7e3e0ec199df5548f49abd6c377 2022-08-11 13:00:29 +01:00
Kyle Altendorf bd5a7e4163 just raise, not raise e when reraising (#12825) 2022-08-10 15:39:51 -05:00
dustinface 7c9fb654a3 server|tests: Delay the harvester -> farmer connection in some tests (#12862)
* Introduce `Service.add_peer`

* Delay the harvester/farmer connection in some tests to avoid flakiness
2022-08-10 04:57:34 -05:00
Amine Khaldi 5b8451303a Reflect the change on datalayer related code. 2022-08-08 19:13:07 +01:00
Amine Khaldi ea9e94982c Merge commit 'f2061d86d62dd8ed4a09ea4f63404784e1726adc' into checkpoint/long_lived_atari_from_main_f2061d86d62dd8ed4a09ea4f63404784e1726adc 2022-08-08 18:46:26 +01:00
dustinface 820889a316 rpc: Improve RPC server start/stop (#12768)
* rpc: Improve RPC server start/stop

* `RpcServer.stop` -> `RpcServer.close`
2022-08-06 01:07:41 -05:00
dustinface e5297f411d util: Unify lockfile handling (#12541)
* util: Unify lockfile handling

This patch:
- Renames `util.lock.lock_by_path` to `util.lock.lock_path` and improves it by using `FileLock`
- Unifies all places where we create lock files to use `chia.util.lock.lock_path`
- Changes the daemon lockfile from `start-daemon.launching` to `daemon.lock` like we have it with all `<service>.lock` in the `run` folder
- Drops the `fastener` dependency

* Move the not keyring related sync tests into `test_lockfile.py`
2022-08-04 12:59:39 -05:00
Kyle Altendorf 135ca6e990 fixup
https://github.com/Chia-Network/chia-blockchain/commit/9a4316f96eeb92a06d5ceaf94387d349c5ea9537
2022-08-03 11:00:19 -07:00