Commit Graph
615 Commits
Author SHA1 Message Date
J. Nick KostonandGitHub 327edabb64 Fix checking if a package is installed on py3.11 (#88768)
pkg_resources is abandoned and we need to move away
from using it https://github.com/pypa/pkg_resources

In the mean time we need to keep it working. This fixes
a new exception in py3.11 when a module is not installed
which allows proper fallback to pkg_resources.Requirement.parse
when needed

```
2023-02-25 15:46:21.101 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/components/http/ban.py", line 80, in ban_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/components/http/auth.py", line 235, in auth_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/components/http/view.py", line 146, in handle
    result = await result
             ^^^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/components/config/config_entries.py", line 148, in post
    return await super().post(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/components/http/data_validator.py", line 72, in wrapper
    result = await method(view, request, data, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/helpers/data_entry_flow.py", line 71, in post
    result = await self._flow_mgr.async_init(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/config_entries.py", line 826, in async_init
    flow, result = await task
                   ^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/config_entries.py", line 844, in _async_init
    flow = await self.async_create_flow(handler, context=context, data=data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/config_entries.py", line 950, in async_create_flow
    await async_process_deps_reqs(self.hass, self._hass_config, integration)
  File "/Users/bdraco/home-assistant/homeassistant/setup.py", line 384, in async_process_deps_reqs
    await requirements.async_get_integration_with_requirements(
  File "/Users/bdraco/home-assistant/homeassistant/requirements.py", line 52, in async_get_integration_with_requirements
    return await manager.async_get_integration_with_requirements(domain)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/requirements.py", line 171, in async_get_integration_with_requirements
    await self._async_process_integration(integration, done)
  File "/Users/bdraco/home-assistant/homeassistant/requirements.py", line 186, in _async_process_integration
    await self.async_process_requirements(
  File "/Users/bdraco/home-assistant/homeassistant/requirements.py", line 252, in async_process_requirements
    await self._async_process_requirements(name, missing)
  File "/Users/bdraco/home-assistant/homeassistant/requirements.py", line 284, in _async_process_requirements
    installed, failures = await self.hass.async_add_executor_job(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/requirements.py", line 113, in _install_requirements_if_missing
    if pkg_util.is_installed(req) or _install_with_retry(req, kwargs):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bdraco/home-assistant/homeassistant/util/package.py", line 40, in is_installed
    pkg_resources.get_distribution(package)
  File "/opt/homebrew/lib/python3.11/site-packages/pkg_resources/__init__.py", line 478, in get_distribution
    dist = get_provider(dist)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/pkg_resources/__init__.py", line 354, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
                                            ~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
``
2023-02-25 22:47:18 -05:00
epenetandGitHub aa20c902db Add typed helpers and improve type hints in util/json (#88534)
* Add type hints to load_json

* Adjust ios

* Adjust nest

* Add use of load_json_array

* Add tests

* Adjust test patch

* Add test_load_json_os_error
2023-02-22 10:09:28 +01:00
J. Nick KostonandGitHub 5bc0636905 Replace fire_coroutine_threadsafe with asyncio.run_coroutine_threadsafe (#88572)
fire_coroutine_threadsafe did not hold a reference to the asyncio
task which meant the task had the risk of being prematurely
garbage collected
2023-02-21 21:16:18 -05:00
Paulus SchoutsenandGitHub db3cd8e276 Fix danglin task for util.timeout (#88366)
Fix danglin task for util timeout
2023-02-18 00:11:35 -05:00
epenetandGitHub 8c821c8969 Add JSON support to load_fixture (#88076)
* Add JSON support to load_fixture

* More tests

* Remove lru_cache on load_json
2023-02-16 19:40:47 +01:00
Jan BouwhuisandGitHub 57738fbb8c Replace deprecated SSLContext constant PROTOCOL_TLS in mqtt (#88214)
Replace deprecated SSLContext constants
2023-02-16 13:01:28 -05:00
epenetandGitHub ba23816a0c Inverse json import logic (#88099)
* Fix helpers and util

* Adjust components

* Move back errors

* Add report

* mypy

* mypy

* Assert deprecation messages

* Move test_json_loads_object

* Adjust tests

* Fix rebase

* Adjust pylint plugin

* Fix plugin

* Adjust references

* Adjust backup tests
2023-02-16 11:37:57 +01:00
epenetandGitHub a1b7842df2 Fix incorrect reference to json WriteError (#88161) 2023-02-15 10:40:26 +01:00
J. Nick KostonandGitHub 03eea7bd3f Avoid subprocess memory copy when c library supports posix_spawn (#87958)
* use posix spawn on alpine

* Avoid subprocess memory copy when c library supports posix_spawn

By default python 3.10 will use the fork() which has to
copy all the memory of the parent process (in our case
this can be huge since Home Assistant core can use
hundreds of megabytes of RAM). By using posix_spawn
this is avoided.

In python 3.11 vfork will also be available
https://github.com/python/cpython/issues/80004#issuecomment-1093810689
https://github.com/python/cpython/pull/11671 but we won't
always be able to use it and posix_spawn is considered safer
https://bugzilla.kernel.org/show_bug.cgi?id=215813#c14

The subprocess library doesn't know about musl though
even though it supports posix_spawn https://git.musl-libc.org/cgit/musl/log/src/process/posix_spawn.c
so we have to teach it since it only has checks for glibc
https://github.com/python/cpython/blob/1b736838e6ae1b4ef42cdd27c2708face908f92c/Lib/subprocess.py#L745

The constant is documented as being able to be flipped here:
https://docs.python.org/3/library/subprocess.html#disabling-use-of-vfork-or-posix-spawn

* Avoid subprocess memory copy when c library supports posix_spawn

By default python 3.10 will use the fork() which has to
copy memory of the parent process (in our case
this can be huge since Home Assistant core can use
hundreds of megabytes of RAM). By using posix_spawn
this is avoided and subprocess creation does not
get discernibly slow the larger the Home Assistant
python process grows.

In python 3.11 vfork will also be available
https://github.com/python/cpython/issues/80004#issuecomment-1093810689
https://github.com/python/cpython/pull/11671 but we won't
always be able to use it and posix_spawn is considered safer
https://bugzilla.kernel.org/show_bug.cgi?id=215813#c14

The subprocess library doesn't know about musl though
even though it supports posix_spawn https://git.musl-libc.org/cgit/musl/log/src/process/posix_spawn.c
so we have to teach it since it only has checks for glibc
https://github.com/python/cpython/blob/1b736838e6ae1b4ef42cdd27c2708face908f92c/Lib/subprocess.py#L745

The constant is documented as being able to be flipped here:
https://docs.python.org/3/library/subprocess.html#disabling-use-of-vfork-or-posix-spawn

* missed some

* adjust more tests

* coverage
2023-02-13 09:02:51 -05:00
J. Nick KostonandGitHub 7aa1359c4a Cache try_parse_enum (#87911) 2023-02-12 03:57:36 -06:00
J. Nick KostonandGitHub ea356ad260 Optimize fetching weekly/monthly/yearly statistics (#87747)
* Optimize fetching statistics

* speed up

* avoid double groupby

* avoid another loop

* tweak flow

* fixes

* tweak

* avoid a new dt object in the cache for week/month

* avoid a new dt object in the cache for week/month
2023-02-09 21:22:32 -05:00
Erik MontnemeryandGitHub 45ac8c515e Update unit system unit conversions (#87632)
* Update unit system unit conversions

* Tweak tests
2023-02-08 10:52:54 +01:00
Erik MontnemeryandGitHub 4cecc28f8c Update unit system unit conversions (#87621) 2023-02-07 13:19:11 +01:00
Marc MuellerandGitHub ea4e2ab4aa Add Self typing (4) [mypy 1.0] (#87601) 2023-02-06 22:30:22 -06:00
Franck NijhofandGitHub ca1a12898c Enable Ruff D212 (#87347) 2023-02-03 23:08:48 +01:00
epenetandGitHub 61734f04b8 Re-enable Ruff D418 (#87302) 2023-02-03 16:33:03 +01:00
CrisicusandGitHub 934bb3e0c1 Fix an extra quote typo in the percentage util (#87161) 2023-02-02 19:59:08 -06:00
epenetandGitHub 1225c5e97d Add enum util (#87082)
* Add enum helper

* docstring

* Move to util

* Add use case
2023-02-01 15:00:27 +01:00
Franck NijhofandGitHub 3b5fd4bd06 Enable Ruff TRY004 (#86811) 2023-01-30 14:06:52 +01:00
epenetandGitHub 561fc2d771 Remove deprecated unit system properties (#86643)
* Remove deprecated unit system properties

* Fix tests
2023-01-27 12:09:26 +01:00
0ccab19d2c Add Mega Joule as valid unit of energy (#86055)
* Add Mega joule

* Reorder valid energy types

Alphabetical

* Add Mega Joule

* Add Mega Joule as valid energy unit

* Add Mega Joule

* Add Mega Joule as a Unit of Measurement to Energy

* Update tests

* Update tests

* Update number docstring

Co-authored-by: Roving Ronin <108674933+Roving-Ronin@users.noreply.github.com>
Co-authored-by: Erik Montnemery <erik@montnemery.com>
2023-01-25 14:13:42 +01:00
Marc MuellerandGitHub 8abce25948 Update Union typing (4) [Py310] (#86427) 2023-01-23 09:04:40 +01:00
Marc MuellerandGitHub 45b4b0e990 Import ParamSpec from typing [Py310] (#86413)
* Import ParamSpec from typing [Py310]

* Update additional imports
2023-01-23 07:28:43 +01:00
Franck NijhofandGitHub 79b52a2b41 Stricter pylint message control (#86154) 2023-01-20 13:47:55 +01:00
Erik MontnemeryandGitHub f93bbd55ba Improve logging shutdown (#85812)
* Improve logging shutdown

* Update test
2023-01-17 20:06:25 +01:00
Erik MontnemeryandGitHub b0d4b73874 Add unitless unit converter (#85694)
* Add unitless unit converter

* Adjust type hints

* Adjust tests

* Rename to UnitlessRatioConverter
2023-01-12 09:20:00 +01:00
Franck NijhofandGitHub 7adb8d5ddc Code styling tweaks to core utils & YAML loader (#85433)
Code styling tweaks to core utils
2023-01-08 22:01:55 -08:00
J. Nick KostonandGitHub d81febd3f4 Small speed up to frequently called datetime functions (#85399) 2023-01-08 09:42:29 -10:00
SukramJandGitHub 005bc8994d Add mA to SensorDeviceClass.CURRENT units (#84492)
fixes undefined
2022-12-30 13:55:14 +01:00
Jan BouwhuisandGitHub b2388b74d9 Add mV as a unit for voltage and enable conversions (#84805)
fixes undefined
2022-12-30 12:51:05 +01:00
epenetandGitHub 6261994fcf Enable unit conversion for DATA_SIZE (#84699) 2022-12-29 09:58:15 +01:00
epenetandGitHub 46e02ebaff Enable unit conversion for DATA_RATE (#84698) 2022-12-28 21:49:00 +01:00
d4f69a3652 String formatting and max line length - Part 7 (#84532)
Co-authored-by: Martin Hjelmare <marhje52@gmail.com>
2022-12-27 11:18:56 +01:00
Marc MuellerandGitHub 19acbf0d2a Update types packages (#84381) 2022-12-21 19:15:55 +01:00
epenetandGitHub d6fc2d9452 Prevent use of deprecated units (#83384)
* Prevent use of deprecated units

* Adjust obsolete util

* More units

* Add FREQUENCY

* Adjust pylint ignore

* Add ELECTRIC (current/potential)

* Add TIME

* Adjust units
2022-12-21 15:24:11 +01:00
Michaël ArnautsandGitHub 7142b4ecac Fixes some grammar mistakes (#84283) 2022-12-20 11:10:31 +01:00
epenetandGitHub f4c13645d2 Add support for CCF volume unit (#84029) 2022-12-19 14:57:04 +01:00
epenetandGitHub 4a848e8222 Add stones to mass units (#83933) 2022-12-13 17:35:14 +01:00
Marc MuellerandGitHub 0e95875912 Improve ignore_variance typing (#83927) 2022-12-13 15:22:23 +01:00
epenetandGitHub 583b4aef07 Enable automatic conversion for pressures (#83525)
* Enable automatic conversion between bar and psi

* Fix tests

* Fix mazda tests

* Fix oncue tests

* Adjust US pressures

* Adjust metric pressures

* Adjust tests

* Adjust tests
2022-12-12 12:42:35 +01:00
epenetandGitHub b172abaeeb Add new atmospheric pressure device class (#83455)
* Add new atmospheric pressure device class

* Translations

* Automatic conversion

* Convert all pressure units
2022-12-09 08:02:53 +01:00
epenetandGitHub 91d6d620c2 Use new unit enums in alexa (#83409)
* Use new unit enums in alexa

* Adjust for mypy
2022-12-06 17:31:40 +01:00
epenetandGitHub eb13f2b724 Use UnitOfVolume in unit conversion (#83292) 2022-12-05 16:56:05 +01:00
epenetandGitHub 4ad9633dfb Use new unit enums in weather entity (#82937) 2022-11-29 18:10:31 +01:00
epenetandGitHub dd960c4e62 Add precipitation device class (#81145) 2022-11-18 12:41:33 +01:00
Aarni KoskelaandGitHub 8038485ca4 Use partition instead of split where possible in core (#81806) 2022-11-15 21:45:48 +01:00
Aarni KoskelaandGitHub fa9a51e528 Use os.path.dirname() for getting the directory name of a path (#81504) 2022-11-14 15:21:35 +01:00
Marc MuellerandGitHub 0c8eeaa643 Update mypy to 0.990 (#81783)
* Update mypy to 0.990

* Remove type ignore - overriding attr with property (13475)

* Remove type ignores - hasattr (13544)

* Adjust type ignore - assignment (13549)

* New error code - type-abstract (13785)

* Disable annotation-unchecked (13851)
2022-11-08 14:41:39 +01:00
J. Nick KostonandGitHub 1589c06203 Significantly reduce clock_gettime syscalls on platforms with broken vdso (#81257) 2022-10-31 09:35:08 -04:00
epenetandGitHub 2a2e097e17 Use unit enums in unit utilities (#81030) 2022-10-26 20:47:17 +02:00