* fix: apply the SSRF checks to redirect targets on every web fetch path
Two guards protect server-side fetches: a private-IP check and the operator's `WEB_FETCH_FILTER_LIST`. Neither reached a redirect hop on the aiohttp paths, and the filter list never reached one on the requests paths either.
aiohttp answers IP-literal hosts itself without consulting a resolver, so `_SSRFSafeResolver` was never invoked for a hop such as `http://169.254.169.254/` and the private-IP check simply did not run. With redirect following enabled, a submitted public URL that redirects to an IP literal reached loopback, RFC1918 and cloud-metadata addresses, and the response body was returned to the caller. The filter list was consulted only in `validate_url`, on the originally submitted URL, so a redirect to a filter-listed host was fetched without it ever being applied.
`_SSRFSafeResolver` is replaced by `_SSRFSafeConnector`, which hooks `_resolve_host` so the IP check also covers the IP-literal shortcut and both DNS cache paths. The filter list moves to a per-request hook on each transport, `connect()` for aiohttp and `send()` for the requests adapter, because those see the request destination: at the connection layer a proxied request presents the proxy's host, and a pooled connection skips resolution entirely. This covers every hop, including redirects, on all five aiohttp call sites and both requests sessions. The Playwright loader already validated each hop and is unchanged.
Both gaps required `AIOHTTP_CLIENT_ALLOW_REDIRECTS=true`, which is not the default.
Two behaviour changes for operators. The filter list now applies to redirect targets rather than only to submitted URLs. Under a forward proxy it is evaluated against the request destination instead of the proxy, which also fixes allowlist entries rejecting every fetch in proxied deployments.
* refac: match the web fetch filter list against resolved addresses
The filter list is now evaluated against the hostname together with the addresses it resolves to, at URL validation and on each connection, on both transports. An IPv6 address is also matched by the IPv4 address it carries.
* refac: screen outbound fetch addresses against reserved ranges ipaddress misses
`ipaddress.is_global` was the only test behind the web-fetch address check, and it answers a narrower question than "may we fetch this". Several special-purpose ranges are globally routable by registry while nothing on them is a legitimate destination, so they passed. Classification now screens those ranges on top of `is_global`, and applies the same screen to the IPv4 address embedded in an IPv6 transition encoding rather than only to the literal. All three checkpoints share the predicate, so they all inherit it.
The range list is the exact complement of what CPython's `ipaddress` already models, checked entry by entry against both IANA special-purpose registries. Prefixes IANA marks globally reachable are deliberately left out, so no real destination changes behaviour. Verified against 31 addresses covering every entry, their transition-encoded forms, and public controls in both families: 31/31 expected after, 18/31 before.
* refac: match web fetch filter entries that name an address or a range
A filter entry that parses as an address or a CIDR range is matched by containment rather than by DNS label suffix, so a range covers the addresses inside it and an address matches however it is spelled. A range entry previously matched nothing at all, silently.
The built-in list gains the special-purpose networks that ipaddress.is_global reports as reachable while nothing on them is a legitimate destination, so taking an address out of reach is a WEB_FETCH_FILTER_LIST change rather than a release. Those entries hold whether or not local web fetch is enabled; the private-address rule still follows the toggle.
Raising GLOBAL_LOG_LEVEL to WARNING buys quieter output but not less work: 241 INFO call sites interpolate their payload into an f-string before the logging call gets to drop it. The heaviest is get_doc, which logs every chunk id and metadata dict in a collection, so on the full-context retrieval path that is the entire knowledge base, once per chat request.
That one line at WARNING, CPython 3.12:
| knowledge base | payload | before | after |
| -------------- | ------- | -------- | ------- |
| top-k of 3 | 1.2 kB | 3.8 us | 0.07 us |
| 500 chunks | 201 kB | 583.6 us | 0.08 us |
| 5000 chunks | 2.0 MB | 5.8 ms | 0.15 us |
The lazy form log.info('query_doc:result %s %s', result.ids, result.metadatas) hands the payload to record.getMessage(), which the InterceptHandler only reaches once a record has passed the level check. Output at INFO is byte-identical. Two sites that already built their message eagerly, one str concat and one % operator, move to the same lazy form.
Add self-hosted OpenSERP as a web search engine option. OpenSERP
provides browser-rendered search across Google, Bing, Yandex, Baidu,
DuckDuckGo, and Ecosia with no API keys required.
- New module: retrieval/web/openserp.py (async, uses aiohttp session pool)
- Config: OPENSERP_BASE_URL env var (defaults to http://localhost:7070)
- Routing: search_web() dispatch for 'openserp' engine
- Follows existing patterns (searxng, brave)
Co-authored-by: crustopher-lgtm <crustopher-lgtm@users.noreply.github.com>
The OAuth / OIDC section in Admin Settings > Authentication had no
enable/disable switch, unlike the LDAP section above it. Add one that
persists via the existing Save flow and actually gates OAuth sign-in,
mirroring how the LDAP toggle works.
- config: new ENABLE_OAUTH persistent config ('oauth.enable'), defaulting
to True so existing deployments with a provider configured keep working.
- oauth: expose ENABLE_OAUTH via the OAuth runtime config and reject the
login and callback handlers with 404 when it is disabled.
- /api/config: report no OAuth providers when disabled so the login page
hides the OAuth buttons (and cannot auto-redirect), without clearing the
admin's provider configuration.
- auths: expose ENABLE_OAUTH through the admin OAuth config get/update
endpoints (OAuthConfigForm + OAUTH_CONFIG_KEYS).
- Authentication.svelte: bind the OAuth / OIDC header Switch to the
persisted oauthConfig.ENABLE_OAUTH and collapse the section when off,
matching the LDAP header (size, weight, alignment).
With ENABLE_PERSISTENT_CONFIG=False the admin's model order is reset on every restart because ui.model_order_list falls back to its DEFAULT_CONFIG default, and unlike every other Models setting (DEFAULT_MODELS, DEFAULT_PINNED_MODELS, DEFAULT_MODEL_METADATA and DEFAULT_MODEL_PARAMS) that default was hardcoded to an empty list with no environment variable to source it from. This adds a MODEL_ORDER_LIST environment variable parsed as a JSON array using the same guarded pattern as the neighbouring DEFAULT_MODEL_METADATA and DEFAULT_MODEL_PARAMS defaults, falling back to an empty list on parse errors. Behaviour when the variable is unset is unchanged.
Fixes#27206
