Feat/upstream proxy hostname passthrough

[fchinch]

Builds on #2142 — that PR routes the egress proxy through an upstream, this one covers the case where the upstream must resolve the name itself.

Problem

The egress proxy resolves and pins every target before dialing, then asks the upstream to CONNECT to the pinned IP. That is the right default, but it assumes the name resolves here and that the upstream accepts an address. Two deployments where it does not:

• the upstream fronts a network whose hostnames do not resolve outside it, so resolve_and_pin fails before the upstream is ever consulted;
• the upstream enforces hostname ACLs and refuses CONNECT to a bare IP.

Today the only way through is CRAWL4AI_ALLOW_INTERNAL_URLS=true, which drops the entry-point check for every destination.

Change

CRAWL4AI_UPSTREAM_PROXY_DNS_SUFFIXES — a comma-separated allowlist of DNS suffixes (e.g. .corp.example,.internal). A name matching a listed suffix is handed to the upstream unresolved; the upstream then decides where it lands. Everything else is unchanged.

The same list exempts those names from the entry-point URL check, which runs before the browser starts — without that they would be rejected there and never reach the proxy.

Wildcard entries are rejected. Only explicit DNS suffixes are accepted. Delegating every name would turn any caller-supplied URL into a lookup performed by the upstream, which is the one shape this must not allow.

Behaviour

Configuration	Behaviour
Variable unset	Unchanged — resolve_and_pin() for everything
Explicit suffix, port 80/443	Hostname delegated to the upstream
Host outside the list	resolve_and_pin()
IP literal	Never delegated, whatever is listed
Port outside the allowed set	Never delegated — falls back to resolve_and_pin()
Wildcard entry	Ignored, with a warning
Invalid suffix	Ignored, with a warning

Names are matched on the DNS label boundary and in normalised form — case-folded, root dot stripped, IDNA-encoded, length checked before and after encoding — so .corp.example covers wiki.corp.example and wiki.corp.example. but never corp.example.attacker.com or notcorp.example.

Delegated names are reachable on ports 80 and 443 only, configurable via CRAWL4AI_UPSTREAM_PROXY_DNS_PORTS. On the pinned path the port is unconstrained but the address must be global, so a non-web port still only reaches the public internet. Passthrough gives up exactly that address check, which leaves the port as the only thing still narrowing where a listed name can land — without it, a listed suffix could reach :9200, :5432 or :2375 rather than a web server. An explicit list replaces the default rather than extending it.

What you are trusting

For a delegated name there is no pin, so the rebinding and non-global checks do not apply to it — that is the point of the mode, and worth being explicit about. A subdomain under a listed suffix reaches whatever its DNS resolves to, including a private address, because we neither resolve nor pin it. The connection is still only ever handed to your configured upstream and never dialed directly, so this cannot reach the crawler's own network — but it can reach whatever the upstream is willing to reach, on the allowed ports.

This is strictly narrower than CRAWL4AI_ALLOW_INTERNAL_URLS=true, which an operator in this situation has had to reach for anyway: that drops the entry-point check for every destination, while the suffix list confines the relaxation to named hosts, keeps IP literals blocked, restricts ports, and never dials a target directly.

Tests

deploy/docker/tests/test_security_egress_proxy.py — 60 passing.

• TestHostnamePassthrough — a listed name reaches the upstream unresolved; a name outside the list stays pinned; an IP literal never bypasses the pin; an unset variable changes nothing.
• TestSuffixMatching — equivalent spellings (case, trailing root dot, IDNA) all match; near-misses (corp.example.attacker.com, notcorp.example) stay pinned; DNS label boundaries; wildcards dropped rather than honoured, without poisoning the rest of the list; listing an IP literal does not bypass the pin.
• TestPassthroughPorts — 80/443 delegated by default; 22/2375/5432/6379/8080/9200 fall back to the pin; an operator can widen the list; a set-but-unusable list closes rather than opens; blank means unset.
• Cross-layer consistency — the entry-point check and the proxy are asserted together on the same names and ports, so they cannot drift into disagreeing. Both import the same normalize_host() / passthrough_suffixes() / passthrough_ports() rather than repeating the parsing.

The one failure in that file (test_enforce_egress_sets_proxy) is pre-existing and unrelated — it needs the crawl4ai package installed.