Add upstream proxy

[eraow]

Note: This PR was generated with AI assistance.

Summary

Adds support for routing httpjail's outbound requests through an upstream proxy using the standard HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables.

Motivation

I want to run httpjail behind a corporate proxy. Without upstream proxy support, httpjail connects directly to destination hosts, which fails in environments where direct egress is blocked.

What this does

• Uses HTTP_PROXY for HTTP destinations and HTTPS_PROXY for HTTPS destinations.
• Uses NO_PROXY to select destinations that are contacted directly, following curl 8.14.1: domains match at a label boundary along with their subdomains, a list of exactly * disables proxying, and IP or CIDR entries apply to destinations written as an IP literal. Deliberate divergences are listed in the docs.
• Also accepts the lowercase http_proxy, https_proxy and no_proxy variants. The uppercase spelling wins, consistent with the rest of the codebase; curl prefers the lowercase one and this is documented as a divergence.
• Supports http:// and bare host:port proxy addresses, with Basic authentication through the proxy URL. Reaching the proxy itself over TLS (https://proxy) is not supported and is rejected with an explicit error.
• Sends HTTPS requests through a CONNECT tunnel and forwards plain HTTP requests in absolute-form.
• Never sends Proxy-Authorization to a destination that NO_PROXY bypasses, nor to an HTTPS destination, whose request travels inside the tunnel to the origin server.
• Applies timeouts to proxy setup operations without limiting established long-running connections such as WebSocket or gRPC.
• Removes the parent's proxy environment (HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, NO_PROXY, both spellings) from jailed processes. Inheriting NO_PROXY in particular let a jailed process reach the named hosts directly, with no rule evaluation at all.
• Keeps upstream proxy initialization logs at debug level so normal CLI output is unaffected.

Proxy configuration is environment-only; no additional command-line option is introduced. Behavior is unchanged when no proxy environment variable is configured.

Usage

Route HTTPS requests through a corporate proxy:

HTTPS_PROXY=http://proxy.example.com:8080 \
  httpjail --js-file rules.js -- curl https://github.com

Route both HTTP and HTTPS requests through the same proxy:

HTTP_PROXY=http://proxy.example.com:8080 \
HTTPS_PROXY=http://proxy.example.com:8080 \
  httpjail --js-file rules.js -- ./my-app

Proxy credentials are supported:

HTTPS_PROXY=http://user:pass@proxy.example.com:8080 \
  httpjail --js-file rules.js -- curl https://github.com

Reach internal hosts without going through the proxy:

HTTPS_PROXY=http://proxy.example.com:8080 \
NO_PROXY=api.internal.example,10.0.0.0/8 \
  httpjail --js-file rules.js -- ./my-app

Manual verification

Given rules allowing only github.com:

const DOMAINS = [
  "github.com",
];

(function () {
  const h = (r.host || "").toLowerCase();
  for (const d of DOMAINS) {
    if (d[0] === ".") {
      if (h === d.slice(1) || h.endsWith(d)) return true;
    } else if (h === d) {
      return true;
    }
  }
  return false;
})();

Allowed request

$ sudo env -u SUDO_UID -u SUDO_GID \
    HTTPS_PROXY=http://proxy.example.com:8080 \
    target/debug/httpjail -vv \
    --js-file rules.js \
    --request-log jail.log \
    curl -s -o /dev/null -w "%{http_code}\n" https://github.com
...
DEBUG httpjail: Routing httpjail upstream requests through the proxy environment
DEBUG httpjail::proxy: Upstream client initialized to route through the upstream proxy
...
200
...

Blocked request

$ sudo env -u SUDO_UID -u SUDO_GID \
    HTTPS_PROXY=http://proxy.example.com:8080 \
    target/debug/httpjail -vv \
    --js-file rules.js \
    --request-log jail.log \
    curl -s -o /dev/null -w "%{http_code}\n" https://example.com
...
403
...

NO_PROXY bypass

This is a sanitized transcript from a reachable internal service; the proxy and
service host names have been replaced with reserved example domains.

$ HTTPS_PROXY=http://proxy.example.com:8080 \
    NO_PROXY=api.internal.example \
    ./target/debug/httpjail --weak -vv --js "true" -- \
    curl -s -o /dev/null -w "%{http_code}\n" https://api.internal.example
...
DEBUG httpjail::upstream: Bypassing upstream proxy for api.internal.example
...
302

The 302 is our internal service's redirect response. Together with the
debug line, it confirms that httpjail reached the service directly instead of
sending the request to the configured upstream proxy.

The jailed process does not inherit that value:

$ NO_PROXY=api.internal.example ./target/debug/httpjail --weak --js "true" -- \
    env | grep -i no_proxy
NO_PROXY=localhost,127.0.0.1,::1
no_proxy=localhost,127.0.0.1,::1

[ammario]

Why shouldn't httpjail itself respect the standard HTTP_PROXY variables? I think it's clear on its face it wouldn't pass that down to children (as that would invalidate the whole point of the jail).

[eraow]

Agreed. For the first release, I’ll remove --upstream-proxy and HTTPJAIL_UPSTREAM_PROXY and use the standard HTTP_PROXY / HTTPS_PROXY environment variables for httpjail’s own egress.
Adding a dedicated CLI option and httpjail-specific env var creates an extra configuration path before we have a concrete need for it. If we later need an explicit per-invocation override, we can add it in a follow-up change.

[eraow]

Done in 701d116.

[eraow]

@ammario following up on 701d116, which drops --upstream-proxy / HTTPJAIL_UPSTREAM_PROXY in favor of the standard HTTP_PROXY / HTTPS_PROXY for httpjail's own egress.

One question: did you intend for the dedicated flag and env var to be removed entirely, or kept as an explicit override alongside HTTP_PROXY? I went with full removal, but it's easy to bring back either way. Let me know how this looks.

[ammar-agent]

I reviewed this against the latest main (f66518b). The core approach—a Hyper connector that preserves streaming, pooling, and unbounded established connections—is reasonable, so I don't think the raw line count alone warrants a rewrite. I do think a few changes are needed before merging:

1. NO_PROXY / no_proxy is not honored. With HTTP_PROXY or HTTPS_PROXY set, every destination of that scheme is proxied, including internal hosts that a standard corporate proxy environment expects to connect to directly. Please either implement bypass matching or explicitly narrow the documented semantics.
2. Proxy configuration is silently process-global through HTTPS_CLIENT.get_or_init. If one ProxyServer initializes the client, a later call to the public new_with_upstream_proxies constructor can ignore its proxy argument. Keeping the client on ProxyServer / ProxyContext would make the configuration honest and remove hidden global state.
3. The most intricate path lacks an end-to-end test: HTTPS destination through an upstream CONNECT proxy, followed by destination TLS and an HTTP response. The existing tests separately cover plain HTTP proxying and the CONNECT exchange.
4. read_connect_status performs one async read per byte. That avoids swallowing tunnel payload, but is contrary to the project's minimal-latency goal. Please use buffered reads while preserving any bytes read beyond the CONNECT headers.

For complexity cleanup, the manual RunArgs Debug implementation appears to be residue from the removed dedicated proxy option and can return to a derive. Test-only/public helpers such as ProxyConnector::new and UpstreamProxies::all could also be narrowed. If TLS-to-proxy (https://proxy) is not a demonstrated requirement for the first release, deferring it would remove a meaningful portion of the stream-erasure and second-TLS-config machinery while still supporting HTTPS destinations through an ordinary HTTP proxy.

Local verification: formatting and cargo clippy --all-targets -- -D warnings pass; all unit tests pass. The full suite encountered the unchanged parallel fixed-port collision in weak_integration_max_tx_bytes.

— Codex, AI review agent