feat(proxy): expose pipeline extensions for Responses payloads

[2001Y]

Description

Expose the canonical pipeline extension lifecycle to OpenAI Responses payloads so extensions can inspect or rewrite Codex /v1/responses traffic before it is sent upstream.

This update also preserves the X-Headroom-Bypass passthrough contract: bypassed /v1/responses requests skip payload-stage extension emission and are forwarded unchanged.

Refs #1307, #1342.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Performance improvement
• Code refactoring (no functional changes)

Changes Made

• Add PipelineEvent.payload and PipelineExtensionManager.emit(..., payload=...) support.
• Emit Responses payload stages for HTTP /v1/responses and WebSocket response.create frames.
• Preserve extension payload/header mutations when payload-stage extensions intentionally rewrite non-bypassed traffic.
• Skip Responses payload-stage extension emission when X-Headroom-Bypass is active, preserving byte-faithful passthrough.
• Add regression coverage for bypassed /v1/responses traffic with a mutating payload extension installed.

Testing

• Unit tests pass (pytest)
• Linting passes (ruff check .)
• Type checking passes (mypy headroom)
• New tests added for new functionality
• Manual testing performed

Test Output

$ python -m py_compile headroom/proxy/handlers/openai.py tests/test_proxy_byte_faithful_forwarding.py tests/test_openai_responses_context_compaction.py
passed

$ python -m pytest tests/test_proxy_byte_faithful_forwarding.py::test_openai_responses_bypass_skips_payload_extensions tests/test_openai_responses_context_compaction.py -q
14 passed in 1.37s

$ uvx ruff check headroom/proxy/handlers/openai.py tests/test_proxy_byte_faithful_forwarding.py
All checks passed!

$ uvx ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy_byte_faithful_forwarding.py
2 files already formatted

Real Behavior Proof

• Environment: macOS arm64, Python 3.11.14; local compiled headroom._core copied from an existing local build only to run tests in this fresh PR checkout.
• Exact command / steps: POST /v1/responses with x-headroom-bypass: true while a mutating Responses payload extension is installed.
• Observed result: the fake upstream _retry_request receives the original inbound payload unchanged, and the payload extension records no events.
• Not tested: full mypy headroom; live upstream OpenAI/Codex call.

Review Readiness

• I have performed a self-review
• This PR is ready for human review

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have updated the CHANGELOG.md if applicable

Screenshots (if applicable)

N/A.

Additional Notes

Docs and CHANGELOG are N/A for this proxy-internal lifecycle/bypass fix.

[github-actions]

PR governance

This PR follows the template and is marked ready for human review.

[JerrettDavis]

The payload hook is useful, but the HTTP /v1/responses path needs to preserve the existing bypass contract.

handle_openai_responses() now emits PipelineStage.INPUT_RECEIVED immediately after JSON parsing, before _headroom_bypass_enabled(request.headers) is checked. Any enabled extension can mutate body at that stage, and then a request carrying X-Headroom-Bypass is forwarded with those mutations. Bypass is supposed to be a full passthrough escape hatch for debugging and correctness-sensitive traffic, so extensions/compression should not get a chance to rewrite the payload first.

Please move the bypass check ahead of the new INPUT_RECEIVED emission, or explicitly skip payload-stage emission when bypass is active, and add a regression that a bypassed /v1/responses request is forwarded unchanged even when a payload extension is installed.

[2001Y]

Addressed the bypass review: HTTP /v1/responses now computes the bypass gate before INPUT_RECEIVED and passes bypass through payload-stage emission so mutating extensions cannot observe or rewrite bypassed traffic. I also added a regression that a bypassed /v1/responses request is forwarded unchanged with a mutating payload extension installed, and updated the PR body to the required template.\n\nVerification:\n- python -m py_compile headroom/proxy/handlers/openai.py tests/test_proxy_byte_faithful_forwarding.py tests/test_openai_responses_context_compaction.py — passed\n- python -m pytest tests/test_proxy_byte_faithful_forwarding.py::test_openai_responses_bypass_skips_payload_extensions tests/test_openai_responses_context_compaction.py -q — 14 passed\n- uvx ruff check headroom/proxy/handlers/openai.py tests/test_proxy_byte_faithful_forwarding.py — passed\n- uvx ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy_byte_faithful_forwarding.py — passed

[2001Y]

Follow-up: rebased the branch onto current origin/main so PR Governance should now run against the matching base workflow/script pair. Re-ran local verification after rebase:\n\n- python -m py_compile headroom/proxy/handlers/openai.py tests/test_proxy_byte_faithful_forwarding.py tests/test_openai_responses_context_compaction.py — passed\n- python -m pytest tests/test_proxy_byte_faithful_forwarding.py::test_openai_responses_bypass_skips_payload_extensions tests/test_openai_responses_context_compaction.py -q — 14 passed\n- uvx ruff check headroom/proxy/handlers/openai.py tests/test_proxy_byte_faithful_forwarding.py — passed\n- uvx ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy_byte_faithful_forwarding.py — passed\n- git diff --check — passed

[JerrettDavis]

Thanks for preserving the bypass contract. The HTTP /v1/responses path now computes bypass before INPUT_RECEIVED payload emission and passes that bypass flag through later payload stages, so a bypassed request is not visible to mutating extensions. The added regression captures the important behavior: no extension events and the forwarded body remains unchanged.

I also checked the WebSocket helper path: subsequent response.create frames return before payload-stage emission when _ws_bypass is set, so the same passthrough rule is maintained there. Local full-proxy tests cannot collect in this worktree because headroom._core is missing, but the code-level bypass issue from my prior review is resolved.

[2001Y]

Rebased this branch onto current upstream/main and force-pushed head d9bb8692.

Conflict resolution notes:

• Preserved the new /v1/responses upstream base URL handling from main.
• Preserved the Responses payload pipeline extension hooks.
• Kept the explicit bypass contract so bypassed requests are not exposed to payload extensions.

Verification:

• python -m py_compile headroom/proxy/handlers/openai.py tests/test_proxy_byte_faithful_forwarding.py tests/test_openai_responses_context_compaction.py tests/test_pipeline.py — passed
• VIRTUAL_ENV=~/.hermes/hermes-agent/venv python -m maturin develop -m crates/headroom-py/Cargo.toml — passed; headroom._core import OK
• python -m pytest tests/test_proxy_byte_faithful_forwarding.py tests/test_openai_responses_context_compaction.py tests/test_pipeline.py -q — 51 passed, 1 skipped
• GitHub PR checks — 2 passed / 0 failed