conformance harness for mcp

[filintod]

setup mcp conformance harness check. not wired to CI yet but there are make targets that would be easy to incorporate.

Here we are using typescript everything server as official conformance server still have a non-conformant ndjson issue that we are not lenient for nor need to.

it will be good to use golang sdk everything server as an additional server (same one used by github for example).

Part of #2707 . I started it before the issue was created but it can serve as base and add follow ups to complete 2707 requirements. This also need more development as more tests are available upstream (pending tasks https://github.com/agentgateway/agentgateway/pull/2790/changes#diff-066ad6e1750d98b93b7fed37a6591440f221d333f0623d675788937024118641R58)

last run status: https://github.com/filintod/agentgateway/blob/4737db1671870e960955f0411fe4543e71919047/crates/agentgateway/tests/conformance/status.md

Current status July-31st-2026 (status.md)

Scenario	Direct	Gateway	Details	Rationale
caching	pass (8 checks)	pass (8 checks)	—	—
http-custom-header-server-validation	pass (5 checks)	pass (5 checks)	—	—
http-header-validation	pass (5 checks)	pass (5 checks)	—	—
input-required-result-basic-elicitation	pass (3 checks)	pass (3 checks)	—	—
input-required-result-basic-list-roots	pass (3 checks)	pass (3 checks)	—	—
input-required-result-basic-sampling	pass (3 checks)	pass (3 checks)	—	—
input-required-result-capability-check	pass (2 checks)	pass (2 checks)	—	—
input-required-result-ignore-extra-params	pass (2 checks)	pass (2 checks)	—	—
input-required-result-missing-input-response	pass (2 checks)	pass (2 checks)	—	—
input-required-result-multi-round	pass (4 checks)	pass (4 checks)	—	—
input-required-result-multiple-input-requests	pass (3 checks)	pass (3 checks)	—	—
input-required-result-non-tool-request	pass (3 checks)	pass (3 checks)	—	—
input-required-result-request-state	pass (3 checks)	pass (3 checks)	—	—
input-required-result-result-type	pass (2 checks)	pass (2 checks)	—	—
input-required-result-tampered-state	pass (2 checks)	pass (2 checks)	—	—
input-required-result-unsupported-methods	pass (2 checks)	pass (2 checks)	—	—
input-required-result-validate-input	pass (3 checks)	pass (3 checks)	—	—
sep-2164-resource-not-found	pass (4 checks)	pass (4 checks)	—	—
server-stateless	pass (28 checks)	gap (1/28 checks)	server-stateless:sep-2575-server-unsupported-version-error (gap; gateway-attributed)	intentional proxy behavior (#2417): The gateway rejects unsupported versions before opening upstream connections, so it returns gateway-supported versions rather than probing every target in the 400 path.

[Copilot]

🟡 Not ready to approve

The tools/mcp-conformance.sh run entrypoint currently drops the provided OUT argument due to not shifting/passing script args into run(), so the documented OUT=... behavior won’t work as intended.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR introduces an opt-in MCP conformance harness that runs the official MCP Conformance Framework against a pinned TypeScript SDK “everything” server both directly and through AgentGateway, and generates versioned status artifacts for tracking gaps/regressions over time.

Changes:

• Add a make mcp-conformance* workflow (shell driver + Rust ignored tests) to run pinned official suites in direct and gateway topologies.
• Add inventory generation, expected-failure parsing, and a Python reporter (with unit tests) that produces status.json, status.md, and appends status-history.json.
• Pin upstream framework + TypeScript SDK revisions and commit initial inventory/status/baseline files.

File summaries

File	Description
tools/mcp-conformance.sh	Shell entrypoint for running/capturing/reporting conformance runs via cargo test + reporter.
Makefile	Adds mcp-conformance* convenience targets.
crates/agentgateway/tests/mcp_conformance.rs	Rust harness that orchestrates direct vs gateway suite runs and validates complete outputs.
crates/agentgateway/tests/conformance/framework.sha	Pins the MCP conformance framework revision.
crates/agentgateway/tests/conformance/typescript-sdk.sha	Pins the TypeScript SDK revision used as the reference server.
crates/agentgateway/tests/conformance/suite-inventory.json	Records suite/scenario inventory at the pinned framework revision (incl. gated pending).
crates/agentgateway/tests/conformance/generate-inventory.ts	Generates inventory using the framework’s own selectors/APIs.
crates/agentgateway/tests/conformance/parse-expected-failures.ts	Parses expected-failures files using the pinned framework parser.
crates/agentgateway/tests/conformance/expected-failures-direct-2025-11-25.yml	Baseline for accepted direct failures (empty at this pin).
crates/agentgateway/tests/conformance/expected-failures-direct-2026-07-28.yml	Baseline for accepted direct failures (empty at this pin).
crates/agentgateway/tests/conformance/expected-failures-direct-pending-json-schema-2020-12.yml	Baseline for accepted direct failures for the gated pending scenario (empty).
crates/agentgateway/tests/conformance/expected-failures-gateway-2025-11-25.yml	Baseline for known gateway gaps in the 2025-11-25 suite.
crates/agentgateway/tests/conformance/expected-failures-gateway-2026-07-28.yml	Baseline for known gateway gaps in the 2026-07-28 suite.
crates/agentgateway/tests/conformance/expected-failures-gateway-pending-json-schema-2020-12.yml	Baseline for known gateway gaps for the gated pending scenario (empty).
crates/agentgateway/tests/conformance/expected-failure-rationales.json	Adds human-readable rationale metadata for select baselined gaps.
crates/agentgateway/tests/conformance/report.py	Builds aggregated status outputs from suite results + baselines + rationales.
crates/agentgateway/tests/conformance/report_tests.py	Unit tests for report classification and rendering logic.
crates/agentgateway/tests/conformance/status.json	Committed example output of the reporter for the current pins/run.
crates/agentgateway/tests/conformance/status.md	Human-readable rendered status summary for the current pins/run.
crates/agentgateway/tests/conformance/status-history.json	Historical ledger of prior generated statuses (appended by reporter).
crates/agentgateway/tests/conformance/README.md	Documentation for setup, running suites, capture mode, inventory, and pin updates.
crates/agentgateway/tests/common/gateway.rs	Exposes wait_for_port with a configurable timeout for reuse by the harness.
.gitignore	Ignores Python __pycache__/ outputs from reporter/tests.

Review details

• Files reviewed: 22/23 changed files
• Comments generated: 2
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[howardjohn]

This PR is too complex to review or consider merging. We are ultimately running npx mcp-conformance but somehow we have thousands of lines of python, typescript, bash, and json.

I don't mind a simple script to solve the ~3-5 line of bash it takes to run this but this doesn't seem worth the complexity.

[howardjohn]

why do we have 2 places to put the same 'expected failures'?

[danehans]

This SHA is no not reachable from or fetchable from your fork.