conformance

httpsuite ↔ ijhttp conformance harness

This harness answers one question empirically: do httpsuite and the JetBrains HTTP Client CLI (ijhttp) produce the same pass/fail results for the same .http files?

It runs every .http file under tests/ through both tools, converts each tool's output to JUnit XML, and diffs the two reports by test name. A green run means httpsuite matches JetBrains on that suite.

Verified results

Run against real ijhttp (IntelliJ HTTP Client CLI, GraalJS engine, JDK 25):

• tests/ — CONFORMANT. All tests agree between httpsuite and ijhttp across:
  • status codes, JSON body access, cross-request client.global state
  • response.headers.valueOf/valuesOf, response.contentType
  • ES language features (regex lookahead, named capture groups, per-iteration let, destructuring, base64 btoa/atob)
  • pre-request scripts (< {% %}) and request.variables.set
  • response.cookies() / cookiesByName()
  • XML response.body as a DOM (getElementsByTagName, …)
  • the request object inside a response handler (request.method, body())
  • the jsonPath(body, "$.path") helper
• extensions/ — httpsuite-only. crypto.* and jwt are not defined in stock ijhttp (ReferenceError: crypto is not defined), verified against the GraalJS engine. These are additive httpsuite features, kept out of the parity verdict in extensions/.

Bottom line: across the JetBrains HTTP Client scripting surface — including the DOM and jsonPath, which turned out to be shared, not httpsuite-specific — httpsuite is a faithful match. It adds crypto/jwt on top.

ijhttp emits per-request lifecycle testcases (Response, Response Handler, Pre-request Handler) that have no httpsuite equivalent; the harness filters these so only real client.test assertions are compared.

How it works

1. Builds httpsuite and the bundled example server, and starts the server on 127.0.0.1:8799 (the @base hard-coded in the test files).
2. For each tests/*.http:
   • runs httpsuite --report <xml> <file> and parses the JUnit report;
   • runs ijhttp <file> --report and parses its JUnit report;
   • compares the two by test name.
3. Prints per-test agreement and a final verdict, exiting non-zero on any divergence.

Both tools emit JUnit XML natively, so the comparison is structured, not text-scraping: httpsuite grew a --report flag for exactly this, matching ijhttp's --report.

Running

# from the httpsuite module root
go run ./conformance

Without ijhttp on PATH, the harness runs the httpsuite side only and prints its results, then notes that the comparison was skipped (exit 0). Once ijhttp is available it performs the full side-by-side diff.

Flags:

Flag	Default	Purpose
--dir	conformance/tests	directory of .http files
--ijhttp	ijhttp on PATH	path to the ijhttp binary or a wrapper; empty skips the comparison
--addr	127.0.0.1:8799	example-server address (must match @base in the files)
--httpsuite	(build from source)	path to a prebuilt httpsuite binary
--keep	false	keep the temp work dir (reports, binaries) for inspection

Getting ijhttp

ijhttp requires a JDK (25+). Two common ways to provide it:

Homebrew (macOS/Linux) — simplest; runs natively and can reach the local server directly:

brew install ijhttp
go run ./conformance          # ijhttp is auto-detected on PATH

Docker — via JetBrains' published image. Wrap it so it presents the ijhttp <file> --report interface the harness expects, then:

go run ./conformance --ijhttp ./ci/ijhttp-docker.sh

Note that in a container the host server at 127.0.0.1:8799 may not be reachable without host networking (--network host on Linux, or host.docker.internal on macOS/Windows — which would also require the test files' @base to point there). The Homebrew path avoids this entirely.

Interpreting the output

• ✓ name httpsuite=PASS ijhttp=PASS — the tools agree. Good.
• ✗ name ... <-- MISMATCH — same test, different verdict. This is the signal that matters: a likely httpsuite bug (or a genuine behavioural difference worth investigating).
• ! name ... <-- HTTPSUITE ONLY / IJHTTP ONLY — a test produced by one tool and not the other. Usually a naming/coverage difference (e.g. httpsuite exposes an API JetBrains doesn't, or a script errored in one engine so its tests never registered), not necessarily a bug — but worth a look.

Writing conformance files

Keep the test files to the JetBrains HTTP Client API that both tools support so a green run is meaningful:

• Register checks with client.test(name, fn) + client.assert(...) — not httpsuite's # @expect (which ijhttp doesn't understand).
• Use @base = http://127.0.0.1:8799 and {{base}} — in-file variables work in both tools, and the hard-coded port matches the example server.
• Prefer deterministic assertions (known response fields, known crypto vectors).

The current suite covers status codes and JSON body access, cross-request client.global state, response headers/content type, a spread of ES language features (regex lookahead, named groups, per-iteration let, destructuring, base64), and HMAC test vectors. Add files as coverage grows; each is compared independently.

CI

The harness exits non-zero on any divergence (or run failure), so it drops straight into a pipeline step once ijhttp is provided:

go run ./conformance --ijhttp "$(command -v ijhttp)"