feat(provider-tck): emit a machine-readable conformance report

[aepfli]

Makes the TCK emit a machine-readable conformance report.

Stacked on #940, which adds the suite this reports on. Part of open-feature/spec#424; the schema is open-feature/spec#425.

$ PROVIDER_TCK_REPORT_DIR=./reports go test ./...
$ jq '.scenarios | group_by(.outcome) | map({(.[0].outcome): length}) | add' reports/in-memory.json
{
  "passed": 24,
  "not-declared": 5
}

Why Go first

Because Go is the language that needs it most. godog counts a capability-gated skip in its passed tally:

29 scenarios (29 passed)

Five of those twenty-nine did not run. Appendix F is unambiguous — a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed — and the harness does say so in a separate log line. But the headline number still says something false, and the number is what gets read. pytest and jest-cucumber both report skips correctly, so this is a property of the runner, not of the suite's design.

This does not fix godog's summary. It makes the summary stop mattering: every scenario's outcome is recorded individually, so a consumer can check the rule instead of trusting the runner to have applied it. The same run above accounts for all twenty-nine and calls five of them not-declared, each with its reason.

What identifies a report

tck.specRevision and tck.assetsTree come from a generated revision.go, which sync_assets.go writes from the submodule beside the embedded artifacts. Generating both in one command is what keeps them honest — make provider-tck-assets-check regenerates and fails on any difference, so a revision that disagrees with the artifacts next to it cannot be committed. The check is widened here to cover the generated file, which it would otherwise have missed.

Both the commit and the tree hash are recorded. The tree identifies the artifacts alone: unrelated edits elsewhere in the specification leave it untouched, so two runs that executed identical artifacts agree even when pinned to different commits, and it is checkable because git rev-parse <specRevision>:specification/assets/provider-tck must reproduce it.

What identifies a scenario

feature and name together do not. Every row of a Scenario Outline shares one name, and the type-mismatch matrix in errors.feature is eleven rows, so the report emitted eleven entries that differed only in durationMs:

{ "feature": "errors", "name": "Requesting the wrong type returns the code default", "outcome": "passed", "durationMs": 0.21 }

If one row failed and ten passed, that report could not say which failed, and a consumer keying on the pair kept whichever row it read last.

A row is identified by its parameters, which the schema now carries in example — the Examples row it came from, keyed by column header:

$ jq -c '.scenarios[] | select(.name | startswith("Requesting the wrong type")) | .example' reports/in-memory.json
{"default":"false","key":"string-flag","requested":"Boolean"}
{"default":"1","key":"string-flag","requested":"Integer"}
{"default":"0.1","key":"string-flag","requested":"Float"}
...

The values are the cells verbatim, as strings. Gherkin has no types, so "1" stays "1": coercing it would be this implementation inventing a fact the feature file did not state, and four implementations would each invent a different one. The field is present only for outline rows and omitted otherwise.

A field rather than a naming convention, because the parameters are the identity and they come from the feature file rather than from any runner. Mandating a mangled name instead would put a separator, an ordering and an escaping rule into normative text that four languages have to reproduce byte for byte, and drift there is invisible until two reports silently fail to line up. That is not hypothetical: before the field existed, implementations were observed diverging on exactly this — one emitting the bare scenario name for all eleven rows, another appending its runner's example id, a third its runner's expanded title.

A skipped row carries it too. The capability gate records its outcome before the scenario starts, so the four rows of the @object outline would otherwise be four not-declared entries differing in nothing — exactly as ambiguous as four failures.

How the row is recovered

godog's hooks receive an already-expanded pickle: the parameters are substituted into the step text and the row is otherwise gone. What survives is AstNodeIds, whose last entry, for a pickle compiled from an outline, is the id of the Examples TableRow. So the row is recovered by parsing the embedded feature files a second time and indexing every TableRow by that id.

Those ids are not intrinsic to a document. They come from a counter godog creates once per run and shares across every file it parses, so reproducing them means reproducing godog's parse — the same files in the same order, with pickle compilation in between, because compiling pickles draws from the same counter. That is a coupling to godog's internals, and it is a deliberate one; the alternative is forking the pickle compiler.

It is not left to be trusted. A pickle that came from an outline and did not resolve fails the run, so a godog release that renumbers nodes breaks the build loudly instead of quietly returning to the ambiguity the field exists to remove. Deliberately desynchronising the counter locally produced exactly that failure, which is how the guard was checked.

gherkin/go/v26 moves from an indirect requirement to a direct one. It is the same module and version godog already builds against, so no dependency is added and no go.sum entry changes.

A related fix

The capability gate's own bookkeeping was keyed by scenario name and is now keyed by pickle id, for the same reason. Gherkin allows an Examples block to carry its own tags, so two rows of one outline can differ in whether the gate stops them; keyed by name, gating one row suppressed the after hook for every row, and the rows that did run would have vanished from the report entirely. No feature file in the current assets triggers this, so it was latent rather than observed.

Smaller decisions

• Opt-in through the environment, not Config. Emitting a report is a property of the run rather than of the code: CI sets the variable, a local run does not, and no adopter changes a line to publish one. Several suites in one binary each write their own file, so flagd's two resolvers do not collide.
• The provider is identified by its own metadata name, with Config.Name recorded as the configuration. Config.Name is chosen to read well in a failure message (flagd-rpc), which makes it the configuration; a provider with two materially different modes produces two reports that are not interchangeable.
• How the backend was driven is read through an optional interface, not a new BackendControl method, so adding it breaks no existing implementation and a control that does not implement it simply omits the field. (HTTPControl lives on the OFREP branch and gains it there.)
• Outline rows are listed in table order, not sorted by their parameter values, so a report reads next to the Examples table it came from.

The tests earned their place immediately

TestReportNeverCallsASkippedScenarioPassed asserts that no gate-stopped scenario is ever reported as passed, and that every scenario is accounted for exactly once — a report that silently omitted what it skipped would satisfy the first half while still misleading a reader.

It caught this emitter recording every skipped scenario twice, the second time as passed, because godog does not deliver the before hook's ErrSkip to the after hook: err arrives nil there, indistinguishable from a scenario that ran and passed.

TestOutlineRowsAreDistinguishable asserts the property example exists to establish: (feature, name, example) is unique across every scenario in a report. TestSkippedOutlineRowsCarryTheirExample covers the gate path separately, since it fills the field in from a different place.

Verification

• go build, go vet, gofmt clean; all self-tests pass.
• All emitted reports validate against the schema in feat: add a schema for machine-readable provider conformance reports spec#425 with a Draft 2020-12 validator (jsonschema 4.10.3), including one whose outline rows are not-declared.
• The eleven rows of the type-mismatch matrix carry eleven distinct example objects, matching the three Examples tables in errors.feature cell for cell and in table order.
• Non-outline scenarios omit example entirely; the four skipped @object rows each carry theirs.
• make provider-tck-assets-check passed on a clean regenerate when the report was first added. The example change touches no embedded asset and no generated file, so nothing about that check moves.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}