feat: add Appendix F, provider conformance (TCK)

[aepfli]

Implements the spec-repo half of #417, the cross-language tracking issue for the provider conformance suite. Java implementation: java-sdk-contrib#1829.

What

Adds Appendix F: Provider Conformance (TCK) — a language-agnostic conformance suite for validating OpenFeature providers, plus the three artifacts it rests on.

Marked experimental and explicitly non-normative. This is a proof of concept and the scenario set is a representative subset, not exhaustive coverage.

Why

OpenFeature's central promise is that swapping providers does not change application behaviour, and nothing currently verifies it. Every provider tests itself differently, against its own backend, with its own harness — so "implements the provider contract" is an unverified claim, and a behavioural difference between two providers gets discovered by the application that trips over it.

Why a new appendix rather than extending Appendix B

They answer different questions:

	Appendix B	Appendix F
Subject under test	the SDK	the provider
Harness	an in-memory provider as a stand-in	the provider under test, against a real backend

Appendix F also needs more than Gherkin. The scenarios, the flag set they assume, and the control API that produces outages are three co-dependent artifacts — a feature file that evaluates boolean-flag is meaningless without the flag definition, and a disconnect scenario is meaningless without the endpoint that produces the disconnect. An OpenAPI document has no home under "Gherkin Suites".

Both appendices now cross-reference each other.

Contents

• specification/appendix-f-provider-conformance.md — the appendix, written for implementers in other languages: what the artifacts are, the control-API contract and its two invariants, the capability tag vocabulary, and the six things a language implementation owns
• specification/assets/provider-tck/gherkin/ — four feature files (evaluation, errors, events, lifecycle)
• specification/assets/provider-tck/flags/canonical-flags.json — the flag set every scenario assumes
• specification/assets/provider-tck/openapi/control-api.yaml — what a backend under test must expose
• specification/assets/provider-tck/README.md — a landing page for the directory, matching the convention of assets/gherkin/README.md

Assets live under assets/provider-tck/ rather than alongside the existing SDK Gherkin, because assets/gherkin/evaluation.feature already exists and means something different.

A .gitattributes normalises these files to LF. They are consumed byte for byte by every language's TCK and copied verbatim into published build artifacts, so a Windows checkout must not produce a different packaged file than a Linux one.

Design decisions worth reviewing

Capabilities. Not every provider implements every optional part of the contract. Scenarios exercising an optional capability carry a tag; a provider declares what it supports and the rest are reported as skipped with a reason, never as passed. The rule behind it: a conformance suite that quietly goes green on scenarios it did not run is worse than no suite at all.

@strict-numeric-typing is not really optional. The spec requires TYPE_MISMATCH when the requested type cannot be satisfied, and narrowing 0.5 to 0 loses information silently. It is a capability only so a provider with this defect can adopt the suite today and see the gap reported explicitly. Not declaring it is an admission of a known bug.

No container restarts. Unavailability is simulated inside the running stack via the control API. Orchestrators assign host ports dynamically and cannot reliably preserve them across a restart, so restarting silently invalidates every provider pointed at the old port — and the failure looks like a flaky provider.

In-process control is a narrow carve-out. Providers with no backend (in-memory, environment-variable, file-based) may be driven in-process. Providers with a backend must use the control API; reaching an external backend through a side channel produces a suite that passes while proving nothing.

Open questions

• Normative status. Nothing here is expressed as a numbered requirement. Whether the control-API contract and the capability vocabulary should become normative sections is a TSC decision, and is recorded in the appendix as an open question rather than assumed.
• Evaluation context passthrough is not covered — it needs an echo operation on the control API (GET /last-evaluation or similar). Until then a provider that silently drops the context passes. The @targeting tag is reserved for those scenarios.
• Finer-grained flag manipulation (set/remove an individual flag) would need new control-API endpoints.
• Hooks, flag metadata and caching are not covered.

Reference implementation

tools/provider-tck in open-feature/java-sdk-contrib#1830, adopted by the flagd provider for both its RPC and in-process resolvers. A follow-up PR there switches that module to consume these files from this repo via submodule, and verified the artifacts round-trip byte-identical.

Which of #417's open questions this answers

Q1, directory layout — proposes specification/assets/provider-tck/{gherkin,flags,openapi}/. The subdirectory is forced rather than chosen: assets/gherkin/evaluation.feature already exists and tests an SDK, not a provider.

Q4, @strict-numeric-typing as capability vs hard requirement — keeps it a capability so providers with the defect can adopt today, but the appendix says in prose that it is not an optional feature and that not declaring it is an admission of a known bug. The "known deviations" idea from #417 is left open.

Left open, unchanged by this PR: Q2 versioning, Q3 canonical flag-set format, Q5 context passthrough, Q6 reason values, Q7 conformance claims. Q5 is also recorded in the appendix's own "Open questions" section.

What the suite has found so far

Two real defects, from the outside, which is the argument for the whole exercise:

• flagd silently narrows a float flag to an integer — the finding already described in [Tracking] A cross-language conformance suite for OpenFeature providers #417.
• The Java SDK's MultiProvider swallows child provider events — it extends EventProvider but never subscribes to its children, so PROVIDER_CONFIGURATION_CHANGED, PROVIDER_ERROR and PROVIDER_STALE never reach the client. Independently reproduced by running the suite against it; already tracked as open-feature/java-sdk#1882 (gap 1, High), where it had been found by hand-comparing against the js-sdk reference.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}