Skip to content

test(flagd): run the provider conformance suite against both resolvers - #1608

Draft
aepfli wants to merge 2 commits into
feat/provider-tckfrom
feat/provider-tck-flagd
Draft

test(flagd): run the provider conformance suite against both resolvers#1608
aepfli wants to merge 2 commits into
feat/provider-tckfrom
feat/provider-tck-flagd

Conversation

@aepfli

@aepfli aepfli commented Aug 24, 2026

Copy link
Copy Markdown
Member

Stacked on #1606. Base is feat/provider-tck, so this diff shows only the new work. Review #1606 first.

Part of #1607 and open-feature/spec#417.

Nothing here has been built or run — no Node and no Docker on the authoring machine. See Verification.

What

The flagd provider under the conformance suite, both resolvers, plus HttpControl — the client for the backend control API — added to the TCK library.

HttpControl implements BackendControl and ConnectionControl over the global fetch, so it adds no dependency. It mirrors the Go client: prepareScenario() prefers POST /reset and falls back to POST /start?config=default on 404/501, caching that decision for the suite; after a disconnect it always uses /start, because /reset restores flag state and is not specified to start a stopped backend. flagd-testbed's launchpad serves only /start, /restart, /stop and /change, so the fallback is the normal path rather than an edge case.

baseUrl accepts a thunk, and that is necessary rather than decorative: runProviderTck is called at module load, but the launchpad's host port does not exist until beforeAll has started the stack.

Two spec files, because jest-cucumber accumulates step definitions in module state and two runProviderTck calls in one file would make every step ambiguous.

The finding: no Go-style asymmetry here

Go's flagd RPC resolver never emits PROVIDER_STALE while its in-process resolver does (go-sdk-contrib#939), so the Go adoption has to withhold the capability for RPC. JavaScript has no such split, and the reason is structural rather than lucky.

Neither resolver decides anything about staleness. Both report a lost connection through a disconnectCallbackservice/grpc/grpc-service.ts:274 and service/in-process/grpc/grpc-fetch.ts:197 — and the provider owns the whole transition in one place, flagd-provider.ts:130-148: emit Stale, arm a timer, escalate to Error when retryGracePeriod expires.

Because the contract exists once, the two resolvers cannot diverge. That is the shape Go needs, and it has been noted on go-sdk-contrib#939.

Both suites therefore declare Events, Stale, ConfigurationChange, Object, UnavailableInit. StrictNumericTyping is withheld from both, as it must be for every JavaScript provider — the language has no integer type, so the scenario is unsatisfiable by construction rather than through any defect. See the TCK README.

Verification

Check Result
build / lint / test not run — no Node, npm or npx available
the flagd suites not run — no Docker
SDK and provider APIs pinned against source: FlagdComposeContainer's start/stop/getLaunchpadUrl/getPort, the provider options used by the neighbouring e2e suite, and the disconnect seams cited above
launchpad endpoints confirmed against flagd-testbed/launchpad/main.go — only /start, /restart, /stop, /change; the /reset fallback is live
httpControl.spec.ts pins the request sequences against a stubbed fetch — the /reset/start fallback and its caching, and the start-after-disconnect rule — so the control client has coverage that needs no container

No CI will run on this PR while it is stacked: the workflow triggers on pull_request: branches: [main].

One deliberate omission: I did not add jest.retryTimes to the TCK specs, though the neighbouring flagd suites use 3. A conformance result that only holds on the third attempt is not a conformance result, so contention will surface as a failure rather than being retried away.

Worth filing against the JS flagd provider

  1. The offline/file mode has no staleness contract at all. service/in-process/file/file-fetch.ts never receives or calls disconnectCallback — its connect ignores the reconnect callback — so a provider using offlineFlagSourcePath emits neither PROVIDER_STALE nor a reconnect PROVIDER_READY. That may be intended for a file source, but it is undocumented and invisible.
  2. Doc/code mismatch on retryGracePeriod: configuration.ts:118-123 documents @default 5; constants.ts:6 sets DEFAULT_RETRY_GRACE_PERIOD = 10. Small, but it is the knob deciding how long a provider stays STALE before claiming ERROR.

@coderabbitai

coderabbitai Bot commented Aug 24, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

aepfli added 2 commits August 24, 2026 22:07
The suite's normative control path for any provider with a real backend, and
the piece the README listed as the one blocking a containerised adopter.

HttpControl implements both BackendControl and ConnectionControl over the
global fetch, so it adds no dependency. It mirrors the Go client's behaviours:
prepareScenario prefers POST /reset and falls back to POST /start?config=... on
a 404 or 501, caching that decision once per suite; and because /reset restores
flag state rather than starting a stopped backend, the scenario following a
disconnect always uses /start. The fallback is the normal path rather than an
edge case, since flagd-testbed's launchpad serves only /start, /restart, /stop
and /change.

The base URL may be a thunk. A control service's host port is mapped
dynamically and does not exist until the stack is up, while runProviderTck has
to be called at module load, so a plain string is usually impossible to supply.

httpControl.spec.ts pins the request sequence against a stubbed fetch, with no
container involved: the fallback rules are invisible from inside a scenario, so
a client that gets them wrong produces failures that look like provider
defects.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
flagd resolves flags two quite different ways, and they are separately
conformant: any difference between the two results is a difference an
application would see when it switches resolver. Two spec files rather than
one, because jest-cucumber accumulates step definitions in module state and two
runProviderTck calls in one file would make every step ambiguous.

The TCK drives flagd-testbed's launchpad through the standardised control API,
which the launchpad already implements, so neither the testbed nor the existing
e2e suites in this directory change. The stack is started once per suite and
never restarted; isolation comes from the control API instead.

Both resolvers declare Events, Stale, ConfigurationChange, Object and
UnavailableInit. The Stale declaration is the notable one: in Go the RPC
resolver never emits PROVIDER_STALE while in-process does
(open-feature/go-sdk-contrib#939), but here both report a lost connection
through the same disconnectCallback seam -- grpc-service.ts:274 and
grpc-fetch.ts:197 -- and the single handler behind it, flagd-provider.ts:130-148,
emits PROVIDER_STALE before escalating to PROVIDER_ERROR. The contract is
implemented once, in the provider, so it cannot differ between resolvers.

StrictNumericTyping is left undeclared, as it must be for every JavaScript
provider: the language has no integer type, so the scenario is unsatisfiable by
construction rather than by defect. Targeting and Caching are undeclared
because no scenario carries their tags yet.

Closes #1607

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants