feat(provider-tck): add the Go conformance suite for OpenFeature providers

[aepfli]

Closes #938
Part of open-feature/spec#417

Draft. Written without a Go toolchain on the machine, so every API was pinned against the source of the exact pinned versions — godog v0.15.1 and go-sdk v1.18.0 — rather than against a build, and CI was the first execution. It has now run green: the module compiles, golangci-lint passes, and all three self-test suites pass. See Verification.

What this is

A conformance suite any Go provider can adopt to verify it implements the provider contract of the specification — the Go implementation of Appendix F, running the same Gherkin scenarios, against the same canonical flag set, driven through the same control API as every other language's TCK. That shared basis is the point: "conformant" only means something if the question is identical everywhere.

Java is the reference (java-sdk-contrib#1830 and its stack). This is not a transcription of it — Go has no inheritance, no JUnit Platform Suite and no exceptions, and each of those forced a decision rather than a translation.

The adoption surface

One function call, one struct literal. Three required fields.

func TestMyProviderConformance(t *testing.T) {
	control := myBackendControl()

	tck.Run(t, tck.Config{
		Name:    "my-provider",
		Control: control,
		NewProvider: func(ctx context.Context) (openfeature.FeatureProvider, error) {
			return myprovider.New(control.Address()), nil
		},
		Capabilities: []tck.Capability{tck.Events, tck.Object, tck.StrictNumericTyping},
	})
}

The TCK owns the whole lifecycle: registering the provider, waiting for readiness, awaiting events, resetting the backend between scenarios, releasing the provider at the end. Each scenario becomes a Go subtest, so -run selects one and failures name a scenario. The canonical Gherkin and flag set are go:embed-ed, so consumers need no git submodule.

Config is a struct rather than an interface because Go has no inheritance and an interface with six optional methods is worse on every axis that matters here. The trade is compile-time enforcement for discoverability; Run validates up front and names what is missing.

Three Go-specific decisions

1. "no exception should have been thrown" asserts the evaluation did not panic. Go has no exceptions, and the returned error is not one — an errored evaluation correctly returns the code default alongside a non-nil error, which is the normal shape of the API. The behaviour the feature files forbid — an unhandled failure escaping a flag evaluation and taking the host application down — is a panic here. Provider registration is guarded the same way.

2. Providers are registered under a suite-scoped domain, not a per-scenario one. Registering in a domain replaces and shuts down the previous provider. A fresh domain per scenario reads cleaner but would leave every provider of the suite registered and running — for a provider holding a network connection, one leaked connection per scenario.

3. Capability gating is godog.ErrSkip returned from the Before hook. godog treats that as "skip this scenario and all its steps" rather than a failure, and errors.Is still matches after godog wraps it. Every skip is also collected and printed with its reason at the end of the suite, because a conformance suite that quietly goes green on scenarios it did not run is worse than no suite at all.

There is a non-obvious constraint the design depends on: the SDK compares providers with reflect.DeepEqual unless the dynamic type is a pointer. A value-typed provider holding an equal flag set would compare as the same provider, and the per-scenario replacement the TCK relies on for isolation would silently not happen. ControllableProvider is therefore pointer-only, and says so.

Self-tests: three suites, no Docker

Suite	Subject	Why
TestInMemoryProvider	memprovider.InMemoryProvider	reference adoption for a backend-less provider
TestControllableProvider	tck.ControllableProvider	the only suite that exercises the configuration-change path
TestMultiProvider	multi.Provider wrapping one child	delegation must be transparent

TestMultiProvider wraps exactly one child deliberately. That is the interesting configuration rather than a degenerate one: the correct answer is precisely what TestControllableProvider already asserts about the child alone, so any difference between the two suites is attributable to the multi-provider and nothing else — a variant that does not survive the hop, a reason rewritten to DEFAULT, an error code flattened to GENERAL, an event that never reaches the client.

Alongside them, gate_internal_test.go pins the capability gate directly rather than by inference from a scenario count, including that the error returned from the Before hook is one godog treats as a skip.

Finding: the Go SDK's in-memory provider cannot update its flag set

Appendix A is unambiguous about what an SDK's in-memory provider must do:

The provider must support a means of updating the flag set, resulting in the emission of PROVIDER_CONFIGURATION_CHANGED events.

memprovider.InMemoryProvider is Metadata + five evaluations + Hooks + Track. No update method, not an EventHandler, not even a StateHandler.

SDK	update method	emits PROVIDER_CONFIGURATION_CHANGED
JavaScript	putConfiguration()	yes
Java	updateFlag()	yes
Go	none	no

The knock-on is not confined to this suite: Appendix A also requires SDK end-to-end tests to use the in-memory provider, so go-sdk's own Appendix B suite cannot cover configuration-change events either.

This is handled honestly rather than worked around. TestInMemoryProvider leaves ConfigurationChange undeclared, so the scenario is reported as skipped with the reason, and plainMemoryControl.ChangeFlag returns an error stating the gap in case anyone declares it anyway. tck.ControllableProvider supplies the missing behaviour by wrapping the SDK's provider rather than reimplementing it — every resolution decision is still made by memprovider — so it doubles as a reference for what the SDK's provider should grow. Tracked as open-feature/go-sdk#530.

Finding it before the suite had run once is a reasonable advertisement for what a conformance suite is for.

Verification

Being precise about this, because the usual table would be misleading.

Authored without a toolchain; these are CI's results, not local ones.

Check	Result
make e2e (go test -tags=e2e, all modules)	pass — the module compiles and all three self-test suites are green
golangci-lint	pass, after fixing one finding: ST1008: error should be returned as the last argument in registerProvider
godog v0.15.1 API pinned against source	yes — TestSuite, Options{FS, TestingT, Concurrency, Strict}, ErrSkip semantics in suite.go, (context.Context, error) step returns in internal/models/stepdef.go (including that the context must not be nil)
go-sdk v1.18.0 API pinned against source	yes — SetNamedProviderWithContextAndWait, Client.AddHandler/RemoveHandler/State, EventCallback as *func(EventDetails), ResolutionDetail fields, ErrorCode constants, emitOnRegistration replay-on-registration, providerReference.equals pointer-vs-DeepEqual behaviour
Assets byte-identical to spec#423	verified with cmp for all six files, and .gitattributes pins LF so a Windows checkout cannot embed different bytes
go.sum	assembled from hashes already present in this repo's lockfiles plus sum.golang.org for the rest of the closure, computed by walking each dependency's go.mod from proxy.golang.org. It resolved cleanly in CI. Still worth a go mod tidy before merging, to normalise it to what the toolchain would write.
Delimiter balance, import hygiene, dupword/errcheck hazards	checked mechanically

I had expected a missing go.sum entry to be the likeliest failure. It was not — the assembled checksums resolved and the build ran clean. The single real defect was the return-value ordering above.

Known gaps

• The assets are vendored, not submoduled. pkg/tck/assets/ is a copy of specification/assets/provider-tck/ from spec#423, marked as such in assets.go. A follow-up will source them from a submodule at build time, the way providers/flagd already consumes the spec repo — the equivalent of java-sdk-contrib#1838.
• Evaluation context passthrough is unverifiable without an echo endpoint on the control API. A provider that silently drops the context passes.
• POST /restart is unused — no scenario needs a bounded outage, so ConnectionControl has no DisconnectFor.
• Caching, hooks and flag metadata are not covered.

Open questions

1. Is the Config struct the right adoption surface for Go, or should this be an interface with a BaseHarness to embed?
2. Is tools/provider-tck the right home, given tests/ in this repo already means "shared test harness module" (tests/flagd)? It was chosen to mirror the Java module path.
3. Should the containerised-backend helper live in this module (adding a testcontainers dependency for every consumer) or stay with adopters until a second one shows what to factor out?
4. Is shipping ControllableProvider here right, or should the fix land in go-sdk first and this module depend on it?

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[erka]

Hey @aepfli

Could we rename provider-tck to just tck? It isn't a provider, and the current name is a bit confusing.

Also, could we make it a flat package instead of putting it under pkg? pkg isn't really common practice anymore.

[aepfli]

Hey @aepfli

Could we rename provider-tck to just tck? It isn't a provider, and the current name is a bit confusing.

Also, could we make it a flat package instead of putting it under pkg? pkg isn't really common practice anymore.

Hey,

Sure thing, this is just me drafting and testing with Claude code, for multiple languages, and I try to stick to how the repo is handling things.

The naming is another question. What if there is a tck for other areas (hooks, listeners - bad examples as they are just interfaces currently) should we be ready for this?

[erka]

The naming is another question. What if there is a tck for other areas (hooks, listeners - bad examples as they are just interfaces currently) should we be ready for this?

It really depends on the implementation. If tck.Run(t, ....) has options instead of one struct it may give freedom to run suites for different areas.