diff --git a/specification/README.md b/specification/README.md index 6772899d..ef803fda 100644 --- a/specification/README.md +++ b/specification/README.md @@ -22,6 +22,7 @@ sidebar_position: 0 - [Appendix C: OFREP](./appendix-c/index.md) - [Appendix D: Observability](./appendix-d-observability.md) - [Appendix E: Migrations](./appendix-e-migrations.md) +- [Appendix F: Provider Conformance (TCK)](./appendix-f-provider-conformance.md) ## Conformance diff --git a/specification/appendix-b-gherkin-suites.md b/specification/appendix-b-gherkin-suites.md index 2e046dd4..70df67ea 100644 --- a/specification/appendix-b-gherkin-suites.md +++ b/specification/appendix-b-gherkin-suites.md @@ -11,6 +11,9 @@ This section contains a set of language-agnostic end-to-end tests (defined in gh These tests can be used to validate the behavior of an OpenFeature implementation. "Features" (test suites) can be used in conjunction with an [in-memory provider](./appendix-a-included-utilities.md#in-memory-provider) and a cucumber test-runner for the language in question. +The suites below validate an **SDK**, using an in-memory provider as a stand-in for a real one. +To validate a **provider** against a real backend instead, see [Appendix F: Provider Conformance (TCK)](./appendix-f-provider-conformance.md). + ## Evaluation Feature The [evaluation feature](./assets/gherkin/evaluation.feature) contains tests for the basic functionality of the [Evaluation API](./sections/01-flag-evaluation.md). diff --git a/specification/appendix-f-provider-conformance.md b/specification/appendix-f-provider-conformance.md new file mode 100644 index 00000000..941d3f40 --- /dev/null +++ b/specification/appendix-f-provider-conformance.md @@ -0,0 +1,232 @@ +--- +id: appendix-f +title: "Appendix F: Provider Conformance (TCK)" +description: A language-agnostic conformance suite for validating OpenFeature providers +sidebar_position: 7 +--- + +# Appendix F: Provider Conformance (TCK) + +[![experimental](https://img.shields.io/static/v1?label=Status&message=experimental&color=orange)](https://github.com/open-feature/spec/tree/main/specification#experimental) + +> **Status: proof of concept.** The artifacts in this appendix are under active development and are +> not yet normative. The scenario set is a representative subset covering each architectural +> mechanism once, not exhaustive coverage. Breaking changes should be expected. + +OpenFeature's central promise is that swapping providers does not change application behaviour. +Nothing currently verifies that. Every provider tests itself differently, against its own backend, +with its own harness — so "implements the provider contract" is an unverified claim, and a +behavioural difference between two providers is discovered by the application that trips over it. + +This appendix defines a **provider conformance suite**: a shared set of scenarios, a shared flag set, +and a shared way to manipulate a backend under test, so that the same conformance question can be +asked of every provider in every language and get comparable answers. + +## Relationship to Appendix B + +[Appendix B](./appendix-b-gherkin-suites.md) also contains Gherkin, and the distinction matters: + +| | Appendix B | Appendix F | +|---|---|---| +| Subject under test | the **SDK** | the **provider** | +| Harness | an [in-memory provider](./appendix-a-included-utilities.md#in-memory-provider) | the provider under test, against a real backend | +| Answers | does this SDK implement the Evaluation API, hooks and events correctly? | does this provider map its backend onto the provider contract correctly? | + +They are complementary. An SDK passing Appendix B and a provider passing Appendix F are different +claims, and a language needs both suites to make both. + +## What it tests, and what it does not + +**In scope — the provider contract:** + +- mapping backend responses onto typed resolution details: value, variant, reason, error code +- keeping the integer and float types distinct rather than coercing between them +- error handling: a type mismatch and an unknown flag return the code default, report the right + error code, and never throw +- lifecycle: reaching `READY`, and settling into `ERROR` against an unreachable backend +- events: `PROVIDER_READY`, `PROVIDER_ERROR`, `PROVIDER_STALE`, `PROVIDER_CONFIGURATION_CHANGED` +- that a signalled configuration change is actually **applied** on re-evaluation, not merely + signalled + +**Out of scope:** + +- **Backend evaluation logic**, targeting and bucketing correctness. Every flag in the canonical set + resolves to its default variant with no targeting involved, so what is under test is the + provider's mapping of a response, not the backend's decision. +- **The provider↔backend wire protocol.** How a provider talks to its backend is its own business. +- **SDK behaviour.** That is Appendix B. + +## The three artifacts + +Conformance rests on three files, and they **travel together by necessity**. A feature file that +evaluates `boolean-flag` is meaningless without the flag definition, and a disconnect scenario is +meaningless without the control endpoint that produces the disconnect. Changing one without the +others breaks the suite in every language at once. + +| Artifact | Location | What it defines | +|---|---|---| +| Gherkin scenarios | [`assets/provider-tck/gherkin/`](./assets/provider-tck/README.md) | the test cases themselves | +| Canonical flag set | [`assets/provider-tck/flags/canonical-flags.json`](./assets/provider-tck/flags/canonical-flags.json) | the flags those cases assume | +| Control API | [`assets/provider-tck/openapi/control-api.yaml`](./assets/provider-tck/openapi/control-api.yaml) | what a backend under test must expose | + +### Gherkin scenarios + +Four feature files: + +- [`evaluation.feature`](./assets/provider-tck/gherkin/evaluation.feature) — resolving each type with + the right value, variant and reason +- [`errors.feature`](./assets/provider-tck/gherkin/errors.feature) — the type-mismatch matrix and the + unknown-flag case +- [`events.feature`](./assets/provider-tck/gherkin/events.feature) — configuration change, and the + stale/ready transition across an outage +- [`lifecycle.feature`](./assets/provider-tck/gherkin/lifecycle.feature) — initialisation against a + healthy backend and against an unreachable one + +The step vocabulary is inherited from the +[flagd test harness](https://github.com/open-feature/test-harness) wherever it was already +provider-neutral, so an existing suite ports with a near-zero diff. + +### The canonical flag set + +A backend under test must serve an equivalent set under the configuration named `default`. The file +is expressed in the flagd flag-definition format because that is the only widely implemented +vendor-neutral format today — **the format is not what matters**, the keys, types, variant names and +resolved values are. Seed them however your backend seeds flags. + +Two properties are load-bearing and easy to break by accident: + +- **`missing-flag` must not exist.** Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding + it turns that scenario green for the wrong reason. +- **No flag has targeting rules.** Every scenario expects reason `STATIC`, because the suite tests + the provider's mapping of a response, not the backend's decision. + +### The control API + +Scenarios need to change flags and simulate outages, and they need to do it identically across +vendors. The control API is a small HTTP surface the backend under test exposes for that purpose: + +| Endpoint | Required | Purpose | +|---|---|---| +| `POST /start?config=` | yes | start the backend, seeding flags to a named baseline | +| `POST /stop` | yes | make the backend unreachable | +| `POST /restart?seconds=` | yes | simulate an outage of a bounded duration | +| `POST /change` | yes | mutate flag configuration so the provider observes a change | +| `POST /reset` | no | restore the baseline without an availability blip | +| `GET /healthz` | no | readiness of the control API | + +See the [OpenAPI document](./assets/provider-tck/openapi/control-api.yaml) for the normative detail. + +Two invariants are worth stating here because they are the ones a TCK implementation gets wrong: + +- **No container is ever stopped or restarted between scenarios.** Unavailability is simulated + *inside* the running stack. Container orchestrators assign host ports dynamically and cannot + reliably preserve them across a restart, so restarting would silently invalidate every provider + already pointed at the old port. The failure looks like a flaky provider. +- **Scenario isolation comes from the control API**, not from cycling the stack. The backend is + started once per suite and reset before each scenario. + +## Capabilities: how a provider says what it cannot do + +Not every provider implements every optional part of the contract. A provider backed by a static +file has no meaningful notion of going stale; a provider without a streaming transport cannot emit +configuration-change events. Forcing those providers to fail scenarios they were never going to +satisfy makes the suite unadoptable. + +Instead, each scenario that exercises an optional capability carries a **tag**, and a provider +declares which capabilities it supports. Scenarios whose tag is not declared are reported as +**skipped, with the reason** — never as passed. + +| Tag | Meaning | +|---|---| +| `@events` | emits lifecycle events at all | +| `@lifecycle` | performs an initialisation that reaches its backend, with an observable outcome | +| `@stale` | enters `STALE` and emits `PROVIDER_STALE` on backend loss | +| `@configuration-change` | detects configuration changes and emits `PROVIDER_CONFIGURATION_CHANGED` | +| `@object` | supports structured flag values | +| `@unavailable` | reports an error state instead of hanging against a dead backend | +| `@strict-numeric-typing` | does not coerce between integer and float | +| `@targeting` | reserved; no scenarios yet | +| `@caching` | reserved; no scenarios yet | + +Untagged scenarios are mandatory and always run. + +`@lifecycle` and `@events` are deliberately separate, and conflating them is the mistake this +vocabulary exists to prevent. Every SDK synthesises `PROVIDER_READY` for a provider that has no +initialisation step -- the Go SDK's comment says so outright, *"a provider without state handling +capability can be assumed to be ready immediately"* -- so a provider with no lifecycle passes the +readiness scenario without demonstrating anything, exactly as a no-op provider would. A stateless +HTTP provider such as OFREP is the common case: it observably emits nothing of its own and cannot +fail initialisation, yet its client still reports `READY`. Such a provider declares neither tag, and +the lifecycle scenarios are reported as skipped rather than passing vacuously. + +The design rule behind this: **a conformance suite that quietly goes green on scenarios it did not +run is worse than no suite at all.** A TCK implementation must report unsupported capabilities as +skipped and surface the reason, not silently pass or silently omit them. + +`@strict-numeric-typing` deserves a note, because unlike the others it is **not** an optional +feature. The specification requires a provider to report `TYPE_MISMATCH` when the requested type +cannot be satisfied, and narrowing `0.5` to `0` to satisfy an integer request loses information +silently — the worst failure mode for a feature flag, because the application sees a plausible value +and no error. It is a capability only so that a provider with this defect can adopt the suite today +and see the gap reported explicitly rather than being unable to adopt at all. Not declaring it is an +admission of a known bug. + +## Implementing the suite in a language + +A TCK implementation is the language-specific harness around these three artifacts. What it owns: + +1. **Ship the artifacts.** Package the Gherkin, the flag set and the control API document with the + library so that adopting providers need no submodule of their own. +2. **Implement the step definitions** against the language's OpenFeature SDK, using its Cucumber (or + equivalent) runner. +3. **Own the lifecycle** — start the backend stack once, register the provider under test with the + SDK, await events, tear down — so that an adopting provider writes no test infrastructure. If a + provider author finds themselves adding lifecycle code, that is a defect in the TCK. +4. **Drive the backend only through the control API.** This is the part that makes the conformance + claim portable: another language's TCK drives the same endpoints against the same stack and must + get the same answers. +5. **Gate on capabilities** and report undeclared ones as skipped with a reason. +6. **Run scenarios serially.** Backend state is global to the suite; concurrent scenarios corrupt + each other, and the symptom looks like a flaky provider rather than a broken test. + +### Providers with no backend + +An in-memory, environment-variable or file-based provider has nothing to connect to and no control +API to expose. A TCK implementation may offer an **in-process** control path for these, where flag +operations are direct manipulations of the provider's own state rather than HTTP calls. + +This is a narrow allowance, and worth being explicit about, because it is the obvious thing to abuse. +**A provider with an external backend must use the control API.** Reaching into an external backend +from inside the test process — a test-only admin client, a shared database handle, a hook inside the +provider — produces a suite that passes while proving nothing, because the path it exercised is not +the path the contract describes. + +Connection-dependent scenarios (`@stale`, `@unavailable`) have no meaning without a connection, so a +backend-less provider leaves those capabilities undeclared and they are skipped. An in-process +control path should **fail loudly** if a connection operation is reached anyway — that means a +capability was declared that the harness cannot back up, which is a test-configuration bug rather +than a provider defect. + +## Reference implementation + +The first implementation is `tools/provider-tck` in +[open-feature/java-sdk-contrib](https://github.com/open-feature/java-sdk-contrib), adopted by the +flagd provider for both its RPC and in-process resolvers. It is in review alongside this appendix. + +## Open questions + +This appendix is a proof of concept. Known gaps, all of which affect every language equally and so +belong here rather than in any one implementation: + +- **Evaluation context passthrough.** The scenarios build evaluation contexts but cannot assert the + context *reached* the backend intact. That needs an echo operation on the control API — something + like `GET /last-evaluation` returning the request the backend last received. Until then a provider + that silently drops the context passes. The `@targeting` tag is reserved for these scenarios. +- **Setting and removing individual flags.** The control API can reset to a baseline and mutate one + designated flag. Finer-grained flag manipulation would need new endpoints. +- **Caching.** Whether a stale provider keeps serving last-known values during an outage depends on + whether it holds a local copy of the ruleset. The `@caching` tag is reserved; no scenarios yet. +- **Hooks and flag metadata.** Not covered. +- **Normative status.** Nothing in this appendix is currently expressed as a numbered requirement. + Whether the control API contract and the capability vocabulary should become normative sections is + a decision for the TSC. diff --git a/specification/assets/provider-tck/.gitattributes b/specification/assets/provider-tck/.gitattributes new file mode 100644 index 00000000..c617195d --- /dev/null +++ b/specification/assets/provider-tck/.gitattributes @@ -0,0 +1,6 @@ +# These artifacts are consumed byte for byte by every language's provider TCK, and several +# are copied verbatim into published build artifacts. Normalise to LF so a checkout on +# Windows does not produce a different packaged file than one on Linux. +*.feature text eol=lf +*.json text eol=lf +*.yaml text eol=lf diff --git a/specification/assets/provider-tck/README.md b/specification/assets/provider-tck/README.md new file mode 100644 index 00000000..292946c8 --- /dev/null +++ b/specification/assets/provider-tck/README.md @@ -0,0 +1,27 @@ +# Provider Conformance Assets + +Test assets for the provider conformance suite described in [Appendix F](../../appendix-f-provider-conformance.md). + +These validate a **provider** against a real backend. For assets that validate an **SDK**, see [`../gherkin/`](../gherkin/README.md) and [Appendix B](../../appendix-b-gherkin-suites.md). + +## Contents + +| Path | What it is | +| --- | --- | +| [`gherkin/evaluation.feature`](./gherkin/evaluation.feature) | resolving each type with the right value, variant and reason | +| [`gherkin/errors.feature`](./gherkin/errors.feature) | the type-mismatch matrix and the unknown-flag case | +| [`gherkin/events.feature`](./gherkin/events.feature) | configuration change, and the stale/ready transition across an outage | +| [`gherkin/lifecycle.feature`](./gherkin/lifecycle.feature) | initialisation against a healthy backend and against an unreachable one | +| [`flags/canonical-flags.json`](./flags/canonical-flags.json) | the flag set every scenario assumes | +| [`openapi/control-api.yaml`](./openapi/control-api.yaml) | the HTTP surface a backend under test must expose | + +## These three travel together + +A feature file that evaluates `boolean-flag` is meaningless without the flag definition, and a disconnect scenario is meaningless without the control endpoint that produces the disconnect. Changing one without the others breaks the suite in every language at once. + +## Two properties that are load-bearing + +- **`missing-flag` must not exist** in the flag set. Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding it turns that scenario green for the wrong reason. +- **No flag has targeting rules.** Every scenario expects reason `STATIC`, because the suite tests the provider's mapping of a backend response, not the backend's evaluation logic. + +The flag set is expressed in the flagd flag-definition format because that is the only widely implemented vendor-neutral format today. The format is not what matters — the keys, types, variant names and resolved values are. Seed them however your backend seeds flags. diff --git a/specification/assets/provider-tck/flags/canonical-flags.json b/specification/assets/provider-tck/flags/canonical-flags.json new file mode 100644 index 00000000..343b3ae5 --- /dev/null +++ b/specification/assets/provider-tck/flags/canonical-flags.json @@ -0,0 +1,82 @@ +{ + "$comment": [ + "The canonical flag set the TCK's feature files assume. A backend under test MUST serve an", + "equivalent set under the configuration named 'default'.", + "", + "Expressed in the flagd flag-definition format because that is the only widely implemented", + "vendor-neutral format today. The format is not what matters — the keys, types, variant", + "names and resolved values are. Seed them however your backend seeds flags.", + "", + "Two things are load-bearing and easy to get wrong:", + " * 'missing-flag' MUST NOT exist. Its absence is what the FLAG_NOT_FOUND scenario tests.", + " * No flag here has targeting rules. Every scenario expects reason STATIC, because the TCK", + " tests the provider's mapping of a response, not the backend's evaluation logic." + ], + "flags": { + "boolean-flag": { + "state": "ENABLED", + "variants": { + "on": true, + "off": false + }, + "defaultVariant": "on" + }, + "string-flag": { + "state": "ENABLED", + "variants": { + "greeting": "hi", + "parting": "bye" + }, + "defaultVariant": "greeting" + }, + "integer-flag": { + "state": "ENABLED", + "variants": { + "one": 1, + "ten": 10 + }, + "defaultVariant": "ten" + }, + "float-flag": { + "state": "ENABLED", + "variants": { + "tenth": 0.1, + "half": 0.5 + }, + "defaultVariant": "half" + }, + "object-flag": { + "state": "ENABLED", + "variants": { + "empty": {}, + "template": { + "showImages": true, + "title": "Check out these pics!", + "imagesPerPage": 100 + } + }, + "defaultVariant": "template" + }, + "wrong-flag": { + "$comment": "A string flag, evaluated as a boolean by the TYPE_MISMATCH scenario.", + "state": "ENABLED", + "variants": { + "one": "uno", + "two": "dos" + }, + "defaultVariant": "one" + }, + "changing-flag": { + "$comment": [ + "The flag POST /change mutates. The TCK asserts only that its resolved value differs", + "after the change, so which of the two variants you start from does not matter." + ], + "state": "ENABLED", + "variants": { + "foo": "foo", + "bar": "bar" + }, + "defaultVariant": "foo" + } + } +} diff --git a/specification/assets/provider-tck/gherkin/errors.feature b/specification/assets/provider-tck/gherkin/errors.feature new file mode 100644 index 00000000..0346df3d --- /dev/null +++ b/specification/assets/provider-tck/gherkin/errors.feature @@ -0,0 +1,80 @@ +Feature: Provider error handling + + # Every scenario here asserts the same three-part contract, because all three parts matter and + # providers routinely get one of them wrong: + # + # 1. the code default is returned — an application must keep working, + # 2. the correct error code is reported — an application must be able to tell what went wrong, + # 3. nothing is thrown — an unhandled exception from a flag evaluation is never acceptable. + # + # Requires the backend to be seeded with the canonical flag set — see flags/canonical-flags.json. + + Background: + Given a stable provider + + Scenario Outline: Requesting the wrong type returns the code default + # The full non-numeric mismatch matrix. Numeric coercion is a separate question and is covered + # by the @strict-numeric-typing scenarios below, because "is 0.5 an integer?" has a defensible + # wrong answer whereas "is a string a boolean?" does not. + Given a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "ERROR" + And the error-code should be "TYPE_MISMATCH" + And no exception should have been thrown + + Examples: a string flag requested as something else + | key | requested | default | + | string-flag | Boolean | false | + | string-flag | Integer | 1 | + | string-flag | Float | 0.1 | + | wrong-flag | Boolean | false | + + Examples: a boolean flag requested as something else + | key | requested | default | + | boolean-flag | String | fallback | + | boolean-flag | Integer | 1 | + | boolean-flag | Float | 0.1 | + + Examples: a numeric flag requested as a non-numeric type + | key | requested | default | + | integer-flag | Boolean | false | + | integer-flag | String | fallback | + | float-flag | Boolean | false | + | float-flag | String | fallback | + + @object + Scenario Outline: Requesting a structured flag as a scalar returns the code default + Given a -flag with key "object-flag" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "ERROR" + And the error-code should be "TYPE_MISMATCH" + And no exception should have been thrown + + Examples: + | requested | default | + | Boolean | false | + | String | fallback | + | Integer | 1 | + | Float | 0.1 | + + @strict-numeric-typing + Scenario: A float flag is not silently narrowed to an integer + # 'float-flag' resolves to 0.5. Narrowing that to an integer would lose information + # silently, so it must be reported as a type mismatch rather than rounded. + Given a Integer-flag with key "float-flag" and a default value "1" + When the flag was evaluated with details + Then the resolved details value should be "1" + And the reason should be "ERROR" + And the error-code should be "TYPE_MISMATCH" + And no exception should have been thrown + + Scenario: An unknown flag key returns the code default + # 'missing-flag' is deliberately absent from the canonical flag set. + Given a String-flag with key "missing-flag" and a default value "fallback" + When the flag was evaluated with details + Then the resolved details value should be "fallback" + And the reason should be "ERROR" + And the error-code should be "FLAG_NOT_FOUND" + And no exception should have been thrown diff --git a/specification/assets/provider-tck/gherkin/evaluation.feature b/specification/assets/provider-tck/gherkin/evaluation.feature new file mode 100644 index 00000000..e89f174a --- /dev/null +++ b/specification/assets/provider-tck/gherkin/evaluation.feature @@ -0,0 +1,59 @@ +Feature: Provider flag evaluation + + # Verifies that a provider maps backend responses onto typed resolution details correctly. + # + # This does NOT test the backend's evaluation logic. Every flag in the canonical set resolves + # to its default variant with no targeting involved, so what is under test is purely the + # provider's mapping of a backend response to a value, a variant and a reason. + # + # Requires the backend to be seeded with the canonical flag set — see flags/canonical-flags.json. + + Background: + Given a stable provider + + Scenario Outline: Resolve values with variant and reason + Given a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the variant should be "" + And the reason should be "" + And the error-code should be "" + And no exception should have been thrown + + Examples: + | key | type | default | value | variant | reason | + | boolean-flag | Boolean | false | true | on | STATIC | + | string-flag | String | bye | hi | greeting | STATIC | + | integer-flag | Integer | 1 | 10 | ten | STATIC | + | float-flag | Float | 0.1 | 0.5 | half | STATIC | + + Scenario: An integer flag resolves as an integer + # Paired with the float scenario below and with the narrowing scenario in errors.feature. + # Together they pin down that the two numeric types stay distinct rather than both being + # funnelled through one numeric representation. + Given a Integer-flag with key "integer-flag" and a default value "1" + When the flag was evaluated with details + Then the resolved details value should be "10" + And the error-code should be "" + And no exception should have been thrown + + Scenario: A float flag resolves as a float + Given a Float-flag with key "float-flag" and a default value "0.1" + When the flag was evaluated with details + Then the resolved details value should be "0.5" + And the error-code should be "" + And no exception should have been thrown + + @object + Scenario: Resolve a structured value + Given a Object-flag with key "object-flag" and a default value "{}" + When the flag was evaluated with details + Then the variant should be "template" + And the reason should be "STATIC" + And the error-code should be "" + And no exception should have been thrown + And the resolved object value should contain + | key | type | value | + | showImages | Boolean | true | + | title | String | Check out these pics! | + | imagesPerPage | Integer | 100 | diff --git a/specification/assets/provider-tck/gherkin/events.feature b/specification/assets/provider-tck/gherkin/events.feature new file mode 100644 index 00000000..00e7e5ef --- /dev/null +++ b/specification/assets/provider-tck/gherkin/events.feature @@ -0,0 +1,42 @@ +@events +Feature: Provider events + + # Verifies that a provider notices changes in its backend and both signals them and acts on + # them. Signalling alone is not enough: a configuration-change event that is not followed by + # a changed evaluation result is a lie, so each scenario asserts the event AND the behaviour. + # + # Outages here are simulated inside the running stack via the control API. No container is + # ever stopped or restarted — see the invariant in openapi/control-api.yaml. + + Background: + Given a stable provider + + @configuration-change + Scenario: A configuration change is signalled and applied + Given a String-flag with key "changing-flag" and a default value "unset" + And a change event handler + When the flag was evaluated with details + And the resolved value is remembered + And the flag was modified + Then the change event handler should have been executed + And the flag should be part of the event payload + When the flag was evaluated with details + Then the resolved details value should have changed + And no exception should have been thrown + + @stale + Scenario: Losing the backend makes the provider stale, regaining it makes it ready again + Given a ready event handler + And a stale event handler + When a ready event was fired + And the connection is lost + Then the stale event handler should have been executed + And the client should be in stale state + When the connection is restored + Then the ready event handler should have been executed + And the client should be in ready state + + # Deliberately NOT covered here: whether a stale provider keeps serving last-known values + # during the outage. That is caching behaviour, which depends on whether the provider holds a + # local copy of the ruleset, and it belongs behind the @caching capability once those + # scenarios are written. See the "Known gaps" section of the README. diff --git a/specification/assets/provider-tck/gherkin/lifecycle.feature b/specification/assets/provider-tck/gherkin/lifecycle.feature new file mode 100644 index 00000000..338b2c05 --- /dev/null +++ b/specification/assets/provider-tck/gherkin/lifecycle.feature @@ -0,0 +1,40 @@ +@lifecycle +Feature: Provider lifecycle + + # Verifies the two terminal outcomes of provider initialisation: reaching READY against a + # healthy backend, and settling into ERROR against one that cannot be reached. + # + # Gated by @lifecycle rather than @events, and the distinction is load-bearing. Every SDK + # synthesises PROVIDER_READY for a provider that has no initialisation step, so a provider + # without a lifecycle passes the readiness scenario below without demonstrating anything -- + # a NoOpProvider passes it identically. @lifecycle asserts that the provider actually reaches + # its backend during initialisation and that the outcome is observable; a provider that merely + # emits events does not necessarily do that. + # + # The failure case matters more than it looks. A provider that blocks forever, or throws out + # of provider registration, takes the host application down with it — so the requirement is + # not merely that initialisation fails, but that it fails observably and promptly. + + Scenario: A provider reaching its backend becomes ready + Given a stable provider + And a ready event handler + Then the ready event handler should have been executed + And the client should be in ready state + + @unavailable + Scenario: A provider that cannot reach its backend reports an error + Given a unavailable provider + And a error event handler + Then the error event handler should have been executed within 10000ms + And the client should be in error state + + @unavailable + Scenario: A provider that cannot reach its backend still returns code defaults + Given a unavailable provider + And a error event handler + And a Boolean-flag with key "boolean-flag" and a default value "false" + Then the error event handler should have been executed within 10000ms + When the flag was evaluated with details + Then the resolved details value should be "false" + And the reason should be "ERROR" + And no exception should have been thrown diff --git a/specification/assets/provider-tck/openapi/control-api.yaml b/specification/assets/provider-tck/openapi/control-api.yaml new file mode 100644 index 00000000..d2119106 --- /dev/null +++ b/specification/assets/provider-tck/openapi/control-api.yaml @@ -0,0 +1,388 @@ +openapi: 3.0.3 + +info: + title: OpenFeature Provider TCK — Backend Control API + version: 0.0.1 + description: | + The control API that a **backend under test** must expose so the OpenFeature + Provider TCK can drive it. + + The TCK verifies the *provider contract*: how a provider maps backend + responses to typed resolution details, lifecycle states and events. To do + that it must be able to put the backend into specific states on demand — + running, unreachable, reconfigured. This document standardises how. + + This specification is derived from the control endpoints already implemented + by [`flagd-testbed`](https://github.com/open-feature/flagd-testbed)'s + "launchpad" server, which is the reference implementation. + + ## Where this document should live + + This file currently ships inside the Java `provider-tck` artifact, but it is + not a Java artifact: it is a language-agnostic contract that every language's + TCK must implement identically, and that backend vendors implement in + whatever language their testbed is written in (Go, for flagd). + + It therefore belongs in the OpenFeature **spec** repository + (`open-feature/spec`), alongside the canonical Gherkin feature files and the + canonical flag set. Those three artifacts are a single unit — a feature file + that evaluates `boolean-flag` is meaningless without the flag definition, and + a disconnect scenario is meaningless without the endpoint that produces the + disconnect. Splitting them across repositories would let them drift. + + Each language's TCK then vendors the spec repo (git submodule or equivalent) + and packages these files into its own distribution format, so that adopting a + TCK never requires a consumer to check out a submodule of their own. + + ## Conformance language + + The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT and MAY are to be + interpreted as described in RFC 2119. + + Each operation below is tagged **REQUIRED** or **OPTIONAL**. A backend that + implements every REQUIRED operation can run the full TCK. OPTIONAL operations + have a defined fallback that the TCK applies automatically, so omitting them + costs nothing but precision. + + --- + + ## Normative requirement 1 — the no-container-restart invariant + + > **Container lifecycle operations MUST NOT be used to simulate backend + > unavailability. Backend unavailability MUST be simulated from inside the + > running stack.** + + The TCK starts the vendor's Docker Compose stack **once per test suite** and + reads the dynamically mapped host ports. Testcontainers cannot reliably + preserve mapped ports across a container stop/start in all language + bindings — a restarted container generally comes back on a *different* host + port, which silently invalidates every provider instance already pointed at + the old one. Any TCK implementation in any language hits this, so the + constraint is part of the contract rather than a Java detail. + + Therefore an implementation of `/stop`, `/restart` or any other outage + simulation MUST achieve the outage by one of: + + * killing or suspending the backend **process** inside its container + (the reference behaviour — this is what flagd-testbed does); + * a proxy in the stack refusing or blackholing connections + (e.g. a toxiproxy toxic, an envoy `direct_response`); + * an in-container firewall or socket-level block. + + An implementation MUST NOT `docker stop`, `docker kill`, `docker rm` or + recreate any container in the stack while the suite is running. The stack is + brought up before the first scenario and torn down after the last one, and + the mapped ports MUST remain stable for that entire window. + + --- + + ## Normative requirement 2 — flag state semantics across outages + + Outage simulation and flag-state seeding are orthogonal, and the TCK relies + on that separation for scenario isolation: + + * `POST /start` **MUST** (re)seed flag state to the baseline defined by the + named configuration. Any mutation previously applied by `POST /change` + MUST be discarded. This is what makes `/start` usable as a reset. + * `POST /restart` and a `POST /stop` followed by a `POST /start` **of the + same configuration** MUST leave the backend serving the same baseline + flag state it served before the outage. An outage MUST NOT be observable + as a change in flag *values* — only as a change in *availability*. + * `POST /change` mutations persist until the next `/start` or `/reset`. + + --- + + ## Normative requirement 3 — compose stack conventions + + The backend under test is delivered as a **Docker Compose stack**, not a + single image, so vendors can compose proxies, edge services or several + containers. The TCK only relies on these conventions: + + * One service — by default named `backend`, overridable by the provider + author — exposes the control API on container-internal port `8080` + (also overridable). + * The same stack exposes whatever port(s) the provider connects to. + * **All external ports are dynamically mapped.** A stack MUST NOT pin host + ports; the TCK discovers them after startup and hands them to the + provider factory. + * The stack MAY contain any number of additional services. + + --- + + ## Known gap — evaluation context passthrough + + There is currently no operation for asserting that an evaluation context sent + by the provider actually reached the backend intact. Verifying that requires + an echo mechanism (e.g. `GET /last-evaluation` returning the most recent + request the backend received). Until such an operation exists, context + passthrough is out of scope for the TCK. + + license: + name: Apache 2.0 + url: https://www.apache.org/licenses/LICENSE-2.0 + +servers: + - url: http://{host}:{port} + description: | + Resolved at runtime from the Compose stack. `host` is the Docker host and + `port` is the dynamically mapped host port for the control service's + internal port 8080. + variables: + host: + default: localhost + port: + default: "8080" + +tags: + - name: lifecycle + description: Start and stop the backend process. + - name: availability + description: Simulate outages without touching containers. + - name: flags + description: Seed and mutate flag configuration. + - name: health + description: Readiness of the control API itself. + +paths: + + /start: + post: + tags: [lifecycle] + operationId: start + summary: "[REQUIRED] Start the backend and seed flags to a named baseline" + description: | + Starts the backend process using the named configuration and seeds flag + state to that configuration's baseline. + + MUST be idempotent in the sense that calling it while the backend is + already running is not an error: the implementation restarts the process + (or otherwise ensures it is running) with the requested configuration. + + **MUST NOT return until the seeded flag state is actually being served.** + A 200 is a promise that the very next evaluation will resolve against the + new baseline. Returning as soon as the process reports healthy is not + enough: a backend can accept connections and answer a readiness probe + while its flag store is still empty, and an evaluation in that window + gets `FLAG_NOT_FOUND` for a flag the configuration plainly defines. + + This is easy to get wrong and easy to miss. A provider that blocks during + initialisation -- streaming, or syncing a ruleset -- absorbs the window + and never sees it. A **stateless** provider, which evaluates over HTTP + with no initialisation at all, has nothing to hide it behind and fails + essentially every scenario, which reads as a catastrophically broken + provider rather than as a racing testbed. The reference implementation + exhibits this: its `/start` returns roughly 40ms before flagd's file + sources reach the flag store. + + A TCK MAY defensively probe after `/start`, but it should not have to, + and requiring every stateless adopter to reimplement that probe is worse + than stating the requirement here. + + Because this operation resets flag state, the TCK uses it as its default + scenario-isolation mechanism when `/reset` is not implemented. + + The set of valid configuration names is vendor-defined. Every + implementation MUST support the name `default`, which MUST serve the + canonical flag set the TCK's feature files assume. + + Reference implementation: flagd-testbed launches the `flagd` binary with + the config file of that name from `launchpad/configs` and rewrites + `/flags/allFlags.json`. + parameters: + - name: config + in: query + required: false + description: | + Name of the configuration to start with. Defaults to `default`. + schema: + type: string + default: default + example: default + responses: + "200": + description: Backend started and flag state seeded. + "400": + description: Unknown configuration name. + content: + application/json: + schema: + $ref: "#/components/schemas/Error" + + /stop: + post: + tags: [availability] + operationId: stop + summary: "[REQUIRED] Make the backend unreachable" + description: | + Makes the backend unreachable to the provider, simulating an outage. + + **MUST NOT stop the container.** See normative requirement 1. The + reference implementation kills the flagd process while its container + keeps running. + + The backend stays unreachable until a subsequent `POST /start`. Calling + `/stop` when the backend is already stopped MUST succeed. + + The TCK uses this to drive providers into `STALE` and `ERROR` states and + to assert `PROVIDER_STALE` / `PROVIDER_ERROR` events. + responses: + "200": + description: Backend is now unreachable; container still running. + + /restart: + post: + tags: [availability] + operationId: restart + summary: "[REQUIRED] Simulate an outage of a bounded duration" + description: | + Makes the backend unreachable, waits `seconds`, then starts it again with + the configuration currently in effect. + + Flag state MUST be preserved across the outage — see normative + requirement 2. This is what distinguishes `/restart` from + `/stop` + `/start`: the former is an availability event, the latter is + also a reset. + + This operation MAY return as soon as the outage has begun rather than + blocking for the full duration; the TCK does not rely on the response + being delayed. It awaits provider events instead. + + The TCK uses this for the disconnect/reconnect scenarios: `STALE` → + `PROVIDER_STALE`, then back to `READY` → `PROVIDER_READY`. + parameters: + - name: seconds + in: query + required: false + description: | + How long the backend stays unreachable. Defaults to 5. + + Providers differ enormously in how fast they notice an outage — + a streaming provider may see it in milliseconds while a polling + provider needs up to a full poll interval. Feature files therefore + parameterise this value and provider authors tune the matching + await timeouts. + schema: + type: integer + format: int32 + minimum: 0 + default: 5 + example: 5 + responses: + "200": + description: Outage started (and, for blocking implementations, ended). + + /change: + post: + tags: [flags] + operationId: change + summary: "[REQUIRED] Mutate flag configuration so the provider observes a change" + description: | + Mutates the flag configuration such that a conforming provider observes a + configuration change and, on re-evaluation, resolves a **different value** + for the affected flag. + + The implementation MUST: + + * change the resolved value of the flag with key `changing-flag`; + * do so without restarting the backend process, so that a provider sees + a configuration-change signal rather than a reconnect; + * make the change durable until the next `/start` or `/reset`. + + The implementation SHOULD toggle between exactly two known values so that + repeated calls are meaningful and the test remains deterministic + regardless of how many times it has run against the same stack. The + reference implementation toggles `changing-flag`'s `defaultVariant` + between `foo` and `bar`. + + The TCK uses this to assert `PROVIDER_CONFIGURATION_CHANGED`, that the + changed flag key appears in the event payload, and that a subsequent + evaluation returns the new value. + responses: + "200": + description: Flag configuration mutated. + + /reset: + post: + tags: [flags] + operationId: reset + summary: "[OPTIONAL] Restore the seeded baseline without an outage" + description: | + Restores flag state to the baseline of the configuration currently in + effect, discarding any mutation applied by `/change`, **without** making + the backend unreachable at any point. + + This is the preferred scenario-isolation primitive: unlike `/start` it + causes no availability blip, so it cannot inject spurious lifecycle + events into the next scenario. + + **Scope.** This operation resets flag state only. It MUST NOT be + expected to start a backend that is currently stopped — that is what + `/start` is for. A TCK therefore uses `/reset` only when the backend is + known to be running, and `/start` otherwise. The reference client tracks + this: `/stop` and `/restart` mark the backend as possibly-unreachable, so + the scenario that follows either of them is prepared with `/start`. + + **Fallback when not implemented.** A backend that does not implement this + operation MUST respond `404` or `501`. The TCK then falls back to + `POST /start?config={defaultConfig}`, which resets flag state at the cost + of a process restart. The fallback is detected once per suite and cached. + + Implementing `/reset` is RECOMMENDED for providers whose reconnect + behaviour makes the `/start` blip hard to distinguish from a real event. + responses: + "200": + description: Flag state restored to the baseline. + "404": + description: Not implemented; the TCK falls back to `/start`. + "501": + description: Not implemented; the TCK falls back to `/start`. + + /healthz: + get: + tags: [health] + operationId: health + summary: "[OPTIONAL] Readiness of the control API" + description: | + Reports whether the control API is ready to accept commands. + + **Fallback when not implemented.** Readiness defaults to "the control + port accepts a TCP connection", which the TCK establishes with a + Testcontainers listening-port wait strategy before the first scenario. A + `404` here is therefore not a failure, and the reference implementation + does not serve this path. + + Note this reports the health of the **control API**, not of the backend. + The backend is deliberately unhealthy during outage scenarios while the + control API must stay reachable — otherwise the TCK could not end the + outage. + responses: + "200": + description: Control API ready. + content: + application/json: + schema: + $ref: "#/components/schemas/Health" + "404": + description: Not implemented; readiness falls back to a TCP port check. + "503": + description: Control API not ready yet. + +components: + schemas: + + Health: + type: object + properties: + status: + type: string + enum: [ok] + description: Present and equal to `ok` when the control API is ready. + required: [status] + + Error: + type: object + properties: + message: + type: string + description: Human-readable explanation. Never interpreted by the TCK. + required: [message]