private links: model links as rows with controller-observed status

[jshearer]

Reworks private networking so each private link is a first-class row with a system-assigned id and a controller-owned observed status, replacing the flat data_planes.private_links JSON array paired with the separate *_link_endpoints output columns.

The controller now sends each link's id to est-dry-dock and records each link's status from the link_results export it echoes back, with every result recorded against the row its id addresses. The est-dry-dock counterpart adds the optional id field, the link_results export, and per-link handling of input validation errors; its fleet rollout gates this deploy (see Rollout).

Background

The original API exposed the raw primitives we had: the private_links column as desired config, and the three *_link_endpoints columns as the controller's output. Building a UI on top surfaced two problems.

• An endpoint can only be tied back to its link by matching service_name/service_attachment inside a blob whose shape est-dry-dock owns. The correspondence is exact at apply time, discarded, and reconstructed downstream by every consumer; if the shape drifts, a healthy link reads as unprovisioned.
• Per-link status (pending, provisioned with details, failed with a reason) was never persisted. The only signal was "did an endpoint show up", and a single failing link tends to wedge the whole converge.

The fix is the usual reconcile split: user-owned desired config next to a controller-owned observed status, per link. The id travels the full loop: controller to est-dry-dock in the stack model, echoed back per link in control.link_results, recorded against the row it addresses.

What changed

DB (new migration). New internal.data_plane_private_links table, one row per link.

• Columns: system id, data_plane_id, provider, the polymorphic config, a generation counter, a generated service_identity, controller-owned status / details / error / observed_at, and timestamps.
• generation versions the desired config: a before update of config, provider trigger bumps it and resets the observation columns (status to pending) on any desired edit, whatever the writer. The controller pins each link's (id, generation) when it reads desired state and writes status only against a matching generation, so an edit racing a converge is never stamped with a pre-edit status.
• Unique on (data_plane_id, provider, service_identity): a same-provider duplicate would collide on a Pulumi resource name and wedge the converge. The old column path had no guard.
• A wake-only trigger sends the parent's controller task a converge on any desired change (insert, delete, or config/provider edit). The status write touches only observation columns, so it fires neither trigger and cannot loop.
• Lives in internal, unreachable via PostgREST, with no RLS or grants: every reader connects as postgres, and the resolvers gate on the private-networking capabilities. Mirrors internal.invite_links.
• Backfills one row per existing array element, after a pre-flight that aborts on legacy data the table cannot represent (missing identity, undecodable shape, duplicate identity).

data-plane-controller.

• Reads desired links from the table, ordered by created_at, id. Each model private_links entry carries its row id as a sibling field of the link config (PrivateLinkEntry); est-dry-dock parses it as optional, treats it as opaque, and echoes it back in control.link_results.
• After a converge, records each link's status from link_results, matched to the pinned links by id: provisioned with the result's details, failed with its error.
• Absent results are no observation: null-id entries (from a converge which pre-dated sending ids) and unknown statuses are skipped, and a pinned link with no matching entry keeps its prior status.
• link_results is optional in ControlExports: outputs from an est-dry-dock which pre-dates the export parse as None and write no statuses. Pinned by a parse test.
• The config column write-back strips private_links alongside the existing name/fqdn stripping: links are re-hydrated from the table on every read, so a persisted copy is dead data that drifts.

control-plane-api.

• privateLinks returns { id, config, status, details, error, observedAt } from the table; the link's cloud is the config union variant.
• Per-link addDataPlanePrivateLink / updateDataPlanePrivateLink / removeDataPlanePrivateLink replace the wholesale mutation, gated by ModifyDataPlanePrivateNetworking, with a clear duplicate-identity error.
• Schema rename: the config union becomes PrivateLinkConfig; the new row type takes the PrivateLink name.

Rollout

A single deploy with two coordination steps: the est-dry-dock fleet gate and an edit pause.

1. Precondition: every data plane's deploy_branch must contain the est-dry-dock change before the new controller deploys. Sending id to an older checkout fails config parsing (extra="forbid") and wedges that plane until its branch updates. Planes tracking main pick it up automatically (the worker checks out fresh origin/<branch> every action); audit select distinct deploy_branch from data_planes and rebase or retire divergent branches first. The reverse order is safe: without incoming ids, est-dry-dock emits null-id results, which both old and new controllers ignore.
2. Pause private-link edits (internal-only; the API has no external consumers yet). Pause before migrating: the backfill snapshots the legacy column, so an edit in between would reach the column but never the table.
3. Apply the migration. It is additive and touches nothing the running binaries use, so old binaries keep working.
4. Roll out the new agent-api and controller binaries in any order. The legacy private_links column is left frozen: an old controller keeps reading it until replaced, and a new controller reads the table. The pause keeps the two from disagreeing.
5. Resume edits.
6. Cleanup (separate PR): drop the private_links and *_link_endpoints columns, recreate data_planes_overview, remove the endpoint resolvers.

Because id is now part of the composed stack config, each plane with links gets a one-time config-diff converge on deploy. It produces no resource changes, and the stack-history check short-circuits the DNS waits.

A deploy landing mid-converge is safe: the new binary may read outputs from the old binary's pulumi up, whose results carry null ids, which is exactly the no-observation case. Rollback is redeploying the old binaries: the additive migration stays, and est-dry-dock keeps exporting the legacy endpoint arrays the old controller consumes.

Status semantics

pending means not yet observed for the current config: the initial state, and what any desired edit resets to.

failed today covers per-link input validation errors est-dry-dock catches in its Pulumi program (for example an invalid AWS availability-zone id). These no longer abort the update: the failed link registers no resources while the rest of the plane converges.

provisioned is est-dry-dock's program-level claim, trusted because the controller only reads stack outputs after a successful pulumi up. Provider-side errors during the update still fail the whole pulumi up and the converge retries as before, writing no statuses; attributing those engine-level errors to links is tracked follow-up work.

Editing or deleting a link is allowed in any status: the desired-edit trigger re-pends it and bumps its generation, and the generation guard makes an edit that races a converge safe, so there is no remove-and-add-only correction path.

Testing

• models: unit tests for service_identity() and provider_str().
• control-plane-api: a snapshot of the table-backed privateLinks, plus add/update/remove and authorization tests including the duplicate-identity guard and updating a link that is already pending; the full suite is green.
• data-plane-controller: a #[sqlx::test] on write_private_link_statuses covering a provisioned result, a failed result, no-observation for an absent link, skipped entries (null id, unknown status), and both sides of the generation guard. Parse tests pin the PrivateLinkEntry wire shape and that ControlExports without link_results parses as None. Integration snapshots show the id in the serialized model and link_results in the consumed exports.
• pgTAP tests for the two triggers: insert / config-update / delete each wake the controller; a config update bumps the generation and clears the observation columns; a status write fires neither trigger.
• Regenerated the flow-client GraphQL SDL and the .sqlx offline cache.

Follow-ups (not in this PR)

• The est-dry-dock PR (required before this deploys; see Rollout).
• Drop the legacy private_links and *_link_endpoints columns and recreate data_planes_overview.
• Engine-level failure isolation: pulumi up --continue-on-error, attribute provider errors to links from the engine event log, and converge the rest of the plane past a failing link.
• Surface provider acceptance state (AWS pendingAcceptance, GCP psc_connection_status, Azure connection approval) in details, since endpoint creation succeeds while a manual-accept connection is still pending.