private-networking: controller reads links from the table, retire the projection#3070
Closed
jshearer wants to merge 7 commits into
Closed
private-networking: controller reads links from the table, retire the projection#3070jshearer wants to merge 7 commits into
jshearer wants to merge 7 commits into
Conversation
The flat `private_links` array plus separate `*_link_endpoints` output columns gave no reliable way to tie an endpoint back to its link and persisted no per-link status. Promote each link to a `data_plane_private_links` row with a stable id and controller-owned status, so the API reads status instead of reconstructing it from an opaque, externally-owned shape. * New table + migration: per-link `config`, generated `service_identity`, controller-owned `status`/`details`/`error`, unique on `(data_plane_id, service_identity)`; a trigger wakes the controller and projects rows back into `private_links` during the transition. * dpc records `pending`/`provisioned` by matching provisioned endpoints to links on `(provider, service_identity)`, and keeps reading desired links from the projected `private_links` column so it has no deploy-ordering dependency on the agent-api cutover. * agent-api exposes `privateLinks` as rows and replaces the wholesale mutation with per-link add/update/remove; `failed` awaits a follow-up est-dry-dock change.
jshearer
force-pushed
the
jshearer/privatelinks_fast_follow
branch
from
be57daf to
eb61a36
Compare
jshearer
force-pushed
the
jshearer/privatelinks-contract
branch
from
1bfac72 to
a282e05
Compare
…er-provider link uniqueness * dpc pins `links_read_at` (the DB instant its desired links were read) at the `PulumiUp1` poll and skips status writes to rows changed after it, so an edit landing mid-converge stays `pending` instead of briefly reading `provisioned` with pre-edit details; the wake trigger has already queued the converge that settles it. A NULL instant (task state from a prior binary) disables the guard. * The unique constraint becomes `(data_plane_id, provider, service_identity)`: identities are only meaningful per provider, and a same-provider duplicate is the case worth preventing, since it wedges the converge on colliding Pulumi resource names in est-dry-dock. Cross-provider Pulumi identities cannot collide, so the old data-plane-wide scope was stricter than anything requires. * The migration pre-flight also rejects legacy elements that would not decode as a `models::PrivateLink` variant (possible via historical hand-edits), aborting at migration time instead of nulling the whole `dataPlanes` query at read time. The redundant standalone `data_plane_id` index is dropped; the unique index leads with it.
jshearer
force-pushed
the
jshearer/privatelinks-contract
branch
from
a282e05 to
ad1ef96
Compare
The table is read and written only by the agent-api resolvers (which gate on the View/Modify private-networking capabilities) and the data-plane controller, both connecting as `postgres`. PostgREST exposes only the `public` schema, so moving the table to `internal` makes it unreachable over REST and lets the RLS policy, the `authenticated` column grants, and the `reporting_user` grant go away rather than maintaining a second authorization surface that duplicates the capability gate. This mirrors `internal.invite_links`, another agent-api-mediated resource. Query sites are schema-qualified accordingly.
jshearer
force-pushed
the
jshearer/privatelinks-contract
branch
from
ad1ef96 to
aa25b31
Compare
A `pending` link has not yet converged to its current desired config, so editing it moves the goalposts mid-flight and reopens the stale-status window. Reject `updateDataPlanePrivateLink` when the link is `pending`; `provisioned` and `failed` links stay editable in place. Deletion is still permitted in any status, so a mistyped link that never provisions is corrected by remove-and-add rather than being stuck. The resolver already reads the link row for authorization, so it now returns the status too at no extra query.
jshearer
force-pushed
the
jshearer/privatelinks-contract
branch
from
aa25b31 to
5c64cf3
Compare
The post-converge status write and the change trigger were only validated by sqlx at compile time; this adds runtime coverage for both. * Extract the per-link status write into `write_private_link_statuses` and cover it with a `#[sqlx::test]`: a matched endpoint records `provisioned`, an unmatched one within a published provider demotes to `pending`, a provider that published nothing is left untouched, and a row edited after `links_read_at` is skipped (with a NULL instant disabling that guard). * pgTAP test for `on_data_plane_private_links_change`: an insert, a config update, and a delete each reproject `data_planes.private_links` and enqueue a `converge`, while a controller-owned status update fires neither (the scoping that prevents a reconverge loop).
jshearer
force-pushed
the
jshearer/privatelinks-contract
branch
from
5c64cf3 to
41473a2
Compare
…atus observation to the cutover The post-converge status write guarded against mid-converge edits with a read-timestamp comparison, which infers whether a row changed after the converge read its desired state. Replace that inference with an explicit per-link `generation` so a converge's observed status lands only on the exact configuration it applied. * Add a `generation` column bumped by a `before update of config, provider` trigger that also clears the observation columns, so any desired edit (from the API or a direct support edit) re-pends the link and invalidates its status. * Drop the controller status write from this additive change: it returns to reading the projected column unchanged, and the controller reads and pins `(id, generation)` only in the cutover change where it queries the table directly. * Allow editing a `pending` link: the generation bump makes an edit that races a converge safe, so remove-and-add is no longer the only correction path.
…atus by generation, retire the projection The controller now reads desired links directly from `data_plane_private_links` and records their observed status there, so the `data_planes.private_links` projection has no remaining reader and the change trigger becomes wake-only. Roll out this controller binary before applying the cutover migration, so a controller still reading the column keeps seeing projected updates until it is replaced. * Read desired links with their `id` and `generation` from the table instead of the projected column, ordered deterministically; only `config` is handed to est-dry-dock, preserving the prior wire shape into the provisioner. * Pin each link's `(id, generation)` at the `PulumiUp1` poll and land the post-converge status write only on rows whose generation still matches, so a link edited mid-converge is skipped and settled by the converge its own edit queued. * Replace the change trigger with a wake-only version: it still sends a prompt `converge` on any link change but no longer projects into `data_planes.private_links`. That column and the `*_link_endpoints` columns are left in place for a single follow-up cleanup that also recreates `data_planes_overview`.
jshearer
force-pushed
the
jshearer/privatelinks-contract
branch
from
41473a2 to
e48e8d3
Compare
jshearer
force-pushed
the
jshearer/privatelinks_fast_follow
branch
from
22384ed to
b1522ed
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3063 (base branch
jshearer/privatelinks_fast_follow); review and merge that first.The cutover that follows the additive private-networking change. In #3063 desired links live in
data_plane_private_linkswhile the controller still reads them from the projecteddata_planes.private_linkscolumn, so it has no deploy-ordering dependency on the agent-api cutover. This PR moves the controller onto the table directly and retires the projection.What changed
data_plane_private_links(ordered bycreated_at, id); the id is still withheld from est-dry-dock, so the Pulumi input shape is unchanged. Thelinks_read_atguard instant continues to come from thedata_planesfetch in the same poll, which reads no later than the links query it now guards.data_plane_private_linkstrigger drops its projection intodata_planes.private_linksand becomes wake-only, still sending the controller a converge promptly on any link change.Deploy ordering
Roll out the controller binary before applying the migration. A controller still reading the column keeps getting projected updates until it is replaced; once it reads the table, the projection is gone.
Not in this PR
The now-unused
private_linkscolumn is left in place. It and the legacy*_link_endpointscolumns are all projected by thedata_planes_overviewview and exposed by the agent-api endpoint resolvers, so dropping them (recreating the view once, and removing those resolvers) is a single follow-up rather than recreating the view per column. Until that follow-up lands,data_planes_overview.private_links(and the column itself) is frozen at its last projected value; anyone troubleshooting link config from the overview view or the raw column should readdata_plane_private_linksinstead. The endpoint columns stay fresh, since the controller still writes them.Testing
supabase db reset; sqlx cache regenerated.data_plane_private_linkstrigger pgTAP test is updated to the wake-only contract: insert / config-update / delete each enqueue aconvergeand a status update does not, with the retired projection assertions dropped.data_planessuites green; offline build clean.