capture+materialize: Support Nonsensitive Field Overlays by willdonnelly · Pull Request #3119 · estuary/flow

willdonnelly · 2026-07-06T15:05:46Z

Description:

This PR:

Adds a new nonsensitive: true JSON schema annotation for endpoint configs.
Adds an unseal::decrypt_with_overlay helper function which validates overlay non-sensitivity and does the extract-decrypt-merge dance around overlay properties and the main config.
Adds sealed_config_json fields to the Open RPCs in the capture and materialization protocols, which will be used to provide the encrypted endpoint config to connectors. Currently they only receive the decrypted config, but in order to produce configUpdates which modify their own configuration via the overlays they'll need access to the raw encrypted one too.
Refactors the capture and materialization task startup process (in both the old and new runtimes) so that endpoint config decryption is deferred until after the initial Spec RPC. This is necessary because in order to properly validate that an overlay only modifies nonsensitive fields we need the config schema from the spec response. This is mostly a behavior-preserving refactor except that invalid configs would now produce an error after the initial spec RPC where previously the error would occur before the connector was launched at all.
Swaps out unseal::decrypt_sops for unseal::decrypt_with_overlay in those places. This is basically the followup to the previous refactor, I just found it cleaner to make them separate commits.
Populates the sealed_config_json property of Open RPCs.

Since I'm not super familiar with the runtime codebase I've broken this down into fairly small, atomic commits so that I could review them individually and be pretty confident they're correct.

Part of #2985

Workflow steps:

Nothing really changes. In theory someone could manually produce a task with an endpoint overlay but nothing will be generating them automatically yet, because we need the runtime to fully support this before we start using the feature.

willdonnelly · 2026-07-06T21:18:03Z

I believe this is ready for review now. I've tested locally that with these changes a source-hello-world->materialize-postgres pipeline can be set up as normal when not using an overlay, and that an overlay can be added without re-encrypting a task config and (with a connector modified to mark those fields as nonsensitive) will be applied to the effective task config.

I have not specifically tested this with both the v1 and v2 runtimes because I don't know how to toggle between those, but the actual code changes are basically identical between the two so I think that's probably sufficient testing just to make sure I didn't break something dumb but subtle.

willdonnelly · 2026-07-07T15:35:12Z

Hmm, the "Dekaf Test" CI failure appears to be because the source-http-ingest connector it uses hasn't been bumped to have #3059 yet, I think?

This is necessary so that builds of this connector pick up the "tolerate unknown JSON fields" change from Flow commit 184db0e, which in turn is necessary so that Flow CI tests of estuary/flow#3119 can pass since the Dekaf test uses `source-http-ingest:dev` and currently it errors out because that PR adds a new field which it doesn't recognize.

williamhbaker

LGTM, no blockers, a couple of things to consider as comments, and also concurring deployment:

As is, this PR will start to populate the new sealed config field right away. We've been bitten numerous times by new field additions where consumers use the strict pbjson structs that disallow unknown fields. As you noted with source-http-ingest, it hadn't yet been updated with the recent move to more lenient pbjson structs that do allow unknown fields. Anything else that hasn't yet been updated will similarly crash if it sees these populated fields.

Off the top of my head, that would be the other rust connectors: source/materialize-kafka, and all the ATF connectors (at least, I don't remember seeing us having updated those recently, but I might have missed it).

Since the new field only shows up on Open messages it is just limited to consumers of Open messages, which is only connectors.

I'm betting this would also break flowctl preview and flowctl raw materialize-fixture unless an up-to-date flowctl is used, built from at least this commit or later. This is a practical concern for connectors repo integration tests, unless they are always pulling/building the latest flowctl.

williamhbaker · 2026-07-09T15:11:01Z

+    Ok(())
+}
+
+fn display_ptr(ptr: &json::Pointer) -> String {


Any reason not use the Display already implemented by json::Pointer?

The only functional difference I can see is that using Display an overlay which attempts to modify the root object would produce an error message like overlay modifies location , which is not marked nonsensitive (since the RFC 6901 representation of the root is the empty string) whereas this helper function special-cased the root as / for readability in the error message.

That probably isn't sufficient reason to reimplement a hackier version of JSON Pointer stringification so I've removed this helper.

Actually that's a dumb objection anyway, I've added single-quotes so that error would be overlay modifies location '', which is not marked nonsensitive which makes it pretty unambiguous to me.

williamhbaker · 2026-07-09T15:57:07Z

+    fn empty_overlay_is_a_noop() {
+        check(json!({})).unwrap();
+        // An empty object writes nothing even at an otherwise-sensitive location.
+        check(json!({"credentials": {}})).unwrap();


I can't think of a reason why this would need fixing, but if I'm understanding this correctly an empty object would kind of write something to a sensitive location if it were not of type object (like a string), replacing it instead with an object. In any sane world the resulting config would fail to parse in the connector, or otherwise not have any kind of exposure risk.

The worst I can think of is a connector that had something like tls: true as a sensitive field and then somehow it thought tls: {} meant false 🤷

Good catch, yeah, an empty object (or even a nested chain of objects as long as the final one is empty) passes the current nonsensitive-property validation but it probably shouldn't since merge(scalar, object) = merge({}, object).

I think to close that gap the validation rule needs to be something like "we only recurse into an object if the target schema permits an object at this location" and an object in a scalar location needs to be rejected.

mdibaiee · 2026-07-09T20:36:15Z

Am I right in my understanding that this allows connectors to emit an update to a non-encrypted field (e.g. advanced.feature_flags without having to call the decryption endpoint? I love the feature

willdonnelly · 2026-07-10T15:24:21Z

A couple of things to consider as comments

Thanks, I believe those should be addressed now.

As is, this PR will start to populate the new sealed config field right away. We've been bitten numerous times by new field additions where consumers use the strict pbjson structs that disallow unknown fields. As you noted with source-http-ingest, it hadn't yet been updated with the recent move to more lenient pbjson structs that do allow unknown fields. Anything else that hasn't yet been updated will similarly crash if it sees these populated fields.

Off the top of my head, that would be the other rust connectors: source/materialize-kafka, and all the ATF connectors (at least, I don't remember seeing us having updated those recently, but I might have missed it).

Is your recommendation to split this up into separate pieces, adding the field and making sure the connectors pick that up and then populating it in a separate PR once that's done, or are you just calling out that we should make sure everything's picked up the recent allow-unknown-fields change before merging this?

If it's the latter, I think the current state of things is:

All Go connectors would be fine, because they've never forbidden unknown fields
All Python connectors would be fine for the same reason
Rust Connectors
- source-http-ingest was just updated yesterday to have the allow-unknown-fields change
- source-kafka was updated a week or two ago
- materialize-kafka uses protobuf rather than JSON so it should be unaffected AFAIK
ATF Connectors: Current Flow pin before the allow-unknown-fields change, so still vulnerable

TL;DR it looks like only ATF connectors still need any work, we would need to bump their Flow version pin and make sure they're all released before merging this.

I'm betting this would also break flowctl preview and flowctl raw materialize-fixture unless an up-to-date flowctl is used, built from at least this commit or later. This is a practical concern for connectors repo integration tests, unless they are always pulling/building the latest flowctl.

The integration tests do actually pull down the latest flowctl (fetch-flow.sh pulls it from the latest dev-next release at the start of each CI run). But I'm not sure I understand the mechanism here though, what exactly would break when using an older flowctl build?

williamhbaker · 2026-07-10T15:39:51Z

Is your recommendation to split this up into separate pieces, adding the field and making sure the connectors pick that up and then populating it in a separate PR once that's done, or are you just calling out that we should make sure everything's picked up the recent allow-unknown-fields change before merging this?

Yeah I think making sure everything's picked up the allow-unknown-fields change would be the practical way to go about it. Doesn't seem like there's much additional needed for that.

I'm not sure I understand the mechanism here though

I was probably wrong about this actually. I was thinking that since flowctl preview etc. read these values from connector RPCs, that they would be at risk of not having them in their pbjson structs. But it is flowctl itself that would have added them, so they should actually be in-sync by construction.

willdonnelly · 2026-07-10T17:34:08Z

Yeah I think making sure everything's picked up the allow-unknown-fields change would be the practical way to go about it. Doesn't seem like there's much additional needed for that.

Okay, thanks. I've uploaded estuary/airbyte#360 for the ATF connectors and will make sure that's merged and pushed to all 40-something of them before this gets merged.

Add the field to the capture Open RPC and thread it through all construction and destructure sites, which default or ignore it. It's actually populated in a later commit. The regenerated protobuf bindings follow in the next commit.

Checked-in output of the Rust and Go protobuf generators for the preceding capture Open change. No hand-written edits.

Defer endpoint config decryption until after the connector's spec response is available, capturing the sealed config from the matched endpoint. This prepares for overlay-aware decryption, which needs the connector's config schema to validate overlay property non-sensitivity.

Decrypt endpoint configuration with unseal::overlay::decrypt_with_overlay, which applies sops.overlay properties subject to config schema validation.

Thread the sealed endpoint configuration into the Open request's sealed_config_json field, so running capture connectors can emit a configUpdate which adjusts their nonsensitive sops.overlay without re-encrypting the whole configuration.

Add the field to the materialize Open RPC and thread it through all construction and destructure sites, which default or ignore it. It's actually populated in a later commit. The regenerated protobuf bindings follow in the next commit.

Checked-in output of the Rust and Go protobuf generators for the preceding materialize Open change. No hand-written edits.

Defer endpoint config decryption until after the connector's spec response is available, capturing the sealed config from the matched endpoint. This prepares for overlay-aware decryption, which needs the connector's config schema to validate overlay property non-sensitivity. Dekaf decrypts its own nested config outside this path, so its sealed config is None and nothing is decrypted here.

Decrypt endpoint configuration with unseal::overlay::decrypt_with_overlay, which applies sops.overlay properties subject to config schema validation.

Thread the sealed endpoint configuration into the Open request's sealed_config_json field, so running materialization connectors can emit a configUpdate which adjusts their nonsensitive sops.overlay without re-encrypting the whole configuration. Dekaf has no sealed config on this path, so its Open is left unpopulated.

Removes an unnecessary `display_ptr` helper function and tweaks the nonsensitivity validation logic to handle an edge case around empty objects in the overlay correctly.

Part of estuary/flow#2985, we need all connectors to have picked up the "tolerate unknown fields" change from Flow commit 184db0e before merging estuary/flow#3119 which introduces a new sealed endpoint config property to the Open RPCs. As far as I'm aware everything else either has this change or didn't need it, but airbyte-to-flow connectors still need it.

This could probably be folded into a prior commit but I'm just keeping up with codebase churn so the PR can land cleanly.

willdonnelly force-pushed the wgd/2026-06-04-overlay-aware-configs branch from 6f50475 to 3fd8931 Compare July 6, 2026 21:13

willdonnelly marked this pull request as ready for review July 6, 2026 21:13

willdonnelly requested a review from a team July 6, 2026 21:14

willdonnelly added the change:planned This is a planned change label Jul 6, 2026

willdonnelly mentioned this pull request Jul 7, 2026

source-http-ingest: Update Flow dependencies estuary/connectors#4769

Merged

willdonnelly force-pushed the wgd/2026-06-04-overlay-aware-configs branch from 3fd8931 to 9dc610a Compare July 9, 2026 14:58

williamhbaker self-requested a review July 9, 2026 15:02

williamhbaker previously approved these changes Jul 9, 2026

View reviewed changes

willdonnelly dismissed williamhbaker’s stale review via fb74e33 July 10, 2026 15:05

williamhbaker previously approved these changes Jul 10, 2026

View reviewed changes

willdonnelly mentioned this pull request Jul 10, 2026

airbyte-to-flow: Update Flow version pin estuary/airbyte#360

Merged

willdonnelly added 12 commits July 10, 2026 17:39

doc: Add nonsensitive schema annotation

bcd9ad5

unseal: Add decrypt_with_overlay operation

b0c3483

capture: Add sealed_config_json to Open protocol

7ba1b2d

Add the field to the capture Open RPC and thread it through all construction and destructure sites, which default or ignore it. It's actually populated in a later commit. The regenerated protobuf bindings follow in the next commit.

capture: Regenerate protobuf bindings for sealed_config_json

a39215e

Checked-in output of the Rust and Go protobuf generators for the preceding capture Open change. No hand-written edits.

capture: Overlay-aware endpoint decryption

22f2864

Decrypt endpoint configuration with unseal::overlay::decrypt_with_overlay, which applies sops.overlay properties subject to config schema validation.

materialize: Regenerate protobuf bindings for sealed_config_json

bf28223

Checked-in output of the Rust and Go protobuf generators for the preceding materialize Open change. No hand-written edits.

materialize: Overlay-aware endpoint decryption

0ba7bdd

Decrypt endpoint configuration with unseal::overlay::decrypt_with_overlay, which applies sops.overlay properties subject to config schema validation.

unseal: Fixes from PR feedback

9a4c11c

Removes an unnecessary `display_ptr` helper function and tweaks the nonsensitivity validation logic to handle an edge case around empty objects in the overlay correctly.

runtime: Fix sealed_config_json default

3ed5f64

This could probably be folded into a prior commit but I'm just keeping up with codebase churn so the PR can land cleanly.

willdonnelly dismissed williamhbaker’s stale review via 3ed5f64 July 10, 2026 19:07

willdonnelly force-pushed the wgd/2026-06-04-overlay-aware-configs branch from fb74e33 to 3ed5f64 Compare July 10, 2026 19:07

williamhbaker approved these changes Jul 10, 2026

View reviewed changes

Uh oh!

Conversation

willdonnelly commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

willdonnelly commented Jul 6, 2026

Uh oh!

willdonnelly commented Jul 7, 2026

Uh oh!

williamhbaker left a comment

Choose a reason for hiding this comment

Uh oh!

williamhbaker Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

willdonnelly Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

willdonnelly Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

williamhbaker Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

williamhbaker Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

willdonnelly Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mdibaiee commented Jul 9, 2026

Uh oh!

willdonnelly commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

williamhbaker commented Jul 10, 2026

Uh oh!

willdonnelly commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

willdonnelly commented Jul 6, 2026 •

edited

Loading

willdonnelly Jul 9, 2026 •

edited

Loading

willdonnelly commented Jul 10, 2026 •

edited

Loading