The union's absent-column contract is undefined-or-throws, not null (#731)

[philcunliffe]

Corrects a documented invariant that was never true of the code, and pins the real contract with tests. No runtime behaviour changed.

What the docs claimed

LLP 0015#multi-partition-union and the comment at src/core/query/union-source.js:28 both said:

columns is always forwarded: projecting an absent column reads as null, never throws.

Neither half was ever true.

What actually happens, measured

Built a fixture of two real parquet partitions unioned by unionSources, with additive drift: partition A {id, score}, partition B {id, score, extra}. Instrumented the sub-scans to see the options each partition received and the row.columns it yielded.

The issue's headline claim is directionally right but wrong in one important detail: the projected key is not missing. It is present with the value undefined. JSON.stringify drops undefined-valued keys, which is how the reporter saw [{}, {extra:"x"}].

SELECT extra FROM t
  rows=[{},{},{"extra":"x"}]   ownKeys=[["extra"],["extra"],["extra"]]
  hasOwnProperty('extra')=[true,true,true]  ===undefined=[true,true,false]  ===null=[false,false,false]
SELECT extra FROM t WHERE score > 1          -> identical
SELECT extra AS e / LIMIT 1 / subquery / ORDER BY id  -> same, no throw

SELECT extra FROM t WHERE extra IS NOT NULL  -> THROW ColumnNotFoundError (row 1)
SELECT extra FROM t WHERE extra IS NULL      -> THROW
SELECT id FROM t WHERE extra = 'x'           -> THROW
SELECT coalesce(extra,'none') AS e FROM t    -> THROW
SELECT extra || 'y' AS e FROM t              -> THROW
SELECT id, extra FROM t ORDER BY extra       -> THROW
SELECT DISTINCT extra FROM t                 -> THROW
SELECT extra, count(*) FROM t GROUP BY extra -> THROW
SELECT max(extra) / count(extra) FROM t      -> THROW

SELECT * FROM t   -> OK, hasOwnProperty('extra')=[false,false,true]  (key truly absent)
SELECT extra FROM t, single partition A      -> THROW at plan time

The boundary is copy-versus-evaluate. executeProject takes a fast path for bare identifier projections: it copies row.resolved[name], which is undefined for a partition lacking the column, so nothing ever invokes the cell that would throw. Any other use goes through evaluateExpr, whose identifier lookup misses row.cells and throws at the first row from the lacking partition.

Mechanically: parquetDataSource builds each row's advertised columns from Object.keys(data[0]) (what hyparquet actually returned), and hyparquet returns {} for a file lacking a requested column, so the drifted partition's rows never carry an extra cell at all. The column is addressable in the first place only because union.columns advertises the superset; with no partition having it, planning fails.

The choice made

Option (a) from the issue: correct the doc and the comment to describe the real contract. Not option (b), changing the union layer to null-pad.

The reasoning: the code throws loudly rather than returning silently-wrong data, so current behaviour is defensible; correcting a doc that states a false invariant is strictly an improvement regardless of which option is ultimately chosen; and null-padding is a behaviour change a maintainer should make deliberately, not one slipped in under a doc-accuracy issue. If you would rather have (b), this PR does not foreclose it - it makes the current contract explicit, which is what a behaviour change would then have to supersede.

The LLP 0015 edit is an in-place factual repair rather than a superseding doc, since it corrects a statement that was never true of the code rather than changing anything the spec settled. Minimal and surgical; the section is not restructured.

Also fixed

The inline import('squirreling/src/types.js') type CLAUDE.md forbids. Note it is at line 366 on current master, not the 439 the issue cites - the file has shifted since filing. ScanColumnResults moved into the file's existing @import block.

Tests

Three added, two carrying @ref LLP 0015#multi-partition-union [tests], all over real parquet partitions:

• a projected column one partition lacks reads as undefined, never null (asserts own-key present, values, !== null, and the exact JSON.stringify output)
• evaluating such a column throws ColumnNotFoundError (five shapes: WHERE, WHERE-not-projected, coalesce, ORDER BY, aggregate)
• SELECT * keeps each partition's row shape, so a drifted key is absent rather than undefined

Suite 3978 pass / 0 fail / 1 pre-existing skip (+3 new); typecheck clean; llp-ref-hygiene 11/11, resolving both new anchors.

Fixes #731

[philcunliffe]

Review round 1 of 76cc970. Verdict: findings - 2 major, 3 minor. All five fixed and pushed as 763c931.

The reviewer re-derived the contract on its own fixtures rather than re-running the PR's, probed roughly 40 SQL shapes, and read executeProject/evaluateExpr/collect in the installed squirreling. Every observable claim held. The documented mechanism did not - and the precondition that mechanism hid is load-bearing.

1. major - the mechanism was backwards, and it hid a load-bearing precondition. FIXED

Both texts said a bare identifier projection "copies the value through and yields undefined". Nothing copies anything, and the projected cell does throw.

executeProject takes the copy path only if (sourceName in row.cells). For a partition lacking the column that is false, so control goes to the else branch, which installs a throwing evaluator thunk and writes no resolved[alias]. squirreling's own comment says why: "fall through to the evaluator so suffix-search and the proper ColumnNotFoundError apply instead of emitting an undefined cell."

projected AsyncRow for SELECT extra FROM t
  resolvedHasKey=false  cells.extra()= THROW ColumnNotFoundError (row 1)
  resolvedHasKey=false  cells.extra()= THROW ColumnNotFoundError (row 2)
  resolvedHasKey=true   cells.extra()= "x"

The undefined and the present own-key come entirely from collect(), whose all-rows-pre-materialized fast path builds item[col] = row.resolved[col] over row.columns, creating the own key from a missing one and never invoking the cell.

Two consequences the old text denied:

• The real boundary is consumer reads resolved versus invokes cells, not copy-versus-evaluate. A caller iterating AsyncRow.cells gets ColumnNotFoundError on a bare projection - the shape the doc promised never throws.
• collect()'s fast path requires every row to carry resolved. Built from partitions yielding hand-rolled AsyncRows without it - a legal AsyncDataSource, and this very test file already exercises legacy bare-iterable sources - SELECT extra FROM t, SELECT extra AS e FROM t and LIMIT 1 all throw. The claim is true today only because every in-repo partition goes through squirreling's asyncRow. unionSources accepts arbitrary sources, so the next partition implementation that skips asyncRow silently converts "reads undefined" into "throws".

Fixed in both places, now naming the resolved-versus-cells seam and stating the asyncRow precondition explicitly. The fixer independently re-derived it before writing and agreed in full, adding one precision: the projected row does carry a resolved object; what is missing is the key.

2. major - the same false invariant survived on the flagship dataset. FIXED

ai-gateway/src/dataset.js:167 still read: "a row object that lacks the key simply reads as null, which is the correct value for 'this partition predates the column'."

withSchemaColumns wraps unionSources and its scan delegates straight through, so this is the identical row path. Measured: key-present/undefined, not null, and WHERE extra IS NULL throws.

After this PR the repo would have held two comments about the same code path flatly contradicting each other, with the surviving wrong one on ai_gateway_messages - the dataset a user chasing this symptom actually queries. Issue #731's option (a) said "correct the doc and comment"; this was the second, more consequential comment. Fixed, now also pointing at scanColumn as the layer that genuinely does normalize holes to null, which is the whole reason that forwarding exists.

3. minor - the plan-failure claim was false of the shipped path. FIXED

"When no partition has it, planning fails with the same error" is true of a bare unionSources and false of every production dataset, each of which is wrapped by a layer advertising the declared schema precisely so planning does not fail:

bare      SELECT extra FROM t   THROW ColumnNotFoundError
wrapped   SELECT extra FROM t   OK  [{},{}]  own=[true,true]

Fixed with the withSchemaColumns qualification.

4. minor - the LLP edit needed a provenance note. FIXED

LLP 0015 is Status: Active, and CLAUDE.md allows in-place edits for changes that do not change meaning; this reverses a stated invariant. The in-place repair is still right (the sentence described the code, wrongly - it was never a decision the spec settled), but as landed the record read as if the spec always said this. Fixed with a one-line corrected-by note.

5. minor - test helper duplication, and "Pinned by" was not honest. FIXED

asyncBufferFromBytes was a byte-identical copy of the one in parquet-source.test.js. test/helpers/ already exists, so both now import a shared parquet_source_fixture.js. And the LLP named "with or without an alias" and "a LIMIT" as part of the pinned contract while neither shape was in the test; both are now in the case table, alongside two new tests - one inspecting the unresolved cell and throwing thunk directly, one proving a partition without resolved makes all three bare shapes throw.

Also checked, clean

• Roughly 40 shapes re-derived independently. Non-throwing, all key-present/undefined/not-null: bare projection, alias, LIMIT 1, LIMIT 0, subquery, ORDER BY id, ORDER BY score DESC, WHERE score > 1, LIMIT 2 OFFSET 1, SELECT extra, extra, SELECT t.extra, SELECT a.extra FROM t a, SELECT id, extra, SELECT *, extra, SELECT count(*), UNION ALL.
• Shapes the PR did not list, all consistent with the rule - so the rule is complete even though the enumeration is not: CASE WHEN extra IS NULL, NOT extra, extra IN (...), cast(extra AS TEXT), HAVING max(extra) IS NULL, GROUP BY id, extra, DISTINCT id, extra, row_number() OVER (PARTITION BY extra ...), and a JOIN projecting a.extra. Every one throws. No shape contradicts the documented rule.
• SELECT * genuinely distinguishes absent from undefined (hasOwnProperty [false,false,true]), and that assertion fails under a null-padding implementation.
• Test discrimination is honest, not loose. assert.deepEqual here is node:assert/strict, so [undefined, undefined, 'x'] does not match [null, null, 'x']; combined with the explicit === null check, the exact JSON.stringify string, and the hasOwnProperty array, the tests fail against a null-padding union and against the old claims.
• Consumer behaviour for undefined versus null. format.js:238 maps both to '', so table and markdown render identically - no bug. json/jsonl, and therefore the MCP query_sql surface, do differ visibly: JSON.stringify drops the undefined key, producing ragged objects, which is exactly the reported symptom. The LLP text already calls that out. rowBytes, truncateRow and inferColumns all tolerate undefined. In-repo AsyncRow-level consumers are safe: visibility.js:187 prefers resolved and falls back only when the key is genuinely present, and iceberg/store.js:565 runs against a single source, never a union projection.
• The @import fix: zero import( occurrences remain in that file, and tsconfig.json includes test, so typecheck genuinely covers it.
• Conventions clean in all changed files; the LLP edit is minimal and the #multi-partition-union anchor resolves.
• Ran: npm test 3980 pass / 0 fail / 1 skip; typecheck clean; llp-ref-hygiene 11/11.

One nuance from the fix worth recording: typecheck initially failed on the new hand-rolled source because AsyncCell is () => Promise<SqlPrimitive>, not sync. Caught by npm run typecheck, which npm test alone does not cover.

The head has moved to 763c931, so the next tick reviews that head (round 2).

[philcunliffe]

Review round 2 of 763c931. Verdict: findings - 1 major, 3 minor. All fixed and pushed as a08857e. The fixer also corrected two claims in the review itself, with evidence; both corrections are recorded below because they changed what landed.

1. major - the resolveable gate is a missing third precondition. FIXED

The corrected text named two preconditions for the undefined read and stopped. There is a third, in the very function the text cites. executeProject computes up front:

const resolveable = plan.columns.every(col =>
  col.type === 'star' || col.type === 'derived' && col.expr.type === 'identifier'
)

and emits no resolved map at all when that fails. So one non-identifier output column - an expression, a function, even a literal - collapses collect()'s fast path for the whole result, and the drifted column's thunk is invoked and throws:

SELECT extra FROM t                   -> extra=UNDEF | UNDEF | "x"
SELECT id, extra FROM t               -> fine (all identifiers)
SELECT *, extra FROM t                -> fine (star + identifier)
SELECT extra, 1 AS lit FROM t         -> THROW ColumnNotFoundError
SELECT extra, score * 2 AS d FROM t   -> THROW
SELECT extra, coalesce(score,0) AS c  -> THROW

Why it mattered: the text's dichotomy was "bare identifier projection reads undefined" versus "anything that evaluates the absent column throws", and everything it listed on the throwing side was an operation on the absent column. SELECT extra, 1 AS n evaluates nothing on extra and throws. A maintainer using the paragraph to judge whether a query is safe would get the wrong answer for a common shape. And the corrected text is more likely to be trusted than the old one precisely because it cites internals.

Fixed in both places with the three-condition formulation, plus tests for both throwing shapes and a SELECT score, extra FROM t case so the gate's boundary is pinned on both sides.

2. minor - "only while every partition yields rows built by asyncRow" was a false necessary condition. FIXED

asyncRow is not required. resolved is originated only by asyncRow and propagated by cachedDataSource and executeProject, but nothing checks provenance - collect() tests if (!rows[i].resolved) and nothing more. Verified: a hand-rolled source that never calls asyncRow but attaches its own resolved keeps the fast path. That is the difference between a rule a new partition implementation must satisfy and an implementation detail it happens to satisfy today. Reworded to name the real rule.

3. minor - the false invariant survived in three more places. FIXED

Round 1's major #2 was this exact thing one file over, and the sweep stopped at the instance it named:

• message_projector.js:25 - the comment a developer reads while adding a nullable column, i.e. the moment the claim is load-bearing.
• llp/0055:77-80 - the parenthetical "the same additive schema-drift rule withSchemaColumns already applies to row reads", which dataset.js:202 @refs straight at, so a reader following the corrected comment's own reference landed on the uncorrected claim.
• test/core/ai-gateway-dataset.test.js:287 - assert.equal(seen[0].git_remote ?? null, null, 'absent column reads as null'), where ?? null launders undefined into null and the message asserts the untrue half.

All three fixed. For LLP 0055 the fixer took both offered options - dropped the parenthetical and appended a corrected-by note - reasoning that dropping alone would leave a reader arriving via the @ref unable to tell the doc was ever wrong, while a note alone leaves the false claim as the first thing they read. The note states explicitly that what the decision settled is unchanged.

4. minor - dataset.js overclaimed why scanColumn is forwarded. FIXED

"...which is the whole reason the scanColumn forwarding exists" contradicted the block comment 30 lines below in the same function. The forwarding exists for LLP 0055's streaming aggregates; the null normalization is a correctness duty inside it, not its motivation. Corrected.

Two corrections the fixer made to this review, both verified and both material

The flagship-wrapper example in finding 1 was wrong. SELECT git_remote, 1 AS n FROM ai_gateway_messages does not throw. createDataSource builds only from storage.dataSourceForTable, which is icebird, not parquetDataSource (used solely by the s3 plugin). Icebird answers a scan for a column it lacks with row.columns containing the name and a cell resolving to undefined - it never omits the cell, so there is no throwing thunk. Measured over two drifted cache partitions:

SELECT git_remote FROM t                     -> NULL | g     (scanColumn path)
SELECT git_remote, 1 AS n FROM t             -> NULL,n=1 | g,n=1
SELECT date, git_remote, 1 AS n FROM t       -> git_remote=UNDEF | g
WHERE / ORDER BY / DISTINCT / coalesce / max / count on git_remote -> all OK

The resolveable gate still fires (hence the UNDEF), it just cannot produce a throw when the underlying cell exists.

Consequence: the PR's own dataset.js text was a fourth wrong mechanism. Its claim that "anything that evaluates it throws ColumnNotFoundError" is false for that stack. The fixer corrected it beyond the ask rather than leave it, and the comment now distinguishes the two backends explicitly: icebird-backed partitions resolve to undefined, and the throwing half of LLP 0015 belongs to parquet-backed partitions, which omit the cell entirely.

Finding 3's proposed message_projector.js wording repeated that same error ("reads as undefined or throws"). The fixer applied the finding's intent with verified wording instead - hole reads undefined on the row path, JSON.stringify drops it, only scanColumn normalizes to null, and SQL predicates treat it as NULL either way (verified: WHERE git_remote IS NULL returns the old row).

That is the right call. This is the third attempt at describing this mechanism, and writing prescribed text that had been measured false would have made it the fourth.

Verified from round 1

• Major 1 (mechanism backwards) - landed, and re-derived from the squirreling source rather than accepted. The else-branch installs the throwing thunk and writes no resolved entry; evaluateExpr throws after exact-match and suffix-search both miss; collect()'s fast path creates the own-key-with-undefined. Incomplete only in the way finding 1 describes.
• Major 2 (dataset.js) - landed; the scanColumn normalization claim is true (dataset.js:216-218 rewrites undefined holes to null). Its rationale clause was wrong (finding 4) and its throws clause was wrong (the correction above).
• Minor 3 ("planning fails" qualified) - landed correctly, both halves verified empirically.
• Minor 4 (provenance note) - landed as a blockquote note on an Active doc rather than a silent rewrite.
• Minor 5 (helper extraction) - a faithful merge. parquet-source.test.js keeps rowGroupSize: 2; the union test's implicit undefined hits hyparquet-writer's default parameter, so behaviour is byte-identical. The no-resolved test does discriminate: the negative control (identical hand-rolled source with resolved added, nothing else changed) returns undefined instead of throwing, so it measures resolved and not an artifact of the row shape.

Also checked, clean

• One non-resolved partition really does disable the fast path for the whole result - utils.js:62-67 breaks out of the allMaterialized loop on the first such row and takes the slow path for every row. The doc's "for the whole result" is exact.
• collect() is the consumer on the shipped path (query/verb.js, commands/local_only.js, commands/clients.js, query/overview.js).
• ~20 further SQL shapes beyond round 1's set all agree with the rule as stated, except the mixed-projection class in finding 1.
• One pre-existing observation, not a finding: withSchemaColumns.scan forwards where verbatim, so only the scanColumn path strips a predicate naming a declared-but-absent column. Predates this PR and the corrected text describes it correctly; flagged only so it is not mistaken for a regression later.
• Conventions clean across all seven changed files; npm test 3980 pass / 0 fail / 1 skip; typecheck clean; llp-ref-hygiene 11/11.

The head has moved to a08857e, so the round budget (2) is spent at an unreviewed head: the next tick triages rather than opening a round 3.

[philcunliffe]

Triage after the review budget (LLP 0017). Two rounds ran; both cleared. The triage re-derived the whole contract itself and found the icebird-facing text added in round 2's fix is false. This is a doc-accuracy PR, so a false documented claim is its entire defect surface.

The blocker

Three sentences are demonstrably false at this head:

• dataset.js:170-171 - "a projection of it yields undefined (key present, not null)"
• message_projector.js:28-31 - "the value a projection hands back is undefined ... which JSON.stringify drops instead of rendering as null. Only the scanColumn aggregate path normalizes the hole to null"
• llp/0015:102-104 - "unless a wrapper advertises the declared schema ... in which case the same undefined-or-throws contract applies"

Measured on a real icebird-backed ai_gateway_messages fixture:

SELECT git_remote FROM t          -> [{"git_remote":null}, ...]   null, rendered, NOT undefined
SELECT git_remote, 1 AS n FROM t  -> {"git_remote":null,"n":1}    no throw
SELECT id, git_remote FROM t      -> undefined, key dropped        (matches the doc)

The cause: squirreling routes any single-column scan through scanColumn when the source offers one - execute.js:284 gates only on plan.hints.columns?.length === 1, no aggregate required - and withSchemaColumns normalizes holes to null on exactly that path (dataset.js:222-224). So calling it "the scanColumn aggregate path" tells a maintainer plain projections are unaffected, when the most common projection of an absent column is precisely the one that contradicts the doc.

A maintainer acting on the LLP sentence would predict undefined-or-throws for the flagship dataset and observe null-or-null.

Why two rounds missed it

The PR's icebird tests pin only the raw scan() rows and the scanColumn chunks (ai-gateway-dataset.test.js:254-296, 298+), never a full executeSql + collect SELECT on the icebird fixture. SQL-surface tests exist only for the parquet union. The false sentences live precisely in the untested gap - and were written late, in response to a reviewer error, which is exactly the condition under which a plausible-but-wrong sentence gets in.

What is verified good

The parquet half is right, and every sentence of it survived independent re-derivation: the undefined read with alias, LIMIT, WHERE on a shared column and a bare sibling; the SELECT extra, 1 AS n sibling collapse; every evaluating shape throwing; SELECT * keeping per-row shape. That is the counterintuitive trap the old "reads as null, never throws" line would have set, and documenting it earns its keep.

The merge is clean: #724's narrowed projection is intact, a parquet partition asked for a column it lacks still emits its rows rather than throwing, #730's NULL-guard tests pass through the shared fixture refactor, and both comment-block edits to union-source.js coexist correctly. Suite 4000 tests, 0 fail.

The decision needed

How to dispose of the false icebird text before merge.

1. Fix in place. A fifth revision stating the measured icebird contract: single-column projection reads null via the single-column scanColumn fast path (which is not aggregate-only); multi-column bare projection reads undefined; nothing throws, because icebird supplies a resolving cell. Amend the LLP 0015 wrapper sentence to say the wrapper changes the contract to null-or-undefined-never-throws. Add executeSql + collect SQL-surface tests on the icebird fixture so the backend split is pinned where it was previously untested.
2. Descope. Revert the icebird-facing additions to neutral wording ("stays addressable; exact value depends on the read path, see LLP 0015") and ship only the parquet-union correction, which is verified true. Smaller and safer, but leaves the question that spawned round 2's dispute unanswered in the docs.

Neutral is not choosing between these unilaterally. This PR has now described the mechanism wrongly five times - the original claim, and four successive corrections, each of which passed a review round before the next one caught it. Whether the icebird half is worth a fifth attempt or should be descoped is a judgement about how much more effort this document deserves, and that is yours.

Option 3, shipping as-is, is rejected on neutral's own standard: for a documentation PR a false documented claim is a production defect, and this one sits on the dataset users actually query.

How to unstick

Reply with a comment on this PR (or push to the branch). Neutral monitors this thread and will re-engage with your guidance on its next tick.