Resolve column renames on the ATTACH channel import path - #15142
Merged
rtibbles merged 3 commits intoAug 4, 2026
Conversation
Contributor
Build Artifacts
Smoke test screenshot |
rtibblesbot
force-pushed
the
issue-15098-9ca749
branch
from
August 4, 2026 08:29
e6caf71 to
668b73b
Compare
rtibblesbot
marked this pull request as ready for review
August 4, 2026 08:47
A per_row mapping whose value names a column on the source table is a rename, and a rename is expressible directly in the generated SELECT. Both can_use_sqlite_attach_method and raw_attached_sqlite_table_import previously resolved a mapping only against attributes of the import class, so any rename disqualified the whole table from the single statement transfer and sent it down the row-by-row Python mapper. Give both a shared resolver that mirrors generate_row_mapper's order — source column first, then non-callable class attribute — returning the SQL fragment, or None when the mapping has no SQL equivalent. The guard reads None as reject; the SELECT builder reads it as raise. Callable mappers and per_table mappers still take the row-by-row path. Only content_localfile newly qualifies, under the schema 5 and earlier import classes, via the file_size to file_size_bigint rename. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard unit tests prove the decision; these prove a real schema 5 import acts on it. One asserts content_localfile reaches raw_attached_sqlite_table_import, the other that the file_size_bigint values it lands — a set size and four NULLs — match a re-import forced down the row-by-row path. Both flush before re-importing: check_and_delete_existing_channel cancels a same-version re-import, so without an empty database set_content_fixture imports nothing and both assertions pass vacuously. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comment above BENCHMARK_CHANNELS predicted the two import phases would land near-flat, on the grounds that the mapping class maps ContentNode with a "post" key. That half still holds, but content_localfile now ATTACHes, so the phases diverge by roughly that table's cost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rtibblesbot
force-pushed
the
issue-15098-9ca749
branch
from
August 4, 2026 09:20
668b73b to
89e5d4f
Compare
rtibbles
approved these changes
Aug 4, 2026
rtibbles
left a comment
Member
There was a problem hiding this comment.
All changes make perfect sense - no changes to the core channel import tests and they still pass, and the benchmark results show a modest improvement in performance.
The reduction in number of issued queries is the most significant.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Importing a channel published before schema 6 transferred
content_localfilea row at a time through a Python mapper, because the destination'sfile_size_bigintcolumn reads from a differently-named source column and that disqualified the whole table from the single-statement ATTACH path. The ATTACH guard now accepts aper_rowmapping whose value names a column on the source table, and the generatedSELECTreads that column directly.Only
LocalFilenewly qualifies, underNoIncludedPresetsChannelImportand theNoLearningActivitiesChannelImportthat inherits its entry. The three other renames named in the issue stay on the row path for reasons this change does not touch:File.local_file_idandChannelMetadata.root_idsit beside a"post"key and a callable mapper respectively, andLocalFileunderNoVersionChannelImportcarries aper_tablemapper.References
Fixes #15098. Benchmark harness: #15110.
Reviewer guidance
channel_import.py:482— the resolver prefers a source column over a same-named class attribute, mirroringgenerate_row_mapper:405-426. Check the two orders cannot drift apart, since a disagreement would silently insert a different value on each path.channel_import.py:724—can_use_sqlite_attach_methodhas a second caller in_can_use_optimized_pre_deletion, where aTruemakesdelete_old_channel_tree_dataskip rows.LocalFileis filtered out ofmodels_to_deletebymerge_models, so it cannot reach that caller today; check that argument still holds if another model later qualifies.sqlite_table_importtruncates strings tomax_lengthand substitutes column defaults forNone, and the ATTACH path does neither. Confirm the schema 5content_localfilecolumns admit no difference —extensionisString(40)on both sides,availableis a constant under both paths, andfile_size_bigintis nullable with no default.Measured gain
Two captures of
integration_testing/scripts/content_operations_benchmark.py, one harness copy, same machine andKOLIBRI_HOME,--runs 3, one sitting, againstc9d7f950ab6b5a1199e3d6c10d7f0103(Khan Academy English US, 34,488 content nodes, 77,315 local files).fresh_importfresh_import_no_attachannotationupgrade_importWithin the branch capture, which shares a sitting and a preceding file copy and so beats the cross-capture delta,
fresh_importcomes out at 12.210s againstfresh_import_no_attachat 13.059s.The statement counts carry more than the times:
fresh_importdrops from 405 SQLAlchemy statements to 327. That −78 is exactly the row path's 78 batched inserts over 77,315 local files, plus the sourceSELECTthey were read from, collapsing into oneINSERT OR IGNORE … SELECT.annotationanddrive_enumerationare flat, so no capture-wide contention is in play.Acceptance criteria
Ticked here rather than on the issue: this account cannot edit the issue body.
per_rowmappings are constants or renames uses the ATTACH path.SELECTreads the renamed destination column from its source column.per_tablemapper still uses the row-by-row path.content_localfileimports via ATTACH for a schema 5 channel.can_use_sqlite_attach_methodreturnsTruefor a rename mapping andFalsefor a callable mapper.test_channel_import.pygreen — 289 passed on both SQLite and PostgreSQL.file_size_bigintvalues imported via ATTACH match those imported via the row path for the same source.AI usage
Used Claude Code to relax the guard, write the tests and take both benchmark captures, following a plan whose correctness argument I directed — in particular the NULL-parity and column-compatibility checks that make moving this one table safe. Verified with the channel import suite on SQLite and PostgreSQL, and with the content operations benchmark.
@rtibblesbot's comments are generated by an LLM, and should be evaluated accordingly
How was this generated?
🟡 Waiting for feedback
Last updated: 2026-08-04 09:28 UTC