feat(transaction): compound append + add-index via action transactions#7674
Draft
wjones127 wants to merge 6 commits into
Draft
feat(transaction): compound append + add-index via action transactions#7674wjones127 wants to merge 6 commits into
wjones127 wants to merge 6 commits into
Conversation
Extend the experimental action-transaction types for the first compound-commit slice. `AddFragments` now carries `NewFragment`s, each with an operation-local `local_id` placeholder so a later action can reference a fragment before its real id is assigned. Adds the `AddIndex` action, which registers an index segment and expresses coverage as the union of already-committed fragment ids and same-op placeholders. All behind the non-default `unstable-action-transactions` feature. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an ExperimentalUserOperation message + oneof arm (field 115) to the canonical transaction.proto. The action list is carried as an opaque `bytes` payload (a serialized experimental UserOperation) so the PMC-voted schema stays free of the unstable action structure. Builds that cannot interpret it reject the transaction; a real Operation variant is wired up in the next commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the experimental action-based transaction through the existing commit path. Adds a feature-gated `Operation::UserOperation` variant and an `apply_user_operation` step in `build_manifest` that: - mints fragment ids from the manifest counter (late binding) and records each `NewFragment.local_id` -> assigned id, and - registers `AddIndex` metadata whose fragment bitmap is the union of `covers_existing` and the placeholders resolved above. Committing a UserOperation declares the `action-transactions` experimental writer feature (sets FLAG_EXPERIMENTAL). Conflict resolution treats the op like an Append (commutes with concurrent appends); retries re-run `build_manifest` against the latest manifest, so ids and placeholders are recomputed each attempt. The wire payload is encoded into the opaque `experimental_user_operation` transaction field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Feature-gated integration tests: (1) append two fragments and register an index covering only the first in one action-based transaction, asserting fragment ids, the index fragment bitmap (placeholder-resolved), FLAG_EXPERIMENTAL, and row count; (2) a stale UserOperation rebases past a concurrent plain Append and still commits, confirming action-based appends don't conflict with appends. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add gated LanceOperation.UserOperation / UserAction / AddFragments / AddIndex / NewFragment dataclasses, committed through the existing LanceDataset.commit path. The PyO3 layer parses them into the Rust Operation::UserOperation under the non-default `unstable-action-transactions` feature; without it, committing one raises a clear error. A `lance.lance._action_transactions_enabled` capability flag lets callers/tests detect whether the extension was built with the feature. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Feature-gated (skips unless the extension was built with `unstable-action-transactions`): a pylance user constructs and commits a UserOperation that appends two fragments and registers an index over only the first, in one transaction, then asserts the row count, fragment count, and the index's resolved fragment coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
A-python
Contributor
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #7671 (experimental action-transaction skeleton). Please review after it.
The first vertical slice of the "Action-based Transactions" milestone (#5960):
a caller can now construct and commit a
UserOperationthat both appends newfragments and registers an index covering those fragments, in a single
atomic transaction — the compound-commit capability that motivates the milestone.
Everything stays behind the non-default
unstable-action-transactionsCargofeature (absent from released artifacts), and committing one declares the
action-transactionsexperimental writer feature, which setsFLAG_EXPERIMENTALso libraries that don't understand the format refuse to commit. The resulting
manifest and data are standard, so stock readers are unaffected.
What's here
lance-table):AddFragmentsnow carriesNewFragments, eachwith an operation-local
local_idplaceholder so a later action can referencea fragment before its real id is assigned. New
AddIndexaction registers anindex segment; coverage is the union of already-committed fragment ids and
same-operation placeholders.
opaque
experimental_user_operationpayload in the canonicaltransaction.proto, keeping the PMC-voted schema free of the unstable actionstructure. The structured messages live in
transaction_experimental.proto.lance): a feature-gatedOperation::UserOperationvariant flows through the existing
CommitBuilder/commit-retry path. Applymints fragment ids from the manifest counter (late binding), records
local_id -> id, and resolves eachAddIndexbitmap. Conflict resolutiontreats the op like an
Append(commutes with concurrent appends); retriesre-run apply against the latest manifest, so ids/placeholders are recomputed
each attempt.
LanceOperation.UserOperation/UserAction/AddFragments/AddIndex/NewFragment, committed through the existingLanceDataset.commit. Alance.lance._action_transactions_enabledflag letscallers detect whether the extension was built with the feature.
Testing
FLAG_EXPERIMENTAL, row count) and non-conflict-with-concurrent-append.pylance acceptance flow end-to-end.
unaffected.
Notes
with it). The Rust feature-on tests are the real gate; the Python test
documents/verifies the acceptance path only in a feature-enabled build.
written); a real index-build path is out of scope.
🤖 Generated with Claude Code