feat(format): general experimental-feature mechanism

[wjones127]

Proposes and implements a general mechanism for shipping experimental
format features without spending a scarce, permanent feature-flag bit per
experiment. The policy proposal and the proto/format change are in one PR so the
PMC can review and vote on them together.

The problem

Feature flags are two u64 bitmaps with a linear "first unknown bit" boundary.
That's fine for stable features but hostile to experimentation:

• Every persisted bit is burned forever — even an abandoned experiment's
  bit can't be reused, because some dataset may have written it. Experimental
  churn permanently pollutes a scarce, shared namespace.
• Claiming a bit is high-ceremony, which chills experimentation.

The mechanism

• Reserve one persisted bit, FLAG_EXPERIMENTAL (1 << 6, reusing the old
  first FLAG_UNKNOWN bit — safe to repurpose, since writing it always made a
  dataset unreadable so none exist in the wild). It means "this dataset uses
  experimental feature(s)" and is the fail-closed anchor: libraries that
  predate an experiment reject the dataset on the bit alone.
• Carry the actual feature identities in free-form manifest string lists,
  experimental_reader_features / experimental_writer_features. New libraries
  admit a dataset iff they understand every declared name.
• A compile-time registry (known_experimental_features) lists what a build
  understands, gated per experiment by its Cargo feature. A default build
  understands none, so it rejects any experimental dataset — the conservative
  default.

Result: experiments are free-flowing (unbounded string namespace, mint and
abandon at will) while the bitmap stays conservative (a permanent bit is
spent only when a feature graduates).

The bit is load-bearing precisely because a string list alone is unsafe: an old
reader silently ignores an unknown manifest field and would misread experimental
data. The bit forces rejection; the strings do precise capability negotiation.

Contents

• rust/lance-table/design/experimental_feature_flags.md — the proposal:
  design, admission algorithm, graduation/lifecycle, alternatives considered
  (dedicated-bit-per-experiment, string-list-without-bit, feature branch), and
  prior art (Delta Lake table features).
• protos/table.proto — experimental_reader_features = 22 /
  experimental_writer_features = 23 on Manifest, plus doc of bit 1 << 6.
  Backwards compatible.
• feature_flags.rs — FLAG_EXPERIMENTAL, the registry, and admission checks;
  can_read_dataset / can_write_dataset now take the declared feature list.
• Manifest struct + proto conversions + apply_feature_flags wiring; the two
  internal admission callers updated.

Relationship to #7641

The action-based-transactions skeleton (#7641) currently claims a dedicated
FLAG_ACTION_TRANSACTIONS bit. Once this lands, #7641 should adopt this
mechanism instead — set FLAG_EXPERIMENTAL + list "action-transactions",
register the name under its Cargo feature, and free the dedicated bit. This PR
should land first.

Open questions

• Feature-name convention (proposed: stable lowercase-kebab).
• Whether to expose active experimental features through the Python/Java
  bindings for introspection.
• Whether graduation dual-accepts the old experimental name during a transition
  window or requires a rewrite.

🤖 Generated with Claude Code

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[codecov]

Codecov Report

❌ Patch coverage is 96.70330% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-table/src/feature_flags.rs	95.89%	1 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[wjones127]

Superseded by #7670 (same branch, now hosted on lance-format so it can be a proper base for the stacked #7671).