Skip to content

IN LIST: add direct-probe hash filter for large primitive lists#23015

Draft
geoffreyclaude wants to merge 3 commits into
apache:mainfrom
geoffreyclaude:perf/in_list_direct_probe_filter
Draft

IN LIST: add direct-probe hash filter for large primitive lists#23015
geoffreyclaude wants to merge 3 commits into
apache:mainfrom
geoffreyclaude:perf/in_list_direct_probe_filter

Conversation

@geoffreyclaude

@geoffreyclaude geoffreyclaude commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

#23014 handles tiny primitive IN lists by comparing against each constant. That stops being a good tradeoff once the list gets larger.

For larger primitive lists, this PR uses a purpose-built lookup table. The mental model is:

  1. Precompute a table from the constants in x IN (...).
  2. For each input value, compute a cheap table slot from the value.
  3. Check that slot, and move forward if there was a collision.

This is still a hash-table style lookup, but it is simpler than the generic fallback because primitive values are fixed-width and can be stored directly. There is no need for the generic Arrow comparator path for each candidate.

The earlier bitmap and branchless filters remain in place for the cases where they are cheaper.

What changes are included in this PR?

  • Renames the branchless primitive helper trait/alias to PrimitiveFilterType and PrimitiveFilterNative, so branchless and direct-probe filters share the same logical-to-comparison type mapping.
  • Adds DirectProbeFilter, a compact open-addressing lookup table with linear probing.
  • Routes larger 4-, 8-, and 16-byte primitive IN lists to direct probing after the branchless thresholds.
  • Uses the same-width comparison representation from PrimitiveFilterType, so signed integers, floats, and temporal values compare by exact bit pattern without building a converted Arrow array.
  • Keeps slice and null handling on the raw-buffer fast path.

Are these changes tested?

Yes.

  • cargo fmt --all -- --check
  • cargo test -p datafusion-physical-expr expressions::in_list --lib
  • cargo test -p datafusion-physical-expr --bench in_list_strategy --no-run
  • cargo clippy --all-targets --all-features -- -D warnings

Are there any user-facing changes?

No. This is an internal performance optimization only.

Local benchmark snapshot

Benchmark command:

cargo bench -p datafusion-physical-expr --profile release-nonlto --bench in_list_strategy -- --save-baseline <name>

Method: compare adjacent saved baselines using raw Criterion sample minima (min(time / iters)). Lower is better; changes within +/-5% are treated as noise.

Compared baselines: #23014 -> #23015

Relevant scope: large primitive-list rows.

Summary: 13 relevant rows, 13 faster, 0 slower, 0 within +/-5%.

Benchmark Before After Change
f32/large_list/list=64/match=0% 18.83 us 7.91 us -58.0% (2.38x faster)
f32/large_list/list=64/match=50% 33.00 us 10.27 us -68.9% (3.21x faster)
nulls/primitive/i32/large_list/list=64/match=50%/nulls=20% 25.79 us 11.26 us -56.3% (2.29x faster)
primitive/i32/large_list/list=256/match=0% 17.80 us 7.99 us -55.1% (2.23x faster)
primitive/i32/large_list/list=256/match=50% 27.31 us 10.25 us -62.5% (2.67x faster)
primitive/i32/large_list/list=64/match=0% 18.15 us 8.05 us -55.7% (2.26x faster)
primitive/i32/large_list/list=64/match=50% 27.85 us 10.25 us -63.2% (2.72x faster)
primitive/i64/large_list/list=128/match=0% 18.93 us 8.00 us -57.7% (2.37x faster)
primitive/i64/large_list/list=128/match=50% 24.24 us 10.08 us -58.4% (2.41x faster)
primitive/i64/large_list/list=32/match=0% 19.82 us 8.51 us -57.1% (2.33x faster)
primitive/i64/large_list/list=32/match=50% 26.01 us 11.31 us -56.5% (2.30x faster)
timestamp_ns/large_list/list=32/match=0% 19.38 us 8.49 us -56.2% (2.28x faster)
timestamp_ns/large_list/list=32/match=50% 45.82 us 10.03 us -78.1% (4.57x faster)

@github-actions github-actions Bot added the physical-expr Changes to the physical-expr crates label Jun 18, 2026
@geoffreyclaude geoffreyclaude force-pushed the perf/in_list_direct_probe_filter branch 2 times, most recently from 12ca843 to 0111ce5 Compare June 18, 2026 09:12
@geoffreyclaude geoffreyclaude changed the title Implement Direct Probe (Hash) Filter for large primitive lists IN LIST: add direct-probe hash filter for large primitive lists Jun 18, 2026
@geoffreyclaude geoffreyclaude force-pushed the perf/in_list_direct_probe_filter branch 3 times, most recently from a109166 to 2e20173 Compare June 19, 2026 05:55
@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jun 19, 2026
@geoffreyclaude geoffreyclaude force-pushed the perf/in_list_direct_probe_filter branch from 2e20173 to f0c8d4d Compare June 22, 2026 13:50
@github-actions github-actions Bot removed the auto detected api change Auto detected API change label Jun 22, 2026
@geoffreyclaude geoffreyclaude force-pushed the perf/in_list_direct_probe_filter branch 9 times, most recently from 436582f to 57a8e6f Compare June 26, 2026 16:21
@geoffreyclaude geoffreyclaude force-pushed the perf/in_list_direct_probe_filter branch 4 times, most recently from 2c36a1f to 6d59f07 Compare July 2, 2026 08:54
@geoffreyclaude geoffreyclaude force-pushed the perf/in_list_direct_probe_filter branch 2 times, most recently from c7dfb7a to 218844d Compare July 8, 2026 12:05
@geoffreyclaude geoffreyclaude force-pushed the perf/in_list_direct_probe_filter branch from 218844d to a6f0778 Compare July 8, 2026 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-expr Changes to the physical-expr crates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant