Adaptive (runtime, stats-based) conjunct reordering for FilterExec - #22698
Open
adriangb wants to merge 18 commits into
Open
Adaptive (runtime, stats-based) conjunct reordering for FilterExec#22698adriangb wants to merge 18 commits into
adriangb wants to merge 18 commits into
Conversation
github-actions
Bot
added
documentation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
adriangb
force-pushed
the
lift-selectivity-stats
branch
3 times, most recently
from
June 2, 2026 02:29
a24471d to
4d7b733
Compare
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Runner log (last 40 lines)Kubernetes messageFile an issue against this benchmark runner |
Documentation-only pass over the adaptive filter, applying alamb's review nits and correcting the stated baseline. - Module doc: describe what conjunct evaluation already does today — `reorder_predicates` orders conjuncts cheap-before-expensive by a static cost class, and `BinaryExpr`'s `AND` pre-selects when the conjuncts so far keep <= 20% of the rows and produce no nulls. Spell out what pre-selection cannot do (gate on a later conjunct, fire through nulls, carry survivors compacted across a nested chain) instead of claiming the `AND` evaluates every conjunct on every row regardless of order. - Add an intra-doc link to `BinaryExpr`, drop the "left-deep fused" jargon, expand the `regexp_like` example into a before/after evaluation order, and leave the flag's default value documented on the flag itself. - Drop the unsupported "compact-once is itself a win even without reordering" claim; point at the PR for the measurements rather than quoting numbers. - Fold the side-effect caveat, the conditional-statistics caveat and the one-shot caveat into a single "Known limitations" list instead of repeating them across the module; settle on one vocabulary (a decision is *settled*, a stream *adopts* it) and remove the leftover "publishes", "frozen" and "A/B-validated" wording. - Config doc: shorter, and the side-effect caveat is now bidirectional (a divide-by-zero can appear *or* disappear). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
Author
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 54a177f (54a177f) to 54a177f diff Run configurationrun benchmark predicate_eval
env:
CARGO_BUILD_JOBS: "1"
baseline:
ref: "54a177f5de788ef7399f74f7da0e16f7221bb276"
env:
DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING: "false"
changed:
ref: "54a177f5de788ef7399f74f7da0e16f7221bb276"
env:
DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING: "true"Results will be posted here when complete File an issue against this benchmark runner |
… AND Once the warm-up adopts a reorder, materialise the learned order once as a right-nested `AND` chain, `(c_first AND (c_second AND (... AND c_last)))`, and let `BinaryExpr` evaluate it like any other predicate, instead of running the settled path through the per-conjunct compact-once loop. Right-nesting is what makes this cheap: `BinaryExpr`'s pre-selection filters the batch it is handed before evaluating its right-hand side, so the survivors of the first (most selective) conjunct stay compacted for the entire remainder of the chain. A left-nested chain -- what `conjunction()` builds -- re-filters the original batch and scatters at every level, which measures materially slower than the flag off on cheap 4-8 conjunct predicates; right-nested is within noise of the compact-once loop. The measuring path keeps the per-conjunct loop with compaction: it has to time each conjunct on exactly the rows that reach it. `Settled` now carries the expression to evaluate alongside the order, and `settle` builds it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… of a private evaluation loop The warm-up used to walk the conjuncts itself: a private `eval_conjuncts` loop that AND-ed the masks, compacted the working batch past its own selectivity threshold, tracked live row indices and scattered the result back to full length. That was a second conjunction-evaluation engine living next to `BinaryExpr`'s, with its own compaction policy to keep in step and its own null and index handling to get right. Delete it. Each conjunct is now wrapped in a small measuring `PhysicalExpr` that times the call and counts the rows it was handed and the rows it kept, and the wrapped conjuncts are assembled into the written order as the same right-nested `AND` chain the settled path uses. `BinaryExpr` evaluates and pre-selects exactly as it would for the plain predicate, so the compaction is its own and every conjunct is measured on precisely the rows it hands over. The wrapper returns the conjunct's result unchanged, nulls included; three-valued logic stays `BinaryExpr`'s business. The module now holds no evaluation logic of its own on any path: stats and the shared pool, the ranking and cost model, `settle` plus the right-nested chain builder, the measuring wrapper, and the per-stream glue. Behaviour is unchanged where it was observable: the pooled counts for the first measured batch are identical, both divide-by-zero side-effect tests still hold (the warm-up's pre-selection keeps `1 / b` away from the zeros exactly as the old loop's compaction did), and the sqllogictests are unaffected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
adriangb
commented
Sep 7, 2026
…ith_adaptive_reorder_metrics Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
Author
Contributor
Author
|
run benchmark tpch10 clickbench_partitioned tpcds |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing lift-selectivity-stats (12c9a05) to 262936e (merge-base) diff Run configurationrun benchmark tpch10Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing lift-selectivity-stats (12c9a05) to 262936e (merge-base) diff Run configurationrun benchmark clickbench_partitionedResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing lift-selectivity-stats (12c9a05) to 262936e (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
Contributor
Author
|
@alamb @xudong963 sorry for the delay on this. I've reworked so that there is no custom |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 12c9a05 (12c9a05) to 12c9a05 diff Run configurationrun benchmark clickbench_partitioned
baseline:
ref: "12c9a05490090905317181368e0915b02e7d6924"
env:
DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING: "false"
changed:
ref: "12c9a05490090905317181368e0915b02e7d6924"
env:
DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 12c9a05 (12c9a05) to 12c9a05 diff Run configurationrun benchmark tpch10
baseline:
ref: "12c9a05490090905317181368e0915b02e7d6924"
env:
DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING: "false"
changed:
ref: "12c9a05490090905317181368e0915b02e7d6924"
env:
DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING: "true"Results will be posted here when complete File an issue against this benchmark runner |
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing lift-selectivity-stats (12c9a05) to 262936e (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing lift-selectivity-stats (12c9a05) to 262936e (merge-base) diff Run configurationrun benchmark tpch10CPU Details (lscpu)Details
Resource Usagetpch10 — base (merge-base)
tpch10 — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 12c9a05 (12c9a05) to 12c9a05 diff Run configurationrun benchmark tpch10
baseline:
ref: "12c9a05490090905317181368e0915b02e7d6924"
env:
DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING: "false"
changed:
ref: "12c9a05490090905317181368e0915b02e7d6924"
env:
DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING: "true"CPU Details (lscpu)Details
Resource Usagetpch10 — base (merge-base)
tpch10 — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 12c9a05 (12c9a05) to 12c9a05 diff Run configurationrun benchmark tpcds
baseline:
ref: "12c9a05490090905317181368e0915b02e7d6924"
env:
DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING: "false"
changed:
ref: "12c9a05490090905317181368e0915b02e7d6924"
env:
DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing lift-selectivity-stats (12c9a05) to 262936e (merge-base) diff Run configurationrun benchmark clickbench_partitionedCPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 12c9a05 (12c9a05) to 12c9a05 diff Run configurationrun benchmark clickbench_partitioned
baseline:
ref: "12c9a05490090905317181368e0915b02e7d6924"
env:
DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING: "false"
changed:
ref: "12c9a05490090905317181368e0915b02e7d6924"
env:
DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING: "true"CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 12c9a05 (12c9a05) to 12c9a05 diff Run configurationrun benchmark tpcds
baseline:
ref: "12c9a05490090905317181368e0915b02e7d6924"
env:
DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING: "false"
changed:
ref: "12c9a05490090905317181368e0915b02e7d6924"
env:
DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING: "true"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
Benchmark for this request hit the 7200s job deadline before finishing. Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
Contributor
Author
Contributor
Author
|
Benchmark summary for head 12c9a05 (no custom evaluator; learned order evaluated by Flag off → on
Main vs PR, flag off both sides
|
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 12c9a05 (12c9a05) to 12c9a05 diff Run configurationrun benchmark tpch10
baseline:
ref: "12c9a05490090905317181368e0915b02e7d6924"
changed:
ref: "12c9a05490090905317181368e0915b02e7d6924"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 12c9a05 (12c9a05) to 12c9a05 diff Run configurationrun benchmark tpch10
baseline:
ref: "12c9a05490090905317181368e0915b02e7d6924"
changed:
ref: "12c9a05490090905317181368e0915b02e7d6924"CPU Details (lscpu)Details
Resource Usagetpch10 — base (merge-base)
tpch10 — branch
File an issue against this benchmark runner |
Contributor
Author
|
A/A control on tpch10 (trigger: same PR head, same config on both sides) shows ±7–15% per-query swings (Q7 1.14x, Q9 1.13x, Q17 1.15x, Q22 1.12x) and a 4% total difference between two runs of the identical binary. So the flag-off tpch "regression" in the main-vs-PR run above is noise; nothing in the flag-off path shows up above the floor. The same floor applies to single per-query flags in the other suites, so the tpcds +1.8% net with the flag on is suggestive, not proven; the microbenchmark result behind it (cheap-predicate reorders adopted with no payoff) is the more reliable evidence. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Predicate evaluation order matters: a selective conjunct run first gates the
work of the conjuncts after it. Two mechanisms already order and gate
conjuncts, and both decide statically:
cost class (perf: Reorder predicates in conjuncts via simple heuristic #22343). It is blind to selectivity, so a cheap-but-unselective
conjunct still sorts ahead of an expensive-but-very-selective one, and
conjuncts within one class keep their written order;
BinaryExpr'sANDpre-selects: when the conjuncts evaluated so far keepat most 20% of rows (and produce no nulls) it filters the batch before
evaluating the next one. It cannot gate a conjunct on a more selective one
written after it.
This PR adds runtime, statistics-based reordering for
FilterExec: eachconjunct's selectivity and per-row cost are measured on the rows that reach
it, the conjuncts are ranked by rows discarded per nanosecond, and the ranking
is adopted only if it is materially cheaper than the written order. Once
adopted, the learned order is materialised once as an ordinary
ANDchain andevaluated by
BinaryExprlike any other predicate. The module contains noconjunction-evaluation logic of its own on any path; it only measures, ranks,
and builds the chain. It is off by default
(
datafusion.execution.adaptive_filter_reordering).What changes are included in this PR?
Everything lives in a new private module
datafusion/physical-plan/src/adaptive_filter.rs;FilterExecgains a sharedper-execution state field, a metric, and a two-arm match in the stream poll
loop.
operator) the written order is evaluated by
BinaryExpras a right-nestedANDchain in which each conjunct is wrapped in a small measuringexpression that records rows seen, rows passed and time.
BinaryExpr'spre-selection does the compaction, so each conjunct is measured on exactly
the rows
BinaryExprhands it. The wrappers disappear once the order issettled.
(1 - pass_rate) / cost_per_row; adopt the ranking onlyif its expected cost is at least 5% below the written order's. If not, the
written predicate is evaluated unchanged, exactly as with the flag off.
AND: the learned order is built onceas
c1 AND (c2 AND (... AND cn))and handed toBinaryExpr. Right-nestingis what makes this work:
BinaryExprpre-selection filters the batch it isgiven, so the survivors of the first conjunct stay compacted through the rest
of the chain. A left-nested chain (what
conjunction()builds) re-filters theoriginal batch and scatters back at every level; see the measurements below.
adaptive_reordersonFilterExec(per partition, only presentwhen the flag is on) shows in
EXPLAIN ANALYZEwhether a reorder wasadopted.
reset_stategives re-executions fresh measurements; predicate rewrites reset the pooled
state; results are order-independent.
configs.md/information_schema.predicate_evalbenchmark shapes (expensive-selective-first, andk8 over 64 columns).
Known limitations (documented in the module): measurements are
conditional on the written order, so correlated conjuncts can be misjudged;
the settle is one-shot with no drift re-measurement; the settle cost model
does not yet include evaluation overhead, so on very cheap predicates a
reorder can be adopted that buys nothing (see k4 below).
Measurements behind the settled-path design
Same binary, settled path selected by an environment switch, 8 interleaved
rounds × 40 iterations on
predicate_eval, ratios of medians. A = flag off,B = a dedicated compact-once evaluation loop (an earlier revision of this PR),
C = learned order rebuilt as a left-nested
AND, D = learned order rebuilt asa right-nested
AND(this PR).The right-nested rebuild matches the dedicated loop everywhere except a
~10% residual on the 0.6 ms k4 query, which is evaluator fixed cost on a
reorder that buys nothing there; tightening the settle guard to account for
evaluation overhead is a follow-up.
tpch_sf10 (same binary, flag off → on): Q6 1.19× faster, Q12 1.45× faster,
all other queries unchanged; tpcds_sf1 and clickbench neutral within the A/A
noise floor. See the bot runs in the PR comments.
Are these changes tested?
the right-nested shape of the rebuilt chain, the metric transition, lazy
pool init, and both directions of the fallible-predicate side effect (an
adopted reorder can introduce or avoid a divide-by-zero).
FilterExectest (4 partitions, nullable column,repeated execution with and without
reset_state).adaptive_filter.slt: results identical on and off,EXPLAINidentical onand off, and an
EXPLAIN ANALYZEassertion thatadaptive_reordersfireson a predicate written selective-last.
Are there any user-facing changes?
One new config option,
datafusion.execution.adaptive_filter_reordering(experimental, default false), and one new
FilterExecmetric,adaptive_reorders. When enabled, query results never change, but theobservable side effects of fallible predicates can, in either direction:
reordering
b <> 0 AND 1/b > 2can make a divide-by-zero error appear ordisappear. Predicates containing volatile expressions are never reordered.