Skip to content

perf: reuse column normalization context across expressions - #25010

Open
goutamadwant wants to merge 1 commit into
apache:mainfrom
goutamadwant:fix-normalization-context-24777
Open

perf: reuse column normalization context across expressions#25010
goutamadwant wants to merge 1 commit into
apache:mainfrom
goutamadwant:fix-normalization-context-24777

Conversation

@goutamadwant

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Column normalization repeatedly collects fallback schemas and traverses the input plan for USING columns. Wide expression lists and projections repeat that work for the same immutable plan.

What changes are included in this PR?

  • Introduce a private, lazy normalization context reused across columns and expression lists.
  • Share the context in sort normalization and validated projection construction, including wildcard expansion.
  • Keep already-qualified columns and expressions that do not need normalization on the existing fast path.
  • Add benchmarks for qualified and unqualified expressions and projection construction at several schema widths.

What is the testing strategy for this PR?

  • Add normalize_batch_schema_precedence, normalize_batch_using_join, and normalize_batch_skips_unused_plan_context to cover schema precedence, USING joins, ambiguity/error order, sort options, and lazy handling of qualified columns and literals.
  • In balanced local release-nonlto runs, constructing a 2,000-column unqualified projection falls from about 100 ms to 35 ms. This measures projection construction, not full protobuf decoding; small controls remain noisy.
  • Reproduce with cargo bench -p datafusion-expr --bench normalize_columns --profile release-nonlto.
  • Focused expression and SQL tests pass. The required extended workspace test command also passes, including all 511 SQL logic-test files.
  • Expression-crate Clippy passes with all targets and features enabled. The complete documented dev/rust_lint.sh also passes, including strict workspace documentation checks.
  • Full-workspace Clippy with all features enabled hits the existing PostgreSQL decimal-formatting lint in Full workspace clippy fails in PostgreSQL decimal formatting #24974; the affected source is unchanged here.

Are there any user-facing changes?

No public API or name-resolution behavior changes are intended. Normalization reuses plan context instead of collecting it for each column; existing schema lookup costs remain.

@github-actions github-actions Bot added the logical-expr Logical plan and expressions label Sep 7, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.45455% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.70%. Comparing base (e1ca94f) to head (2fcda6c).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/expr/src/expr_rewriter/mod.rs 77.14% 1 Missing and 23 partials ⚠️
datafusion/expr/src/logical_plan/builder.rs 40.00% 0 Missing and 3 partials ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #25010    +/-   ##
========================================
  Coverage   81.69%   81.70%            
========================================
  Files        1127     1127            
  Lines      415471   415743   +272     
  Branches   415471   415743   +272     
========================================
+ Hits       339424   339672   +248     
+ Misses      56108    56107     -1     
- Partials    19939    19964    +25     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants