[FLINK-39986][table-planner][python] Support Common Sub-expression Elimination (CSE) for Python UDFs#28638
Open
raoraoxiong wants to merge 5 commits into
Open
[FLINK-39986][table-planner][python] Support Common Sub-expression Elimination (CSE) for Python UDFs#28638raoraoxiong wants to merge 5 commits into
raoraoxiong wants to merge 5 commits into
Conversation
…uplication for Python UDFs This commit implements Phase 1 of Python UDF Common Sub-expression Elimination (CSE). Identical deterministic Python UDF calls in the projection list are deduplicated at the ExecNode level, reducing redundant cross-process (JVM <-> Python Worker) communication. Key changes: - Modify CommonExecPythonCalc to deduplicate top-level calls and append a ref-reuse projection operator to restore output schema - Add ProjectionCodeGenerator.generateProjectionOperator for the expansion projection - Add unit tests and integration tests Generated-by: Claude-4.6-Opus
…ion-projection deduplication for Python UDFs This commit implements Phase 2 of Python UDF CSE. It extends the deduplication to nested sub-expressions (full-tree CSE) and adds a new optimizer rule to deduplicate Python UDF calls shared between WHERE conditions and SELECT projections. Key changes: - Add PythonCallDeduplicator and PythonCallCseResult for structural deduplication with nested sub-expression reference maps - Add PythonFunctionInfo.ResultRef for referencing pre-computed results - Extend protobuf Input message with refIndex field - Modify Python worker (operations.py) to support sequential execution with result references - Add RemoteCalcConditionProjectionCseRule optimizer rule - Register the new rule in stream and batch rule sets - Add plan tests and integration tests Generated-by: Claude-4.6-Opus
Contributor
Author
|
hi @liuyongvs @snuyanzin ，please take a look. |
Contributor
|
…moteCalcCallFinder to RemoteCallFinder Fix typo in RemoteCalcConditionProjectionCseRule constructor parameter type. The correct interface name is RemoteCallFinder, not RemoteCalcCallFinder. Generated-by: Claude-4.6-Opus
…d use RelRule API - Convert RemoteCalcConditionProjectionCseRule from Scala to Java using the modern RelRule<Config> pattern instead of deprecated RelOptRule - Convert PythonCalcConditionCseTest from Scala to Java - Update PythonCalcSplitRule to use Config-based rule instantiation Generated-by: Claude-4.6-Opus
…avoid projection pushdown Use addTableSource with Schema instead of CREATE TABLE with values connector to prevent projection pushdown from changing the plan output. Generated-by: Claude-4.6-Opus
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
This pull request implements Common Sub-expression Elimination (CSE) for Python UDFs in Flink SQL. When a query contains identical deterministic Python UDF calls (e.g.,
SELECT udf1(x), udf1(x)orSELECT udf1(x), udf2(udf1(x))), the current implementation sends each call independently to the Python worker, resulting in redundant cross-process (JVM ↔ Python Worker) communication and computation. This PR deduplicates such calls so that each unique expression is computed only once, and duplicate positions reference the pre-computed result.Brief change log
Phase 1: Top-level projection deduplication (Commit 1)
CommonExecPythonCalcto deduplicate identical deterministic Python UDF calls in the projection list at the ExecNode levelProjectionCodeGenerator.generateProjectionOperatorfor the expansion projectionPhase 2: Full-tree CSE and condition-projection deduplication (Commit 2)
PythonCallDeduplicatorandPythonCallCseResultfor structural deduplication with nested sub-expression flattening and reference mapsPythonFunctionInfo.ResultReffor referencing pre-computed UDF results by indexInputmessage withrefIndexfieldoperations.py) to support sequential execution with result referencesRemoteCalcConditionProjectionCseRuleoptimizer rule to deduplicate Python UDF calls shared between WHERE conditions and SELECT projectionsVerifying this change
This change added tests and can be verified as follows:
CommonExecPythonCalcRefReuseTest(Java unit test) to verify deduplication logic, ref-map resolution, and detail name generation with parameterized test casesPythonCalcConditionCseTest(Scala plan test) to verify the optimizer rule produces correct plans for condition-projection CSE scenariostest_scalar_function_cse.py(Python unit test) to verify the sequential execution codegen path (with refIndex) and the traditional lambda path (without refIndex)test_udf.pyto verify end-to-end correctness of CSE with real Python UDFsDoes this pull request potentially affect one of the following parts:
@Public(Evolving): noDocumentation
Was generative AI tooling used to co-author this PR?