[CALCITE-7505] RelToSqlConverter fails to alias outer relation for correlated sub-queries in Filter

[Dwrite]

Jira Link
CALCITE-7505

This PR fixes incorrect SQL generation for correlated sub-queries in a Filter's WHERE clause.

The root cause is that Filter did not preserve the correlation variables it binds, so RelToSqlConverter could not reliably determine when the current node was the binding point of a correlation variable. As a result, correlation alias handling could be applied at the wrong level, leading to incorrect correlated references in generated SQL.

The fix adds variablesSet to Filter / LogicalFilter and only resets the correlation alias when the current Filter / Project actually binds the variable, the input row type matches, and the input is not a BiRel.

A follow-up change will cover similar correlated cases in Join conditions.

[mihaibudiu]

The JavaDoc above says: "a correlated variable defined by the given input"
Where is this check validating this fact?
(I am not sure how "defined by an input" is defined)

[Dwrite]

Sorry， that's my english fault, The wording is inaccurate. The method does not actually verify that the correlation variable is defined by the given input; it only checks for a correlated reference with a matching row type. I'll update the Javadoc to reflect what it actually does.

[mihaibudiu]

There can be many variables with the same type, please explain why your approach is correct.

[Dwrite]

Hi @mihaibudiu

Thanks for the question. During our test case execution, we observed that planner rules like FilterIntoJoinRule( test case with testFilterCorrelateMissingVariableCor) can reposition or push down a Filter. This optimization often causes the Filter's variablesSet to become stale, meaning the variable's rowType no longer matches the schema of the new underlying input.

Therefore, relying solely on correlIds.contains(v.id) is insufficient. We need the rowType equality check to ensure that this Filter remains the true binding point of the correlation variable post-optimization. If the types mismatch, it indicates the Filter has been relocated, and we should not trigger resetAliasForCorrelation.

Regarding the structure check, rather than a generic BiRel, I specifically restrict this validation to Join and LogicalCorrelate operators. These two operators represent the exact semantic boundaries where correlation aliases are defined and resolved. Checking for Join and LogicalCorrelate allows us to correctly determine whether the alias needs to be appended, ensuring correctness without interfering with other relational nodes.

Let me know if this clarifies the design choice!

[mihaibudiu]

It looks to me that you are describing a bug, I think having a wrong type on the correlated variables should be normally impossible - whoever rewrote the filter should have adjusted the variable types too. Am I wrong? If not, can you make sure that there is an issue filed for this?

Can a variable have the right type and also be obsolete at the same time?

Also, you can probably add an explanation here as a comment about why the type is needed.

[Dwrite]

You are correct, and I appreciate the sharp observation.

The rowType matching check is not a reliable discriminator. It was
introduced as an empirical workaround based on observed behavior in
specific test cases, not derived from a sound semantic invariant. As
you pointed out, a variable can have the correct type and still be
obsolete, so the check is not sound in general.

After mirroring the same variablesSet approach from Project to Filter,
all tests pass except for two cases that involve HepPlanner rules
(FilterIntoJoinRule-- testFilterCorrelateMissingVariableCor() and FilterCorrelateRule testFilterIntoJoinMissingVariableCor()
). I will investigate why
these two rules produce incorrect behavior and follow up.

[Dwrite]

Hi @mihaibudiu

I add new code !(definedHere.isEmpty()) && e.getInput().getInputs().size() <= 1. and here's explaination.
The purpose of calling resetAliasForCorrelation in visit(Filter) / visit(Project)
is narrow: give e.getInput() a canonical alias so that a correlated
sub-query embedded in this node's condition/projects can resolve
$corX.field against that alias.

This is only safe/necessary when e.getInput()'s rendered SQL already
represents a single addressable relation:

• e.getInput().getInputs().size() <= 1 (TableScan, Values, Project,
  Filter, Aggregate, ...): these nodes always render as a single
  identity -- either a bare table reference or a freshly wrapped SELECT
  derived table. Assigning an alias here is safe, and is exactly the
  step the original bug was missing.

• e.getInput().getInputs().size() > 1 (Join, Correlate): visit(Join)
  already builds a multi-alias context via
  joinContext(leftContext, rightContext) (e.g. {EMP: ..., DEPT: ...}),
  keeping each side independently addressable -- which is exactly what a
  correlated sub-query needs to resolve $corX.field. Calling
  resetAliasForCorrelation here would collapse that multi-alias context
  into a single alias, destroying field resolution that was already
  correct (this is the root cause of the DEPTNO0 leakage observed in
  the FilterIntoJoinRule / FilterCorrelateRule regression cases).

Scope: this check only governs whether visit(Filter) / visit(Project)
should invoke resetAliasForCorrelation for the purpose of correlation
variable binding. It says nothing about the general contract of
resetAliasForCorrelation itself, and does not affect other call sites
(e.g. visitAntiOrSemiJoin, which legitimately calls it on a
multi-alias Result for an unrelated purpose -- wrapping a complex
anti-join input). That is why the check belongs at this call site rather
than as an assertion inside resetAliasForCorrelation: an assertion
inside the shared method would incorrectly impose this call site's
precondition on every caller.

Verified against three concrete scenarios:

• EXISTS in WHERE: e.getInput() = TableScan (0 inputs) -> pushed=true, correct
• FilterIntoJoinRule / FilterCorrelateRule residual filter: e.getInput() = Join
  (2 inputs) -> pushed=false, correct (avoids DEPTNO0 leakage)
• foodmart correlated IN, e.getInput() = Project over Join: Project itself
  has 1 input -> pushed=true, correct

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
59.1% Coverage on New Code
90.9% Duplication on New Code

See analysis details on SonarQube Cloud