[CALCITE-6104] Aggregate function that references outer column should be evaluated in outer query by xuzifu666 · Pull Request #5052 · apache/calcite

xuzifu666 · 2026-06-26T13:49:57Z

jira: https://issues.apache.org/jira/browse/CALCITE-6104

… be evaluated in outer query

mihaibudiu · 2026-06-26T17:03:48Z


 !ok

+# [CALCITE-6104] Aggregate function that references outer column should be evaluated in outer query


Please add a comment that these are validated using an independent database.

Can we also have a test with 2 outer aggregates?
Showing the plan could also help.

Thanks for the reminder, I had added comment about it and the test case had been added.
But agg.iq to verify the correctness of the numerical results calculated after the Calcite fix within PostgreSQL, the original SQL must be rewritten into standard SQL supported by PostgreSQL.
Validated these sql in https://onecompiler.com/postgresql/44tgmwq6p and result is as expected.

mihaibudiu · 2026-06-26T17:31:15Z

+
+!ok
+
+SELECT (SELECT sum(sal) FROM dept) AS sum_sal


The blog post quoted by @julianhyde has more examples, please add all of the relevant ones

Yes, I review the blog overall and add new case for it. The correspondence is as follows:

SELECT (SELECT sum(1) FROM xx LIMIT 1) FROM aa

Aggregate with no column references aggregates at the innermost level.

SELECT (SELECT sum(a) FROM xx LIMIT 1) FROM aa

Aggregate function that references outer column should be evaluated in outer query.

SELECT (SELECT sum(x) FROM xx LIMIT 1) FROM aa

Aggregate referencing only inner column aggregates at the inner level.

mihaibudiu · 2026-06-26T17:32:15Z

+   * <p>For example,
+   * <blockquote><pre>SELECT (SELECT sum(a) FROM t LIMIT 1) FROM aa</pre></blockquote>
+   * is rewritten to
+   * <blockquote><pre>SELECT sum(a) * (SELECT 1 FROM t LIMIT 1) FROM aa</pre></blockquote>


From this example it's not obvious to me at all what the general rewrite rule is.
Can you describe the algorithm?

Okay, I’ve added the logic for the overall process; I hope that clarifies things.

mihaibudiu · 2026-06-27T02:29:05Z

+   * </ol>
+   *
+   * <p>For example,
+   * <blockquote><pre>SELECT (SELECT sum(a) FROM t LIMIT 1) FROM aa</pre></blockquote>


We cannot read this example because we don't know where column a is.
Can you also add tests where one or both tables are empty or contain only nulls in the aggregated column?

OK, The Javadoc examples for rewriteOuterAggregatesInSelectList have been improved, explicitly stating that a comes from aa and x comes from t.
Boundary tests have been added to agg.iq: Inner table empty → 1 row NULL
Outer table empty → 1 row NULL (because the outer aggregation has no input rows)
and postgresql test also updated (result is also keep the same with agg.iq, empty results for VALUE are not currently supported, so I am using the SQL statement SELECT 1 FROM emp WHERE 1 = 0 as a substitute.) https://onecompiler.com/postgresql/44tgmwq6p

mihaibudiu · 2026-06-27T03:46:01Z

 !ok

+# [CALCITE-6104] Aggregate function that references outer column should be evaluated in outer query
+# The expected results below have been verified in PostgreSQL by running


I am confused, I thought these queries are standard SQL, but, indeed, I cannot run them on postgres or other databases I tried. Isn't this PR supposed to implement a standard SQL feature? Why can't these queries use standard SQL then?

Yes, this is indeed a confusing point. These correlated-aggregate queries are not standard SQL, and CALCITE-6104 should not implement a SQL standard feature. Instead, it makes Calcite support a non-standard extension that some databases already implement.

The SQL standard does not allow aggregate functions to reference outer columns
In standard SQL, an aggregate function like sum(a) can only reference columns from the SELECT level where it appears. So a query like:

SELECT (SELECT sum(a) FROM xx LIMIT 1) FROM aa;

will fail on PostgreSQL, Oracle, and MySQL/MariaDB because a belongs to the outer table aa, not the sub-query xx.

But PostgreSQL and others support this extension.
These databases allow an aggregate to "climb up" to the nearest SELECT that contains its free variables. Calcite previously handled this pattern incorrectly (it aggregated inside the sub-query, grouped by the outer rows). The fix makes Calcite behave consistently with those databases.
This test is provided in Jira (julianhyde@3f4cab5) is supported—at least with the current PR applied; these are not standard SQL either.
Additionally, I tested converting these SQL statements to standard SQL; they passed even without this PR, which is why I hold the view mentioned above.

Thank you for the clarification. Maybe this SQL should not be accepted by default? There are these conformance modes, I think only modes like LENIENT or BABEL should accept this construct.

You say that postgres accepts these queries, then why do you have to modify the queries to run them on postgres? I couldn't run them as written in Postgres.

Yes, thanks for pointing this out. The current comment is incorrect, should state "These SQL queries are non-standard and their successful execution is only guaranteed within Calcite for now" makes perfect sense.

Regarding the limitation where correlated aggregate rewriting is only available under modes like LENIENT or BABEL, could log a separate jira to resolve this later（If ok I would create a new jira）? This way, we can track these changes independently and easily gather any additional feedback. From my perspective, this limitation bears little relevance to the current Jira ticket. I’d like to hear if you agree with my view on this.

I think this will require 3 lines of code and should be done as part of this PR.

Okay, I'll implement this restriction directly in this PR.

mihaibudiu · 2026-06-27T22:47:12Z

+    // Rewrite scalar sub-queries whose single select item is an aggregate
+    // over outer columns. The aggregate belongs to the outer query per SQL
+    // standard.
+    rewriteOuterAggregatesInSelectList(select);


call this handleOuterAggregate, and make the function reject it if not in a suitable conformance mode.

Yes, the latest commit implements controls in this area.

mihaibudiu · 2026-06-27T22:48:09Z

 !ok

+# [CALCITE-6104] Aggregate function that references outer column should be evaluated in outer query
+# The expected results below have been verified in PostgreSQL by running


You say that postgres accepts these queries, then why do you have to modify the queries to run them on postgres? I couldn't run them as written in Postgres.

mihaibudiu · 2026-06-27T22:49:08Z

+   * <blockquote><pre>
+   * WITH aa(a) AS (VALUES 1, 2, 3),
+   *      t(x) AS (VALUES 10, 20, 30)
+   * SELECT (SELECT sum(a) FROM t LIMIT 1) FROM aa


Is the LIMIT 1 needed here?

Not need, I had updated it.

mihaibudiu · 2026-06-27T22:52:32Z

+   * <blockquote><pre>
+   * WITH aa(a) AS (VALUES 1, 2, 3),
+   *      t(x) AS (VALUES 10, 20, 30)
+   * SELECT sum(a) * (SELECT 1 FROM t LIMIT 1) FROM aa


This rewrite will only work for numeric aggregates on types which have a multiplication operation

I've added this message to the comments first.

xuzifu666 · 2026-06-28T00:45:18Z

I have addressed the comments; please take a look when you have a moment to see if the current changes are appropriate～ @mihaibudiu

mihaibudiu · 2026-06-28T01:05:46Z

+    // over outer columns. The aggregate belongs to the outer query per SQL
+    // standard, but only if the current conformance allows this non-standard
+    // correlated-aggregate construct.
+    if (config.conformance().isCorrelatedAggregateAllowed()) {


where is an error reported if the conformance does not allow this?

Thank you for the reminder, I added handling and tests for cases where conformance is not allow, and also implemented custom exception messages for these scenarios. PTAL

Please read my review comments carefully. I had already suggested error handling in the previous iteration.
My energy for reviews is limited. This is not the only thing that could have been done in fewer steps.

Apologies, and thank you for pointing this out. You are right — I should have caught the error-handling requirement from your earlier comment instead of needing a second reminder.
I will be more careful reading review comments in full going forwar.

xuzifu666 · 2026-07-03T03:56:42Z

These days, I’ve reviewed the comments more on this PR and made further simplifications and revisions.

changes：

Added deriveType(subScope, aggCall) + SqlTypeUtil.isNumeric(...) check after operand validation. Non-numeric aggregates (e.g. max(ename) returning VARCHAR) now return null and fall back to old behavior, avoiding the ill-typed max(ename) * (SELECT 1 ...) rewrite that threw "Cannot apply '*' to VARCHAR". Used deriveType (active resolution) instead of getValidatedNodeTypeIfKnown (cache read), since the cache is empty on the DEFAULT check path.
Fixed Javadoc: rewrote to accurately describe the method.
Removed atomic types: replaced AtomicBoolean/AtomicInteger with boolean[] holders in referencesOnlyOuterColumns and containsSubQuery（confirmed that there are no multi-threaded race conditions）.
Added assertions in testCorrelatedAggregateConformance verifying max(ename) no longer errors under LENIENT/BABEL.

When you have the time, please take a look and see if it is suitable, very thanks~ @mihaibudiu

mihaibudiu

What happens at runtime for a non-numeric aggregate?
I don't see a quidem test covering this case.

xuzifu666 · 2026-07-04T00:46:01Z

What happens at runtime for a non-numeric aggregate?
I don't see a quidem test covering this case.

Originally the rewrite multiplied the aggregate by the scalar sub-query (agg * (SELECT 1 ... LIMIT 1)), which only type-checks for numeric aggregates — a non-numeric aggregate like MAX() would have thrown "Cannot apply '*' to VARCHAR".
I've changed the rewrite to a type-agnostic form and added a quidem test for it which result is as expected. @mihaibudiu

mihaibudiu · 2026-07-04T18:22:07Z

-    return SqlStdOperatorTable.MULTIPLY.createCall(
-        aggCall.getParserPosition(), aggCall, scalarSubQuery);
+    // Rewrite to CASE WHEN (SELECT 1 FROM ... LIMIT 1) IS NOT NULL THEN <agg> END.
+    // This is type-agnostic (unlike multiplication, which only works for


why mention here multiplication?
no one cares about prior incorrect implementations

Please, please, review the changes you submit for review before submitting them.

Sorry for the unnecessary content in this change. I have removed the whole comment here (this comment doesn't mean much).

mihaibudiu · 2026-07-06T16:47:44Z

+        continue;
+      }
+      // Ensure the sub-query's scope is established before inspecting it.
+      deriveType(selectScope,


Why do you need to derive the type?

You're right, this deriveType call is only being used for its side effect of establishing the sub-query's scope, and the return value is thrown away which is a misuse. The scope is needed by findCorrelatedAggregate, but the detection of "is this a scalar sub-query wrapping a single aggregate" is purely syntactic and needs no type at all. I had removed this deriveType and move the scope-dependent check to after the sub-query has been validated normally.

mihaibudiu · 2026-07-06T16:53:02Z

+  /**
+   * Returns whether an expression contains a sub-query.
+   */
+  private static boolean containsSubQuery(SqlNode node) {


is there no other helper function doing this in the Calcite codebase?

There's RexUtil.SubQueryFinder, but it operates on the Rex layer (RexSubQuery), whereas here we're still at the SqlNode layer, so it doesn't apply. I didn't find an existing SqlNode-level "contains sub-query" helper. That said, the hand-written switch over every sub-query SqlKind is fragile (easy to miss a new kind) — I had replaced it with the predefined SqlKind.QUERY set (SqlKind.QUERY.contains(call.getKind())) instead of enumerating each case.

mihaibudiu · 2026-07-06T16:54:03Z

    // First pass, ensure that aliases are unique. "*" and "TABLE.*" items
    // are ignored.

+    // Rewrite scalar sub-queries whose single select item is an aggregate


could this be done after validating all arguments?
Then you would not need to call deriveType early.

Yes, and this is the root cause behind the first question you asked this time.
The sub-query's SELECT scope is already established during the registration phase (registerSubQueries for the SELECT list), so the early deriveType calls were redundant. I've removed them; getSelectScope(subQuery) works directly.

mihaibudiu · 2026-07-07T21:40:33Z

+  }
+
+  /**
+   * Returns whether an identifier refers to a scope outside the current select.


I am not sure exactly what "outside" means, but given this function definition I think that the javadoc could be "in a scope which is not the supplied one". There's certainly no "current select" in sight.

That makes sense; I’ve made the relevant changes to the comment.

mihaibudiu · 2026-07-07T21:43:25Z

+   * arguments reference only columns from the outer query. Such an aggregate
+   * belongs to the outer query per the SQL standard.
+   *
+   * <p>Returns {@code null} unless the sub-query has exactly one select item,


This raises the question: what happens if the sub-query has two such aggregates, e.g., SUM(a1) + SUM(a2)? Can you write a test case for that so we see what happens?

Thank you for pointing this issue, the previous implementation could not correctly handle this scenario.

So I've extended the rewrite to handle this(For the sake of clarity, I have also modified this comment): Instead of requiring the select item to be a single aggregate call, findCorrelatedAggregate now lifts any select item that contains an aggregate and references only outer columns (checked by applying referencesOnlyOuterColumns to the whole item). So sum(a) + sum(b) over outer columns is now lifted as a whole to the outer query.

The mixed case is handled safely: if the expression references any inner column (e.g. sum(a) + sum(x) where x is inner), referencesOnlyOuterColumns returns false and we don't rewrite, keeping the pre-existing behaviour rather than silently producing a wrong result.

I've added test cases for all three: sum(a) + sum(b) (lifted, single row), sum(a) + sum(x) (mixed, not lifted), and an empty-inner-table guard case, plus validator-level conformance tests.

sonarqubecloud · 2026-07-08T03:15:10Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
93.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

mihaibudiu · 2026-07-08T17:52:55Z

+# result is a single row 66.
+WITH aa (a, b) AS (VALUES (1, 10), (2, 20), (3, 30)),
+  xx (x) AS (VALUES 100, 200, 300)
+SELECT (SELECT sum(a) + sum(b) FROM xx LIMIT 1) AS s


Can we also have a test for SUM(a + b)?

[CALCITE-6104] Aggregate function that references outer column should…

106b216

… be evaluated in outer query

xuzifu666 force-pushed the calcite-6104 branch from d2ff7dd to 106b216 Compare June 26, 2026 14:10

xuzifu666 added 2 commits June 26, 2026 22:41

Addressed

cd8346a

Addressed

47e152f

mihaibudiu reviewed Jun 26, 2026

View reviewed changes

xuzifu666 added 2 commits June 27, 2026 09:55

Addressed

7555c70

Addressed

e3c977e

mihaibudiu reviewed Jun 27, 2026

View reviewed changes

xuzifu666 added 2 commits June 27, 2026 11:19

Addressed

c9fe712

Addressed

7f81a12

mihaibudiu reviewed Jun 27, 2026

View reviewed changes

Addressed

d92b678

mihaibudiu reviewed Jun 27, 2026

View reviewed changes

xuzifu666 added 2 commits June 28, 2026 07:42

Addressed

fb6ac70

Addressed

3b449e4

mihaibudiu reviewed Jun 28, 2026

View reviewed changes

xuzifu666 added 5 commits June 28, 2026 09:50

Addressed

f0544c5

Addressed

6969a5a

Addressed

a0b7478

Addressed

2822919

Addressed

4706e6c

mihaibudiu reviewed Jul 3, 2026

View reviewed changes

Addressed

2bdf0e6

mihaibudiu reviewed Jul 4, 2026

View reviewed changes

xuzifu666 added 2 commits July 5, 2026 06:47

Addressed

6ce5af2

Addressed

9cb2ffc

mihaibudiu reviewed Jul 6, 2026

View reviewed changes

Addressed

050a953

mihaibudiu reviewed Jul 7, 2026

View reviewed changes

Addressed

5a714dd

mihaibudiu reviewed Jul 8, 2026

View reviewed changes


		!ok

		# [CALCITE-6104] Aggregate function that references outer column should be evaluated in outer query

Uh oh!

Conversation

xuzifu666 commented Jun 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 commented Jun 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 commented Jul 3, 2026

Uh oh!

mihaibudiu left a comment

Choose a reason for hiding this comment

Uh oh!

xuzifu666 commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 Jun 27, 2026 •

edited

Loading

xuzifu666 Jun 27, 2026 •

edited

Loading

xuzifu666 Jun 28, 2026 •

edited

Loading

xuzifu666 commented Jul 4, 2026 •

edited

Loading

xuzifu666 Jul 4, 2026 •

edited

Loading

xuzifu666 Jul 8, 2026 •

edited

Loading