Skip to content

[CALCITE-7647] Support SELECT * in GROUP BY ALL and ORDER BY ALL #5089

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tisyabhatia wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[CALCITE-7647] Support SELECT * in GROUP BY ALL and ORDER BY ALL #5089

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions core/src/main/java/org/apache/calcite/runtime/CalciteResource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -389,9 +389,6 @@ ExInst<SqlValidatorException> naturalOrUsingColumnNotCompatible(String a0,
@BaseMessage("Windowed aggregate expression is illegal in {0} clause")
ExInst<SqlValidatorException> windowedAggregateIllegalInClause(String a0);

@BaseMessage("GROUP BY ALL requires an explicit SELECT list; ''*'' is not supported")
ExInst<SqlValidatorException> groupByAllRequiresExplicitSelectList();

@BaseMessage("Aggregate expressions cannot be nested")
ExInst<SqlValidatorException> nestedAggIllegal();

Expand Down Expand Up @@ -783,9 +780,6 @@ ExInst<CalciteException> illegalArgumentForTableFunctionCall(String a0,
@BaseMessage("Streaming ORDER BY must start with monotonic expression")
ExInst<SqlValidatorException> streamMustOrderByMonotonic();

@BaseMessage("ORDER BY ALL requires an explicit SELECT list; ''*'' is not supported")
ExInst<SqlValidatorException> orderByAllRequiresExplicitSelectList();

@BaseMessage("Set operator cannot combine streaming and non-streaming inputs")
ExInst<SqlValidatorException> streamSetOpInconsistentInputs();

Expand Down
32 changes: 28 additions & 4 deletions core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5256,7 +5256,25 @@
}
}

/** Expands a single "*" or "t.*" select item into its underlying columns,
* for a GROUP BY ALL / ORDER BY ALL rewrite.
*
* <p>Calls the private {@code expandStar} core directly (not the public
* {@code expandStar(SqlNodeList, SqlSelect, boolean)} wrapper), with fresh
* collections: the wrapper would derive types over every select item and
* mark the expanded list, poisoning {@link AggregatingSelectScope}'s
* memoized grouping set with the not-yet-rewritten placeholder. The fresh
* {@code items}/{@code fields} must stay paired for NATURAL/USING index
* alignment. */
private List<SqlNode> expandStarForAllRewrite(SqlSelect select, SqlNode starItem) {

@mihaibudiu mihaibudiu Jul 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how does this differ from expand star?
Can this be avoided by doing the rewriteOrderByAll after the star expansion?

@tisyabhatia tisyabhatia Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

expandStarForAllRewrite just runs the normal star expansion on the *. I don't call the public expandStar because it expands the whole SELECT list before GROUP BY ALL is replaced and that makes the real columns look "not grouped."

@xiedeyantu xiedeyantu Jul 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you give an example? I don't quite understand either.

@tisyabhatia tisyabhatia Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's say I had a query like SELECT *, UPPER(ename) FROM emp GROUP BY ALL. When we first type-check a column, we figure out which columns we're grouping by and cache that answer for good. So if I use the public expandStar, it type-checks UPPER(name) before I've replaced GROUP BY ALL with the actual columns. So internally, that stores "we don't have to group by anything." After that, I can't change the that memory. We get the error "Expression EMP.EMPNO is not being grouped."
It reports EMPNO (first column), not UPPER(ename).
Note: SELECT * FROM emp GROUP BY ALL works fine, but the extra UPPER(ename) makes Calcite type-check early.
This helper only touches the * so nothing gets locked in before we've decided the grouping columns.

final SelectScope scope = (SelectScope) getWhereScope(select);
final List<SqlNode> items = new ArrayList<>();
expandStar(items, catalogReader.nameMatcher().createSet(), PairList.of(),
false, scope, starItem, false);
return items;
}

protected void rewriteOrderByAll(SqlSelect select) {

Check failure on line 5277 in core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Refactor this method to reduce its Cognitive Complexity from 16 to the 15 allowed. 


See more on https://sonarcloud.io/project/issues?id=apache_calcite&issues=AZ9DCpo_22eGBhYISFaR&open=AZ9DCpo_22eGBhYISFaR&pullRequest=5089
final SqlNodeList orderList = select.getOrderList();
if (orderList == null || orderList.size() != 1) {
return;
Expand Down Expand Up @@ -5288,8 +5306,10 @@
for (SqlNode selectItem : select.getSelectList()) {
final SqlNode expr = SqlUtil.stripAs(selectItem);
if (expr instanceof SqlIdentifier && ((SqlIdentifier) expr).isStar()) {
throw newValidationError(expr,
RESOURCE.orderByAllRequiresExplicitSelectList());
for (SqlNode column : expandStarForAllRewrite(select, expr)) {
keys.add(applyOrderByAllDirection(column, desc, nulls, pos));
}
continue;
}
keys.add(applyOrderByAllDirection(expr, desc, nulls, pos));
}
Expand Down Expand Up @@ -5475,7 +5495,7 @@

/** If GROUP BY clause is the {@code GROUP BY ALL} placeholder, replaces it
* with every non-aggregated expression from the SELECT clause. */
private void rewriteGroupByAll(SqlSelect select) {

Check failure on line 5498 in core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Refactor this method to reduce its Cognitive Complexity from 17 to the 15 allowed. 


See more on https://sonarcloud.io/project/issues?id=apache_calcite&issues=AZ9DCpo_22eGBhYISFaS&open=AZ9DCpo_22eGBhYISFaS&pullRequest=5089
final SqlNodeList groupList = select.getGroup();
if (groupList == null
|| groupList.size() != 1
Expand All @@ -5483,14 +5503,18 @@
return;
}
final List<SqlNode> keys = new ArrayList<>();
for (SqlNode selectItem : select.getSelectList()) {

Check warning on line 5506 in core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Reduce the total number of break and continue statements in this loop to use at most one. 


See more on https://sonarcloud.io/project/issues?id=apache_calcite&issues=AZ9DCpo_22eGBhYISFaQ&open=AZ9DCpo_22eGBhYISFaQ&pullRequest=5089
if (SqlValidatorUtil.isMeasure(selectItem)) {
continue;
}
final SqlNode expr = SqlUtil.stripAs(selectItem);
if (expr instanceof SqlIdentifier && ((SqlIdentifier) expr).isStar()) {
throw newValidationError(expr,
RESOURCE.groupByAllRequiresExplicitSelectList());
for (SqlNode column : expandStarForAllRewrite(select, expr)) {
if (aggOrOverFinder.findAgg(column) == null) {
keys.add(column);
}
}
continue;
}
if (aggOrOverFinder.findAgg(expr) == null) {
keys.add(expr);
Expand Down
2 changes: 0 additions & 2 deletions core/src/main/resources/org/apache/calcite/runtime/CalciteResource.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,6 @@ GroupingInWrongClause={0} operator may only occur in SELECT, HAVING or ORDER BY
NotSelectDistinctExpr=Expression ''{0}'' is not in the select clause
AggregateIllegalInClause=Aggregate expression is illegal in {0} clause
WindowedAggregateIllegalInClause=Windowed aggregate expression is illegal in {0} clause
GroupByAllRequiresExplicitSelectList=GROUP BY ALL requires an explicit SELECT list; ''*'' is not supported
NestedAggIllegal=Aggregate expressions cannot be nested
MeasureIllegal=Measure expressions can only occur within AGGREGATE function
MeasureMustBeInAggregateQuery=Measure expressions can only occur within a GROUP BY query
Expand Down Expand Up @@ -256,7 +255,6 @@ CannotConvertToStream=Cannot convert table ''{0}'' to stream
CannotConvertToRelation=Cannot convert stream ''{0}'' to relation
StreamMustGroupByMonotonic=Streaming aggregation requires at least one monotonic expression in GROUP BY clause
StreamMustOrderByMonotonic=Streaming ORDER BY must start with monotonic expression
OrderByAllRequiresExplicitSelectList=ORDER BY ALL requires an explicit SELECT list; ''*'' is not supported
StreamSetOpInconsistentInputs=Set operator cannot combine streaming and non-streaming inputs
CannotStreamValues=Cannot stream VALUES
CyclicDefinition=Cannot resolve ''{0}''; it references view ''{1}'', whose definition is cyclic
Expand Down
25 changes: 19 additions & 6 deletions core/src/test/java/org/apache/calcite/test/SqlValidatorTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7337,9 +7337,15 @@ public boolean isBangEqualAllowed() {
sql("select deptno, sal from emp order by all").ok();
// direction applies to every expanded key
sql("select deptno, sal from emp order by all desc").ok();
// SELECT * can't be expanded here
sql("select ^*^ from emp order by all")
.fails("(?s).*ORDER BY ALL requires an explicit SELECT list.*");
// SELECT * is expanded to the underlying columns, each of which
// becomes a sort key
sql("select * from emp order by all").ok();
// SELECT * combined with a trailing direction
sql("select * from emp order by all desc").ok();
// Multiple qualified stars are expanded independently (no state leaks
// between them) and the trailing direction still applies to every one of
// the expanded keys, including the duplicated join column
sql("select emp.*, dept.* from emp, dept order by all desc").ok();
// Aliases that shadow other column names must not confuse expansion
sql("select empno as deptno, deptno as empno from emp order by all").ok();
// verify "x" still resolves and the two features coexist
Expand Down Expand Up @@ -7793,9 +7799,16 @@ public boolean isBangEqualAllowed() {
// only aggregates -> global aggregation (one group), still valid
sql("select count(*) from emp group by all").ok();

// SELECT * cannot be expanded at group-validation time -> clear error
sql("select ^*^ from emp group by all")
.fails("(?s).*GROUP BY ALL requires an explicit SELECT list.*");
// SELECT * is expanded to the underlying columns, each of which
// becomes a group key
sql("select * from emp group by all").ok();
// SELECT * alongside an aggregate: star columns become group keys,
// the aggregate is excluded
sql("select *, count(*) from emp group by all").ok();
// SELECT * over a NATURAL JOIN: the star expands via the same path as a
// normal projection, so the shared join column is coalesced (not
// duplicated) among the group keys
sql("select * from emp natural join dept group by all").ok();

// contains-an-aggregate
sql("select deptno, substring(job, 1), count(*) + 1 as c, 'x' as x\n"
Expand Down
15 changes: 15 additions & 0 deletions core/src/test/resources/sql/agg.iq
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4301,4 +4301,19 @@ GROUP BY GROUPING SETS ((deptno), ());

!ok

# [CALCITE-7647] GROUP BY ALL expands SELECT * to every underlying column;
# the star columns become grouping keys and the aggregate is excluded
select *, count(*) as c from (values (1, 'a'), (1, 'a'), (2, 'b')) as t(x, y)

@xuzifu666 xuzifu666 Jul 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to know if this has been verified through specific database testing which @mihaibudiu also metioned in jira.

@tisyabhatia tisyabhatia Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question; yes, it's verified.

GROUP BY ALL originates in DuckDB, and DuckDB, Snowflake, and Databricks/Spark SQL all group by the non-aggregate items in the SELECT list. DuckDB expands * to its underlying columns for this, so a star and an aggregate can coexist.

I ran this exact test case in the DuckDB shell (https://shell.duckdb.org):

SELECT *, count(*) AS c
FROM (VALUES (1, 'a'), (1, 'a'), (2, 'b')) AS t(x, y)
GROUP BY ALL
ORDER BY ALL;
┌───────┬─────────┬───────┐
│   x   │    y    │   c   │
├───────┼─────────┼───────┤
│   1   │    a    │   2   │
│   2   │    b    │   1   │
└───────┴─────────┴───────┘

Same result as test: * expands to (x, y), those become the grouping keys, and COUNT(*) is excluded

group by all
order by x;
+---+---+---+
| X | Y | C |
+---+---+---+
| 1 | a | 2 |
| 2 | b | 1 |
+---+---+---+
(2 rows)

!ok

# End agg.iq
14 changes: 14 additions & 0 deletions core/src/test/resources/sql/sort.iq
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -563,4 +563,18 @@ order by all;

!ok

# [CALCITE-7647] ORDER BY ALL expands SELECT * to every underlying column
select * from (values (2, 'b'), (1, 'a'), (1, 'c')) as t(x, y)
order by all;
+---+---+
| X | Y |
+---+---+
| 1 | a |
| 1 | c |
| 2 | b |
+---+---+
(3 rows)

!ok

# End sort.iq
6 changes: 6 additions & 0 deletions site/_docs/reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -426,6 +426,9 @@ in the order that they appear in the list; for example:
"SELECT x, y FROM t ORDER BY ALL" is equivalent to
"SELECT x, y FROM t ORDER BY x, y"
An optional trailing ASC / DESC and NULLS FIRST / NULLS LAST applies to all keys.
A `*` in the SELECT clause is expanded to its underlying columns, each of which
becomes a sort key; for example, "SELECT * FROM t ORDER BY ALL" sorts by every
column of `t`.

In *query*, *count* and *start* may each be either an unsigned integer literal
or a dynamic parameter whose value is an integer.
Expand Down Expand Up @@ -460,6 +463,9 @@ GROUP BY ALL on its own groups by every expression in the SELECT clause
that is not an aggregate function; for example,
"SELECT deptno, SUM(sal) FROM emp GROUP BY ALL" is equivalent to
"SELECT deptno, SUM(sal) FROM emp GROUP BY deptno".
A `*` in the SELECT clause is expanded to its underlying columns, each of which
becomes a grouping key; for example,
"SELECT *, COUNT(*) FROM emp GROUP BY ALL" groups by every column of `emp`.

*selectWithoutFrom* is equivalent to VALUES,
but is not standard SQL and is only allowed in certain
Expand Down
Loading