[branch-54] Avoid sort pushdown panic on incomplete file statistics

[geoffreyclaude]

Problem

DF54 added a sort-pushdown fallback that tries to reorder files within each file group by min/max statistics. This is an optional optimization: if the files do not have usable stats, planning should keep the original file order and leave the normal SortExec in place.

One path did not handle incomplete file stats safely.

MinMaxStatistics::new_from_files() first looks for min/max in file.statistics.column_statistics. If the requested column index is past that vector, it assumes the value must be a partition column and reads file.partition_values[...].

That assumption is not always true. A file can have a statistics object with no column statistics and also have no partition values.

Simplified failure:

requested sort column: a@0

file.statistics.column_statistics: []
file.partition_values: []

current code:
  column 0 is not in column_statistics
  so read partition_values[0]
  but partition_values is empty
  panic: index out of bounds

Observed production stack:

PushdownSort::optimize
  DataSourceExec::try_pushdown_sort
    FileScanConfig::try_sort_file_groups_by_statistics
      sort_files_within_groups_by_statistics
        MinMaxStatistics::new_from_files
          statistics.rs: pv[i - column_statistics.len()]

This surfaced on df-executor-canary while planning BlobStore parquet scans for idp-shadow ListEntities requests. The failing query shape sorted service names by LOWER(name) with pagination. This is not response-cache related: the trace had cache_aware_planning=false and connectors_used=[blob].

Patch

• Bounds-check the partition-value lookup in MinMaxStatistics::new_from_files().
• Return the existing statistics not found planning signal when neither column stats nor partition values contain the requested stat.
• Keep the caller behavior unchanged: sort_files_within_groups_by_statistics() already catches missing-stat errors, keeps the original file group order, and skips the optional pushdown.

In other words, this does not turn the panic into a user-visible planning error. It makes the optional optimization gracefully decline:

stats available:
  reorder files by min/max

stats unavailable:
  keep original file order
  keep SortExec
  query still plans normally

Why this belongs in DataFusion

The invalid assumption is in the shared file-statistics helper, not in a Datadog connector. A dd-source-only mitigation, such as forcing BlobStore scans to collect parquet footer stats, would add planning I/O and only cover one caller. The helper should be robust whenever file statistics are incomplete.

Tests

• cargo test -p datafusion-datasource min_max_statistics_errors_when_column_and_partition_stats_are_missing
• cargo test -p datafusion-datasource sort_pushdown_unsupported_source_empty_column_statistics

References

• Datadog trace: https://app.datadoghq.com/apm/trace/5724981991073414002?graphType=flamegraph&shouldShowLegend=true&spanID=3283904308320907482&timeHint=1783001047000.0000&trace=57249819910734140023283904308320907482&traceQuery=
• Upstream activation commit: cdfade51a feat: sort file groups by statistics during sort pushdown (Sort pushdown phase 2) (#21182)

[geoffreyclaude]

Duplicate of upstream PR apache#23277