chore: Update to arrow/parquet 59.1.0 by alamb · Pull Request #23312 · apache/datafusion

alamb · 2026-07-03T18:15:43Z

(DRAFT until arrow is updated, I am using this PR to pre-test the release)

Which issue does this PR close?

related to Release arrow-rs / parquet Minor version 59.1.0 (June 2026) arrow-rs#9878

Rationale for this change

Update to latest arrow

What changes are included in this PR?

Are these changes tested?

Yes by CI

Are there any user-facing changes?

No API change (this is a minor update of Arrow)

github-actions · 2026-07-03T19:37:20Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
error: running 'cargo update' on crate 'datafusion-common' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.1.0"`
candidate versions found which didn't match: 59.0.0, 58.3.0, 58.2.0, ...
location searched: crates.io index
required by package `datafusion-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common)`
    ... which satisfies path dependency `datafusion-common` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_common-54_0_0-x86_64_unknown_linux_gnu-f9cccb49834cc3e3)`

-----
error: failed to update dependencies for crate datafusion-common v54.0.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/common --features backtrace,force_hash_collisions,object_store,parquet,parquet_encryption,recursive_protection,sql,sqlparser &&
          cargo update

error: running 'cargo update' on crate 'datafusion-functions' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.1.0"`
candidate versions found which didn't match: 59.0.0, 58.3.0, 58.2.0, ...
location searched: crates.io index
required by package `datafusion-functions v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions)`
    ... which satisfies path dependency `datafusion-functions` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_functions-54_0_0-x86_64_unknown_linux_gnu-aebe678f280ae0e3)`

-----
error: failed to update dependencies for crate datafusion-functions v54.0.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/functions --features base64,blake2,blake3,chrono-tz,crypto_expressions,datetime_expressions,default,encoding_expressions,hex,math_expressions,md-5,regex,regex_expressions,sha2,string_expressions,unicode_expressions,uuid &&
          cargo update

error: running 'cargo update' on crate 'datafusion-functions-aggregate' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.1.0"`
candidate versions found which didn't match: 59.0.0, 58.3.0, 58.2.0, ...
location searched: crates.io index
required by package `datafusion-functions-aggregate v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate)`
    ... which satisfies path dependency `datafusion-functions-aggregate` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_functions_aggregate-54_0_0-x86_64_unknown_linux_gnu-04e18662548f8c91)`

-----
error: failed to update dependencies for crate datafusion-functions-aggregate v54.0.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/functions-aggregate --features force_hash_collisions &&
          cargo update

error: running 'cargo update' on crate 'datafusion-functions-nested' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.1.0"`
candidate versions found which didn't match: 59.0.0, 58.3.0, 58.2.0, ...
location searched: crates.io index
required by package `datafusion-functions-nested v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-nested)`
    ... which satisfies path dependency `datafusion-functions-nested` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_functions_nested-54_0_0-x86_64_unknown_linux_gnu-4bad501135bc976c)`

-----
error: failed to update dependencies for crate datafusion-functions-nested v54.0.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/functions-nested --features default,sql &&
          cargo update

error: running 'cargo update' on crate 'datafusion-physical-expr-common' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.1.0"`
candidate versions found which didn't match: 59.0.0, 58.3.0, 58.2.0, ...
location searched: crates.io index
required by package `datafusion-physical-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-common)`
    ... which satisfies path dependency `datafusion-physical-expr-common` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_physical_expr_common-54_0_0-x86_64_unknown_linux_gnu-ffd76717a91ae125)`

-----
error: failed to update dependencies for crate datafusion-physical-expr-common v54.0.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/physical-expr-common --features default,proto &&
          cargo update

error: running 'cargo update' on crate 'datafusion-proto-common' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.1.0"`
candidate versions found which didn't match: 59.0.0, 58.3.0, 58.2.0, ...
location searched: crates.io index
required by package `datafusion-proto-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/proto-common)`
    ... which satisfies path dependency `datafusion-proto-common` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_proto_common-54_0_0-x86_64_unknown_linux_gnu-acf4fa010dedbd89)`

-----
error: failed to update dependencies for crate datafusion-proto-common v54.0.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/proto-common --features default,json,pbjson,serde &&
          cargo update

error: running 'cargo update' on crate 'datafusion-spark' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.1.0"`
candidate versions found which didn't match: 59.0.0, 58.3.0, 58.2.0, ...
location searched: crates.io index
required by package `datafusion-spark v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/spark)`
    ... which satisfies path dependency `datafusion-spark` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_spark-54_0_0-x86_64_unknown_linux_gnu-a9517d2f8b476eb8)`

-----
error: failed to update dependencies for crate datafusion-spark v54.0.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/spark --features core,datafusion,default &&
          cargo update

error: aborting due to failure to run 'cargo update' for crate datafusion-common v54.0.0

alamb · 2026-07-03T19:37:12Z

                // add an initial FlightData message that sends schema
                let options = arrow::ipc::writer::IpcWriteOptions::default();
-                let mut compression_context = CompressionContext::default();
+                let mut compression_context = IpcWriteContext::default();


Struct was renamed (old was deprecated) in

rename Compression struct arrow-rs#10221

alamb · 2026-07-03T19:38:09Z


            for (start, end) in valid_or_empty.set_slices() {
-                mutable_array_data.extend(
+                mutable_array_data.try_extend(


Required because extend and extend_null was deprecated

Fix deprecate MutableArrayData::extend and MutableArrayData::extend_nulls in favour of fallible try_extend / try_extend_nulls arrow-rs#9710

(basically now errors are propagated rather than causing panics)

Jefffrey · 2026-07-04T00:23:21Z

things that can now progress after this arrow-rs upgrade:

arrow_cast accepts invalid Time32(Microsecond)/Time32(Nanosecond)/Time64(Second)/Time64(Millisecond), panics on use #22194
Panic when casting [] to FixedSizeList(0, Null) #9158
Use concat_elements_dyn from arrow-rs #23211 cc @pepijnve

alamb · 2026-07-04T09:35:32Z

things that can now progress after this arrow-rs upgrade:

I expect the release to be finalized / approved on Monday at the earliest and then I will mark this PR ready

Jefffrey · 2026-07-05T00:18:23Z

Looks like feat: Add support for MapArray in arrow_row arrow-rs#9486 also made it into the release so select distinct on map types yields ArrowError NotYetImplemented #15428 may also be able to progress

zhuqi-lucas · 2026-07-05T05:49:36Z

Nice — this also pulls in ParquetPushDecoder::peek_next_row_group(). I'll start on #23067 (Per-RG fully_matched RowFilter skip) once this lands!

chore: Update to arrow/parquet 59.1.0

dac7c3c

alamb mentioned this pull request Jul 3, 2026

Release arrow-rs / parquet Minor version 59.1.0 (June 2026) apache/arrow-rs#9878

Open

10 tasks

Update for deprecated APIs

d6e194c

github-actions Bot added physical-expr Changes to the physical-expr crates common Related to common crate proto Related to proto crate functions Changes to functions implementation spark labels Jul 3, 2026

github-actions Bot added the auto detected api change Auto detected API change label Jul 3, 2026

alamb commented Jul 3, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore: Update to arrow/parquet 59.1.0#23312

chore: Update to arrow/parquet 59.1.0#23312
alamb wants to merge 2 commits into
apache:mainfrom
alamb:alamb/update_arrow_59.1.0

alamb commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

alamb Jul 3, 2026

Uh oh!

alamb Jul 3, 2026

Uh oh!

Jefffrey commented Jul 4, 2026 •

edited

Loading

Uh oh!

alamb commented Jul 4, 2026

Uh oh!

Jefffrey commented Jul 5, 2026

Uh oh!

zhuqi-lucas commented Jul 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

alamb commented Jul 3, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

alamb Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Jefffrey commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb commented Jul 4, 2026

Uh oh!

Jefffrey commented Jul 5, 2026

Uh oh!

zhuqi-lucas commented Jul 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Jefffrey commented Jul 4, 2026 •

edited

Loading