Skip to content

Devalue uxarray to make it less expensive#1588

Open
cmdupuis3 wants to merge 11 commits into
UXARRAY:mainfrom
cmdupuis3:cmd/devalue
Open

Devalue uxarray to make it less expensive#1588
cmdupuis3 wants to merge 11 commits into
UXARRAY:mainfrom
cmdupuis3:cmd/devalue

Conversation

@cmdupuis3

@cmdupuis3 cmdupuis3 commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator

Closes #1583

Overview

This PR is to resolve suboptimal usage of .values throughout the repo, primarily by deferring to lazy xarray and dask operations. See the table of usage sites in Issue #1583.

This specific PR is for the first two tables of usage sites; dead code and bugfixes aren't really included here. Aside from simple cases, some cases were solved by branching on whether the data type was already chunked, so scalability for some parts will depend on if you're using numpy or xarray/dask arrays at those points.

PR Checklist

General

  • An issue is linked created and linked
  • Add appropriate labels
  • Filled out Overview and Expected Usage (if applicable) sections

Testing

  • Adequate tests are created if there is new functionality
  • Tests cover all possible logical paths in your function
  • Tests are not too basic (such as simply calling a function and nothing else)

Documentation

  • Docstrings have been added to all new functions
  • Docstrings have updated with any function changes
  • Internal functions have a preceding underscore (_) and have been added to docs/internal_api/index.rst
  • User functions have been added to docs/user_api/index.rst

Examples

  • Any new notebook examples added to docs/examples/ folder
  • Clear the output of all cells before committing
  • New notebook files added to docs/examples.rst toctree
  • New notebook files added to new entry in docs/gallery.yml with appropriate thumbnail photo in docs/_static/thumbnails/

@cmdupuis3 cmdupuis3 self-assigned this Jul 18, 2026
@cmdupuis3 cmdupuis3 added scalability Related to scalability & performance efforts run-benchmark Run ASV benchmark workflow labels Jul 18, 2026
@github-actions

github-actions Bot commented Jul 18, 2026

Copy link
Copy Markdown

ASV Benchmarking

Benchmark Comparison Results

Benchmarks that have improved:

Change Before [b0487c4] <v2026.07.0> After [136177a] Ratio Benchmark (Parameter)
- 578M 391M 0.68 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
- 699M 390M 0.56 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
- 695±2ms 587±3ms 0.84 mpas_ocean.CrossSections.time_const_lat('120km', 1)
- 351±2ms 293±3ms 0.84 mpas_ocean.CrossSections.time_const_lat('120km', 2)
- 183±0.7ms 152±0.7ms 0.83 mpas_ocean.CrossSections.time_const_lat('120km', 4)
- 498M 383M 0.77 mpas_ocean.Gradient.peakmem_gradient('480km')

Benchmarks that have stayed the same:

Change Before [b0487c4] <v2026.07.0> After [136177a] Ratio Benchmark (Parameter)
10.7±0.06μs 10.8±0.2μs 1.02 bench_connectivity.Connectivity.time_edge_face('120km')
11.3±0.1μs 11.2±0.1μs 0.99 bench_connectivity.Connectivity.time_edge_face('480km')
10.8±0.08μs 10.4±0.2μs 0.97 bench_connectivity.Connectivity.time_edge_node('120km')
11.3±0.1μs 11.0±0.1μs 0.97 bench_connectivity.Connectivity.time_edge_node('480km')
10.8±0.07μs 11.0±0.1μs 1.02 bench_connectivity.Connectivity.time_face_edge('120km')
11.1±0.1μs 11.2±0.2μs 1.01 bench_connectivity.Connectivity.time_face_edge('480km')
10.9±0.1μs 10.9±0.04μs 1 bench_connectivity.Connectivity.time_face_face('120km')
11.1±0.1μs 11.0±0.2μs 0.99 bench_connectivity.Connectivity.time_face_face('480km')
21.8±0.4μs 21.5±0.1μs 0.98 bench_connectivity.Connectivity.time_face_node('120km')
22.9±0.2μs 22.3±0.3μs 0.97 bench_connectivity.Connectivity.time_face_node('480km')
10.8±0.2μs 11.0±0.07μs 1.02 bench_connectivity.Connectivity.time_node_edge('120km')
11.2±0.2μs 11.2±0.07μs 1 bench_connectivity.Connectivity.time_node_edge('480km')
10.9±0.1μs 10.7±0.1μs 0.99 bench_connectivity.Connectivity.time_node_face('120km')
11.2±0.08μs 11.1±0.2μs 0.99 bench_connectivity.Connectivity.time_node_face('480km')
391M 389M 0.99 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
421M 421M 1 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
14.6±0.1ms 14.4±0.05ms 0.99 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
3.67±0.03ms 3.64±0.03ms 0.99 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
18.9±0.06ms 19.1±0.04ms 1.01 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
2.20±0.04ms 2.17±0.02ms 0.99 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.00±0s 1.03±0.01s 1.02 import.Imports.timeraw_import_uxarray
899±4ns 932±3ns 1.04 mpas_ocean.CheckNorm.time_check_norm('120km')
917±7ns 900±8ns 0.98 mpas_ocean.CheckNorm.time_check_norm('480km')
808±4ms 832±10ms 1.03 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('120km')
53.4±0.8ms 53.6±0.6ms 1 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('480km')
650±9μs 677±7μs 1.04 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('120km')
582±10μs 587±10μs 1.01 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('480km')
5.31±0.04ms 5.38±0.02ms 1.01 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('120km')
3.85±0.02ms 3.91±0.02ms 1.02 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('480km')
3.45±0.01s 3.44±0.01s 1 mpas_ocean.ConstructFaceLatLon.time_welzl('120km')
224±0.7ms 224±2ms 1 mpas_ocean.ConstructFaceLatLon.time_welzl('480km')
18.1±0.01ms 18.2±0.03ms 1 mpas_ocean.ConstructTreeStructures.time_ball_tree('120km')
1.02±0.03ms 1.05±0ms 1.02 mpas_ocean.ConstructTreeStructures.time_ball_tree('480km')
10.6±0.01ms 10.6±0.02ms 1.01 mpas_ocean.ConstructTreeStructures.time_kd_tree('120km')
686±20μs 740±10μs 1.08 mpas_ocean.ConstructTreeStructures.time_kd_tree('480km')
538±2ms 524±1ms 0.97 mpas_ocean.CrossSections.time_const_lat('480km', 1)
273±1ms 266±3ms 0.97 mpas_ocean.CrossSections.time_const_lat('480km', 2)
141±0.7ms 137±0.7ms 0.97 mpas_ocean.CrossSections.time_const_lat('480km', 4)
24.2±0.2ms 24.5±0.1ms 1.01 mpas_ocean.DualMesh.time_dual_mesh_construction('120km')
3.09±0.04ms 3.12±0.05ms 1.01 mpas_ocean.DualMesh.time_dual_mesh_construction('480km')
946±7ms 951±3ms 1.01 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', False)
51.4±0.4ms 55.6±0.7ms 1.08 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', True)
84.4±0.7ms 85.8±0.6ms 1.02 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', False)
5.45±0.2ms 5.78±0.2ms 1.06 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', True)
404M 404M 1 mpas_ocean.Gradient.peakmem_gradient('120km')
174±0.8ms 176±0.8ms 1.01 mpas_ocean.Gradient.time_gradient('120km')
12.2±0.09ms 12.3±0.08ms 1 mpas_ocean.Gradient.time_gradient('480km')
223±2μs 225±1μs 1.01 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('120km')
131±0.6μs 134±1μs 1.02 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('480km')
351M 351M 1 mpas_ocean.Integrate.peakmem_integrate('120km')
331M 330M 1 mpas_ocean.Integrate.peakmem_integrate('480km')
217±4μs 214±1μs 0.99 mpas_ocean.Integrate.time_integrate('120km')
201±2μs 195±1μs 0.97 mpas_ocean.Integrate.time_integrate('480km')
181±2ms 180±2ms 0.99 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'exclude')
183±2ms 182±0.9ms 1 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'include')
180±2ms 180±1ms 1 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'split')
13.6±0.3ms 13.8±0.08ms 1.02 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'exclude')
13.6±0.04ms 13.6±0.08ms 1.01 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'include')
13.5±0.3ms 13.9±0.1ms 1.03 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'split')
355±2μs 357±1μs 1.01 mpas_ocean.PointInPolygon.time_face_search_lonlat('120km')
355±3μs 353±3μs 1 mpas_ocean.PointInPolygon.time_face_search_lonlat('480km')
335±3μs 334±2μs 1 mpas_ocean.PointInPolygon.time_face_search_xyz('120km')
340±0.8μs 338±2μs 0.99 mpas_ocean.PointInPolygon.time_face_search_xyz('480km')
243±0.8ms 245±2ms 1.01 mpas_ocean.RemapDownsample.time_bilinear_remapping
297±1ms 295±3ms 0.99 mpas_ocean.RemapDownsample.time_inverse_distance_weighted_remapping
1.43±0.01s 1.44±0.01s 1.01 mpas_ocean.RemapUpsample.time_bilinear_remapping
36.6±0.6ms 37.5±0.8ms 1.02 mpas_ocean.RemapUpsample.time_inverse_distance_weighted_remapping
9.45±0.2ms 9.71±0.04ms 1.03 mpas_ocean.RemapUpsample.time_nearest_neighbor_remapping
29.2±0.2ms 28.7±0.05ms 0.98 mpas_ocean.ZonalAverage.time_zonal_average('120km')
6.65±0ms 6.84±0.05ms 1.03 mpas_ocean.ZonalAverage.time_zonal_average('480km')
326M 328M 1.01 quad_hexagon.QuadHexagon.peakmem_open_dataset
326M 326M 1 quad_hexagon.QuadHexagon.peakmem_open_grid
7.02±0.1ms 7.09±0.1ms 1.01 quad_hexagon.QuadHexagon.time_open_dataset
6.03±0.1ms 6.07±0.03ms 1.01 quad_hexagon.QuadHexagon.time_open_grid

Benchmarks that have got worse:

Change Before [b0487c4] <v2026.07.0> After [136177a] Ratio Benchmark (Parameter)
+ 4.26±0.01ms 4.81±0.02ms 1.13 mpas_ocean.RemapDownsample.time_nearest_neighbor_remapping

@cmdupuis3

Copy link
Copy Markdown
Collaborator Author

pre-commit.ci autofix

@cmdupuis3

Copy link
Copy Markdown
Collaborator Author

There's some spurious asv results from machine variability, but the benchmarks consistenly show peak-mem reductions and some speedups for cross-sections.

@cmdupuis3
cmdupuis3 requested a review from erogluorhan July 20, 2026 23:25
@cmdupuis3
cmdupuis3 marked this pull request as ready for review July 20, 2026 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-benchmark Run ASV benchmark workflow scalability Related to scalability & performance efforts

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Census of ".values" and cleanup

1 participant