From 57ba36d8b2357f30eb5ea9c03d193c1e1f534e1d Mon Sep 17 00:00:00 2001 From: Patrick Peglar Date: Mon, 6 Jul 2026 17:33:39 +0100 Subject: [PATCH 1/3] Fill docs placeholders for old netcdf features. --- .../src/user_manual/explanation/netcdf_io.rst | 92 ++++++++++++++++--- 1 file changed, 81 insertions(+), 11 deletions(-) diff --git a/docs/src/user_manual/explanation/netcdf_io.rst b/docs/src/user_manual/explanation/netcdf_io.rst index f5f608f5c8..d021234a3f 100644 --- a/docs/src/user_manual/explanation/netcdf_io.rst +++ b/docs/src/user_manual/explanation/netcdf_io.rst @@ -133,6 +133,87 @@ Iris' optimisation all together, and will take its chunksizes from Dask's behavi (70, 37, 49) +Split Attributes +----------------- + +Since Iris does not provide a "dataset" object, it is not possible to represent netcdf +file-level, aka "global", attributes directly. Instead, the attributes of *cubes*, +specifically, are "split" to discriminate between global (i.e. file) and local (i.e. +variable) attributes. + +Prior to Iris v3.8, this was done "automatically" according to a specific lists of +known attribute names which would normally *only* appear as local (variable) attributes +(e.g. "valid_min") or *only* as global (file) ones (e.g. "history"). +All other names were saved as "global if the same for all cubes, otherwise local". + +Since Iris v3.8, but explicit handling has now been added, so that a ``cube.attributes`` +is a :class:`~iris.cube.CubeAttrsDict` object which combines global and local attributes +within a single dictionary-like object. +Finer-grained control of attribute saving is thus now possible, by setting the +:data:`iris.Future.save_split_attrs` control to ``True``. +Ideally, anyone who is saving cubes to netcdf files should now set this, and +a warning is raised if it is not set. + +.. note:: + The default setting remains ``iris.FUTURE.save_split_attrs = False``, purely for + backwards compatibility, but it is recommended to override this for all new code. + +The :class:`~iris.cube.CubeAttrsDict` provides a regular dictionary access which mimics +the older single dictionary, while also containing separate lists in its ``.global`` and +``.local`` properties which allow the user to specifically choose the nature of each +attribute. +On saving, each cube's ".attributes.local"s become variable-level attributes and its +".attributes.global"s become file-level attributes -- *except only*, if globals +have different values in multiple cubes, then global attributes may be 'demoted' to +local ones. + +Summary +======= + +* You should now always set the ``iris.Future.save_split_attrs=True`` unless you have a + specific need for backwards compatibility. +* all Cube attributes are now stored as specifically "global" or "local" +* you can continue to assign and fetch cube attributes as before, but when required you + can also now access ``.global`` and ``.local`` properties specifically. +* practically, all this is only relevant when **saving to netcdf files** + + +Deferred Saving +---------------- + +In some cases it is useful for performance reasons to defer the actual writing of bulk +data to a NetCDF file's variables, from after the "iris.save" call to a later time. + +You can do this by adding a ``compute=False`` keyword to the ``iris.save`` call. The +call then returns, instead of None, a :class:``dask.delayed`` object, which can be +computed later to complete the writing of variable data to the file. + +Where this can be useful is when you are saving multiple cubes whose lazy data is +computed from shared data, for example multiple statistics over the same data. + +To do this, you specify ``compute=False`` to multiple :func:``iris.save`` calls, and +collect the resulting :class:``dask.delayed`` objects. You can then complete the saves +in parallel, by supplying a list of the 'delayed' results to the :func:`Dask.compute` +function. + +This mechanism allows Dask to calculate the multiple "derived" results while only +fetching the "common" data from which they derive **once**, whereas otherwise the source +data may need to be fetched multiple times. In this respect, the benefit is the same as +the :meth:``iris.cube.CubeList.realise_cubes`` method. However there is an additional +potential benefit in this case, which is that the actual **writing** of data to file can +be performed in parallel. The scope of this, however, depends on the parallel +capabilities of the Python ``netCDF4`` package, the NetCDF C library, the file +system and the operating system : *Typically* you can write data arrays in parallel +to **separate files**, but not to different variables, or variable sections *within a +single file*. + + +Character and String data +------------------------- + +TBC + + Variable-length datatypes ------------------------- @@ -189,17 +270,6 @@ loader so it can be make a more informed decision on lazy loading: False -Split Attributes ------------------ - -TBC - - -Deferred Saving ----------------- - -TBC - .. _save_load_dataless: Dataless Cubes in NetCDF files From 53c1354927a264ee1903e9e39fcc5f1456ddc9b6 Mon Sep 17 00:00:00 2001 From: Patrick Peglar Date: Mon, 6 Jul 2026 17:56:16 +0100 Subject: [PATCH 2/3] Fix title level --- docs/src/user_manual/explanation/netcdf_io.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/user_manual/explanation/netcdf_io.rst b/docs/src/user_manual/explanation/netcdf_io.rst index d021234a3f..47adf88fe2 100644 --- a/docs/src/user_manual/explanation/netcdf_io.rst +++ b/docs/src/user_manual/explanation/netcdf_io.rst @@ -168,7 +168,7 @@ have different values in multiple cubes, then global attributes may be 'demoted' local ones. Summary -======= +^^^^^^^ * You should now always set the ``iris.Future.save_split_attrs=True`` unless you have a specific need for backwards compatibility. From 2b4c926c953d82843dc743f28ba9b915a1fe37b2 Mon Sep 17 00:00:00 2001 From: Patrick Peglar Date: Mon, 6 Jul 2026 18:36:25 +0100 Subject: [PATCH 3/3] Improves --- .../src/user_manual/explanation/netcdf_io.rst | 93 +++++++++++-------- 1 file changed, 52 insertions(+), 41 deletions(-) diff --git a/docs/src/user_manual/explanation/netcdf_io.rst b/docs/src/user_manual/explanation/netcdf_io.rst index 47adf88fe2..ec7c86300e 100644 --- a/docs/src/user_manual/explanation/netcdf_io.rst +++ b/docs/src/user_manual/explanation/netcdf_io.rst @@ -136,46 +136,57 @@ Iris' optimisation all together, and will take its chunksizes from Dask's behavi Split Attributes ----------------- -Since Iris does not provide a "dataset" object, it is not possible to represent netcdf -file-level, aka "global", attributes directly. Instead, the attributes of *cubes*, -specifically, are "split" to discriminate between global (i.e. file) and local (i.e. -variable) attributes. +Since Iris does not provide a "dataset" object, it is not possible to represent netCDF +file-level, aka "global", attributes directly. Instead, when saving to NetCDF, +the attributes of saved *cubes*, specifically, may be saved as either global +(i.e. file-level) or local (i.e. variable-specific) attributes. -Prior to Iris v3.8, this was done "automatically" according to a specific lists of -known attribute names which would normally *only* appear as local (variable) attributes -(e.g. "valid_min") or *only* as global (file) ones (e.g. "history"). -All other names were saved as "global if the same for all cubes, otherwise local". +This makes no difference to Iris behaviour, but it is relevant when saving cubes to +netCDF files, as to how the decision is made which kingd of attribute to create. -Since Iris v3.8, but explicit handling has now been added, so that a ``cube.attributes`` -is a :class:`~iris.cube.CubeAttrsDict` object which combines global and local attributes -within a single dictionary-like object. -Finer-grained control of attribute saving is thus now possible, by setting the -:data:`iris.Future.save_split_attrs` control to ``True``. -Ideally, anyone who is saving cubes to netcdf files should now set this, and -a warning is raised if it is not set. +**Prior to Iris v3.8** this was done automatically, using specific lists of known +attribute names defined in the `CF Conventions `_ +as normally *only* appearing as local (variable) attributes +(e.g. "valid_min") or *only* as global (file) ones (e.g. "history"). +For other names, the rule is _"global if the same for all cubes, otherwise local"_. + +**Since Iris v3.8**, all cube attributes now have explicit 'global' or 'local' identity. +This is assigned automatically on creation using similar rules as described above, or +type-specifically when loading from NetCDF, or explicitly controlled by the user. +A ``cube.attributes`` +is now a :class:`~iris.cube.CubeAttrsDict` object, which combines global and local +attributes within a single dictionary-like object. + +On saving to NetCDF, finer-grained type-specific control of attribute saving is now +possible, but this requires setting the ``.save_split_attrs`` property of +:data:`iris.FUTURE` to ``True`` -- either permanently with :meth:`~iris.Future.set` or +within a code block with :meth:`~iris.Future.context`. +Ideally, anyone who is saving cubes to netcdf files **should now set this** : a warning +is raised if it is not set. .. note:: The default setting remains ``iris.FUTURE.save_split_attrs = False``, purely for - backwards compatibility, but it is recommended to override this for all new code. + backwards compatibility : it is recommended to override this for all new code. The :class:`~iris.cube.CubeAttrsDict` provides a regular dictionary access which mimics -the older single dictionary, while also containing separate lists in its ``.global`` and -``.local`` properties which allow the user to specifically choose the nature of each -attribute. -On saving, each cube's ".attributes.local"s become variable-level attributes and its -".attributes.global"s become file-level attributes -- *except only*, if globals -have different values in multiple cubes, then global attributes may be 'demoted' to +the older single dictionary, while also providing access to separate types via its +``.globals`` and ``.locals`` properties, allowing the user (and NetCDF loads) to +specifically set the type of each attribute. +On saving, with ``iris.FUTURE.save_split_attrs`` enabled, each cube's +``.attributes.locals`` become variable-level attributes and its +``.attributes.globals`` become file-level attributes -- *except only that* if globals +have different values across multiple cubes, then global attributes may be 'demoted' to local ones. Summary ^^^^^^^ -* You should now always set the ``iris.Future.save_split_attrs=True`` unless you have a +* All Cube attributes are now stored as specifically "global" or "local" +* You should now always set the ``iris.Future.save_split_attrs = True`` unless you have a specific need for backwards compatibility. -* all Cube attributes are now stored as specifically "global" or "local" -* you can continue to assign and fetch cube attributes as before, but when required you - can also now access ``.global`` and ``.local`` properties specifically. -* practically, all this is only relevant when **saving to netcdf files** +* You can continue to assign and fetch cube attributes as before, but when required you + can also now access ``.globals`` and ``.locals`` properties specifically. +* Practically, all this is only relevant when **saving to netcdf files** Deferred Saving @@ -185,27 +196,27 @@ In some cases it is useful for performance reasons to defer the actual writing o data to a NetCDF file's variables, from after the "iris.save" call to a later time. You can do this by adding a ``compute=False`` keyword to the ``iris.save`` call. The -call then returns, instead of None, a :class:``dask.delayed`` object, which can be -computed later to complete the writing of variable data to the file. +call then returns, instead of None, a Dask :class:`~dask.delayed.Delayed` object, which +can be "compute"-d later to complete the writing of variable data to the file. -Where this can be useful is when you are saving multiple cubes whose lazy data is -computed from shared data, for example multiple statistics over the same data. +This can be when you are saving multiple cubes whose lazy data is computed from shared +data, for example multiple statistics over the same large data array. -To do this, you specify ``compute=False`` to multiple :func:``iris.save`` calls, and -collect the resulting :class:``dask.delayed`` objects. You can then complete the saves -in parallel, by supplying a list of the 'delayed' results to the :func:`Dask.compute` +To do this, you specify ``compute=False`` to multiple :func:`iris.save` calls, and +collect the resulting :class:``dask.delayed`` objects. You can then "complete" the +saves, in parallel, by supplying a list of 'delayed' results to the :func:`Dask.compute` function. This mechanism allows Dask to calculate the multiple "derived" results while only fetching the "common" data from which they derive **once**, whereas otherwise the source data may need to be fetched multiple times. In this respect, the benefit is the same as -the :meth:``iris.cube.CubeList.realise_cubes`` method. However there is an additional -potential benefit in this case, which is that the actual **writing** of data to file can -be performed in parallel. The scope of this, however, depends on the parallel -capabilities of the Python ``netCDF4`` package, the NetCDF C library, the file -system and the operating system : *Typically* you can write data arrays in parallel -to **separate files**, but not to different variables, or variable sections *within a -single file*. +the :meth:`iris.cube.CubeList.realise_cubes` method. However for saving there is an +additional benefit, in that the actual **writing** of data to file can also be performed +in parallel. The scope of this, however, depends on the parallel capabilities of +several components : the Python ``netCDF4`` package; the NetCDF C library; the file +system; and the operating system : **Typically**, you can write data arrays in parallel +to **separate files**, but not to different variables, or different data sections +*within a single file*. Character and String data