Chardata docs

docs/src/user_manual/explanation/netcdf_io.rst

@@ -133,6 +133,201 @@ Iris' optimisation all together, and will take its chunksizes from Dask's behavi
     (70, 37, 49)
 
 
+Character and String datatypes
+------------------------------
+Summary
+^^^^^^^
+
+*   Iris currently *only* fully supports fixed-width 'char' type data in netCDF variables
+
+    * the 'string' type (variable-width unicode strings) will be added in a future release
+
+*   'char' variable data is represented as numpy string arrays in Iris objects, such as
+    cubes and coordinates.
+
+    *   the numpy dtype is of the type "U<xx>", where <xx> is a character width.
+    *   the character width relates to a string *dimension* of the netCDF variable,
+        which is not in the dimensions of the Iris object or its data array.
+    *   the dtype 'width' controls the length of string dimensions created when saving
+
+*   Iris also uses a variable ``_Encoding`` attribute to enable storage of non-ascii
+    characters in 'char' type arrays.
+
+    *   it appears as a regular attribute of the Iris object
+    *   it is not needed for ascii-only data
+    *   it is not needed to *read* 'utf-8' encoded data correctly
+    *   it **is** required to *save* any non-ascii characters
+
+The following describes the nature of character and string data handling in :
+netCDF itself; the CF conventions; the netCDF4 Python module and the Iris implementation.
+In practice all these are connected.
+
+The details are generally much simpler when strings may contain only ASCII characters.
+When strings may include non-ascii characters, this requires a specific encoding to be
+adopted when translating to and from bytes, and rules for determining what the encoding
+is or was.
+
+Another significant factor is the way in which all the relevant projects have changed
+their features and methods over time, so that historic datasets may often use
+non-standard approaches to record string data.
+
+
+In the NetCDF file format
+^^^^^^^^^^^^^^^^^^^^^^^^^
+In the NetCDF v4 implementation, there are three specific areas where the datatype and
+storage characteristics of character data are relevant:
+
+*   The **names** of file components (variables, dimensions, and attributes) are
+    natively unicode-capable strings of arbitrary (variable) length.
+
+*   **Attributes** with string content likewise *appear* to be natively unicode.  However,
+    the actual datatype of the attribute may vary, being either 'char' or 'string'.
+
+*   The **content of variables** can be either 'char' or 'string'.
+
+    *   'string' type variables contain a variable-length unicode string at each array element.
+
+    *   'char' type variables contain one-byte characters, and generally have a fixed-length
+        "string dimension".  If they contain *only* ascii character values, this is
+        uncomplicated, but they may also be used to contain non-ascii data (i.e.
+        including unicode characters).  There is no universally defined agreement for
+        how to indicate that bytes are encoded non-ascii data, but many older datasets
+        have used a variable attribute ``_Encoding`` indicating the encoding name.
+
+The NetCDF documentation also mentions that an ``_Encoding`` attribute may be used to
+represent non-ascii strings.  However this is described as "reserved for future use",
+and its valid values and effects are not explicitly defined.
+
+However, it is also notable that the standard ``ncgen`` and ``ncdump`` tools *do*
+correctly interpret an ``_Encoding`` attribute in most cases, despite this not being an
+"official" solution.
+
+
+In the netCDF CF Conventions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The `CF Conventions <https://https://cfconventions.org/>`_ define a subset of
+"allowed" datatypes, and various types of data elements represented by variables
+-- such as data variables, auxiliary coordinates, cell methods, etc.
+
+CF currently supports the use of either netcdf 'string' or 'char' arrays for any
+variables.
+However, *historically*, CF has had more limited support, and also "unofficial
+conventions" have been used for string data encoded as bytes, which may be encountered
+in older datasets.
+
+**Prior to v1.8**, CF required to use 'char' type only, and provided
+**no official means** of representing non-ascii data.
+
+**Since v1.8**, CF has allowed the use of 'string' data in all variables.
+However, up to v1.12 there was still no official way of encoding non-ascii data in
+'char' arrays.
+
+**Since v1.12**, CF now mandates a *default* assumption of utf-8 encoding to store
+non-ascii data in 'char' form.  It does also note that some data in the past has used an
+``_Encoding`` attribute -- though this was never an official CF usage.
+
+Characteristics
+~~~~~~~~~~~~~~~
+Where strings are stored as 'char' type, which is the more common traditional approach,
+the array must have a "string dimension", which is a normal file dimension.  Thus, these
+strings always have a *fixed byte width*.  (However, that is not the same as a fixed
+*character* width, since in most encodings non-ascii characters require more bytes to
+store).
+
+Although the variable-length 'string' data is now supported in CF, the use of
+fixed-width 'char' arrays is obviously more efficient for storage and access, and it is
+still the most common approach in practice.
+
+
+In the netCDF4 Python module
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+*   attributes with string content always appear as Python 'str' (i.e. unicode strings).
+    It is not possible to distinguish or control the 'char' and 'string' type in the file,
+    this is hidden from the user by the Python implementation.
+
+*   variable data of type 'string' is presented (read and written) as numpy arrays of dtype
+    "U<xx>", where <xx> is a (maximum) string length.
+
+*   variable data of type 'char' is presented (read and written) as numpy arrays of
+    dtype "S1" -- that is, an array of length-1 Python "bytes" objects.
+
+    .. note::
+
+        The netCDF4 package can also automatically translate this to string arrays of
+        dtype "U<xx>", if the variable has an ``_Encoding`` attribute.  **However,**
+        Iris turns this feature *off*, in order to implement its own wider-ranging
+        encoding support (described below).
+
+
+In Iris
+^^^^^^^
+.. note::
+
+    In Iris, **the 'string' data type is not supported at present**, though this is
+    planned for future releases.  See the following section `Variable-length datatypes`_
+    for an interim solution enabling you at least to *load* variable-length string data.
+
+Iris stores string data in arrays of dtype "U<xx>", where <xx> is a maximum character
+width.  However, this data is currently **only** read and written in netCDF files as
+'char' type variables (i.e. byte arrays).
+
+Iris supports a specific set of valid encodings for non-ascii data :
+"ascii", "utf8", "utf16" and "utf32".  These (or aliases) will appear in the
+``_Encoding`` attribute of a file variable, and likewise as an attribute of the
+corresponding Iris component object (e.g. cube or coordinate).
+
+When loading
+~~~~~~~~~~~~
+If there is a valid ``_Encoding`` attribute this is used to decode the
+data, otherwise a default encoding of "utf8" is applied:  This works transparently when
+only ascii characters are present, and also allows the ``_Encoding`` attribute to be
+omitted as long as utf8 was used.  An invalid or unsupported encoding name will be
+ignored, with a warning, but the attribute will still be added to the Iris component
+object.
+
+When saving
+~~~~~~~~~~~
+Any string data with only ascii characters does not require an ``_Encoding`` attribute.
+However if there are any non-ascii characters, and no ``_Encoding``
+attribute, then an error will be raised.  An invalid or unsupported encoding name will
+be ignored, with a warning, but the attribute will still be stored to the file.
+
+So effectively, the **default encoding is 'utf8' for load and 'ascii' for save**.
+
+String width dimensions
+~~~~~~~~~~~~~~~~~~~~~~~
+For each valid encoding there is a definite relation between the string dimension length
+in the file (actually, the number of *bytes*), and the maximum character length, aka
+string width, in the array dtype : i.e. the "<xx>" in the "U<xx>" dtype.
+
+The **lengths of string dimensions created on write** are calculated as follows:
+
+*   ascii : n-bytes = n-characters
+*   utf8 : n-bytes = n-characters
+*   utf16 : n-bytes = 2 * (n-characters + 1)
+*   utf32 : n-bytes = 4 * (n-characters + 1)
+
+For reading, the inverse relations are applied to determine the '"U<xx>"' dtype in which
+the data is presented.  This will always round-trip correctly, i.e. the dimension length
+is unchanged if data is read and then written back.
+
+For 'ascii' and 'utf32' this character-to-byte relationship is simple + fixed, but for
+'utf8' and 'utf16', the number of encoded bytes depends on the actual characters present
+**and can exceed the numbers given above**.  If any string in the *actual* data encodes
+to more bytes than the above-calculated string dimension, then Iris will raise an
+:class:`iris.exceptions.TranslationError`.  In this case, the user must **explicitly
+specify** a longer string dimension, by converting the data to a longer "U<xx>" dtype :
+for example, ``cube.data = cube.core_data().astype("U20")``.
+
+.. warning::
+
+    When processing string arrays, Numpy does not routinely preserve the "<xx>" width part
+    of "U<xx>" type data : instead, some operations will reduce it to the maximum width
+    occurring.  So in these cases also, it may be necessary to explicitly re-assert the
+    desired "string width" before saving -- use ``.astype()``,  as above.
+
+
 Variable-length datatypes
 -------------------------
 

lib/iris/fileformats/netcdf/__init__.py

@@ -48,7 +48,6 @@
     "CFNameCoordMap",
     "CF_CONVENTIONS_VERSION",
     "DEBUG",
-    "DECODE_TO_STRINGS_ON_READ",
     "DEFAULT_READ_ENCODING",
     "DEFAULT_WRITE_ENCODING",
     "MESH_ELEMENTS",