Chardata docs#7194
Conversation
![scitools-ci[bot]](https://avatars.githubusercontent.com/in/212324?s=60&v=4){kind=small}
There was a problem hiding this comment.
Templating
This PR includes changes that may be worth sharing via templating. For each file listed below, please either:
- Action the suggestion via a pull request editing/adding the relevant file in the SciTools/.github
templates/directory. 1 - Raise an issue against the SciTools/.github repo for the above action if you really don't have 10mins spare right now. Include an assignee, to avoid it being forgotten.
- Dismiss the suggestion if the changes are not suitable for templating.
You will need to dismiss this review before this PR can be merged. Recommend the reviewer does this as their final action before merging, as this text will continually update as commits come in.
Templated files
The following changed files are templated:
Footnotes
-
Include this text in the PR body to avoid any notifications about applying the template changes back to the source repo!
@scitools-templating: please no update notification on: iris↩
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## FEATURE_chardata #7194 +/- ##
=================================================
Coverage 90.29% 90.29%
=================================================
Files 92 92
Lines 25171 25171
Branches 4700 4700
=================================================
Hits 22728 22728
Misses 1673 1673
Partials 770 770 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
pp-mo
dismissed
scitools-ci[bot]’s stale review
Changes to pre-commit actually occurred in the target branch 'FEATURE_chardata'.
Only appeared here because I temporarily changed the target to 'main' to get docs building
| Summary | ||
| ^^^^^^^ | ||
|
|
||
| * Iris currently *only* fully supports fixed-width 'char' type data in netCDF variables |
There was a problem hiding this comment.
Are the single quotes intended to be backticks? It doesn't look like the rest of the document uses single quotes for this.
|
|
||
| * Iris currently *only* fully supports fixed-width 'char' type data in netCDF variables | ||
|
|
||
| * the 'string' type (variable-width unicode strings) will be added in a future release |
There was a problem hiding this comment.
A link to an issue to track this might be useful
| have used a variable attribute ``_Encoding`` indicating the encoding name. | ||
|
|
||
| The NetCDF documentation also mentions that an ``_Encoding`` attribute may be used to | ||
| represent non-ascii strings. However this is described as "reserved for future use", |
| .. note:: | ||
|
|
||
| In Iris, **the 'string' data type is not supported at present**, though this is | ||
| planned for future releases. See the following section `Variable-length datatypes`_ |
There was a problem hiding this comment.
Same as previous comment for link to issue
| encoding support (described below). | ||
|
|
||
|
|
||
| In Iris |
There was a problem hiding this comment.
I think the Iris part of this section should be the first under summary, since I imagine users are most interested in that. The other sections are more to provide context for decisions made in Iris
| cubes and coordinates. | ||
|
|
||
| * the numpy dtype is of the type "U<xx>", where <xx> is a character width. | ||
| * the character width relates to a string *dimension* of the netCDF variable, |
There was a problem hiding this comment.
I'm not following what this bullet point means.
| * 'char' variable data is represented as numpy string arrays in Iris objects, such as | ||
| cubes and coordinates. | ||
|
|
||
| * the numpy dtype is of the type "U<xx>", where <xx> is a character width. |
There was a problem hiding this comment.
This is referred to as "string width" later on. I think that term is more correct?
|
|
||
| In the netCDF CF Conventions | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
| The `CF Conventions <https://https://cfconventions.org/>`_ define a subset of |
| * it **is** required to *save* any non-ascii characters | ||
|
|
||
| The following describes the nature of character and string data handling in : | ||
| netCDF itself; the CF conventions; the netCDF4 Python module and the Iris implementation. |
There was a problem hiding this comment.
NetCDF is inconsistently capitalised
|
|
||
|
|
||
| Character and String datatypes | ||
| ------------------------------ |
There was a problem hiding this comment.
I think an introduction might be warranted. Something like:
In NetCDF, text can be represented in 2 ways:
char: the number of characters in the text is fixedstring: the number of characters in the text can vary
Then text can have different encodings, which affects the number of bytes used to store a character. For example:
ASCIIandUTF-32: the number of bytes are fixedUTF-8andUTF-16: the number of bytes can vary
This section discusses how Iris handles the different cases and how the standards and interpretations have changed.
2 participants
Closes #6924
In writing this, I have encountered a bit of a doubt about the implementation : #7195
But for the sake of getting this under review, I'm going to treat that as a separate issue :
I believe it can only simplify the account here.