Initial spec for lists and structs by hadley · Pull Request #113 · tidyverse/data-dict

hadley · 2026-07-09T22:28:18Z

Will eventually fix #56 and fix #57.

hadley · 2026-07-09T22:39:47Z

 * `date`: calendar dates, written as ISO 8601 strings (`YYYY-MM-DD`, e.g. `2024-01-31`).
 * `datetime`: date-times, written as ISO 8601 strings. Without a `time_zone` they carry an offset (e.g. `2024-01-31T09:30:00Z`); with a `time_zone` they're written zoneless and interpreted in that zone (see [Time zones](#time-zones)).
 * `enum`: a column with repeated values from a known set. The allowed values are listed in the `values` property.
+* `list(element_type)`: an ordered sequence of zero or more elements of the given type (see [List element types](#list-element-types)).


Are list and struct the right names?

list definitely. struct is as good as any imo

hadley · 2026-07-09T22:42:33Z


-* `primary_key`: the set of columns with the `primary_key` constraint uniquely identifies each row. Implies `required` and `unique`.
-* `foreign_key`: the column references a primary key in another table (or in the current table, if a self-join). The specific relationship is defined in [`relationships`](#relationships).
+* `primary_key`: the set of columns with the `primary_key` constraint uniquely identifies each row. Implies `required` and `unique`. Not valid on `list` or `struct` columns, or on fields within a `struct`.


We could consider consider other carve outs if you think support is going to be complex

This looks OK for me, although I am not sure what more you would add.

lwjohnst86 · 2026-07-10T07:41:38Z

+  type: list(enum)
+  values: [food, drink, dessert]


This looks the same as an enum. What's different here?

It's a multiselect — an enum must have exactly one value for each row. A list(enum) can have any number of values for each row.

lwjohnst86 · 2026-07-10T07:42:12Z

+  type: list(string)
+  examples: [nature, outdoor, urban, photography, wildlife]


Same comment below about the enum. How would this be different?

lwjohnst86 · 2026-07-10T07:45:06Z

+- name: address
+  type: struct
+  fields:
+    - name: street
+      type: string
+      examples: [123 Main St, 456 Oak Ave, 789 Elm Dr]
+    - name: city
+      type: string
+      examples: [Portland, Austin, Chicago]
+    - name: zip
+      type: string
+      examples: ["97201", "78701", "60601"]
+    - name: country
+      type: enum
+      values: [US, CA, MX]


I have a hard time seeing the difference here with the list(struct). I understand that the data itself might be different, but when the metadata looks the exact same, I'd have a hard time differentiating them both.

lwjohnst86 · 2026-07-10T07:47:31Z

 * `datetime`: date-times, written as ISO 8601 strings. Without a `time_zone` they carry an offset (e.g. `2024-01-31T09:30:00Z`); with a `time_zone` they're written zoneless and interpreted in that zone (see [Time zones](#time-zones)).
 * `enum`: a column with repeated values from a known set. The allowed values are listed in the `values` property.
+* `list(element_type)`: an ordered sequence of zero or more elements of the given type (see [List element types](#list-element-types)).
+* `struct`: a structured record with named fields documented in the required `fields` property (see [Struct fields](#struct-fields)).


Are you thinking of using these for very nested data? I personally haven't encountered these "in the wild". What would be the advantage of a list(struct) rather than as either another table or as a flattened set of columns with specific items appended to their column name.

It's not a data structure that I'd generally recommend, since I would prefer a relation framing with multiple tables, but it is something that you get in parquet files so its important that we can represent it.

gaborcsardi · 2026-07-10T12:14:10Z

+
+#### Struct fields
+
+A `struct` column may include a `fields` property — an ordered list of field descriptors. Each field descriptor uses the same schema as a column descriptor. A field may itself be `list(...)` or `struct` (with its own `fields`), allowing deep nesting.


So if I understand correctly list(..., struct(list(...)), ...) is allowed, but list(list(...)) is not? Is that intentional? Since the list -> struct -> list nesting is already allowed I don't see why the list(list(...)) would not be.

Ah yes, good point.

gaborcsardi · 2026-07-10T12:20:06Z

 * `date`: calendar dates, written as ISO 8601 strings (`YYYY-MM-DD`, e.g. `2024-01-31`).
 * `datetime`: date-times, written as ISO 8601 strings. Without a `time_zone` they carry an offset (e.g. `2024-01-31T09:30:00Z`); with a `time_zone` they're written zoneless and interpreted in that zone (see [Time zones](#time-zones)).
 * `enum`: a column with repeated values from a known set. The allowed values are listed in the `values` property.
+* `list(element_type)`: an ordered sequence of zero or more elements of the given type (see [List element types](#list-element-types)).


list definitely. struct is as good as any imo

gaborcsardi · 2026-07-10T12:21:42Z


-* `primary_key`: the set of columns with the `primary_key` constraint uniquely identifies each row. Implies `required` and `unique`.
-* `foreign_key`: the column references a primary key in another table (or in the current table, if a self-join). The specific relationship is defined in [`relationships`](#relationships).
+* `primary_key`: the set of columns with the `primary_key` constraint uniquely identifies each row. Implies `required` and `unique`. Not valid on `list` or `struct` columns, or on fields within a `struct`.


This looks OK for me, although I am not sure what more you would add.

hadley · 2026-07-10T22:30:14Z

To do:

Carefully review snapshots
Check that list/struct/list recursion is correctly implemented
Support list-list nesting

Initial spec for lists and structs

76910f8

hadley requested a review from gaborcsardi July 9, 2026 22:28

hadley added 4 commits July 9, 2026 17:33

Fix wording

7733b40

Allow lists of structs; refine text

1066f66

Don't support bare lists/structures

69410b7

Clarify constraints

614dccd

hadley commented Jul 9, 2026

View reviewed changes

lwjohnst86 reviewed Jul 10, 2026

View reviewed changes

gaborcsardi approved these changes Jul 10, 2026

View reviewed changes

hadley added 2 commits July 10, 2026 14:58

Merge remote-tracking branch 'tidyverse/main' into list-fields

ee80c62

Start on validation

049b441

		type: list(string)
		examples: [nature, outdoor, urban, photography, wildlife]


		#### Struct fields

		A `struct` column may include a `fields` property — an ordered list of field descriptors. Each field descriptor uses the same schema as a column descriptor. A field may itself be `list(...)` or `struct` (with its own `fields`), allowing deep nesting.

Uh oh!

Conversation

hadley commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hadley commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hadley commented Jul 9, 2026 •

edited

Loading