Skip to content

SDV 2.0: Update metadata auto-detection to the new V2 format #2957

Description

@npatki

Problem Description

Starting from SDV 2.0, we will have an enhanced metadata that includes the ordinal sdtype as well as range information.

The metadata auto-detection should be updated to adhere to this format.

Expected behavior

Update the detect_from_dataframes and detect_from_dataframe functions to be compatible with SDV. This means the following changes:

  1. The METADATA_SPEC_VERSION should be "V2"
  2. There should no longer be a computer_representation field for numerical sdtypes
  3. The ordinal sdtype should be detected. This is the sdtype we should give to low-cardinality columns that are represented as integers or whole numbers (could be floats like 1.0, 2.0, etc.). Currently these are marked as categorical.
  4. The min_range, max_range, range_values, range_is_nullable, and decimal_places should be detected based on the training data.
    • When adding the list of values, we should only add them if there are <500 unique values; otherwise the JSON might get too long.

Additional context

No backwards compatibility is needed since this is for SDV 2.0.

Merge into a feature branch for SDV 2.0, as there will be other issues that are also needed for this version.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions