Problem Description
Starting from SDV 2.0, we will have an enhanced metadata that includes the ordinal sdtype as well as range information.
The metadata auto-detection should be updated to adhere to this format.
Expected behavior
Update the detect_from_dataframes and detect_from_dataframe functions to be compatible with SDV. This means the following changes:
- The
METADATA_SPEC_VERSION should be "V2"
- There should no longer be a
computer_representation field for numerical sdtypes
- The
ordinal sdtype should be detected. This is the sdtype we should give to low-cardinality columns that are represented as integers or whole numbers (could be floats like 1.0, 2.0, etc.). Currently these are marked as categorical.
- The
min_range, max_range, range_values, range_is_nullable, and decimal_places should be detected based on the training data.
- When adding the list of values, we should only add them if there are <500 unique values; otherwise the JSON might get too long.
Additional context
No backwards compatibility is needed since this is for SDV 2.0.
Merge into a feature branch for SDV 2.0, as there will be other issues that are also needed for this version.
Problem Description
Starting from SDV 2.0, we will have an enhanced metadata that includes the ordinal sdtype as well as range information.
The metadata auto-detection should be updated to adhere to this format.
Expected behavior
Update the
detect_from_dataframesanddetect_from_dataframefunctions to be compatible with SDV. This means the following changes:METADATA_SPEC_VERSIONshould be"V2"computer_representationfield for numerical sdtypesordinalsdtype should be detected. This is the sdtype we should give to low-cardinality columns that are represented as integers or whole numbers (could be floats like 1.0, 2.0, etc.). Currently these are marked ascategorical.min_range,max_range,range_values,range_is_nullable, anddecimal_placesshould be detected based on the training data.Additional context
No backwards compatibility is needed since this is for SDV 2.0.
Merge into a feature branch for SDV 2.0, as there will be other issues that are also needed for this version.