Skip to content

Added goldens for USCensuspep imports - #2091

Open
niveditasing wants to merge 15 commits into
datacommonsorg:masterfrom
niveditasing:golden_us_imports
Open

Added goldens for USCensuspep imports#2091
niveditasing wants to merge 15 commits into
datacommonsorg:masterfrom
niveditasing:golden_us_imports

Conversation

@niveditasing

@niveditasing niveditasing commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Overview
This PR updates the Python preprocessing and ingestion scripts across PEP (Population Estimates Program) datasets to ensure compatibility
with Pandas 2.0+ and optimize processing logic.


  1. Pandas Compatibility Updates
  • Replaced Deprecated delim_whitespace: Replaced delim_whitespace=True with sep=r'\s+' inside pd.read_table() calls to support modern
    Pandas versions in:
    • monthly_population_estimate/preprocess.py
    • population_estimates_by_asr/ (county, national, and state scripts)
    • us_pep_sex/process.py
    • us_pep_sexrace/ (county, national, and state scripts)
  • Replaced Direct Indexing: Replaced index-based Series access (e.g., col[0]) with explicit .iloc indexer (col.iloc[0]) in
    monthly_population_estimate/preprocess.py
  1. Added golden to all these 4 Directories

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces validation configurations, golden summary reports, and generated outputs for the US Census PEP monthly population estimate and sex datasets. Additionally, it updates download routines in process.py to include timeouts, retries with exponential backoff, and sorted file processing. Feedback on the changes highlights an incorrect golden file path in the us_pep_sex validation configuration and a potential resource leak in process.py where temporary files are not cleaned up if an exception occurs during file copy or move operations.

Comment thread scripts/us_census/pep/us_pep_sex/validation_config.json
Comment thread scripts/us_census/pep/us_pep_sex/process.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant