Restructure CI/CD pipeline and fix PyPI publishing#667
Draft
Restructure CI/CD pipeline and fix PyPI publishing#667
Conversation
Fixes #638. The versioning workflow used a PAT (POLICYENGINE_GITHUB) to push the "Update package version" commit, which broke when the token expired. Switch to a GitHub App token via actions/create-github-app-token@v1, matching the pattern used in policyengine-api-v2-alpha. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Split tests into unit/ (self-contained, synthetic data, mocks) and integration/ (requires built H5 datasets). Unit sub-folders use no test_ prefix (datasets/, calibration/) to avoid confusion with integration tests. - Move 30+ unit test files to tests/unit/ with calibration/ and datasets/ sub-directories - Move 11 integration test files to tests/integration/ - Merge test_dataset_sanity.py into per-dataset integration files (test_cps.py, test_enhanced_cps.py, test_sparse_enhanced_cps.py) - Rename calibration integration tests to match dataset names (test_source_imputed_cps_masking.py, _consistency.py) - Move top-level tests/ files into appropriate unit/ or integration/ - Add integration conftest.py with skip logic and shared fixtures - Update pyproject.toml testpaths and add pytest-cov dependency - Update modal_app/data_build.py TEST_MODULES - Add make test-unit and make test-integration targets - Fix Path(__file__) references in moved test files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New queue-based H5 build system that replaces the partition-based N-worker model with one-container-per-item processing. - generate_work_items(scope, db_path): auto-generates work item lists filtered by scope (all/national/state/congressional/local/test). Test scope builds national + NY + NV-01 only. - build_single_area(): Modal function (1 CPU, 16GB) that processes exactly one work item per container via worker_script.py - queue_coordinator(): spawns up to 50 single-item workers, collects results. No multi-threading, no chunking. - main_queue entrypoint for CLI access - Wire scope parameter from pipeline.yaml through run_pipeline() - Fall back to legacy coordinate_publish for scope=all Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add testing standards: unit vs integration placement rules, per-dataset naming convention, make test-unit/test-integration commands. Add CI/CD overview documenting the four workflow files and their triggers. Update Python version from 3.11 to 3.12-3.13. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
anth-volk
force-pushed
the
fix/fix-us-data-pypi
branch
from
48abbd4 to
e12990b
Compare
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Delete 7 deprecated/redundant workflow files and replace with 3 consolidated workflows matching the policyengine-api-v2-alpha pattern. pr.yaml: fork check, lint, uv.lock, changelog, unit tests with Codecov (informational), smoke test. Runs in ~2-3 minutes. push.yaml: two paths — version bump commits publish to PyPI; all other commits run per-dataset Modal builds with integration tests after each stage, then manual approval gate, then pipeline dispatch. Also adds: - .codecov.yml with informational-only coverage reporting - --script mode to data_build.py for per-dataset Modal execution - SCRIPT_SHORT_NAMES mapping for human-friendly script names - run_single_script() Modal function for single-dataset builds Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Pass HUGGING_FACE_TOKEN to unit test step in pr.yaml so tests that transitively import huggingface.py can collect without crashing - Fix test_etl_national_targets.py: remove nonexistent TAX_EXPENDITURE_REFORM_ID import, use reform_id > 0 filter instead (mirrors fix from unmerged PR #664) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove test_reproducibility.py: imports modules that never existed (enhanced_cps.imputation, enhanced_cps.reweight, scripts). Broken since PR #117 (July 2025), never caught because old testpaths didn't include top-level tests/. - Fix test_legacy_target_overview_without_reform_id: create builder before installing legacy view so __init__'s create_or_replace_views doesn't overwrite it. Clear column cache so builder re-detects missing reform_id. Mirrors fix from unmerged PR #665. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
4 tasks
download_private_prerequisites.py downloads files (puf_2015.csv, demographics_2015.csv, soi.csv, np2023_d5_mid.csv, policy_data.db) to the local filesystem, which vanishes when the container exits. In --script mode, each script runs in a separate container, so subsequent scripts couldn't find the prerequisites. Fix: save prerequisite files to the checkpoint volume after download, and restore them before running any other script. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #666
Fixes #638
Summary
unit/(270 tests, fast) andintegration/(53 tests, require datasets)pr.yaml,push.yaml,pipeline.yaml,versioning.yamlTest plan
pr.yaml): verify unit tests, lint, changelog check, smoke test all passpipeline.yamlwithscope=testbuilds only national + NY + NV-01pipeline-approvalGitHub environment before mergeCODECOV_TOKENsecret before merge🤖 Generated with Claude Code