[andreatnvidia]

Summary

Rebuilds the bundled US ZIP-to-area-code lookup from Census/Wikidata with auditable provenance and per-ZIP code candidates. The phone sampler chooses a ZIP by its existing population weight, then samples a candidate uniformly, preserving alternatives that a single-code tie-break discarded. Also corrects phone-number generation bounds and skips N11 exchanges.

Related Issue

N/A

Changes

• Record source URLs, dates, SHA-256 hashes, SPARQL queries, curated exclusions, and immutable source references; embed the complete source manifest and sampling policy in parquet metadata.
• Add sorted area_codes lists while retaining all 33,772 ZIPs, their population weights, and the existing representative area_code column. The loader also supports legacy single-code assets.
• Choose the closest qualifying locality point, preferring nearby literal postcode associations, then retain every accepted same-state code at that point, including localities without a postcode statement. Choose codes without giving ZIPs with more candidates extra sampling weight.
• Replace optional PyShp/Shapely tooling with a reader for the pinned Census Polygon/DBF layout and NumPy point-in-polygon assignment; remove the extra requirements file.
• Generate per-candidate lineage and per-code reconciliation, reject codes absent from the pinned input, and check artifact hashes, metadata, coverage, and conserved weights.
• Enforce LF checkout bytes for hashed provenance text files through .gitattributes.
• Include rebuild unit tests in the engine Make targets and CI. Cover candidate ties, weighting, legacy assets, multipart geometry, holes, shared boundaries, and malformed inputs.
• Keep generated exchange codes within 200-999, skip N11 exchanges, and keep line numbers within 0000-9999, with boundary regression tests.

Sources and permissions

Source	Use	Published terms
U.S. Census Bureau, 2023 ACS five-year B01003	ZIP/ZCTA population weights	U.S. government work, public domain in the United States; reproduction permitted.
U.S. Census Bureau, 2023 ZCTA Gazetteer and TIGER/Line state boundaries	Representative coordinates and state assignment	Census government data; TIGER/Line metadata explicitly permits use in products/publications with Census acknowledgment, which is included.
Wikidata structured-data snapshots, retrieved 2026-11-06	Locality identifiers, coordinates, postal codes, and dialing codes	CC0 1.0 public-domain dedication, with fallback permission to copy, modify, and redistribute commercially without attribution or share-alike requirements.

See the Census technical documentation, sections 1.1-1.2, Wikidata licensing policy, and CC0 terms. Source-data permissions are documented separately from the Apache-2.0 rebuild code.

The NANPA NPA report was consulted to screen foreign, inactive, and wrong-state locality associations. No explicit open-data license was established for that report. It is not redistributed or required to rebuild. The pinned exclusions contain Wikidata fields and Census-derived states; the reference URL/date/hash document the screening step.

No third-party GitHub ZIP/area-code compilation is used. The rebuild uses the Python standard library and existing NumPy, pandas, SciPy, and PyArrow dependencies. It adds no build or runtime dependencies, native geometry libraries, or downloaded extensions. Package dependency declarations and the lockfile are unchanged from the base branch.

The provenance README contains the complete method, permissions, and rebuild commands.

Attention areas

• Candidate lists are locality-level approximations for synthetic data, not exact telephone boundaries or measured market shares. A city can contain both overlays and geographic splits; uniform selection does not resolve their intra-city geography.
• The data has 37,239 ZIP/code associations; 2,322 ZIPs have multiple candidates. Retaining alternatives makes 333 codes sampleable instead of 277, recovering 56 previously discarded codes including 332, 726, and Chicago's 773 and 872.
• Of 386 syntactically valid source values, 358 have retained localities and 355 are eligible for the covered geography. There are still 22 eligible codes with no selected locality point. The report lists them explicitly; it does not label them invalid.
• All ZIP identifiers and Census population weights match the original repository asset. The representative column matches the initial PR rebuild, whose values differ from the original repository asset for 16,101 ZIPs (47.7%). Historical agreement is not an accuracy measure.
• There are 18,749 literal-postcode selections and 15,023 nearest-locality fallbacks, including 348 fallback ZIPs more than 50 km from their source point. The snapshots and screening reference are dated, not live assignment checks.

Testing

• Sampling and rebuild tests: 307 passed, with PyShp and Shapely absent from the environment.
• Dependency-free Census reader matches all 56,588 source-point assignments from the previous geometry implementation.
• All 37,239 candidate associations match the pinned country, in-service, and state screening reference. This is not an exact ZIP-boundary validation.
• Repeated offline builds reproduce the parquet, report, and reconciliation byte for byte.
• Isolated Git checkout with CRLF conversion preserves the hashed text files byte for byte.
• ZIP identifiers, representative codes, and population weights compared with the previous PR head.
• Engine wheel and source distribution built; wheel inspected for candidate data, sampler support, provenance, notice, and dependency metadata.
• Changed Python files pass Ruff lint/format checks.
• Full repository suite and E2E tests were not run.

Checklist

• Follows commit message conventions.
• Commits are signed off (DCO).
• External contributor issue requirement: N/A.
• Architecture docs: N/A; package boundaries and public sampler interfaces are unchanged.

[3mei]

@andreatnvidia thanks for putting this together!

Four suggestions, all about making the lineage enforced rather than documented:

1. Assert the output hash. test_cached_source_checksum covers the input-side mechanism, but nothing compares the packaged parquet to output_sha256. A test would keep the asset and its recorded lineage from drifting apart silently.

2. Pin the NOTICE link. It points at tree/main/scripts/person_data, which moves. A tag or commit SHA keeps the lineage resolvable for anyone holding an older wheel.

3. Consider embedding the source hashes in the parquet key-value metadata, so lineage travels with the file if it's ever separated from its NOTICE.

4. Reconcile output against input at build time. The asset uses 277 of the 385 well-formed NANP codes in the pinned snapshot. 15 of the absent ones are toll-free, non-geographic or non-US and are correctly excluded. However, 93 are US geographic codes, and they're predominantly metro overlays: 646/332/917/929 (New York), 424 (Los Angeles), 872 (Chicago), 281/832/346 (Houston), 267/445 (Philadelphia), 857 (Boston), 786 (Miami), 628 (San Francisco), 443/667 (Baltimore).

The effect is that every synthetic resident of those metros receives the single primary code for the area. Across 17 major complexes alone that covers roughly 43 million people, so the realism gap falls hardest on the most populous areas and skews one way rather than averaging out. Anything that treats area code as a feature (geographic inference, record linkage, fraud baselines) sees a distribution that doesn't match reality. Worth noting 773 is also absent, and that one isn't an overlay: it covers Chicago outside the Loop, so it's a genuine geographic hole rather than a collapsed overlay choice.

build_report.json reports locality-level counts but nothing at code level, so this reduction is currently the one step in the chain a reader can't reconstruct from the artifacts. A code-level line in the report, plus an assertion that output codes are a subset of the pinned input, would close it.

Unrelated to lineage, flagging while it's in view: phone_number.py:116 uses random.randint(200, 1000) for the central office code. The upper bound is inclusive, so it returns 1000 roughly 1 in 778 times and emits an 11-digit string like 202-1000-0427, which isn't a phone number. It also produces N11 codes about 1% of the time. N11 is reserved for service dialling, never assigned as a central office code.

This matters because the field's main use is exercising PII detection, redaction and record-linkage pipelines. A detector matching \d{3}-\d{3}-\d{4} silently skips the 11-digit rows, and a validator that correctly rejects non-dialable numbers appears to fail against our own data. So roughly 1.1% of rows quietly undermine the evaluation the field exists to support. validate_digits won't catch either case, since it only checks .isdigit(). Emitting numbers with 911 in the exchange position seems worth avoiding on its own. Fix is randint(200, 999) plus skipping X11.

[andreatnvidia]

Thanks @3mei. Added the output-hash check, pinned source link, embedded provenance, and per-code reconciliation with the input-subset assertion. The phone-number bounds and N11 fixes are also included here.

In dbb8c857, the sampler now retains alternatives at the chosen locality point, samples ZIPs with their existing population weights, then chooses uniformly among the candidates. This restores 54 previously dropped codes, including 773, bringing the asset to 331 represented codes. The remaining 24 eligible but unselected codes are listed explicitly. Geographic boundaries and usage proportions remain approximate.

I also removed PyShp/Shapely and the bundled GEOS requirement. The rebuild now uses only the standard library and existing project dependencies.

[greptile-apps]

Confidence Score: 4/5

[Medium impact] The changes reach U.S. phone-number sampling and its rebuild workflow.

The three previously reported repository requirements still need to be satisfied before merging.

Findings

1.  Heavy libraries load immediately ▶
2.  Helper names omit actions ▶
3.  Weight test skips the loader ▶

Fix with agent prompt

### Issue 1
scripts/person_data/rebuild.py:19-23
The new modules import NumPy, pandas, PyArrow, and SciPy directly. The repository guide requires heavy libraries to load through `lazy_heavy_imports.py`. Apply that pattern here and in `test_rebuild.py` and `test_phone_number_data.py` before merging.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 2
scripts/person_data/rebuild.py:undefined-55
The new helpers `sphere` and `states_for` do not start with action verbs, as the repository guide requires. Rename them, for example to `convert_to_sphere` and `assign_states`, and update their callers before merging.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 3
packages/data-designer-engine/tests/engine/sampling_gen/entities/test_phone_number.py:undefined-119
The new candidate-weight test mocks the internal `_load_zip_area_code_data` function. The testing guide requires mocks at external dependencies, not internal functions. Mock the parquet read instead so the test also checks how the loader passes candidates and weights to `get_area_code`. This repository requirement must be satisfied before merging.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Rebuilds the ZIP-to-area-code asset with source records and multiple candidates per ZIP.

• ZIP sampling now keeps and draws among local area-code alternatives.
• Generated phone numbers stay within standard exchange and line-number ranges.
• The ZIP lookup can be rebuilt from pinned sources and audited.

No new findings were accepted. andreatnvidia explicitly documents locality-level approximations and 22 eligible but unselected codes as known limits.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Pinned Census and Wikidata inputs] --> B[Assign states and remove exclusions]
  B --> C[Choose a same-state locality point]
  C --> D[Retain codes at that point]
  D --> E[Write one row per ZIP with original population]
  E --> F[Sample ZIP by population]
  F --> G[Choose one candidate uniformly]
  G --> H[Generate exchange and line number]

Loading

_{Reviews (2) · Last reviewed commit: "chore: merge slurm execution base and re..." · Reviewed by Greptile}

[greptile-apps]

Heavy libraries load immediately

The new modules import NumPy, pandas, PyArrow, and SciPy directly. The repository guide requires heavy libraries to load through lazy_heavy_imports.py. Apply that pattern here and in test_rebuild.py and test_phone_number_data.py before merging.

Context Used: CLAUDE.md (source)

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/person_data/rebuild.py
Line: 19-23

Comment:
**Heavy libraries load immediately**

The new modules import NumPy, pandas, PyArrow, and SciPy directly. The repository guide requires heavy libraries to load through `lazy_heavy_imports.py`. Apply that pattern here and in `test_rebuild.py` and `test_phone_number_data.py` before merging.

**Context Used:** CLAUDE.md ([source](https://github.com/nvidia-nemo/datadesigner/blob/main/CLAUDE.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Helper names omit actions

The new helpers sphere and states_for do not start with action verbs, as the repository guide requires. Rename them, for example to convert_to_sphere and assign_states, and update their callers before merging.

Context Used: CLAUDE.md (source)

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/person_data/rebuild.py
Line: 55

Comment:
**Helper names omit actions**

The new helpers `sphere` and `states_for` do not start with action verbs, as the repository guide requires. Rename them, for example to `convert_to_sphere` and `assign_states`, and update their callers before merging.

**Context Used:** CLAUDE.md ([source](https://github.com/nvidia-nemo/datadesigner/blob/main/CLAUDE.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Weight test skips the loader

The new candidate-weight test mocks the internal _load_zip_area_code_data function. The testing guide requires mocks at external dependencies, not internal functions. Mock the parquet read instead so the test also checks how the loader passes candidates and weights to get_area_code. This repository requirement must be satisfied before merging.

Context Used: CLAUDE.md (source)

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/data-designer-engine/tests/engine/sampling_gen/entities/test_phone_number.py
Line: 119

Comment:
**Weight test skips the loader**

The new candidate-weight test mocks the internal `_load_zip_area_code_data` function. The testing guide requires mocks at external dependencies, not internal functions. Mock the parquet read instead so the test also checks how the loader passes candidates and weights to `get_area_code`. This repository requirement must be satisfied before merging.

**Context Used:** CLAUDE.md ([source](https://github.com/nvidia-nemo/datadesigner/blob/main/CLAUDE.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[danecor]

Good catch