Fix calibration population overshoot (~6% drift)

[vahid-ahmadi]

Summary

• Adds post-calibration population rescaling: after the optimiser finishes, all weights are uniformly scaled so the weighted UK population matches the ONS target exactly
• The optimiser treats population as 1 of ~556 targets, causing it to drift ~6% high (69M → 74M)
• Extracts rescale_weights_to_population() as a standalone function for testability
• Adds 4 microsimulation-based tests using native microdf weighted operations (MicroSeries.sum()) — population target match (3% tolerance), household count range, inflation guard, and country-sum consistency
• Tightens existing test_population tolerance from 7% to 3%
• Fixes pre-existing ruff lint errors (unused Microsimulation import, ambiguous variable name l)

Closes #217

Test plan

• CI passes (ruff, pytest)
• test_weighted_population_matches_ons_target — weighted population within 3% of 69.5M
• test_household_count_reasonable — total households in 25–33M range
• test_population_not_inflated — population stays below 72M
• test_country_populations_sum_to_uk — country populations sum to UK total

🤖 Generated with Claude Code

[nikhilwoodruff]

Vote against jumping against the calibration, which should be the final step. this would invalidate the calibration dashboard

[vahid-ahmadi]

@nikhilwoodruff Good point — I've replaced the post-hoc rescaling with a fix inside the calibration loss function itself.

What changed: Instead of uniformly scaling all weights after the optimiser finishes (which would invalidate the calibration dashboard), the population target (ons/uk_population) now gets 10x weight in the national loss during training. This means the optimiser treats population accuracy as a near-hard constraint rather than 1-of-556 equally-weighted soft targets.

Specifically:

• Removed rescale_weights_to_population() and the post-calibration call
• Added _build_national_target_weights() — builds a per-target weight vector (all 1.0 except population at 10.0)
• Changed torch.mean(sre(...)) → weighted_mean(sre(...), weights) in the national loss

The calibration output is now the final output — no post-hoc modification. The dashboard stays valid.

We don't have anything similar to per-target weighting in the codebase currently — is this an approach you'd be happy with, or would you prefer a different method?

[nikhilwoodruff]

Don't think we should change our standpoint against weighted targets- should find the root cause of why we can't fit population, given we have hundreds of targets on it

[MaxGhenis]

Rebased onto current main and resolved the loss_val → loss_value rename conflict. Taking the main-branch naming throughout so this merges cleanly. Ready for re-review / CI once you're available @vahid-ahmadi.

[MaxGhenis]

Two things after my recent merges:

1. Conflict — your branch is now DIRTY against current main. The TFC calibration targets PR (Refresh Tax-Free Childcare calibration targets #363) just merged, which conflicts with existing changes here. Needs a rebase.
2. Test failures remaining — the population rescale tests are still failing:
   • test_population: actual 71.6M vs 69.5M target (3.07% off, tolerance 3.0%)
   • test_household_count_reasonable: 33.5M vs max 33.0M
   • test_weighted_population_matches_ons_target: same 3.07% delta
   • test_salary_sacrifice_below_cap_users: 5.1M vs target 4.3M (19% over)

The asymmetric-loss fix narrowed the gap but didn't fully close it. Either the tolerance needs to bump to ~3.5% or the POPULATION_LOSS_WEIGHT needs to go higher. Whichever you prefer.

[MaxGhenis]

Update: the latest push-CI on current main shows UK population settling at 70.97 M vs 69.87 M target = +1.58 % — the data-pipeline merges from yesterday (#362 stage-2 QRF, #363 TFC target refresh, #359 reported-anchor takeup) pulled the overshoot you were fighting from ~6.5 % down to ~1.6 %.

Opened #366 to cherry-pick just your test-tolerance tightening + the four new regression tests from this branch, so the current-state gain is locked in with CI gates. That side-steps Nikhil's concerns about weighted targets / post-hoc rescaling without losing your test coverage.

The asymmetric-loss change (log((1+x)/(1+y))^2) and the Scotland NRS target removal are independent proposals — happy to keep this PR open for those as separate discussion, but they'd benefit from a before/after sweep across all targets (not just population) to confirm the loss swap is net-positive. Want to repurpose this branch for that, close it, or leave it as-is?

Thanks for the tests — they're going into #366 with co-authorship attribution.