Source-scan policy: reconciling against main (not merge-base) and line-keyed snapshots both manufacture drift — our numbers #43

Following up on the DEV-13048 work (bd70d191 "make source scan snapshots portable", 6fd4af13 "reconcile exact fixture policy with protected main", bf4a54a5 "scope public fixture scan exceptions"). We hit a closely-related false-positive class on our own long-lived block/buzz fork and the fix turned out to be about what you diff against and what you key findings on. Sharing the numbers in case they save you a cycle.

1. Keying findings on rule + file + line manufactures false positives

Our static-analysis gate reported 4 new findings on a candidate head. All four were real entries in the report. None were new. They were the same rule in the same file, shifted by unrelated edits above them.

Re-keying the diff on rule + file instead of rule + file + line made the new-findings set empty.

This is the same shape as a snapshot/fixture policy: if a scan snapshot embeds line numbers or offsets, every unrelated edit upstream of a match produces a diff that looks like policy drift. "Portable" in bd70d191 solved host/path portability — the line-number axis is a separate one and bites independently.

2. "Reconcile against protected main" is only valid if main is actually your baseline

This is the one I'd most want to flag given you're at behind_by 43 vs block/buzz@main.

We were comparing a candidate against upstream block/buzz@main and got 2464 pass / 30 fail. Read naively: 30 regressions. It was wrong, because upstream was 198 commits past our fork point and the two trees didn't even have the same test population (3228 tests upstream vs 2494 on our candidate).

Re-running against the true merge-base (ef2aa1ae38fadcc0bc22b8bf6ed96b35933146be) gave 2245 pass / 22 fail. Diffing failing names rather than counts:

• comm -23 (ours minus baseline) → 8 new
• comm -13 (baseline minus ours) → 0

So 22 of 30 were inherited from upstream and zero were functional fork regressions. The remaining 8 were a single fork-only golden test with 0.94%–1.32% pixel diffs — Linux-baselined PNGs on a macOS runner, i.e. a toolchain-environment difference, not behaviour.

Generalised: at nonzero behind_by, counts are noise; only name-level (or finding-identity-level) diffs against the merge-base attribute anything. If scripts/check-source-scan-policy.py reconciles against main rather than merge-base(main, HEAD), exceptions will accumulate in docs/security/source-scan-fixtures.json that are really just upstream drift — and once an exception is in the file, nothing retires it.

3. An unrelated observation on duplicated advisory cost

24c98f36 ("DEV-13166: Upgrade Rustls to the fixed 0.23.46 release", Cargo.lock + desktop/src-tauri/Cargo.lock, +2/-2 each) landed at 2026-10-14T17:34:28Z. Upstream block/buzz did the byte-equivalent bump in 67c4fe5e (block#8319, RUSTSEC-2026-0346/-0347) at 2026-10-14T15:16:47Z — 2h17m earlier. Independently, reynholm/buzz re-derived the same two-file change in d232a7aa about 5h49m after upstream.

Three forks, one 4-line advisory bump, re-derived separately because merging at behind_by 43+ costs more than redoing it. Not a criticism — your response time was good. It's just the clearest price tag we've seen on divergence, and it's why I care whether the fixture policy can tell inherited drift from fork-introduced drift automatically.

Questions

1. Does check-source-scan-policy.py compare against main, or against the merge-base? If it's main, do you already see exceptions in source-scan-fixtures.json that exist purely because upstream moved?
2. Do the snapshots key on line numbers anywhere? If so, the rule + file re-key above is a cheap way to collapse the noise.
3. Is there a retirement path for an exception once upstream absorbs the thing it was covering — or is pruning manual?

Our comm-based attribution is a few lines around test-list output and isn't buzz-specific; happy to paste it if useful. Mostly asking because if your policy script already handles (1), I'd rather adopt your shape than keep carrying ours.