[binaryaaron]

Summary

reconcile_candidates now checks each candidate for overlap against its two neighbours in a start-sorted list of accepted spans, instead of scanning every accepted span. Finding the neighbours is a binary search, O(log M) per candidate. Keeping the list sorted uses list.insert, which shifts O(M) entries per accepted span, so the worst case is still O(M^2) element moves; those are list shifts, not interpreted overlap checks, and are much cheaper in practice. Acceptance decisions and output order are unchanged.

Why

Dense agent traces produce thousands of candidate spans in a single record. The old any(_overlaps(candidate, existing) for existing in accepted) scan runs O(M^2) Python-level overlap checks: going from 2k to 8k spans made reconciliation about 16x slower.

What changed

• src/anonymizer/engine/detection/candidates.py: accepted bounds are kept in two parallel lists (starts and ends) ordered by start. Because accepted spans are disjoint and non-empty (enforced by AdmittedCandidate), ordering by start also orders by end. A candidate [s, e) overlaps an accepted span iff the predecessor (greatest start below s) ends after s, or the successor (smallest start at or above s) starts before e. That is two comparisons per candidate after the binary search.
• The iteration order over ranked candidates is untouched, so _rank_candidates and _rank_identical_geometry still decide ties exactly as before. The neighbour check keeps the same half-open semantics as _overlaps (touching spans do not overlap); _overlaps is now used only by the test reference.
• No API change, no behaviour change, no new dependency.

Equivalence test

tests/engine/test_detection_candidates.py adds a differential test. It keeps the old full-scan algorithm as a reference inside the test and compares it with reconcile_candidates over 300 seeded random inputs (nested, touching, and identical-geometry spans; several source texts; mixed labels, source kinds, explicit and fallback admissions, and scores), both in generated order and shuffled. A separate test uses many identical-geometry candidates across overlapping starts so identical-bounds ranking still picks the winner. A plain test reconciles 5000 disjoint spans and checks all are kept in order.

Measurements

One source text, M disjoint spans, best of 3 runs of reconcile_candidates. "Append" accepts spans in start order, so every insertion is at the end. "Front insertion" is the worst case from review: the left half has length 1 and the right half length 2, so longest-first ranking accepts the right half first and every left-half span then inserts near the front.

M	Shape	Before	After
2000	append	233 ms	16 ms
8000	append	3741 ms	64 ms
32000	append	not run	277 ms
2000	front insertion	287 ms	16 ms
8000	front insertion	4616 ms	71 ms
32000	front insertion	not run	390 ms

These are single-machine timings, not a bound. At 32k spans the worst-case shifts add about 40% over the append case.

Testing

Beyond CI: the timing probe above, which also asserts that the new and full-scan results are identical for both shapes at 2000 and 8000 spans.

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues were found.

Summary

reconcile_candidates replaces the full overlap scan with checks against the two nearest accepted spans.

• Candidate reconciliation checks nearby spans to speed up dense records.

_{Reviews (2) · Last reviewed commit: "perf(detection): reconcile candidates wi..." · Reviewed by Greptile}

[lipikaramaswamy]

Thanks for the focused optimization. I reviewed the intended two-file change at 21cbc72 separately from the older #305 history currently included in this PR. The neighbour check looks correct under the validated, non-empty/disjoint interval invariants; I found no acceptance, provenance, or publication-order regression.

Requesting changes for two items before merge:

1. Restack only the performance commit onto rebased #305. This head still has 28f8439 as its direct parent, while the current target is 6352cba. GitHub currently shows 31 commits / 54 changed files and a non-mergeable PR; local merge analysis reports six conflicting paths. Please replay only 21cbc72 onto current #305, preserve the newer base behavior, and rerun relevant tests. The intended change should then be reviewable as the small candidates.py/test patch, without replaying obsolete #305 commits.

2. Correct the total/worst-case complexity claim. At candidates.py:176-177, each accepted candidate can perform two O(K) Python list insertions. Binary-search overlap lookup is O(log K), but the complete algorithm remains O(M²) in the worst case. For a concrete valid input, use M disjoint spans at start=4*i: left half length 1, right half length 2. Length ranking accepts the right half first; each left-half insertion then shifts M/2 entries in each list, totaling M²/2 shifted entries. The existing 5000-span equal-length/increasing-start test exercises the append-shaped case instead. Please distinguish logarithmic lookup, linear insertion, and the substantial practical speedup in the summary/measurements. I am not requesting a new tree structure or an admission/policy rewrite.

Independent validation of this head:

• 415 candidate/lifecycle/admission/schema tests passed.
• 4,440 exhaustive/random scenarios, including iterator/reversed/coalesced and shuffled variants, matched the original full-scan implementation loaded directly from Git.
• All 2,155 full-suite tests passed across an initial run (2,150 pass, five environment-blocked failures) and a targeted rerun of those five with local socket access and short temp paths; this was not one uninterrupted green full-suite run.
• Changed-file lint/format and diff checks passed.

Local synthetic timings support the optimization: 8k disjoint spans took about 32–36 ms versus 1.63–2.39 s for the old scan. These are machine-specific observations, not a worst-case bound. Happy with the direction once the stack and complexity wording are corrected.

[binaryaaron]

agent (review-pr): Both items are addressed at 9240ccd.

1. Restack. The branch is now one commit on 6352cba, touching candidates.py and test_detection_candidates.py only. The older refactor(detection): establish deterministic entity candidate lifecycle #305 history is gone from the PR.
2. Complexity. Agreed: list.insert shifts O(M) entries per accepted span, so the worst case is still O(M^2) element moves. The commit message and description now separate the O(log M) lookup, the O(M) insertion and the measured speedup. Timing the input from the review (left half length 1, right half length 2, so every left-half span inserts near the front) next to the append-shaped input, one source text, best of 3:

M	Shape	Before	After
8000	append	3741 ms	64 ms
8000	front insertion	4616 ms	71 ms
32000	append	not run	277 ms
32000	front insertion	not run	390 ms

At 32k spans the shifts add about 40% over the append case. Both stay far below the full scan.