[binaryaaron]

This PR is the long-lived base of the Graph SDK feature line. It is not a merge candidate for main. Later steps land on this branch as merge commits, without rewriting history, and main is merged in as it moves. The branch contains main at 7605c13c.

Against main, Anonymizer's pipelines run on a small private graph kernel, behind the Anonymizer API and config. anonymizer.graph is provisional: its names, signatures and defaults may change before it is published. The compatibility contract is the Anonymizer API and config of 0.4.0; this branch only adds to them: Substitute(consistency=...), declared detector facts on Detect (detector_labels, detector_labels_fixed, detector_offset_unit), result.outcome(row) and result.detection_scope.

What changes against main

118 files, +10,269 / -1,442: src/ 35 files, +4,186 / -874; tests/ 55 files, +4,437 / -491; docs/ +88 / -10; skills/ +62 / -54. Non-blank, non-comment, non-docstring lines under src/: +2,438. The branch has 180 commits, 36 of them merges.

• The earlier graph SDK is gone. At its previous head (9a8369cb, +100,340 / -1,237) this PR carried engine/graph_sdk, engine/harness, their adapters and their test suite. All of it is removed, so none of it appears in the diff against main.
• A private kernel, src/anonymizer/engine/graph_core/ (1,593 lines in six files), with five modules:
  • kernel.py (419): kind registry, static plan, the executor over its state, cancel token;
  • authoring.py (379): operations (per target or phase-scoped), workflows, graph values;
  • execution.py (697): prepare and protect, on the caller's thread, and the request forecast;
  • observe.py (44): the observed-request channel (below);
  • stages.py (45): runs a fixed sequence of stages as one invocation. Detection and the rewrite judges use it.
• anonymizer.graph (66 lines) re-exports exactly 24 names from the kernel: Assessment, CancelToken, ContextScope, ContextSource, ContextView, Datum, Dependency, GraphRejection, GraphScope, InspectionSummary, Limits, Mention, Operation, OutputGroup, PreparedProtection, ProtectionGraph, ProtectionResult, Reason, Requests, TargetOutcome, Workflow, prepare, protect, register_kind. tests/graph/test_public_api.py pins the list and checks that importing the module loads only the kernel.
• Replace routes on the kernel (engine/replace/route.py). Redact, Annotate and Hash run one target per row. Substitute's replacement map is a phase-scoped step of the same per-row workflow, so a row DataDesigner drops is a failed target in the same record as every other row.
• Detection and Replace as one invocation. In Replace mode, one protect() call runs detect, then substitute-map (Substitute only), replace and verify-patches. A row the detector drops fails at detect and blocks its later steps. A caller-supplied runner that overrides run keeps the previous path.
• Execution backend seam (engine/ndd/backend.py). DataDesignerBackend runs a DataDesigner workflow and returns the surviving frame, each surviving row's input position and the observed request count. At this head only Substitute's map step uses it; detection, Rewrite and the judges call NddAdapter.run_workflow() directly. NddAdapter.run_workflow() is still the only caller of DataDesigner.create() and preview().
• Observed-request channel (engine/graph_core/observe.py). Code inside a step calls report_requests(n). ProtectionResult.summary() shows the total as exact only when every step that can make requests reported. report_requests is not exported.
• Also added: docs/concepts/graph-sdk.md, a provider-free example in examples/graph_sdk/, the spec programs in tests/spec_programs/, and a dogfood kit in tools/dogfood/ that uses only the public anonymizer API.

Since #345-#347 folded in, Rewrite runs as one invocation with detection as its first step, and evaluate() runs its judges as one invocation. Inside those invocations, detection's own stages (detect, tag-latent, finalize) and the rewrite judges still run through run_stages.

Reading order

1. docs/concepts/graph-sdk.md (61 lines): what anonymizer.graph declares, prepares, protects and reports.
2. The kernel, src/anonymizer/engine/graph_core/: kernel.py, authoring.py, execution.py, then observe.py and stages.py.
3. src/anonymizer/engine/replace/route.py (375 lines): the Replace pipeline. Then engine/ndd/backend.py and the facade in interface/anonymizer.py.
4. tests/spec_programs/: numbered programs, each pinning one kernel behaviour. run_all runs them all.

Folded step history

Each step below was a draft PR stacked on this one. This branch was fast-forwarded to 44924e2e and then, step by step, to 13fb7470, which contains all of them unchanged. Each is folded into this PR; closed, description kept as the record.

• chore: run Redact, Annotate, Hash and Substitute on a private kernel #324 (f736a427): Redact, Annotate, Hash and Substitute run on the private kernel, and the graph SDK code only those routes used is deleted.
• refactor: run Rewrite as 0.4.0's stages on the graph kernel #325 (b68adb88): Rewrite runs 0.4.0's stages on the kernel, and the graph SDK's own Rewrite path is deleted.
• refactor: run detection on the kernel and remove the old graph SDK #326 (0cea5923): detection runs on the kernel, the old graph SDK and its tests are removed, and anonymizer.graph exports the kernel's 23 names.
• feat(graph): invocation-scoped operations in the kernel #332 (e8ea6481): an operation can run once over every target in a protect() call and still record one outcome per target.
• refactor(replace): run Substitute as one invocation #336 (ba93da53): Substitute runs as one invocation, with its replacement map as a step of the per-row workflow. It also carries the kernel executor rewrite from refactor(graph): simplify the kernel executor #337.
• refactor(replace): run detection and replacement as one invocation #342 (78815482): in Replace mode, one invocation runs detection and replacement.
• feat(graph): count observed requests and name the DataDesigner backend #343 (44924e2e): steps report the requests they make, and Substitute's map step runs through a named execution backend.
• chore(graph-sdk): run Rewrite as a per-target operation with a covering verifier #345 (16ce9a80): Rewrite as a per-target operation with a covering verifier, and needs_human_review projected from the execution result.
• chore: run detection as the first step of the Rewrite invocation #346 (f0980f6c): detection as the first step of the Rewrite invocation.
• chore: run evaluate() as one invocation of judge bodies and covering verifiers #347 (13fb7470): evaluate() as one invocation of the judge bodies and covering verifiers.

Since then, PRs into this branch merge with merge commits, and main was merged in five times, most recently at 7605c13c:

• refactor(graph): name the operation scope phase_scoped #361: name the operation scope phase_scoped.
• fix(graph): keep admitted form, check produced mentions at acceptance, make release linear #362: keep the admitted form, check produced mentions at acceptance, make release linear.
• feat(graph): typed context bindings and assembly ordering #363: typed context bindings and assembly ordering.
• feat(graph): declared checks and derived reasons #364: declared checks and derived reasons.
• feat(graph): read the final candidate from the last text producer #365: read the final candidate from the last text producer.
• fix(rewrite): keep each repair's output in the repair loop #371: keep each repair's output in the repair loop (main's fix(rewrite): keep every repair iteration's output on the staged path #354 fix, at this branch's own repair join).
• docs: settle graph SDK and config terminology #386: settle graph SDK and config terminology.
• fix(rewrite): report rows that fail detection as failed records #392: report rows that fail detection as failed records.
• fix(graph): keep callback, context-ceiling, release and cancel contracts #391: keep callback, context-ceiling, release and cancel contracts.
• feat(graph): report whether observed requests stayed within the plan #383: report whether observed requests stayed within the plan.
• feat(graph): assemble region replacements and record boundary measurement #384: assemble region replacements and record boundary measurement.
• feat(detect): validate declared detector facts and report the resolved scope #385: validate declared detector facts and report the resolved scope.
• fix(replace): keep run-mode substitutes unique across the run #394: keep run-mode substitutes unique across the run.
• feat(replace): measure substitute collision fallbacks #397: measure substitute collision fallbacks.

How to test

uv sync --frozen --group dev
make check
uv run --group dev pytest
uv run --group dev python -m tests.spec_programs.run_all

Please test at this PR's head and name the SHA you ran.

CI's Test job runs only on PRs to main, so this PR is where the full suite runs on GitHub.

Known gaps

• docs/development/graph-sdk-cutover.md (the M0 stage plan, requirement map and route inventory) is removed on this branch: every path it named is gone and every route it listed has moved to the kernel. It stays in history at 44924e2e.
• skills/anonymizer/ differs from main in 6 files (+62 / -54). require-nvskills-ci passes at this head; if the skill changes again it needs a /nvskills-ci comment, and a signature run may add a bot commit to this branch.
• Earlier review comments were made on a much larger head. The two inline review comments from 2026-10-28 sit on engine/graph_sdk/_execution_publication.py and _workflow_admission.py, which no longer exist.
• Behaviour differences from main are described in the steps' PRs: trace-column dtypes in refactor(replace): run Substitute as one invocation #336, and measurement-stage and custom-adapter notes under "Known differences" in refactor(replace): run detection and replacement as one invocation #342 and feat(graph): count observed requests and name the DataDesigner backend #343.
• Nothing was rewritten, so the removed package, including about 9 MB of reference data, stays in this branch's history. How this branch reaches main is decided when it lands. This repo merges by squash only.
• One item from the 2026-10-29 conformance review is unchanged and now documented in docs/concepts/graph-sdk.md: prepare(request_budget=...) checks the planned request count and does not enforce a ceiling on actual requests while the workflow runs. The other four (checks on mentions produced by a step, content() after a failed text producer, a missing leakage number in the final checks, protect() rechecking the prepared combination) are fixed on this branch.
• The effective label names were removed from the detection-scope log lines in this branch as part of its content-safe logging change; the resolved scope, labels included, is now reported as result.detection_scope (feat(detect): validate declared detector facts and report the resolved scope #385); whether the labels should also be logged is an open question.

🤖 Generated with Claude Code

[andreatnvidia]

I’d keep record as the default and run opt-in. I’m good with the placeholder fallback too. A less natural result is better than making two different entities look like the same one.

Codex spotted a catch with run: collision checks are per record, so Alice and Carl can both become “Maya” in different records. Placeholder counters reset per record too. I’d either enforce uniqueness across the run or drop the recommendation to use it for joins.

I’d also let an existing run-wide mapping win over another entity’s new proposal. Falling back because it matches a protected original makes sense; losing the mapping because another proposal picked the same name doesn’t.

[binaryaaron]

@andreatnvidia Both points are right. #394 fixes them: in run mode a substitute is now unique across the whole run, placeholders are numbered run-wide, and an established mapping keeps its substitute when a later record proposes the same one. Record mode is unchanged. Thanks for the catch.

[greptile-apps]

Fold measurements cannot be imported

generate_map_only() now records LlmReplaceWorkflow.fold on every call, but StageMeasurement.stage does not accept that name. The strict model also rejects the new consistency, entry_count, records_with_placeholder, and fallback_counts fields.

Measured Substitute and non-combined Rewrite runs therefore produce files that read_measurement_snapshot() rejects, blocking completion seals and W&B imports. Update the measurement readers alongside this record and add a test that passes it through the strict reader.

[andreatnvidia]

This still reproduces at 5b764cab, including when there are no entities and no model calls. The emitted fold record fails strict validation on the stage name and all four new fields. The reader schema needs to land with the producer change.

[greptile-apps]

Agreed—this confirms the finding, including the no-entity fast path. generate_map_only() emits LlmReplaceWorkflow.fold via record_stage() even when there are no entities or model calls; the strict StageMeasurement schema rejects that stage name and the additional fields (consistency, entry_count, records_with_placeholder, and fallback_counts). The reader schema and a strict-reader regression test need to land with the producer change.

[binaryaaron]

@andreatnvidia Fixed by #398, merged into this branch at cb37b27. The measurement reader and the W&B stage table accept the fold stage and its four fields, including on the no-entity path, and a test passes the record through the strict reader.

[andreatnvidia]

At 5b764cab, calling Anonymizer.run() from a graph callback after cancelling the outer token hits this fallback even with nonempty input. Detection then runs with target=(), and resolving the returned positions raises IndexError. A custom detector also executes despite cancellation. The same guard appears in Rewrite. Could we restrict this fallback to genuinely empty graphs and handle cancelled invocations separately?

[binaryaaron]

@andreatnvidia Confirmed in Replace and Rewrite, and fixed in 7e83c01. After the outer token was cancelled the nested detect step was skipped, and the fallback meant for empty input then ran the detector outside the cancel check. The fallback now runs only when the graph has no targets (7015150), and a nested run() or preview() that the cancellation stops raises AnonymizerWorkflowError without calling the detector. The same rule now applies at every entry into the kernel: both Replace routes, evaluate(), which used to return unjudged rows after a cancel, and the staged detection and rewrite-evaluation runs (tests in 56389bc).

[andreatnvidia]

A custom replace_runner.evaluate() returning one surviving row and one FailedRecord from two inputs now raises TypeError: invocation operation returned list of length 1, expected 2 entries. The same public Anonymizer.evaluate() call returns the partial result on 7605c13c. The graph keeps the original target count, but this publishes only the surviving rows. Keeping the previous path for custom evaluators would preserve compatibility; otherwise the outputs need explicit source alignment.

[binaryaaron]

@andreatnvidia Confirmed, and fixed in 82e4500. The judge step returned one entry per returned row instead of one per input row. It now maps returned rows back to their inputs, so a custom evaluate() that drops a row returns the partial result with its failed record, as on main. A test covers Replace and Rewrite.