Skip to content

Freeze released run-report directories with a preflight block (#271) - #315

Merged
memtomem merged 4 commits into
mainfrom
test/frozen-envelope2-271
Nov 15, 2026
Merged

memtomem merged 4 commits into
mainfrom
test/frozen-envelope2-271

Conversation

@memtomem

Copy link
Copy Markdown
Owner

Refs #271. This is the second half of the Run envelope family. The issue stays open: the Memory backends and TUI jobs families remain.

What

tests/fixtures/persisted/envelope/<release>/preflight/ is a reports root of its own, so the first half's bytes and frozen listings are untouched. It holds the run directories that v0.1.2, v0.2.0 and v0.3.0 wrote through their own syncmill run, with report recording and toolgraph preflight on. freeze.py --part preflight ran each release's CLI against an embedded fake toolgraph:

  • The fake answers with the release tree's vendored goldens (tests/fixtures/toolgraph, same blob ids in all three releases and HEAD) and substitutes only what the consumer binds to the request: run_id, and for control-plan replies the plan digest and one evaluation per plan principal. A request whose agent, profile or candidates differ from the golden is refused (exit 3), and every reply is validated against the vendored producer schema before it is written.
  • Four replies are synthetic, and index.json names them: a second identity answered with the allow golden (strict-multi-allow, strict-gen-mismatch, which also answers at generation 43), and one added control-plan finding (control-finding, control-strict-finding).
  • Persistence failures come from the release's own writer, with syncmill.preflight._atomic_write made to fail for one destination name. Nothing else is patched, and index.json records the destination.

Scenarios: 20 in v0.1.2 and 30 in v0.2.0 and v0.3.0 (control-plan preflight exists from v0.2.0; v0.1.2 silently ignores control_plan_enabled):

  • advisory: ok, drift, unresolved identity, unmapped agent, error, timeout;
  • strict: allow, warn, block, timeout, unresolved, unavailable under both unavailable policies, two identities, two identities with one unavailable, a generation mismatch, an unmapped participant;
  • strict persistence failures, where the releases differ: when an eval file cannot be written, v0.1.2 leaves nothing and v0.2.0+ an aggregate without that reference; when the aggregate cannot be written, v0.1.2 leaves the eval file and v0.2.0+ nothing;
  • control-plan (v0.2.0+): allow, findings, error, error with the warn policy, timeout, a persistence failure, each in the modes measured;
  • a SIGKILL after a strict preflight: the skeleton envelope (preflight: disabled) beside the preflight evidence.

Each scenario's block and preflight files were measured on the release CLIs first; freeze.py stops if a release writes anything else.

tests/test_frozen_preflight_envelope.py (331 cases). Expected values come from raw fixture bytes, the release's own output in index.json, or measurements written in the test, never from render_report, _is_run_dir or run_preflight:

  • Schema: HEAD's contracts/run-envelope.schema.json accepts every released preflight envelope (the ok branch, the strict allOf, graphState, controlPlanArtifact, the no-artifact error shapes); so does the release's own schema.
  • load_envelope returns the stored document unchanged.
  • report show prints byte for byte what the release printed; each line matches the raw values; a blocked run shows failed (preflight), exit code 1.
  • report list prints what the release printed, newest first.
  • The TUI Reports tab and the dashboard REPORT pane list every run and show the release's text for the blocked and smallest blocks.
  • Retention removes released preflight directories whole, nested preflight/ included (v0.1.2 owned through the envelope fallback), and keeps a newer one byte for byte.
  • Fixture integrity, labelled as such because no HEAD code reads the preflight files back: each block's exact keys and digests; each strict aggregate's decision, block reasons and evaluation rows; each referenced eval reply's agent, run id, decision and generation matching its row; the control plan's digest in both places.

docs/architecture/persisted-contracts.md: the Run envelope row names these.

Observed, unchanged: report show prints Preflight: ok for a strict run that enforcement blocked (strict-block, control-strict-finding); only the Result: line shows the block, and mode, enforcement decision and counts appear nowhere in HEAD's UI. Codex suggested a separate usability issue for this; not filed yet.

No src/ change. No current reader misread a released run directory.

Verification

  • suite-run is CLEAN and ruff check is clean on the changed files.
  • 10 injected regressions, each restored from a byte copy, each failing the intended tests with an assertion: Preflight: read from mode; the block read as {}; load_envelope dropping preflight; six schema changes (two status values removed, graphState without instance_id, a controlPlanArtifact const, strict requiring graph_generation, ok requiring evaluation_count); retention removing files only (fails only because of the nested preflight/).
  • Fixture-integrity bites, run on a scratch copy of the fixtures: invalid JSON in the killed run's aggregate, a reference moved to the failed identity, and two rows' references swapped each fail.
  • Codex review: design 2 rounds (NEEDS-FIX twice; missing strict, control and persistence-failure states, all measured and added). Implementation round 1 NEEDS-FIX (the killed run's aggregate was never checked; no per-row expectation; a schema refusal raised instead of exiting 3) and round 2 NEEDS-FIX (swapped references passed), all fixed. The final round got the diff only, with no framing, and returned SHIP with one minor (a wrong synthetic label), which is fixed.

🤖 Generated with Claude Code

memtomem and others added 4 commits November 15, 2026 16:41
Part 2 of the Run envelope family. Each release's own `syncmill run`,
with toolgraph preflight on, ran against a fake `toolgraph` that answers
with the release's vendored goldens. The run dirs it left are under
`tests/fixtures/persisted/envelope/<release>/preflight/`, a reports root
of their own, so part 1's bytes and frozen listings are untouched.

`freeze.py --part preflight` covers advisory, strict and (from v0.2.0)
control-plan preflight: each status, both unavailable policies, two
identities, a generation mismatch, an unmapped participant, and a
SIGKILL after a strict preflight. Persistence failures come from the
release's own writer with one `_atomic_write` destination made to fail;
v0.1.2 and v0.2.0+ leave different files there. Five producer replies
are synthetic, and `index.json` names them.

`tests/test_frozen_preflight_envelope.py` checks HEAD's schema, the
loader, `report show` and `report list` against the release's own
output, the TUI Reports tab and the dashboard REPORT pane, and that
retention removes a released dir with its nested `preflight/`
directory. No HEAD code reads the preflight files back, so the checks
on them are labelled fixture integrity. Observed, unchanged:
`report show` prints `Preflight:  ok` for a run that strict enforcement
blocked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The aggregate check keyed on the envelope's strict mode, so the run
killed after a strict preflight, whose skeleton says `disabled`, was
never opened: replacing its aggregate and eval file with invalid JSON
passed. The check now follows the scenario, and asserts each
aggregate's decision, block reasons and, per evaluation, the identity,
status and whether it references an eval file, so a reference moved to
the wrong identity fails too.

The fake toolgraph now refuses a reply that fails the producer schema
with exit 3, as documented, instead of a traceback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The per-row check recorded only whether a row referenced an eval file,
so swapping two rows' artifact and digest (and the envelope digest)
passed on every release. Each referenced reply's agent must now be the
row's toolgraph identity, its run id the envelope's and the
aggregate's, and its decision and graph generation the row's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#271)

strict-multi-mixed was labelled synthetic, but its second identity's
toolgraph call fails before any reply is written, so every reply in it
is a golden's. The label is gone from the generator, the test's
expected set and the three index.json files (that key only; the
release-written run dirs are unchanged). Four replies are synthetic,
not the five the first commit counted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@memtomem
memtomem merged commit 51ea3fd into main Nov 15, 2026
12 checks passed
@memtomem
memtomem deleted the test/frozen-envelope2-271 branch November 15, 2026 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant