Repository navigation
Freeze released run-report directories with a preflight block (#271) - #315
Merged
Merged
Conversation
Part 2 of the Run envelope family. Each release's own `syncmill run`, with toolgraph preflight on, ran against a fake `toolgraph` that answers with the release's vendored goldens. The run dirs it left are under `tests/fixtures/persisted/envelope/<release>/preflight/`, a reports root of their own, so part 1's bytes and frozen listings are untouched. `freeze.py --part preflight` covers advisory, strict and (from v0.2.0) control-plan preflight: each status, both unavailable policies, two identities, a generation mismatch, an unmapped participant, and a SIGKILL after a strict preflight. Persistence failures come from the release's own writer with one `_atomic_write` destination made to fail; v0.1.2 and v0.2.0+ leave different files there. Five producer replies are synthetic, and `index.json` names them. `tests/test_frozen_preflight_envelope.py` checks HEAD's schema, the loader, `report show` and `report list` against the release's own output, the TUI Reports tab and the dashboard REPORT pane, and that retention removes a released dir with its nested `preflight/` directory. No HEAD code reads the preflight files back, so the checks on them are labelled fixture integrity. Observed, unchanged: `report show` prints `Preflight: ok` for a run that strict enforcement blocked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The aggregate check keyed on the envelope's strict mode, so the run killed after a strict preflight, whose skeleton says `disabled`, was never opened: replacing its aggregate and eval file with invalid JSON passed. The check now follows the scenario, and asserts each aggregate's decision, block reasons and, per evaluation, the identity, status and whether it references an eval file, so a reference moved to the wrong identity fails too. The fake toolgraph now refuses a reply that fails the producer schema with exit 3, as documented, instead of a traceback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The per-row check recorded only whether a row referenced an eval file, so swapping two rows' artifact and digest (and the envelope digest) passed on every release. Each referenced reply's agent must now be the row's toolgraph identity, its run id the envelope's and the aggregate's, and its decision and graph generation the row's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#271) strict-multi-mixed was labelled synthetic, but its second identity's toolgraph call fails before any reply is written, so every reply in it is a golden's. The label is gone from the generator, the test's expected set and the three index.json files (that key only; the release-written run dirs are unchanged). Four replies are synthetic, not the five the first commit counted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #271. This is the second half of the Run envelope family. The issue stays open: the Memory backends and TUI jobs families remain.
What
tests/fixtures/persisted/envelope/<release>/preflight/is a reports root of its own, so the first half's bytes and frozen listings are untouched. It holds the run directories that v0.1.2, v0.2.0 and v0.3.0 wrote through their ownsyncmill run, with report recording and toolgraph preflight on.freeze.py --part preflightran each release's CLI against an embedded faketoolgraph:tests/fixtures/toolgraph, same blob ids in all three releases and HEAD) and substitutes only what the consumer binds to the request:run_id, and for control-plan replies the plan digest and one evaluation per plan principal. A request whose agent, profile or candidates differ from the golden is refused (exit 3), and every reply is validated against the vendored producer schema before it is written.index.jsonnames them: a second identity answered with the allow golden (strict-multi-allow,strict-gen-mismatch, which also answers at generation 43), and one added control-plan finding (control-finding,control-strict-finding).syncmill.preflight._atomic_writemade to fail for one destination name. Nothing else is patched, andindex.jsonrecords the destination.Scenarios: 20 in v0.1.2 and 30 in v0.2.0 and v0.3.0 (control-plan preflight exists from v0.2.0; v0.1.2 silently ignores
control_plan_enabled):preflight: disabled) beside the preflight evidence.Each scenario's block and preflight files were measured on the release CLIs first;
freeze.pystops if a release writes anything else.tests/test_frozen_preflight_envelope.py(331 cases). Expected values come from raw fixture bytes, the release's own output inindex.json, or measurements written in the test, never fromrender_report,_is_run_dirorrun_preflight:contracts/run-envelope.schema.jsonaccepts every released preflight envelope (theokbranch, the strictallOf,graphState,controlPlanArtifact, the no-artifact error shapes); so does the release's own schema.load_envelopereturns the stored document unchanged.report showprints byte for byte what the release printed; each line matches the raw values; a blocked run showsfailed (preflight), exit code 1.report listprints what the release printed, newest first.preflight/included (v0.1.2 owned through the envelope fallback), and keeps a newer one byte for byte.docs/architecture/persisted-contracts.md: the Run envelope row names these.Observed, unchanged:
report showprintsPreflight: okfor a strict run that enforcement blocked (strict-block,control-strict-finding); only theResult:line shows the block, and mode, enforcement decision and counts appear nowhere in HEAD's UI. Codex suggested a separate usability issue for this; not filed yet.No
src/change. No current reader misread a released run directory.Verification
suite-runis CLEAN andruff checkis clean on the changed files.Preflight:read frommode; the block read as{};load_envelopedroppingpreflight; six schema changes (two status values removed,graphStatewithoutinstance_id, acontrolPlanArtifactconst, strict requiringgraph_generation,okrequiringevaluation_count); retention removing files only (fails only because of the nestedpreflight/).🤖 Generated with Claude Code