Skip to content

Freeze released run-report directories and read them with the current readers (#271) - #314

Merged
memtomem merged 5 commits into
mainfrom
test/frozen-envelope-271
Nov 15, 2026
Merged

memtomem merged 5 commits into
mainfrom
test/frozen-envelope-271

Conversation

@memtomem

Copy link
Copy Markdown
Owner

Refs #271. This is the first half of the Run envelope family. The issue stays open: this family's preflight records and the Memory backends and TUI jobs families remain.

What

tests/fixtures/persisted/envelope/<release>/ holds the run directories that v0.1.2, v0.2.0 and v0.3.0 wrote through their own syncmill run and board run with report recording on. freeze.py beside them used each release's worktree and venv (with its phoenix extra) and fake codex and kimi binaries:

  • runs/<run-id>/: envelope.json, result.json, trace.otlp.json and, from v0.1.3 on, the .syncmill-run-dir sentinel, copied byte for byte.
  • index.json: scenario → run id, plus what the release's own CLI printed for report list and for each report show.
  • run-envelope.schema.json: the release's own schema, with its git blob id.

Scenarios, each in all three releases:

  • completed
  • no change: patch_digest is the sha256 of the empty string, not null
  • route failure: agent_id "none", exit_code -1
  • gate passed, and gate failed: the failure runs with -s pipeline, because route drops a rejected gate report
  • a compete team
  • a file trace, a Phoenix export error, a trace error
  • breaker degraded, and breaker plus Phoenix together
  • a board block
  • the skeleton a SIGKILL leaves
  • a SIGINT's finalized crash
  • a trace-only run with no envelope

tests/test_frozen_envelope.py (189 cases). Expected values come from the raw fixture bytes, from the release's own output in index.json, or from literals:

  • Schema: HEAD's contracts/run-envelope.schema.json accepts every released envelope, with a format checker. So does the release's own schema.

  • load_envelope returns the stored document unchanged.

  • report show:

    • prints byte for byte what the release printed, and each field matches the raw values;
    • a run without an envelope is refused with the release's message.
  • report list:

    • prints what the release printed;
    • across all three releases, sorts newest first by created_at;
    • skips foreign entries.

    A test-made long run (created before every released run, finished after them) tells creation order apart from completion order.

  • The TUI Reports tab and the dashboard REPORT pane:

    • list every released run, plus the long run, in the same order;
    • show the release's own report text as the selection moves, read from the displayed widgets.
  • Retention ownership over released directories, through RunReporter.start and the Supervisor's trace-only sweep:

    • Removable: a directory with the sentinel, or for v0.1.2 one whose envelope has syncmill's kind and names its own directory.
    • Never removed: a v0.1.2 trace-only directory, which has neither.
    • Kept: test-made probes that each break one predicate (renamed, kind changed, kind removed, not an object, torn, a foreign result.json).
    • Asserted: the full set of survivors. The current run survives even when it is the oldest, with its contents intact. A custom trace.dir is never swept.

docs/architecture/persisted-contracts.md: the Run envelope row names these. Not covered:

  • Preflight blocks, which appear only as disabled. The released preflight records are the second half.
  • The meaning of trace.otlp.json, which is tracegraph's reader to test.
  • report list also lists a foreign directory holding any nonempty JSON-object envelope.json. This is recorded as observed and left unchanged.

No src/ change. No current reader misread a released run directory.

Verification

  • suite-run is CLEAN, ruff check . is clean and git diff --check main...HEAD passes.
  • 30 injected regressions across report.py, supervisor.py, tui.py, dashboard.py and the schema. Each is injected exactly once and restored from a byte copy, and each fails a test with an assertion. They cover:
    • each ownership predicate;
    • the Supervisor's sweep call, its custom-trace.dir guard, and the current-run exclusion in both callers;
    • the listing sort (removed, ascending, by finished_at) and a TUI or dashboard re-sort;
    • a field dropped on load;
    • rendered lines and separators;
    • UI labels and selection updates;
    • two schema tightenings that the released skeleton breaks.
  • Two schema mutations do not bite on these fixtures, by design: schema_version strictness and a closed traceBlock. Every release writes version 1 and only declared trace fields.
  • Codex review: design 2 rounds (NEEDS-FIX twice, all adopted after measuring the releases), implementation round 1 NEEDS-FIX (2 surviving regressions) and round 2 NEEDS-FIX (UI ordering), both fixed. The final round got the diff only, with no framing, and returned SHIP with one EOF nit, which is fixed.

🤖 Generated with Claude Code

memtomem and others added 5 commits November 15, 2026 12:59
… readers (#271)

The first half of the Run envelope family of the persisted-contracts D
rows. tests/fixtures/persisted/envelope/<release>/ holds the run
directories v0.1.2, v0.2.0 and v0.3.0 wrote through their own run and
board run with report recording on (freeze.py, a fake codex and kimi,
the release venv with its phoenix extra): envelope.json, result.json,
trace.otlp.json and, from v0.1.3 on, the run-dir sentinel. index.json
holds what each release's own CLI printed for report list and report
show; the release's own schema is frozen beside it.

Scenarios: completed, no change (patch digest of the empty string),
route failure, gate passed and failed (pipeline, since route drops a
rejected gate), a compete team, a file trace, a Phoenix export error, a
trace error, breaker and multiple degradations, a board block, the
skeleton a SIGKILL leaves, a SIGINT's finalized crash, and a trace-only
run with no envelope.

tests/test_frozen_envelope.py:
- HEAD's schema accepts every released envelope (and so does the
  release's own); load_envelope returns the stored document unchanged;
- report show and report list print what the release printed, byte for
  byte, and each shown field matches the raw values; a run without an
  envelope is refused with the release's message;
- the TUI Reports tab and the dashboard REPORT pane list every released
  run newest first and show the release's own report text on selection;
- retention ownership over released dirs, through RunReporter.start and
  the Supervisor's trace-only sweep: a sentinel, or for v0.1.2 an
  envelope with syncmill's kind naming its own directory, makes a run
  dir removable; a v0.1.2 trace-only dir is never removed; one-predicate
  probes (renamed, kind changed or removed, not an object, torn, foreign
  result.json) are kept; a custom trace.dir is never swept.

Preflight blocks appear only as disabled; the released preflight records
are the second half. No src change. 25 injected reader mutations fail
these tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Implementation review round 1 found two regressions the envelope tests
let through:

- a retention sweep that removed the current run's directory passed,
  because the tracer recreated it at finalization. The current run now
  holds a marker written before the sweep, and the reporter and both
  Supervisor tests assert its bytes survive.
- a listing sorted by finished_at passed, because every released run
  finished before the next began. The cross-release listing adds a
  test-made long run (a released envelope started before all of them,
  finished after all of them) that the two orders place apart.

Also: load_envelope reads a copy of the fixtures, not the fixtures
themselves, and every helper enumerates the scenarios a release froze
(including any completed-N run freeze.py adds to break name order).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#271)

Implementation review round 2: the TUI Reports tab and the dashboard
REPORT pane were checked only over released runs, whose creation and
completion orders agree, so a view that re-sorted by finished_at passed.
Both now include the test-made long run (created before every released
run, finished after them) and expect it last.

Schema validation now passes a format checker. jsonschema checks `date`
and similar formats always, `date-time` only with the optional
rfc3339-validator, which this environment does not install.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

Review of PR #314 (agy, SHIP with minors):

- RELEASES came from a literal, so a release directory freeze.py adds
  later would be skipped. It is now read from the fixture directories,
  as the other frozen families do, and a test requires the three present
  releases so discovery cannot hide a missing one.
- index.json records the git blob id of the release's schema, but
  nothing compared it with the frozen copy. Each release now checks
  the file's blob id against it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@memtomem
memtomem merged commit 377f4db into main Nov 15, 2026
12 checks passed
@memtomem
memtomem deleted the test/frozen-envelope-271 branch November 15, 2026 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant