Skip to content

Publish immutable numbered capture journal snapshots and prove their removal - #827

Closed
coneilen wants to merge 2 commits into
mainfrom
coneilen-immutable-capture-journal-snapshots
Closed

coneilen wants to merge 2 commits into
mainfrom
coneilen-immutable-capture-journal-snapshots

Conversation

@coneilen

Copy link
Copy Markdown
Collaborator

Summary

The capture supervisor polled the worker's process-journal.json with a check-then-open (Test-Path, then File.Open) while the worker published each update by replacing that same path (File.Replace). The name can be transiently absent or locked during a replacement, so hosted runs intermittently failed UiCaptureDriver (49 run, 47 passed) with Could not find file ...\process-journal.json (run 38673179020 attempt 2, job 116225253342) and, on another PR, being used by another process (run 38679005657, job 116239062659). This PR replaces the mutable path with immutable numbered snapshots, adds a bounded re-list for the one case that means "list again", binds the terminal snapshot to the final manifest, and has the supervisor remove and prove removal of exactly its snapshots at release.

Refs #826
Refs #822

Changes

  • Worker (Invoke-UiCaptureWorker.ps1): Publish-CapJournalSnapshot writes a complete, closed, hash-chained snapshot (sequence, previousSha256, terminal) to a same-directory unique process-journal.pending-GUID.tmp opened CreateNew, flushes it, then publishes it with the non-overwriting File.Move to process-journal.N.json. No overwrite, replace or pre-delete. A collision throws and keeps the unpublished temporary. A terminal snapshot seals the journal. Write-CapTerminalProcessJournal replaces the inline final write, and the final manifest carries processJournal (terminal sequence, SHA-256, count).
  • Supervisor (Invoke-UiCapture.ps1): the reader lists strict positive numeric names (so 10 sorts above 9), directly opens the highest listed snapshot (no Test-Path before open), and re-lists within a bounded budget (8 attempts, 10 s, no sleeps) only for FileNotFoundException on the just-listed name. Parse, encoding, access/sharing, hash, lease/job/module/worker binding, sequence/name mismatch and foreign names in the journal namespace all fail; a corrupt highest never falls back to an older snapshot; the observed highest sequence/hash never regresses or changes. Unpublished temporaries are never read as snapshots.
  • After worker exit, EOF and no survivors, the supervisor requires (exit 0) or verifies (otherwise) the contiguous hash chain, exactly one terminal snapshot that is last, typed terminal arrays, and the exact terminal sequence/hash/count/identity/cleanup arrays bound to the final manifest. It writes a pre-removal inventory (journal-snapshot-inventory.json, created CreateNew) before removing the exact owned snapshot files, then proves their absence. The official release gains a journal record (state, positive inventory, terminal binding, removed and remaining names, failures). A no-worker refusal reports NO_WORKER_STARTED; verification failures retain every snapshot as forensic evidence; any removal or verification failure stays CLEANUP_UNPROVEN.
  • Tests (UiCaptureDriver.Tests.ps1): all 49 original case names are kept. The historical case actual journal reader permits worker atomic replacement while retaining its snapshot keeps its name and held-snapshot assertions but now exercises immutable publication (it no longer uses Replace). Eleven new real cases, no process, launch, lease, supervisor, worker or release mocks, cover: numeric highest selection, corrupt/foreign/mismatched highest refusal without fallback, controlled just-listed disappearance (supersede, sharing violation not treated as missing, vanish without successor, exhausted budget), a many-publication concurrent worker-publisher/supervisor-reader run with a held old reader and named-event barriers, terminal/manifest binding negatives, publisher collision and post-terminal sealing, and five real-command release cases (clean removal proof, no-worker refusal, planted corrupt highest, colliding foreign name, unremovable snapshot).
  • Untouched: UiCaptureDriver.psm1 (byte-identical, native bindings still 16), the harmless-target hooks are test-only, no product, scene adapter, runner, workflow, budget, provider, pin, bootstrap, README, ledger, schema or WinApp changes. The legacy hash-pinned actualSeedEvidenceReuse journal check is unchanged.

Test plan

Root cause of the unknown part is not claimed: the exact blocker or writer state behind each hosted failure was never measured and no worker journal artifact exists. The reproduction below is against the frozen fde702b sources and shows the same message class.

RED: pwsh -NoProfile -File Tools\windows\Tests\UiCaptureDriver.Tests.ps1 -ScratchRoot C:\gcs\j3\red (test-only commit 39690e5, unfixed production) -> executed 59, passed 49, failed 10 including the concurrent publication case: the reader failed 25 of 217 reads with Could not find file process-journal.json and being used by another process
GREEN: pwsh -NoProfile -File Tools\windows\Tests\UiCaptureDriver.Tests.ps1 -ScratchRoot C:\gcs\j3\final4 (head 7b77706) -> executed 60, passed 59, failed 1; all 11 new cases and all original cases pass except the scene component case, which fails the same way on unmodified origin/main under current local load (48 of 49)
REGRESSION: pending hosted exact-head CI -> to be recorded

Not verified

  • A fully green local 60 of 60 run: the preserved scene component regression (outside the journal path) fails locally with Live PID has incomplete current CIM metadata on both this head and unmodified origin/main (C:\gcs\j3\baseline-load, 48 of 49) while the machine is under load. It passed in the RED run and in an earlier GREEN run. Hosted exact-head CI is the authority.
  • Real GUI, native input, accessibility, live capture, product build, daemon persistence and macOS parity; none were run and no capture is admitted by this change.
  • The root cause of each hosted occurrence (blocking process, writer state, AV or ACL) is unknown; this change removes the writer replacement contention and fails closed on every other reader/ACL fault.
  • A transient sharing violation while the worker moves its own finished temporary into place is not retried; it fails the worker loudly.

Checklist

  • I have read the Contributing Guidelines
  • I have signed off my commits (git commit -s) per the DCO
  • Tests pass locally (make test)
  • Code follows the existing style (make check)
  • I added the test/contract before the implementation and observed the intended RED failure

coneilen and others added 2 commits October 14, 2026 20:59
Tests only: concurrent worker-publisher/supervisor-reader stress, numeric highest
selection, corrupt/foreign highest refusal, bounded FileNotFound re-list, terminal
verification, and real-command release cleanup proof. Production sources are unchanged
so these fail against the check-then-open/Replace journal.

Refs #826
Refs #822

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9f32f381-95ba-4648-a71a-c021bcfa8aa3
Signed-off-by: Colin Neilens <coneilen@microsoft.com>
…removal

The supervisor polled process-journal.json with Test-Path then Open while the worker
replaced it with File.Replace, so the name could be briefly missing and the hosted
UiCaptureDriver cases failed intermittently (49 run, 47 passed).

The worker now writes each complete, hash-chained snapshot to a same-directory unique
temporary and publishes it with a non-overwriting File.Move to process-journal.N.json.
The supervisor lists the numeric snapshots, opens the highest directly, and re-lists
within a bounded budget only for FileNotFoundException on the just-listed name; parse,
binding, access and sequence failures never fall back to an older snapshot. After the
worker exits with EOF it verifies the contiguous hash chain, binds the terminal
snapshot (sequence, hash, count, identity, cleanup arrays) to the final manifest,
preserves a pre-removal inventory, removes exactly the owned snapshots and proves
their absence in the official release. Failures stay cleanup-unproven.

Refs #826
Refs #822

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9f32f381-95ba-4648-a71a-c021bcfa8aa3
Signed-off-by: Colin Neilens <coneilen@microsoft.com>
@coneilen

Copy link
Copy Markdown
Collaborator Author

Superseded under the operator-directed fifth-failure escalation for #826. This PR is closed unmerged; its source, uncommitted review-fix candidate, failed and invalid runs, Sol rejection, and stopped-run handoff are preserved. No complete green local child or accepted corrected journal proof is claimed. A separately authorized fresh owner must first prove deterministic CIM attestation against main, then implement the mandatory immutable-snapshot protocol. No captures are authorized by this closure; #826 and the broader verifier remain open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant