Skip to content

feat: capture files an agent sent by path (images, MP4/MOV video) into the ledger and serve them as assets - #55

Open
jamesyong-42 wants to merge 2 commits into
mainfrom
feat/recorded-video-ingest
Open

jamesyong-42 wants to merge 2 commits into
mainfrom
feat/recorded-video-ingest

Conversation

@jamesyong-42

@jamesyong-42 jamesyong-42 commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

What

When Claude Code sends the user a file by path (SendUserFile), the daemon now copies images and MP4/MOV video into the raw ledger at capture time and serves them through the existing asset route. The History room shows a video as a tile that plays in place.

  • Adapter (Claude Code only). Reads attachments[].path, .size and .media_type from a successful, single-result SendUserFile line whose opener it remembers. Relative paths, and paths with a backslash, ./.. segment or trailing slash, are skipped rather than rewritten.
  • Capture. A new IngestOp::PresentedMedia (a no-op in the store) asks the pipeline to read the file. Bytes are ledgered only when the file's size equals the recorded size and its magic bytes match the declared type. Caps: 32 MiB per image, 64 MiB per video, 128 MiB and 32 files per record. Device and process paths (/dev, /proc, also through a symlinked directory) are refused before any open.
  • Ledger. The Document frame gains an optional fifth metadata element {media_type, expected_len}. ledger verify --json reports a media block (frames, captured, missing, too large, unsupported, changed during read).
  • Rebuild. Reads only the ledger, never the filesystem. A later record that names the same file after it is gone does not downgrade an earlier capture.
  • Serving. /api/assets/<digest>.mp4|.mov returns the file whole to any read grant. No Range support yet.
  • Room. A video tile shows a play glyph, name and size, requests nothing until pressed, then plays and seeks. The app plays an object URL built from readAsset; a host may offer mediaAssetUrl only for addresses that answer range requests.
  • Switch. ROSETTA_PRESENTED_MEDIA=off disables capture (for frozen benchmark pairs).

No new table, column or wire type; no schema or protocol bump. The Claude normalizer version is unchanged, so existing rows gain attachments only after rosettad rebuild; new sends are captured live.

Exposure, stated plainly

The daemon opens any absolute path that a successful SendUserFile result names, and a captured file is then readable by any client holding a read grant that knows its digest. docs/AUTH.md §5.4 is updated to say so.

Known limits (documented in docs/ARCHITECTURE.md)

  • No aggregate cap or dedupe across records: the same large file sent many times is re-read and re-ledgered each time.
  • Two crash-window cases are corrected only by the next replay: a file replaced by different bytes of the same size, and a killed batch that was the source's very first.
  • Sliced (legacy ColdEngine) passes have no ledger fallback.
  • Video members are never garbage-collected; older binaries cannot decode media frames.
  • The video tile departs from locked design entries I-2 and I-6; a dated entry in clients/app/design/transcript-record.html records it.
  • Out of scope: Artifact asset uploads, other agents' video, Range serving, poster frames, duration.

Verification

  • Full CI-shaped gate on the pre-merge tree: cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features --locked -- -D warnings, cargo test --workspace --all-features --locked (2,167 passed, 0 failed, 3 ignored).
  • After merging main: fmt, clippy and tests re-run for the touched crates; ground-truth check, Python tool tests, npm run build, npm test (626 pass), npm run check:packages, docs-site check.
  • New tests: pipeline presented_media (append growth, replay, reread, snapshot), presented_media_e2e, rosettad presented_media, and an Electron test that records an H.264 clip, then plays and seeks it on both the bytes path and a range-capable host.
  • Atlas check: 0 dangling. rule:module-size (warn) is reported because already-oversized files grew slightly; no file newly crossed the limit and the baseline is unchanged.
  • Not done: playback of a real captured video in the app, by hand.

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

jamesyong-42 and others added 2 commits November 12, 2026 21:40
…o the ledger and serve them as assets

A Claude Code `SendUserFile` result names files by path; the bytes are not in
the transcript. The adapter now emits a path-reference attachment on the tool
row plus one pure fact per sent image or MP4/MOV video. The pipeline's capture
seam copies the file into a `Document` ledger frame carrying a media claim,
under an identity check (current size equals the recorded size, magic matches
the declared type; 32 MiB per image, 64 MiB per video, 128 MiB per record),
and upgrades the attachment to a content-addressed member served whole by
`GET /api/assets/<digest>.mp4|.mov`. Rebuild reads only the ledger.

- core: `PresentedMedia` fact and `IngestOp::PresentedMedia` (a no-op in the
  store); mp4/mov member types. No table, column, wire or protocol change.
- capture: optional media claim on the Document frame (frames without one are
  byte-identical); encode refuses what decode rejects; the prefix walk knows
  the Document tag, so such segments verify clean and compact; `ledger verify`
  tallies media frames by status.
- media: pure MP4/MOV header probe (ftyp brands, tkhd size through the track
  matrix). No codec.
- pipeline: ledger-first capture on re-frame, replacement and snapshot passes;
  a ledger fallback after a failed observation on append growth; a `reframing`
  checkpoint flag so an interrupted re-frame resumes as one; replay joins the
  first captured frame; `/dev` and `/proc` refused, also through a symlinked
  directory; `ROSETTA_PRESENTED_MEDIA=off` turns observation off.
- claude-code adapter: reads only `attachments[].path`, `.size`,
  `.media_type`; a path is used as written or skipped. Normalizer stays 39.
- room: a video tile that requests nothing until pressed and plays an object
  URL read through the session (the daemon serves a member whole, which a
  `<video>` cannot seek); the Electron check plays and seeks a clip it records.
- docs: ARCHITECTURE.md, AUTH.md, the endpoint page, ground-truth notes, atlas
  media domain.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant