FEAT: file write surface scorer (scorer phase 10)

[WatchTree-19]

Description

This is phase 10 from the scorer roadmap, the surface prototype. It adds a scorer that answers "did this run write that content to that location?" by reading what the location actually holds, rather than what the response says it did.

It is built on the pieces from the earlier phases, so a surface is just another scorable with its own observation source, and nothing in the existing scorers changes.

What it adds

• SurfaceScorable (scorable_type="surface") with a uri, a surface name, an exact or glob match, and an optional ScoringScope.
• ScoringScope with a time window, labels and an attempt_id, so a source can narrow the question to one run.
• ContentWritten condition (uri, match, optional contains).
• SurfaceObservationPayload (kind="surface") holding one SurfaceEntry per location (size, sha256, modified time, a bounded copy of the text) plus SurfaceCoverage and a count of locations that fell outside the scope.
• LocalFileSurfaceSource, which reads files under one root directory, such as a mounted sandbox workspace.
• FileWriteScorer, a TrueFalseScorer with CONDITION_TYPE = ContentWritten.

On the two open questions in the roadmap

• Pluggable surfaces: the scorer takes any ObservationSource[SurfaceScorable]. The local file source is the first one; a container, bucket or MCP filesystem source would plug into the same slot without touching the scorer.
• Paths vs patterns: a scorable can name an exact path or a glob, so "did anything under /data/ receive this?" works as well as a single file.

How the verdict works

• True when any covered location holds the content.
• False only when the source read every covered location in full and none holds it.
• Undetermined otherwise (root missing, file limit hit, unreadable file, binary or truncated content that cannot rule a match out).

Correlating a write to a run

Given a message, the scorer builds the scope from the run itself: the conversation's attack_result_id (from phase 9) and a window from the first message, less a small clock skew allowance, to the time of scoring. The local source can only check the window, against file modification times, so it records in the observation metadata which scope keys it did not apply. A file planted before the run is counted as outside the scope and cannot make an attack succeed.

Safety of the local source

Locations resolve inside the root only. .. is rejected, and symbolic links that point outside the root are not followed and are reported as a coverage gap. Reads are bounded by max_files and max_content_bytes, and run off the event loop.

Replay

The observation keeps the digest, size, modified time and retained text, so score_observation_async can re-judge it against a new contains after the workspace is gone. Replay against a different location raises NonReplayableObservationError.

Tests and Documentation

• tests/unit/models/test_surface.py (17 tests): model validation, round trips through the scorable and condition registries, and the acquisition and coverage invariants on Observation.
• tests/unit/score/test_file_write_scorer.py (41 tests): the source (window, glob, limits, symlink escape, dangling links, truncation on a multi-byte boundary), the matching table, scorer verdicts, offline replay, and an end to end PromptSendingAttack against a local agent that writes the file, including a planted older file that must not count.
• The existing scorer, models, memory, backend and executor suites pass alongside.
• New notebook doc/code/scoring/6_file_write_scorer (paired .py and .ipynb, executed), linked from the scoring overview, myst.yml and framework.md. It needs no model, service or credentials.

[WatchTree-19]

Thanks Roman Lutz (@romanlutz), all four were real, and the reproductions made them quick to pin down. Pushed in one commit on top of your merge of main.

Unreadable directories: listings now go through an explicit walk, and a directory that cannot be listed is recorded as listing_failed, so the observation is partial and the verdict undetermined. Your denied-nested-directory case is a test.

Bounded work: max_listed_entries caps entries examined, enumeration stops at the first candidate over max_files, and max_read_bytes caps bytes read and hashed per acquisition. A file cut short gets no digest (SurfaceEntry.sha256 is now optional) and records read_limit_exceeded. Cancellation sets a flag the worker checks at every entry and chunk. Tests count the bytes and entries actually touched, and check the worker stops after cancel.

Confinement: each file is now opened first, the open handle's final path is checked against the resolved root (/proc/self/fd on Linux, F_GETPATH on macOS, GetFinalPathNameByHandleW on Windows), and size, timestamp and content all come from that same handle. Opens are non-blocking so a FIFO cannot stall the worker, and directory links are not descended into. Since you already have the junction swap reproduction, would you be up for adding it as the regression test? You can exercise it on Windows directly, which I can't here.

Message reference: the scope now comes from MessageScorableResolver.resolve_async(), with tests for a missing id and for pieces from two runs.

[Roman Lutz (romanlutz)]

This is in a good state IMO but leaving it to Richard Lundeen (@richlundeen) to sign off / merge or provide more feedback since this is part of his roadmap for scoring.