Skip to content

fix(collect): read only regular files, and keep manifest names out of paths - #607

Merged
earakely-scale merged 14 commits into
mainfrom
edgararakelyan/collect-artifacts-hardening
Nov 12, 2026
Merged

earakely-scale merged 14 commits into
mainfrom
edgararakelyan/collect-artifacts-hardening

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Why

collect_artifacts copies what an agent leaves behind onto the operator's side. Two flaws let the agent reach past its own deliverables:

  • A manifest's names became paths. In manifest mode (manifest_step_id), the names in the agent's structured output were stored verbatim as the collected keys, and the explorer's "Download All" zip used each key as a member name. So a name like ../../.bashrc became an archive entry that climbs out of the folder it's extracted into, holding bytes the agent wrote.
  • Anything at a path was read. A file was read with base64 < path until the sandbox closed the stream, whatever sat at the path, and its size was checked only afterwards.
    • A link to /dev/zero streamed onto the operator's disk without end, and a FIFO hung the step.
    • A link was refused only because the size probe measured the link itself. An agent could pad the link's target path until the two sizes matched, and so read a small file such as an SSH key through a path the task names.

What changes

  • Manifest names: an entry's name becomes a key with no separators and no . or .. segments, or the file's name if nothing is left. ../../.bashrc is stored as .bashrc, and names that clean to the same key get key (2), key (3) and so on. An entry under base_path, normalized, is named by its path below it, and any other by its file name, as before.

  • Regular files only: the size probe is stat -c %f:%s, links not followed, and the raw mode bits are checked with stat.S_ISREG; %F's type names are translated by locale. A link, a device, a FIFO, a directory or a socket is refused before it's read.

  • CUA path: the size probe refuses anything but a regular file ([ -f ], which follows links as the old wc -c did).

  • Explorer zip: every "Download All" member name is a clean relative path: backslashes count as separators, drive prefixes, leading separators and . segments are dropped, .. climbs at most to the archive root, and a name with nothing left becomes artifact.

  • A flaky recording test: test_a_scripted_drive_is_recorded_on_the_clock_its_screenshots_carry bounded the video's change to a clicked cell by the click's own screenshot. The app repaints the cell some time after the click, so on a busy runner that screenshot can still show it unclicked, and the test failed by a few tenths of a second, on main too. The drive now records the cell's pixel in each screenshot, and the test bounds the change by the first screenshot that shows it, still to one frame on the computer's clock.

Compatibility

  • Links: a link in an agent's output is refused. On main it failed too, unless its target path was padded to its target's size: the size probe measured the link itself, so the integrity check rejected the read.
  • Devices, FIFOs, directories and sockets: they fail before the read instead of during it; a FIFO used to hang the step.
  • Manifest names with a /: logs/run.json is now keyed logs_run.json.
  • Explorer zip names: keys with backslashes, // or ./ get a different member name.

Tests

  • Unit:
    • manifest names lose their separators and dot segments, names that clean to one key keep a key each, and an entry outside base_path is collected and named by its file;
    • an entry's name below base_path is taken with base_path normalized;
    • a link, device, FIFO, directory or socket is refused by its mode bits, and an empty regular file is sized 0;
    • the CUA size probe, run by a real shell, sizes a regular file and refuses a link to /dev/zero, a FIFO or a directory;
    • on a real local sandbox: a link is never read, and a path the task names, inside the work dir or outside it, is read;
    • the explorer smoke checks ../../x, /abs/../../y, backslashes and drive prefixes.
  • On an agent-environments-admin EC2 box, on an earlier version of this branch that had more in it (a size cap, reads bounded to the reported size, a read deadline, Modal streaming, link-following and a local-host check, since dropped); CI covers the current commits:
    • Unit suite: 14,422 passed and 6 failed; five of those also fail on main on that box, with a missing command.
    • Explorer: the smoke suite passed.
    • Local-provider integration: the collect, verify, run_code named-sandbox and bundle-run tests, 25 passed, no skips.
    • Before and after, on real containers and a real local sandbox, each case cut at 20 s:
Case main this branch
report.pdf is a link to /dev/zero 551 MB streamed to the temp dir when cut refused in 0.2 s
a FIFO hung until cut refused in 0.2 s
a link in the work dir to an operator file refused only because stat sized the link itself refused
manifest names ../../.bashrc, ..\..\evil.sh stored as given stored as .bashrc, evil.sh
the explorer's JSZip archive of those keys plus ../../x, /abs/../../y members ../../.bashrc, ..\..\evil.sh, ../../x, abs/../../y members .bashrc, evil.sh, x, y

Not covered here

  • Manifest paths: a manifest entry is read wherever it points, as before. On a local sandbox's host, which is the operator's machine, that includes the operator's own files; the risk is accepted. Only a link to them is refused.
  • No size cap: a regular file is read whatever its size, as before. A large or sparse file the agent leaves is read in full; on providers whose exec hands back the output whole, Modal among them, it is also held in memory.
  • No read deadline: a read that stalls holds the step, as before.
  • Reads aren't bounded to the reported size: a regular file the agent keeps growing during the read, or a base64 it replaced in its container, can stream past the file's size, as before; the integrity check then fails the read.
  • Swapping the file mid-read: the agent can still swap the file between the size probe and the read, as before.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge within its stated scope.

Summary

The PR limits collection to regular files, cleans manifest keys, and keeps archive names relative. It also adjusts the recording test to wait for a screenshot that shows the repaint.

  • Artifact collection reads only regular files.
  • Manifest entries get safe, distinct artifact names.
  • Download All keeps entries within the ZIP root.
  • Desktop recording checks wait for the clicked cell to repaint.

Reviews (17) · Last reviewed commit: "test(computer): name the repaint thresho..." · Reviewed by Greptile

earakely-scale and others added 2 commits November 10, 2026 16:18
… and time

A manifest entry is read only under base_path, and its name is stored as a
key with no separators or dot segments. On a local sandbox's host, which is
this machine, a file is read only where it resolves inside the sandbox's work
dir or base_path; sandboxes declare that folder as app_dir_on_this_machine.

Only regular files are read, links followed, up to the step's max_bytes
(16 GiB by default). The sandbox sends at most the size it reported, the
decoder refuses more, and a read past its deadline is abandoned. Modal exec
output streams instead of being read whole, and the CUA path applies the same
checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each collected key becomes a clean relative member name: backslashes
separate, drive prefixes, leading separators and '.' segments are dropped,
and '..' climbs no higher than the archive root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 3 commits November 10, 2026 16:25
…n, time the whole read

Under /app only the sandbox's work dir counts as a root: an agent could swap a
folder there for a link out. A base_path outside /app still counts, resolved.
Two manifest names that clean to one key get "key (2)" and on. The deadline
covers exec too, which some providers run to completion. Manifest paths are
matched against base_path normalized.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ranslates

stat's %F names the file type in the system's language, so on a host with a
non-English locale every file would have been refused. %f prints the raw mode,
checked with stat.S_ISREG.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collect reads a regular file of any size again, as before this branch. Reads
stay bounded to the size stat reports and to the deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale earakely-scale changed the title fix(collect): read only the agent's own regular files, capped in size and time fix(collect): read only the agent's own regular files, within a deadline Nov 12, 2026
An entry outside base_path is read and named by its file, as before this
branch. On a local sandbox's host the work-dir check still refuses an entry
that resolves outside it, so a manifest can't reach this machine's files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 3 commits November 12, 2026 10:44
… host check for manifests only

Drops what could change an existing run's outcome: the read deadline, Modal
exec output streaming, and following links (stat without -L now refuses a link,
as main's integrity check effectively did). The local-host work-dir check
applies to manifest entries only, so a path the task names is read wherever it
is, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ll path

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ot a new attribute

The manifest check applies where the sandbox declares ON_THIS_MACHINE, and the
allowed folder is /app as the sandbox itself resolves it: a local sandbox maps
/app to its work dir. Sandbox and LocalSandbox are back as they were.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 2 commits November 12, 2026 11:30
…not every VM on this machine

ON_THIS_MACHINE also covers smol and Lume VMs, whose host commands run inside
the VM; their manifests read anywhere in the VM again, as before. The check
applies where the sandbox has a work dir standing for /app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A manifest entry on a local sandbox's host is read wherever it points, as
before; that risk is accepted. Links stay refused in every mode, so a link the
agent leaves still reaches nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale earakely-scale changed the title fix(collect): read only the agent's own regular files, within a deadline fix(collect): read only regular files, and keep manifest names out of paths Nov 12, 2026
earakely-scale and others added 2 commits November 12, 2026 11:39
…reported size

The read is `base64 < path` again, decoded as before. Only the regular-file
check stays in front of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… shows it

The app repaints a clicked cell some time after the click, so the click's own
screenshot can still show it unclicked on a busy runner, and the recording
test failed by a few tenths of a second. The drive now records the cell's pixel
in each screenshot, and the test bounds the video's change by the first
screenshot showing it, still to one frame on the computer's clock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit a79c719 into main Nov 12, 2026
35 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/collect-artifacts-hardening branch November 12, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant