Repository navigation
examples: add bounded coding audits with HF Sandboxes - #1592
Merged
burtenshaw merged 5 commits intoNov 10, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Nested sandbox mkdir may fail
- Sandbox workdirs and reward directories are now created ancestor-first, with strict-parent regression coverage.
Or push these changes by commenting:
@cursor push ec069c8970
Preview (ec069c8970)
diff --git a/examples/coding_audit/hf.py b/examples/coding_audit/hf.py
--- a/examples/coding_audit/hf.py
+++ b/examples/coding_audit/hf.py
@@ -108,6 +108,14 @@
return path == parent or path.startswith(parent + "/")
+def _mkdir(files, path):
+ path = PurePosixPath(path)
+ for directory in reversed((path, *path.parents)):
+ directory = str(directory)
+ if directory != "/" and not files.exists(directory):
+ files.mkdir(directory)
+
+
def _candidate_script(raw):
lines = raw.lstrip().splitlines(keepends=True)
opening = lines[0].strip()
@@ -395,7 +403,7 @@
raise HFBackendError("sandbox control-plane readiness did not converge")
# The SDK stat/read pair otherwise permits up to 512 MiB per read.
sandbox.files.MAX_READ_BYTES = self.max_bytes
- sandbox.files.mkdir(self.workdir)
+ _mkdir(sandbox.files, self.workdir)
return sandbox
def _recover_startup_jobs(self, record, pending):
@@ -1191,7 +1199,7 @@
sandbox.files.write(path, raw)
# Harbor normally mounts this directory. Fresh Sandbox graders
# need it before a native verifier can write its reward file.
- sandbox.files.mkdir(str(PurePosixPath(self.reward_path).parent))
+ _mkdir(sandbox.files, PurePosixPath(self.reward_path).parent)
# A stale image-provided reward must not become new evidence.
if sandbox.files.exists(self.reward_path):
sandbox.files.delete(self.reward_path)
diff --git a/examples/coding_audit/test_hf.py b/examples/coding_audit/test_hf.py
--- a/examples/coding_audit/test_hf.py
+++ b/examples/coding_audit/test_hf.py
@@ -171,6 +171,8 @@
self.writes.append((path, data, mode))
def mkdir(self, path):
+ if str(PurePosixPath(path).parent) not in self.directories:
+ raise SandboxError("missing parent", status_code=404)
self.directories.add(path)
def delete(self, path):You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit c1fc62a. Configure here.
Read original task files through the descriptor-based reader before writing the frozen copy, so copying cannot erase hard-link evidence. Cover context, instruction, verifier and reference artifacts through the runner with offline SDK fakes.
Require a nonempty tool-call list when the provider finishes with tool_calls. Exercise null, missing and empty calls through collection and scoring after a valid terminal action, and verify failed targets are never graded.
Include the repository root in PYTHONPATH so pytest can import the examples package. Verified the documented command with all 417 offline tests passing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Summary
Adds
examples/coding_auditand its standalone CLI. The scorer checks exact reward thresholds and task/verifier/image/execution bindings. The HF backend keeps inference in the controller, transfers only declared submission files to fresh graders, and verifies terminal cleanup. Candidates can opt into declared, bounded public starter files with recorded hashes. Generated references remain candidates; target difficulty is measured separately from verifier reliability.This is operator-side measurement under RFC 008, based on #1591. It adds no installed validation API. The complete Hub workflow's Git-pinned Harbor installation also depends on #1304, JianYan11's FastMCP 4 migration; that migration is an external dependency.
HF comparison: execution uses
HfApi,SandboxandInferenceClient, with explicit image, flavor and timeout settings corresponding tohf jobsconventions. Provider selection is explicit; model source revisions do not attest provider-served weights. The tested HF SDK is 1.33.0.Evidence
After: 407 offline scoring, collector, actual-SDK serialization, stream completion, context binding, deadline, redaction and cleanup tests pass. Commands are in the example README.
Merge Danger
Door: two-way
Blast Radius: examples
Invoking the live backend launches paid services. Use reviewed tasks: Sandbox RPC exposes its execution bearer to workload-controlled code, and separate verifier transfer does not establish hostile-code isolation. The report explicitly leaves full RFC qualification and trainability unestablished.