Skip to content

feat(validation): run Harbor validation trials on HF Sandbox - #1611

Merged
burtenshaw merged 9 commits into
mainfrom
ben/harbor-hf-sandbox-validation
Nov 12, 2026
Merged

burtenshaw merged 9 commits into
mainfrom
ben/harbor-hf-sandbox-validation

Conversation

@burtenshaw

@burtenshaw burtenshaw commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

openenv validate can now run a native Harbor task's validation trials on Hugging Face Jobs instead of the local Docker daemon:

openenv validate path/to/task --sandbox hf-sandbox --flavor cpu-upgrade --namespace my-org

Until now the harbor-semantic-v1 binding could run only where a Linux Docker daemon was available. It hard-coded Harbor's Docker environment, used the local docker CLI for preflight, image identity and cleanup, and turned any provider into a runtime.startup SKIP. The one cloud option, --space, covers served Spaces and refuses every launch with immutable_image.

Harbor 0.25.0, which the binding pins, already ships an hf-sandbox environment: each trial runs in a dedicated HF Job through huggingface_hub.Sandbox. OpenEnv's Harbor rollout code already offers it, and this PR lets validation use it too. A task that pins environment.docker_image by digest declares its own immutable image, so the immutable_image blocker that stops --space does not apply.

Part of RFC 008 (#778). Builds on the Harbor binding (#1591) and digest-pinned task images (#1607).

What changes

  • CLI and API. New --sandbox {docker,hf-sandbox} (default docker) and --namespace. --flavor now also applies to --sandbox hf-sandbox. run_validation gains matching sandbox, flavor and namespace keywords.
  • Trials. On hf-sandbox, every trial (do-nothing, oracle, oracle replay and each canary) is its own HF Job started from the task's pinned image. Nothing is built, so all trials start at once, on a thread pool sized to the trial count. The Docker path is unchanged.
  • Ownership, identity and cleanup. HFJobsInspector takes the place of DockerInspector on this path.
    • Every job carries openenv-validation-run and openenv-validation-role labels. Harbor's environment does not forward labels or namespace to Sandbox.create, so the binding replaces start on the trial's environment instance, the same per-instance pattern it already uses for Compose.
    • Before allocation, no job may already carry the trial's labels.
    • The actual Sandbox job ID is registered before teardown, including failed setup and cancelled startup. At AGENT_START, that job is read back and must run the pinned image on the requested flavor in the RUNNING stage.
    • After each trial, any known or labelled job still running is cancelled, and every one must reach a terminal stage within 60 seconds. Known IDs omitted from list_jobs are checked through inspect_job; listing absence alone never proves termination. All jobs are rechecked against the labels and namespace before cancellation. Pending creation is drained through cancellation, including overlapping Harbor and validator timeouts, before cleanup runs.
  • Refusals. A task without a pinned image (a sandbox cannot build a Dockerfile), or one that sets an agent or verifier user (the sandbox cannot switch users), gets a named runtime.startup SKIP. A flavor smaller than the contract's CPU, memory or disk gives runtime.startup ERROR before any job starts. A served package with --sandbox hf-sandbox gets a named SKIP.
  • Report. Check IDs, graders and thresholds are unchanged. runtime.startup.measured adds sandbox, flavor, namespace and job_ids. harbor-evidence.json adds sandbox, hf_jobs and a limitations entry, and each trial's container_id holds its job ID. No image is retained on this path.
  • Docs. The CLI reference, the Harbor binding section of RFC 008 and item 6 of the RFC 008 completion plan.

Hugging Face ecosystem

--flavor and --namespace match hf jobs run --flavor / --namespace and huggingface_hub.Sandbox.create(flavor=..., namespace=...), with the same names and defaults (cpu-basic, and the token's user). The value hf-sandbox matches Harbor's EnvironmentType.HF_SANDBOX and the sandbox names in OpenEnv's Harbor rollout code. Jobs are created with Sandbox.create, found with HfApi.list_jobs(labels=...) and stopped with HfApi.cancel_job. No new Hub behavior is introduced.

Limits

  • The image must already be in a registry HF Jobs can pull from. Build and push it first, then pin its digest.
  • The image needs /bin/bash, which Harbor uses to run commands, and sha256sum or openssl, which the sandbox server uses to check its own download.
  • The image identity is the digest requested from HF Jobs and read back from the job, not a digest measured inside the sandbox. The evidence says so.
  • The huggingface_hub Sandbox API is marked experimental. The binding already requires harbor==0.25.0, and the start replacement relies on that version's HFSandboxEnvironment.
  • Served Spaces stay blocked until the Space-to-digest binding in item 6 of the completion plan lands.
  • Admission code that requires Docker-only evidence fields, such as retained_image_tag or a 64-hex container_id, needs a rule for HF evidence before it can accept these reports.
  • The namespace's concurrent HF Jobs limit bounds throughput. The validator does not query it.

Test plan and results

  • Original feature baseline: tests/test_validation and tests/test_cli, 2,218 passed, 66 skipped.
  • Earlier review at 95050bc63: installed-wheel fast suite, 1,872 passed with zero skips under --require-complete; every evidence checksum verifies. The recorded source diff matches that commit, and the installed Harbor binding and runner matched those sources by SHA-256.
  • Latest merge review: focused Harbor, supervision and runner suite, 156 passed. Repository lint hook passes. Startup that raises without a sandbox handle now leaves cleanup unverified unless its labelled job is recovered and confirmed terminal. A bounded wait preserves the cancellation regression and addresses CodeQL's no-effect comment. Flavor resource checks use contract MiB, with exact-capacity and one-MiB-over boundary coverage for RAM and disk.
  • New regressions fail before their fixes and cover delayed creation after one or two cancellations, jobs omitted from listings, unavailable job readback, setup failure with failed teardown and an empty listing, and invalid Python API backend names. Unknown backends now raise ValueError before any validation or allocation.
  • The generic full-repository test hook requires the unavailable timeout command on this macOS host; the focused and pinned suites above completed successfully.
  • New fake-boundary tests cover a full HF run (all five trials overlap; job config, labels and evidence), the refusals, the flavor checks, an image readback mismatch, cleanup with and without a working cancel, the ownership recheck, and eleven trials against a two-CPU default thread pool.
  • One test runs the labelled start against Harbor's real HFSandboxEnvironment with a fake Sandbox.create.
  • CLI tests cover the flag conflicts, including a running URL given as the path, and the served-package SKIP.
  • Mutation checks: removing the ownership recheck, the concurrent start, the image readback or the dedicated thread pool each fails its test.

Live qualification on HF Jobs

The four-task smoke set from #1591, each task pinned by digest to a public python image, run with --sandbox hf-sandbox --flavor cpu-basic:

Task Verdict What decided it Wall time
Correct verifier with a wrong-answer canary WARN All six implemented checks pass; WARN comes from checks this binding does not implement 52 s
Verifier that always rewards FAIL semantic.floor_gap: the do-nothing reward is 1.0 38 s
Weak verifier FAIL semantic.canary_floor: the wrong answer earns 1.0 25 s
No solution/solve.sh FAIL semantic.oracle_max, with no jobs started 2 s

The three runs that started jobs created 12 labelled jobs. All 12 ended CANCELED, and every evidence bundle's checksums verify. Harbor's own teardown stopped every job, so cleanup never had to cancel one. The correct-verifier task was rerun on the original feature head with the same result in 46 s. The review fixes were validated with fake-boundary and installed-wheel tests; no additional paid jobs were launched.

🤖 Generated with Claude Code

burtenshaw and others added 2 commits November 12, 2026 14:34
`openenv validate <task> --sandbox hf-sandbox [--flavor F] [--namespace N]`
runs each harbor-semantic-v1 trial (do-nothing, oracle, oracle replay and
canaries) as its own Hugging Face Job through Harbor's hf-sandbox environment,
instead of on the local Docker daemon. The task must pin
environment.docker_image by digest, so nothing is built and every trial starts
at once.

HFJobsInspector labels each job with the run and trial role, reads it back at
AGENT_START (pinned image, requested flavor, RUNNING), and after each trial
cancels any labelled job still running and verifies that all of them stop.
Check ids, graders and thresholds are unchanged. Docker validation behaves as
before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@burtenshaw burtenshaw added feature size: large Large pull request labels Nov 12, 2026 — with Cursor
burtenshaw and others added 3 commits November 12, 2026 15:18
Drain sandbox creation before teardown, register the actual job handle before setup failure can clear it, and inspect known jobs omitted from listings before verifying cleanup. Reject unsupported sandbox names at the Python API boundary.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
Harbor and validator timeout supervisors can both cancel startup. Preserve the pending worker and its eventual job handle through repeated cancellation, then propagate cancellation for owned teardown.

Co-Authored-By: OpenAI Codex <noreply@openai.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: Flavor check uses wrong memory units
    • Converted advertised HF Jobs RAM and disk from GB to MiB using 1024 before comparing them with the contract fields.
Preview (6d1c97182d)
diff --git a/docs/plans/rfc008-validation-completion.md b/docs/plans/rfc008-validation-completion.md
--- a/docs/plans/rfc008-validation-completion.md
+++ b/docs/plans/rfc008-validation-completion.md
@@ -43,7 +43,8 @@ Three limits apply across bindings:
 - **Providers.** Docker-local enforces `public` and `no-network` networking and
   refuses allowlists and GPUs. The HF Sandbox provider refuses every launch with
   `missing provider capability: immutable_image`, because nothing yet binds a Space
-  revision to an image digest.
+  revision to an image digest. Harbor tasks with a digest-pinned image can run
+  their trials on HF Jobs with `--sandbox hf-sandbox`.
 - **Served oracles.** RFC 008 does not yet say how to execute an oracle against a
   served WebSocket environment, so served packages report their semantic checks as
   named SKIPs.

@@ -43,7 +43,8 @@ Three limits apply across bindings:
 - **Providers.** Docker-local enforces `public` and `no-network` networking and
   refuses allowlists and GPUs. The HF Sandbox provider refuses every launch with
   `missing provider capability: immutable_image`, because nothing yet binds a Space
-  revision to an image digest.
+  revision to an image digest. Harbor tasks with a digest-pinned image can run
+  their trials on HF Jobs with `--sandbox hf-sandbox`.
 - **Served oracles.** RFC 008 does not yet say how to execute an oracle against a
   served WebSocket environment, so served packages report their semantic checks as
   named SKIPs.
@@ -92,7 +93,7 @@ or GPU time. Owners are suggestions for discussion.
 | 3 | Verifier strength | #28, #26 procedure | Harbor canaries done; panel open | M, then L | Validation core and Hub pipeline |
 | 4 | Determinism, replay and leakage graders | #30, #39, #40, #44 | Open | M | Validation core |
 | 5 | Static graders and complete reports | #1, #3, #4, #8, #9, #10, #12, #41, #42 | Reports done; graders open | S to M | Any contributor |
-| 6 | Cloud execution on HF Sandbox | Provider qualification | Open | M | Hub pipeline |
+| 6 | Cloud execution on HF Sandbox | Provider qualification | Harbor done; Spaces open | M | Hub pipeline |
 | 7 | Statistical reference procedure | #13 to #16, #27 | Open | L | Hub pipeline |
 | 8 | Wider task shapes | Workspace handoff, multi-container, credentials | Open | M to L | Hub pipeline and validation core |
 | 9 | Training signal | #17 | Open | L | Validation experiments and RFC 013 |

@@ -92,7 +93,7 @@ or GPU time. Owners are suggestions for discussion.
 | 3 | Verifier strength | #28, #26 procedure | Harbor canaries done; panel open | M, then L | Validation core and Hub pipeline |
 | 4 | Determinism, replay and leakage graders | #30, #39, #40, #44 | Open | M | Validation core |
 | 5 | Static graders and complete reports | #1, #3, #4, #8, #9, #10, #12, #41, #42 | Reports done; graders open | S to M | Any contributor |
-| 6 | Cloud execution on HF Sandbox | Provider qualification | Open | M | Hub pipeline |
+| 6 | Cloud execution on HF Sandbox | Provider qualification | Harbor done; Spaces open | M | Hub pipeline |
 | 7 | Statistical reference procedure | #13 to #16, #27 | Open | L | Hub pipeline |
 | 8 | Wider task shapes | Workspace handoff, multi-container, credentials | Open | M to L | Hub pipeline and validation core |
 | 9 | Training signal | #17 | Open | L | Validation experiments and RFC 013 |
@@ -208,9 +209,14 @@ contributor; these make good first issues.
 ### 6. Cloud execution on HF Sandbox
 
 The HF Sandbox provider refuses every launch because it has no immutable image
-binding. Runtime and semantic checks therefore cannot run in the cloud, or on GPUs,
-through the RFC provider interface. Cloud audits work around this with their own
-runner.
+binding. Runtime and semantic checks for served Spaces therefore cannot run in the
+cloud, or on GPUs, through the RFC provider interface.
+
+Harbor tasks do not have this gap when they pin `environment.docker_image` by
+digest, because the task declares its own immutable image. `openenv validate
+--sandbox hf-sandbox` now runs their trials as HF Jobs through Harbor's
+`hf-sandbox` environment, so cloud audits can call `openenv validate` instead of
+their own runner. The remaining work below covers served Spaces.
 
 **Approach.** Resolve a Space revision to the registry digest of the image it
 builds, verify it, and launch by digest. Qualify public Space images first, then add

@@ -208,9 +209,14 @@ contributor; these make good first issues.
 ### 6. Cloud execution on HF Sandbox
 
 The HF Sandbox provider refuses every launch because it has no immutable image
-binding. Runtime and semantic checks therefore cannot run in the cloud, or on GPUs,
-through the RFC provider interface. Cloud audits work around this with their own
-runner.
+binding. Runtime and semantic checks for served Spaces therefore cannot run in the
+cloud, or on GPUs, through the RFC provider interface.
+
+Harbor tasks do not have this gap when they pin `environment.docker_image` by
+digest, because the task declares its own immutable image. `openenv validate
+--sandbox hf-sandbox` now runs their trials as HF Jobs through Harbor's
+`hf-sandbox` environment, so cloud audits can call `openenv validate` instead of
+their own runner. The remaining work below covers served Spaces.
 
 **Approach.** Resolve a Space revision to the registry digest of the image it
 builds, verify it, and launch by digest. Qualify public Space images first, then add

diff --git a/docs/source/reference/cli.md b/docs/source/reference/cli.md
--- a/docs/source/reference/cli.md
+++ b/docs/source/reference/cli.md
@@ -106,7 +106,8 @@ of strings. This is a minimal complete contract:

The harbor-semantic-v1 binding runs
-fresh Harbor trials in Docker: a do-nothing agent, the oracle (solution/solve.sh),
+fresh Harbor trials, in local Docker unless --sandbox hf-sandbox moves them to
+HF Jobs: a do-nothing agent, the oracle (solution/solve.sh),
an oracle replay and any canaries declared in the contract. A canary is a known-wrong
solve.sh replacement. The results report under runtime.startup,
runtime.reward_well_formed, runtime.episode_determinism, semantic.oracle_max,

@@ -106,7 +106,8 @@ of strings. This is a minimal complete contract:


The `harbor-semantic-v1` binding runs
-fresh Harbor trials in Docker: a do-nothing agent, the oracle (`solution/solve.sh`),
+fresh Harbor trials, in local Docker unless `--sandbox hf-sandbox` moves them to
+HF Jobs: a do-nothing agent, the oracle (`solution/solve.sh`),
an oracle replay and any canaries declared in the contract. A canary is a known-wrong
`solve.sh` replacement. The results report under `runtime.startup`,
`runtime.reward_well_formed`, `runtime.episode_determinism`, `semantic.oracle_max`,
@@ -126,7 +127,7 @@ Dockerfile nor a pinned image, task-authored Compose files and separate verifier
environments. GPU/TPU or external MCP requirements,
credentials, non-public or differing phase networks, task environment variables,
healthchecks, skills, verifier collection and artifact extensions are also
-unsupported. A supplied provider, including `--space`, is unsupported. Each case
+unsupported. A served-package provider, including `--space`, is unsupported. Each case
gives `runtime.startup` SKIP with a named reason and runs no trials. With valid
static declarations and an oracle, the verdict is WARN and the exit code is 0.
A missing `solution/solve.sh` still FAILs `semantic.oracle_max` at the semantic level,

@@ -126,7 +127,7 @@ Dockerfile nor a pinned image, task-authored Compose files and separate verifier
environments. GPU/TPU or external MCP requirements,
credentials, non-public or differing phase networks, task environment variables,
healthchecks, skills, verifier collection and artifact extensions are also
-unsupported. A supplied provider, including `--space`, is unsupported. Each case
+unsupported. A served-package provider, including `--space`, is unsupported. Each case
gives `runtime.startup` SKIP with a named reason and runs no trials. With valid
static declarations and an oracle, the verdict is WARN and the exit code is 0.
A missing `solution/solve.sh` still FAILs `semantic.oracle_max` at the semantic level,
@@ -146,6 +147,38 @@ is removed on every run. Tag removal verifies image identity and uses `--no-prun
A removal failure marks cleanup incomplete and gives a startup ERROR. During an
abort, cleanup preserves the original exception and records the removal failure.

+`--sandbox hf-sandbox` runs the same trials on Hugging Face Jobs instead, one job
+per trial, through Harbor's `hf-sandbox` environment:
+
+```bash
+openenv validate path/to/task --sandbox hf-sandbox
+openenv validate path/to/task --sandbox hf-sandbox --flavor cpu-upgrade --namespace my-org
+```
+
+A sandbox cannot build a Dockerfile, so the task must pin a pullable image as
+`environment.docker_image = "name@sha256:<digest>"`. Build and push the image
+first, then pin its digest. The image needs `/bin/bash`, which Harbor uses to run
+commands, and either `sha256sum` or `openssl`, which the sandbox server uses to
+verify its own download. With nothing
+to build, every trial starts at once. `--flavor` and `--namespace` mean what they
+mean for `hf jobs run`: the flavor sets the hardware, and the namespace is the user
+or organization that owns and pays for the jobs, by default the token's user. The
+flavor must provide the CPU, memory and disk the contract declares, or
+`runtime.startup` reports ERROR before any job starts. Tasks that set an agent or
+verifier user are unsupported, because the sandbox cannot switch users.
+
+Each job carries `openenv-validation-run` and `openenv-validation-role` labels. At
+`AGENT_START` the validator reads the job back and requires the pinned image, the
+requested flavor and a `RUNNING` stage. After each trial it cancels any labelled
+job that is still running and waits up to 60 seconds for all of them to stop.
+Otherwise cleanup is unverified and `runtime.startup` reports ERROR.
+`runtime.startup.measured` records the flavor, namespace and job IDs, and
+`harbor-evidence.json` states that the image is the digest requested from HF Jobs,
+not one measured inside the sandbox. Check IDs, graders and thresholds are the
+same as on Docker, and no image is retained. A served package gives a named
+`runtime.startup` SKIP with `--sandbox hf-sandbox`; validate a served Space with
+`--space` instead.
+
The 16 implemented runtime checks cover startup, rewards, observation schemas,
state continuity, network policy, host containment, resource bounds, episode isolation,
oracle containment, seed forwarding, determinism, emitted trajectory records, tool

@@ -146,6 +147,38 @@ is removed on every run. Tag removal verifies image identity and uses `--no-prun
A removal failure marks cleanup incomplete and gives a startup ERROR. During an
abort, cleanup preserves the original exception and records the removal failure.

+`--sandbox hf-sandbox` runs the same trials on Hugging Face Jobs instead, one job
+per trial, through Harbor's `hf-sandbox` environment:
+
+```bash
+openenv validate path/to/task --sandbox hf-sandbox
+openenv validate path/to/task --sandbox hf-sandbox --flavor cpu-upgrade --namespace my-org
+```
+
+A sandbox cannot build a Dockerfile, so the task must pin a pullable image as
+`environment.docker_image = "name@sha256:<digest>"`. Build and push the image
+first, then pin its digest. The image needs `/bin/bash`, which Harbor uses to run
+commands, and either `sha256sum` or `openssl`, which the sandbox server uses to
+verify its own download. With nothing
+to build, every trial starts at once. `--flavor` and `--namespace` mean what they
+mean for `hf jobs run`: the flavor sets the hardware, and the namespace is the user
+or organization that owns and pays for the jobs, by default the token's user. The
+flavor must provide the CPU, memory and disk the contract declares, or
+`runtime.startup` reports ERROR before any job starts. Tasks that set an agent or
+verifier user are unsupported, because the sandbox cannot switch users.
+
+Each job carries `openenv-validation-run` and `openenv-validation-role` labels. At
+`AGENT_START` the validator reads the job back and requires the pinned image, the
+requested flavor and a `RUNNING` stage. After each trial it cancels any labelled
+job that is still running and waits up to 60 seconds for all of them to stop.
+Otherwise cleanup is unverified and `runtime.startup` reports ERROR.
+`runtime.startup.measured` records the flavor, namespace and job IDs, and
+`harbor-evidence.json` states that the image is the digest requested from HF Jobs,
+not one measured inside the sandbox. Check IDs, graders and thresholds are the
+same as on Docker, and no image is retained. A served package gives a named
+`runtime.startup` SKIP with `--sandbox hf-sandbox`; validate a served Space with
+`--space` instead.
+
The 16 implemented runtime checks cover startup, rewards, observation schemas,
state continuity, network policy, host containment, resource bounds, episode isolation,
oracle containment, seed forwarding, determinism, emitted trajectory records, tool

diff --git a/rfcs/008-environment-auto-validation.md b/rfcs/008-environment-auto-validation.md
--- a/rfcs/008-environment-auto-validation.md
+++ b/rfcs/008-environment-auto-validation.md
@@ -756,8 +756,8 @@ an `environment.docker_image` without a digest, tasks with neither a Dockerfile
a pinned image, task-authored Compose files and separate verifier environments.
GPU/TPU or external MCP requirements, credentials, non-public or differing phase
networks, task environment variables, healthchecks, skills, verifier collection and
-artifact extensions are also unsupported. A supplied provider, including `--space`,
-is unsupported. Each case gives `runtime.startup` SKIP with a named reason and runs
+artifact extensions are also unsupported. A served-package provider, including
+`--space`, is unsupported. Each case gives `runtime.startup` SKIP with a named reason and runs
no trials. With valid static declarations and an oracle, the verdict is WARN and
the exit code is 0. A missing `solution/solve.sh` still FAILs `semantic.oracle_max`
at the semantic level, including with `--skip-build` or `--space`.

@@ -756,8 +756,8 @@ an `environment.docker_image` without a digest, tasks with neither a Dockerfile
a pinned image, task-authored Compose files and separate verifier environments.
GPU/TPU or external MCP requirements, credentials, non-public or differing phase
networks, task environment variables, healthchecks, skills, verifier collection and
-artifact extensions are also unsupported. A supplied provider, including `--space`,
-is unsupported. Each case gives `runtime.startup` SKIP with a named reason and runs
+artifact extensions are also unsupported. A served-package provider, including
+`--space`, is unsupported. Each case gives `runtime.startup` SKIP with a named reason and runs
no trials. With valid static declarations and an oracle, the verdict is WARN and
the exit code is 0. A missing `solution/solve.sh` still FAILs `semantic.oracle_max`
at the semantic level, including with `--skip-build` or `--space`.
@@ -792,6 +792,20 @@ containers' actual mounts, and verifies their absence. The oracle's exit code is
read at `AGENT_END`, before sandbox log synchronization can overwrite it, and a
later change fails the run.

+With `--sandbox hf-sandbox`, each trial runs as its own Hugging Face Job through
+Harbor's `hf-sandbox` environment, on the flavor and in the namespace the operator
+names. The task must declare a digest-pinned `environment.docker_image`. Tasks that
+only ship a Dockerfile, and tasks that set agent or verifier users, give
+`runtime.startup` SKIP. A flavor smaller than the contract's resources gives
+`runtime.startup` ERROR before any job starts. All trials start at once from the
+pinned image. Every job carries run and role labels. At `AGENT_START` the validator
+reads the job back and requires the pinned image, the requested flavor and a
+`RUNNING` stage. After each trial it cancels any labelled job still running and
+verifies that every one reaches a terminal stage. The image identity is the digest
+requested from HF Jobs and read back from the job, not a digest measured inside the
+sandbox, and the evidence states this limitation. Check ids, graders and thresholds
+are unchanged.
+
For `harbor-semantic-v1`, one trial is a one-shot episode with no per-step
observations. A fresh oracle replay that reproduces every reward and the oracle
exit code establishes `runtime.episode_determinism`. This rule is specific to the

@@ -792,6 +792,20 @@ containers' actual mounts, and verifies their absence. The oracle's exit code is
read at `AGENT_END`, before sandbox log synchronization can overwrite it, and a
later change fails the run.

+With `--sandbox hf-sandbox`, each trial runs as its own Hugging Face Job through
+Harbor's `hf-sandbox` environment, on the flavor and in the namespace the operator
+names. The task must declare a digest-pinned `environment.docker_image`. Tasks that
+only ship a Dockerfile, and tasks that set agent or verifier users, give
+`runtime.startup` SKIP. A flavor smaller than the contract's resources gives
+`runtime.startup` ERROR before any job starts. All trials start at once from the
+pinned image. Every job carries run and role labels. At `AGENT_START` the validator
+reads the job back and requires the pinned image, the requested flavor and a
+`RUNNING` stage. After each trial it cancels any labelled job still running and
+verifies that every one reaches a terminal stage. The image identity is the digest
+requested from HF Jobs and read back from the job, not a digest measured inside the
+sandbox, and the evidence states this limitation. Check ids, graders and thresholds
+are unchanged.
+
For `harbor-semantic-v1`, one trial is a one-shot episode with no per-step
observations. A fresh oracle replay that reproduces every reward and the oracle
exit code establishes `runtime.episode_determinism`. This rule is specific to the

diff --git a/src/openenv/cli/commands/validate.py b/src/openenv/cli/commands/validate.py
--- a/src/openenv/cli/commands/validate.py
+++ b/src/openenv/cli/commands/validate.py
@@ -223,9 +223,29 @@ def validate(
        str,
        typer.Option(
            "--flavor",
-            help="HF Jobs hardware flavor for --space, e.g. cpu-basic or t4-small",
+            help=(
+                "HF Jobs hardware flavor for --space or --sandbox hf-sandbox, "
+                "e.g. cpu-basic or t4-small"
+            ),
        ),
    ] = "cpu-basic",
+    sandbox: Annotated[
+        str,
+        typer.Option(
+            "--sandbox",
+            help=(
+                "Where Harbor task trials run: docker (local daemon) or "
+                "hf-sandbox (one HF Job per trial)"
+            ),
+        ),
+    ] = "docker",
+    namespace: Annotated[
+        str | None,
+        typer.Option(
+            "--namespace",
+            help="User or organization that owns the --sandbox hf-sandbox jobs",
+        ),
+    ] = None,
) -> None:
    """
    Validate a local package or a running OpenEnv server.

@@ -223,9 +223,29 @@ def validate(
        str,
        typer.Option(
            "--flavor",
-            help="HF Jobs hardware flavor for --space, e.g. cpu-basic or t4-small",
+            help=(
+                "HF Jobs hardware flavor for --space or --sandbox hf-sandbox, "
+                "e.g. cpu-basic or t4-small"
+            ),
        ),
    ] = "cpu-basic",
+    sandbox: Annotated[
+        str,
+        typer.Option(
+            "--sandbox",
+            help=(
+                "Where Harbor task trials run: docker (local daemon) or "
+                "hf-sandbox (one HF Job per trial)"
+            ),
+        ),
+    ] = "docker",
+    namespace: Annotated[
+        str | None,
+        typer.Option(
+            "--namespace",
+            help="User or organization that owns the --sandbox hf-sandbox jobs",
+        ),
+    ] = None,
) -> None:
    """
    Validate a local package or a running OpenEnv server.
@@ -236,9 +256,11 @@ def validate(
    applies the severity policy, and emits a report.

    Native Harbor tasks require the optional Harbor dependency and an explicit
-    `validation/openenv.json` contract. Fresh Docker trials grade the oracle, the
-    do-nothing floor and any declared canaries under the semantic check ids. Checks
-    without an implementation are named SKIPs, so the report is WARN at best.
+    `validation/openenv.json` contract. Fresh trials grade the oracle, the
+    do-nothing floor and any declared canaries under the semantic check ids. They
+    run on the local Docker daemon, or with `--sandbox hf-sandbox` as one Hugging
+    Face Job per trial from the task's digest-pinned image. Checks without an
+    implementation are named SKIPs, so the report is WARN at best.

    Exit codes: 0 pass/warn · 1 fail · 2 unrecognized/unsupported package · 3
    internal or policy error.

@@ -236,9 +256,11 @@ def validate(
    applies the severity policy, and emits a report.

    Native Harbor tasks require the optional Harbor dependency and an explicit
-    `validation/openenv.json` contract. Fresh Docker trials grade the oracle, the
-    do-nothing floor and any declared canaries under the semantic check ids. Checks
-    without an implementation are named SKIPs, so the report is WARN at best.
+    `validation/openenv.json` contract. Fresh trials grade the oracle, the
+    do-nothing floor and any declared canaries under the semantic check ids. They
+    run on the local Docker daemon, or with `--sandbox hf-sandbox` as one Hugging
+    Face Job per trial from the task's digest-pinned image. Checks without an
+    implementation are named SKIPs, so the report is WARN at best.

    Exit codes: 0 pass/warn · 1 fail · 2 unrecognized/unsupported package · 3
    internal or policy error.
@@ -260,15 +282,38 @@ def validate(

    # Validate a Hub Space on HF Jobs GPU hardware
    openenv validate --space openenv/echo_env --level runtime --flavor t4-small
+
+    # Run a Harbor task's trials as HF Jobs instead of local Docker
+    openenv validate path/to/task --sandbox hf-sandbox --flavor cpu-upgrade
    ```
    """
    if space is not None and (target is not None or url is not None or local):
        typer.echo(
            "Error: --space cannot be combined with a path, --url or --local", err=True
        )
        raise typer.Exit(EXIT_FAIL)
-    if space is None and (revision is not None or flavor != "cpu-basic"):
-        typer.echo("Error: --revision and --flavor require --space", err=True)
+    if sandbox not in ("docker", "hf-sandbox"):
+        typer.echo(
+            f"Error: unknown sandbox {sandbox!r}; expected docker or hf-sandbox",
+            err=True,
+        )
+        raise typer.Exit(EXIT_FAIL)
+    hf_sandbox = sandbox == "hf-sandbox"
+    running_url = url is not None or (target is not None and _looks_like_url(target))
+    if hf_sandbox and (space is not None or running_url):
+        typer.echo(
+            "Error: --sandbox cannot be combined with --space or a running URL",
+            err=True,
+        )
+        raise typer.Exit(EXIT_FAIL)
+    if space is None and revision is not None:
+        typer.echo("Error: --revision requires --space", err=True)
+        raise typer.Exit(EXIT_FAIL)
+    if space is None and not hf_sandbox and flavor != "cpu-basic":
+        typer.echo("Error: --flavor requires --space or --sandbox hf-sandbox", err=True)
+        raise typer.Exit(EXIT_FAIL)
+    if namespace is not None and not hf_sandbox:
+        typer.echo("Error: --namespace requires --sandbox hf-sandbox", err=True)
        raise typer.Exit(EXIT_FAIL)

    runtime_target = url

@@ -260,15 +282,38 @@ def validate(

    # Validate a Hub Space on HF Jobs GPU hardware
    openenv validate --space openenv/echo_env --level runtime --flavor t4-small
+
+    # Run a Harbor task's trials as HF Jobs instead of local Docker
+    openenv validate path/to/task --sandbox hf-sandbox --flavor cpu-upgrade
    ```
    """
    if space is not None and (target is not None or url is not None or local):
        typer.echo(
            "Error: --space cannot be combined with a path, --url or --local", err=True
        )
        raise typer.Exit(EXIT_FAIL)
-    if space is None and (revision is not None or flavor != "cpu-basic"):
-        typer.echo("Error: --revision and --flavor require --space", err=True)
+    if sandbox not in ("docker", "hf-sandbox"):
+        typer.echo(
+            f"Error: unknown sandbox {sandbox!r}; expected docker or hf-sandbox",
+            err=True,
+        )
+        raise typer.Exit(EXIT_FAIL)
+    hf_sandbox = sandbox == "hf-sandbox"
+    running_url = url is not None or (target is not None and _looks_like_url(target))
+    if hf_sandbox and (space is not None or running_url):
+        typer.echo(
+            "Error: --sandbox cannot be combined with --space or a running URL",
+            err=True,
+        )
+        raise typer.Exit(EXIT_FAIL)
+    if space is None and revision is not None:
+        typer.echo("Error: --revision requires --space", err=True)
+        raise typer.Exit(EXIT_FAIL)
+    if space is None and not hf_sandbox and flavor != "cpu-basic":
+        typer.echo("Error: --flavor requires --space or --sandbox hf-sandbox", err=True)
+        raise typer.Exit(EXIT_FAIL)
+    if namespace is not None and not hf_sandbox:
+        typer.echo("Error: --namespace requires --sandbox hf-sandbox", err=True)
        raise typer.Exit(EXIT_FAIL)

    runtime_target = url
@@ -367,6 +412,9 @@ def validate(
                report_target=f"spaces/{space}@{revision}"
                if space is not None
                else None,
+                sandbox=sandbox,
+                flavor=flavor,
+                namespace=namespace,
            )
        report_json = write_report(validation_report, output)
    except (SignatureError, UnsupportedPackageError) as exc:

@@ -367,6 +412,9 @@ def validate(
                report_target=f"spaces/{space}@{revision}"
                if space is not None
                else None,
+                sandbox=sandbox,
+                flavor=flavor,
+                namespace=namespace,
            )
        report_json = write_report(validation_report, output)
    except (SignatureError, UnsupportedPackageError) as exc:

diff --git a/src/openenv/validation/harbor.py b/src/openenv/validation/harbor.py
--- a/src/openenv/validation/harbor.py
+++ b/src/openenv/validation/harbor.py
@@ -1,13 +1,15 @@
-"""Native Harbor execution: fresh Docker trials graded through RFC 008 check ids.
+"""Native Harbor execution: fresh trials graded through RFC 008 check ids.

-No served-protocol evidence is synthesized. A do-nothing trial builds the image, or
-pulls the task's digest-pinned one, and the oracle, its replay and any canaries run
-pinned to that image.
+No served-protocol evidence is synthesized. On local Docker, a do-nothing trial
+builds the image, or pulls the task's digest-pinned one, and the oracle, its replay
+and any canaries run pinned to that image. On `hf-sandbox`, every trial starts at
+once from the task's digest-pinned image, each in its own Hugging Face Job.
"""

import asyncio
import hashlib
import json
+import math
import os
import re
import shutil

@@ -1,13 +1,15 @@
-"""Native Harbor execution: fresh Docker trials graded through RFC 008 check ids.
+"""Native Harbor execution: fresh trials graded through RFC 008 check ids.

-No served-protocol evidence is synthesized. A do-nothing trial builds the image, or
-pulls the task's digest-pinned one, and the oracle, its replay and any canaries run
-pinned to that image.
+No served-protocol evidence is synthesized. On local Docker, a do-nothing trial
+builds the image, or pulls the task's digest-pinned one, and the oracle, its replay
+and any canaries run pinned to that image. On `hf-sandbox`, every trial starts at
+once from the task's digest-pinned image, each in its own Hugging Face Job.
"""

import asyncio
import hashlib
import json
+import math
import os
import re
import shutil
@@ -17,6 +19,7 @@
import tempfile
import time
import uuid
+from concurrent.futures import ThreadPoolExecutor
from dataclasses import replace
from decimal import Decimal
from fractions import Fraction

@@ -17,6 +19,7 @@
import tempfile
import time
import uuid
+from concurrent.futures import ThreadPoolExecutor
from dataclasses import replace
from decimal import Decimal
from fractions import Fraction
@@ -58,6 +61,10 @@
    "RetainedImageDrift",
    "RetainedImageUnverified",
}
+_TERMINAL_STAGES = {"COMPLETED", "CANCELED", "ERROR", "DELETED"}
+_JOB_STOP_TIMEOUT_S = 60
+# Readiness budget inside the 600-second start budget; allocation can take longer.
+_SANDBOX_START_TIMEOUT_S = 540
# Checks this binding measures; every other local check is a named SKIP.
_IMPLEMENTED = {
    "runtime.startup",

@@ -58,6 +61,10 @@
    "RetainedImageDrift",
    "RetainedImageUnverified",
}
+_TERMINAL_STAGES = {"COMPLETED", "CANCELED", "ERROR", "DELETED"}
+_JOB_STOP_TIMEOUT_S = 60
+# Readiness budget inside the 600-second start budget; allocation can take longer.
+_SANDBOX_START_TIMEOUT_S = 540
# Checks this binding measures; every other local check is a named SKIP.
_IMPLEMENTED = {
    "runtime.startup",
@@ -276,7 +283,147 @@ def remove_image(self):
            self._retained_image = None


-def _unsupported(task, contract):
+def _job_labels(trial_name):
+    # openenv-validation-<run id>-<role>
+    run_id, role = trial_name.split("-", 3)[2:]
+    return {"openenv-validation-run": run_id, "openenv-validation-role": role}
+
+
+class HFJobsInspector:
+    """
+    Trusted HF Jobs checks limited to sandbox jobs labelled with this run.
+
+    The image identity is the digest requested from HF Jobs and read back from the
+    job; no runtime digest is measured inside the sandbox.
+    """
+
+    def __init__(self, image, flavor, namespace, resources):
+        self.image = image
+        self.flavor = flavor
+        self.namespace = namespace
+        self._resources = resources
+        self._api = None
+        self._observed_jobs = {}
+
+    def preflight(self):
+        from huggingface_hub import HfApi
+
+        self._api = HfApi()
+        if self.namespace is None:
+            self.namespace = self._api.whoami()["name"]
+        hardware = {item.name: item for item in self._api.list_jobs_hardware()}
+        if self.flavor not in hardware:
+            raise NativeEvidenceError(f"unknown HF Jobs flavor {self.flavor}")
+        item = hardware[self.flavor]
+        # The Hub lists specs as "2 vCPU", "16 GB" and "50 GB".
+        cpu, memory_gb, disk_gb = (
+            int(spec.split()[0])
+            for spec in (item.cpu, item.ram, item.ephemeral_storage)
+        )
+        if (
+            cpu < self._resources.cpu
+            or memory_gb * 1024 < self._resources.memory_mb
+            or disk_gb * 1024 < self._resources.disk_mb
+        ):
+            raise NativeEvidenceError(
+                f"HF Jobs flavor {self.flavor} ({item.cpu}, {item.ram} RAM, "
+                + f"{item.ephemeral_storage} disk) is smaller than the contract resources"
+            )
+        return {
+            "flavor": self.flavor,
+            "namespace": self.namespace,
+            "hardware": {
+                "cpu": item.cpu,
+                "ram": item.ram,
+                "disk": item.ephemeral_storage,
+            },
+        }
+
+    def register_job(self, project, job_id):
+        # The Sandbox handle is authoritative even before AGENT_START or listing.
+        self._observed_jobs.setdefault(project, set()).add(job_id)
+
+    def _jobs(self, project):
+        if not _PROJECT.fullmatch(project):
+            raise NativeEvidenceError("not a run-owned Harbor project")
+        labels = _job_labels(project.removesuffix("__env"))
+        jobs = list(self._api.list_jobs(labels=labels, namespace=self.namespace))
+        # A listing omission does not prove a previously observed job terminated.
+        listed_ids = {job.id for job in jobs}
+        known_ids = self._observed_jobs.get(project, set())
+        jobs.extend(
+            self._api.inspect_job(job_id=job_id, namespace=self.namespace)
+            for job_id in sorted(known_ids - listed_ids)
+        )
+        # Cleanup cancels what this returns, so recheck the server-side filter.
+        if any(
+            not labels.items() <= (job.labels or {}).items()
+            or job.owner.name != self.namespace
+            for job in jobs
+        ):
+            raise NativeEvidenceError("HF Jobs returned a job outside this run")
+        self._observed_jobs[project] = known_ids | listed_ids
+        return jobs
+
+    def inventory(self, project):
+        return {"jobs": sorted(job.id for job in self._jobs(project))}
+
+    def capture(self, project, retained_tag, *, retain=True):
+        jobs = self._jobs(project)
+        if len(jobs) != 1:
+            raise NativeEvidenceError("HF Sandbox trial requires exactly one job")
+        job = jobs[0]
+        if (
+            job.status.stage != "RUNNING"
+            or job.docker_image != self.image
+            or job.flavor != self.flavor
+        ):
+            raise NativeEvidenceError(
+                "HF Sandbox job is not running the pinned image on the requested flavor"
+            )
+        configuration = {
+            "image_ref": job.docker_image,
+            "flavor": self.flavor,
+            "namespace": self.namespace,
+        }
+        return {
+            "container_id": job.id,
+            "project": project,
+            "image_ref": job.docker_image,
+            "configuration": configuration,
+            "configuration_digest": _digest(configuration),
+        }
+
+    def cleanup(self, project):
+        # Startup settles allocation before teardown; inspect until each job ends.
+        deadline = time.monotonic() + _JOB_STOP_TIMEOUT_S
+        cancelled = set()
+        while True:
+            stages = {job.id: job.status.stage for job in self._jobs(project)}
+            running = {
+                job_id
+                for job_id, stage in stages.items()
+                if stage not in _TERMINAL_STAGES
+            }
+            for job_id in running - cancelled:
+                self._api.cancel_job(job_id=job_id, namespace=self.namespace)
+                cancelled.add(job_id)
+            if not running or time.monotonic() > deadline:
+                return {
+                    "verified": not running,
+                    "jobs": stages,
+                    "cancelled": sorted(cancelled),
+                }
+            time.sleep(2)
+
+    def verify_image(self, tag, image_ref):
+        return image_ref == self.image
+
+    def remove_image(self):
+        pass  # Nothing is pulled or tagged on this host.
+
+
+def _unsupported(task, contract, sandbox):
    config = task.config
    env = config.environment
    if task.has_steps:

@@ -276,7 +283,147 @@ def remove_image(self):
            self._retained_image = None


-def _unsupported(task, contract):
+def _job_labels(trial_name):
+    # openenv-validation-<run id>-<role>
+    run_id, role = trial_name.split("-", 3)[2:]
+    return {"openenv-validation-run": run_id, "openenv-validation-role": role}
+
+
+class HFJobsInspector:
+    """
+    Trusted HF Jobs checks limited to sandbox jobs labelled with this run.
+
+    The image identity is the digest requested from HF Jobs and read back from the
+    job; no runtime digest is measured inside the sandbox.
+    """
+
+    def __init__(self, image, flavor, namespace, resources):
+        self.image = image
+        self.flavor = flavor
+        self.namespace = namespace
+        self._resources = resources
+        self._api = None
+        self._observed_jobs = {}
+
+    def preflight(self):
+        from huggingface_hub import HfApi
+
+        self._api = HfApi()
+        if self.namespace is None:
+            self.namespace = self._api.whoami()["name"]
+        hardware = {item.name: item for item in self._api.list_jobs_hardware()}
+        if self.flavor not in hardware:
+            raise NativeEvidenceError(f"unknown HF Jobs flavor {self.flavor}")
+        item = hardware[self.flavor]
+        # The Hub lists specs as "2 vCPU", "16 GB" and "50 GB".
+        cpu, memory_gb, disk_gb = (
+            int(spec.split()[0])
+            for spec in (item.cpu, item.ram, item.ephemeral_storage)
+        )
+        if (
+            cpu < self._resources.cpu
+            or memory_gb * 1024 < self._resources.memory_mb
+            or disk_gb * 1024 < self._resources.disk_mb
+        ):
+            raise NativeEvidenceError(
+                f"HF Jobs flavor {self.flavor} ({item.cpu}, {item.ram} RAM, "
+                + f"{item.ephemeral_storage} disk) is smaller than the contract resources"
+            )
+        return {
+            "flavor": self.flavor,
+            "namespace": self.namespace,
+            "hardware": {
+                "cpu": item.cpu,
+                "ram": item.ram,
+                "disk": item.ephemeral_storage,
+            },
+        }
+
+    def register_job(self, project, job_id):
+        # The Sandbox handle is authoritative even before AGENT_START or listing.
+        self._observed_jobs.setdefault(project, set()).add(job_id)
+
+    def _jobs(self, project):
+        if not _PROJECT.fullmatch(project):
+            raise NativeEvidenceError("not a run-owned Harbor project")
+        labels = _job_labels(project.removesuffix("__env"))
+        jobs = list(self._api.list_jobs(labels=labels, namespace=self.namespace))
+        # A listing omission does not prove a previously observed job terminated.
+        listed_ids = {job.id for job in jobs}
+        known_ids = self._observed_jobs.get(project, set())
+        jobs.extend(
+            self._api.inspect_job(job_id=job_id, namespace=self.namespace)
+            for job_id in sorted(known_ids - listed_ids)
+        )
+        # Cleanup cancels what this returns, so recheck the server-side filter.
+        if any(
+            not labels.items() <= (job.labels or {}).items()
+            or job.owner.name != self.namespace
+            for job in jobs
+        ):
+            raise NativeEvidenceError("HF Jobs returned a job outside this run")
+        self._observed_jobs[project] = known_ids | listed_ids
+        return jobs
+
+    def inventory(self, project):
+        return {"jobs": sorted(job.id for job in self._jobs(project))}
+
... diff truncated: showing 800 of 3000 lines

You can send follow-ups to the cloud agent here.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 95050bc. Configure here.

Comment thread src/openenv/validation/harbor.py Resolved
burtenshaw and others added 4 commits November 12, 2026 15:31
Wait for the cancelled startup task with an explicit deadline while preserving the cancellation assertion and avoiding a CodeQL no-effect finding.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
HF Sandbox creation can raise after allocating a job and failing its best-effort cancellation. Record missing startup handles and require recovery of a labelled job before cleanup can be verified. Exercise both recovered and invisible allocations with fakes.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
Accept the advertised memory and storage capacities expressed in contract MiB, and reject one MiB above each limit before job allocation.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
@burtenshaw
burtenshaw merged commit ca6df26 into main Nov 12, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature size: large Large pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants