Repository navigation
feat(validation): grade Harbor tasks with RFC 008 semantic checks; add coding-audit examples - #1591
Conversation
Ported onto main from the Hub validation pipeline branch. Keeps main's duplicate-key handling in runtime/json_values.py (e4ca57b) instead of this commit's helper, and drops this commit's duplicate-key test, which main's artifact tests already cover. (cherry picked from commit f60512b5ec0f808e8657edbbb052fa67cfc35397)
Static runs reported only static.manifest, so a valid package read as PASS while nine static checks never ran. Every requested local check without a grader is now a named SKIP at every level, which makes such runs WARN (exit code 0). Served packages also state why semantic checks skip: oracle execution is not implemented for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The native Harbor binding now reports its oracle and floor results under the RFC check ids instead of folding them into runtime.startup: - semantic.oracle_max: every oracle run exits 0 and earns max - tolerance. A task without solution/solve.sh fails it, as the RFC requires. - semantic.floor_gap: the do-nothing reward sits the declared margin below the maximum and below the oracle. - semantic.canary_floor: operator-declared canaries (known-wrong solve.sh replacements in validation/canaries/) earn no more than doing nothing. - runtime.episode_determinism: an oracle replay in a fresh container on the same image must reproduce every reward. It gates the semantic checks. runtime.startup now means the trials ran and cleaned up. A malformed verifier reward fails runtime.reward_well_formed rather than erroring startup. The graders read a format-neutral OracleEvidence, so a future served binding can reuse them. The contract profile becomes harbor-semantic-v1 because the trials and report changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run_rollout and run_batch accept on_trial_created, a trusted callback invoked with the native Harbor trial after Trial.create and before trial.run(). An exception fails the rollout before execution, so a caller can verify the trial's image and settings first. Both functions also forward environment_kwargs to Harbor's environment config; run_batch forwards agent_timeout_sec and agent_step_limit to run_rollout. None of these is exposed through MCP. The defaults keep current behaviour. Ported from bf0b78ee on the Hub validation pipeline branch: these two modules and their test only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…008 work Replace the RFC 008 pilot amendment with the harbor-semantic-v1 binding: trial order, image pinning, the check-id mapping, canaries and the floor-gap clarification. Update the CLI reference and command help. Add a plan that records where validation stands and ranks the remaining workstreams. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for all 3 issues found in the latest run.
- ✅ Fixed: Temp cleanup can discard successful trials
- Temporary-directory cleanup errors are now ignored so collected Harbor trials remain available for grading.
- ✅ Fixed: Missing oracle omits semantic level
- A graded missing-oracle failure now records the semantic level even though no runtime trial starts.
- ✅ Fixed: Harbor extra does not pin 0.25.0
- The Harbor extra and dependent lock metadata now pin Harbor exactly to version 0.25.0.
Preview (abf12d0227)
diff --git a/docs/plans/rfc008-validation-completion.md b/docs/plans/rfc008-validation-completion.md
new file mode 100644
--- /dev/null
+++ b/docs/plans/rfc008-validation-completion.md
@@ -1,0 +1,302 @@
+# Completing RFC 008 validation
+
+Planning baseline: 2026-11-10, `openenv/OpenEnv` main at
+`fb36d7d62ff9548ebd44f3323386cbea908e8706`.
+
+Trackers: [RFC 008 #778](https://github.com/openenv/OpenEnv/issues/778),
+[Level 2 umbrella #1177](https://github.com/openenv/OpenEnv/issues/1177) and
+[task packages #898](https://github.com/openenv/OpenEnv/issues/898).
+
+RFC 008 promises that `openenv validate` grades an environment against the 44
+acceptance tests of #778. At the baseline, main ran every Level 2 runtime check for
+served environments on Docker, but no Level 3 semantic check. That includes the
+check the RFC opens with: a correct solution must earn maximum reward, and doing
+nothing must not. The PR that adds this document delivers that check for Harbor
+task packages and makes every report name the checks it did not run. This document
+records where validation stands after that PR, defines what "complete" means, and
+ranks the remaining work as PR-sized workstreams.
+
+## Where validation stands
+
+Each RFC check has a stable id, a level and a lane. The lane says who can run the
+check. `local` checks run inside `openenv validate`. `hub` and `statistical` ids are
+reserved for operators, the teams that run validation hubs or lab intake. The table
+counts the local ids in the committed severity policy and the ids each execution
+binding grades.
+
+| Level | Local ids | Served packages (`openenv_ws`) | Harbor tasks (`harbor_task`) |
+|---|---|---|---|
+| 1 Static | 10 | 1 (`static.manifest`) | 1 (`static.manifest`) |
+| 2 Runtime | 16 | 16, on Docker-local | 3: startup, reward well-formedness, episode determinism |
+| 3 Semantic | 8 | 0 | 3: oracle reaches max, floor gap, canary floor |
+| 4 Statistical | 0 (7 reserved) | Operator-side by design | Operator-side by design |
+| Hub | 0 (5 reserved) | Operator-side by design | Operator-side by design |
+
+Every requested local check that does not run is a named SKIP with its reason, at
+every level. A report therefore reads WARN until all requested checks pass, and a
+PASS on a subset cannot pass for full coverage.
+
+Three limits apply across bindings:
+
+- **Parsers.** `openenv.yaml` and Harbor `task.toml` ship. PostTrain `task.md`
+ packages are refused as unrecognized.
+- **Providers.** Docker-local enforces `public` and `no-network` networking and
+ refuses allowlists and GPUs. The HF Sandbox provider refuses every launch with
+ `missing provider capability: immutable_image`, because nothing yet binds a Space
+ revision to an image digest.
+- **Served oracles.** RFC 008 does not yet say how to execute an oracle against a
+ served WebSocket environment, so served packages report their semantic checks as
+ named SKIPs.
+
+Work outside main measures more than the RFC's local checks. A cloud audit on HF
+Jobs adds generated reference solutions, wrong-answer controls, repeat runs,
+leakage probes and target-model pass rates.
+[#1582](https://github.com/openenv/OpenEnv/pull/1582) adds training and validator
+performance experiments. These results report under their own fields rather than
+RFC check ids, so they do not count toward RFC coverage yet.
+
+## What complete means
+
+The RFC splits responsibility between the repo and operators, so completion has two
+targets. The repo can reach the first one alone. The second one needs an operator
+procedure that the repo documents but does not run.
+
+**Local completeness.** Every local-lane id has a grader for each shipped execution
+binding where it applies, and reports name every inapplicable or unimplemented id
+with its reason. For served packages this means nine static graders, an oracle
+execution contract and eight semantic graders. For Harbor tasks it means the same
+static graders, the remaining semantic graders, and a decision on which runtime ids
+apply to one-shot tasks.
+
+**Operator completeness.** Each reserved `statistical.*` id has at least one
+documented reference procedure that emits results in the shared report schema. The
+repo still ships no hub, as the RFC requires, so the procedure lives in an example or
+an operator repository. Comparable operator results also need the metric namespace
+(`pass@k`, headroom, variance bound) that the RFC defers to a future RFC.
+
+Validation still certifies that an environment can be trained on, not that it should
+be. Measuring training signal comes last, and it belongs with the evaluation work in
+[RFC 013 (#1256)](https://github.com/openenv/OpenEnv/pull/1256).
+
+## Proposed workstreams
+
+The workstreams are ranked by dependency and by how much each one improves
+environment selection. Each one is one to three PRs. Costs are relative: S needs no
+build or inference, M needs Docker builds and replays, and L needs model inference
+or GPU time. Owners are suggestions for discussion.
+
+| Rank | Workstream | #778 tests | Status | Cost | Suggested owner |
+|---|---|---|---|---|---|
+| 1 | Verifier sanity graders | #11, #25 | Harbor done; envelope and served open | M | Validation core |
+| 2 | Harbor binding on main | Harbor parser slice | Done | M | Validation core |
+| 3 | Verifier strength | #28, #26 procedure | Harbor canaries done; panel open | M, then L | Validation core and Hub pipeline |
+| 4 | Determinism, replay and leakage graders | #30, #39, #40, #44 | Open | M | Validation core |
+| 5 | Static graders and complete reports | #1, #3, #4, #8, #9, #10, #12, #41, #42 | Reports done; graders open | S to M | Any contributor |
+| 6 | Cloud execution on HF Sandbox | Provider qualification | Open | M | Hub pipeline |
+| 7 | Statistical reference procedure | #13 to #16, #27 | Open | L | Hub pipeline |
+| 8 | Wider task shapes | Workspace handoff, multi-container, credentials | Open | M to L | Hub pipeline and validation core |
+| 9 | Training signal | #17 | Open | L | Validation experiments and RFC 013 |
+
+### 1. Verifier sanity graders
+
+`semantic.oracle_max` and `semantic.floor_gap` now grade Harbor tasks. The graders
+read a format-neutral `OracleEvidence`, so another binding only has to measure the
+rewards.
+
+**Remaining.** Served packages need an oracle execution contract first: the RFC
+names injected state and executable scripts, but not how either drives a served
+WebSocket session or what doing nothing means there. That needs an RFC amendment.
+`semantic.resource_envelope` also remains: record peak CPU, memory and disk during
+the oracle run and compare them with the declared resources.
+
+**Evidence.** Oracle and do-nothing rewards, the declared margin and tolerance, the
+image digest and the cleanup outcome.
+
+**Cost and owner.** M: Docker builds and short episodes, with no inference.
+Validation core.
+
+### 2. Harbor binding on main
+
+Done. Harbor `task.toml` packages validate through the `harbor-semantic-v1`
+profile and an operator-supplied `validation/openenv.json` contract. Fresh Docker
+trials run a do-nothing agent, the oracle, an oracle replay and declared canaries,
+all pinned to one immutable image.
+
+**Remaining.** Decide which runtime ids apply to one-shot tasks. Containment,
+network policy, resource bounds and oracle containment could apply. Observation
+schema, state contract and tool declaration accuracy do not.
+
+**Cost and owner.** M. Validation core.
+
+### 3. Verifier strength: canaries and an adversarial panel
+
+A verifier can pass the oracle and floor checks and still reward wrong answers.
+`semantic.canary_floor` now runs operator-declared canaries on Harbor tasks: each
+known-wrong `solve.sh` replacement must earn no more than doing nothing.
+
+**Remaining.** The reserved `statistical.adversarial_floor` (#26) still has no
+procedure; the RFC notes that the panel "is not yet specified anywhere". Write an
+RFC 008 amendment that specifies:
+
+- where wrong answers come from: mutations of the reference solution, known-fault
+ corpora, empty and partial outputs, and plausible solutions from a weaker model
+ that an independent check confirms are wrong;
+- how many are needed for a verdict;
+- the pass rule, which is that every wrong answer scores at the floor.
+
+The cloud audit's wrong-answer controls are a working prototype of this panel.
+
+**Evidence.** Each wrong answer's provenance and reward, plus the verifier version.
+
+**Cost and owner.** L for generated panels, which need inference. Validation core
+writes the amendment, and the Hub pipeline team implements the operator panel.
+
+### 4. Determinism, replay and leakage graders
+
+`semantic.verifier_determinism`, `semantic.verifier_portability`,
+`semantic.replayability` and `semantic.no_solution_leakage` have no graders.
+Harbor tasks now establish `runtime.episode_determinism` with an oracle replay in a
+fresh container, which these checks depend on.
+
+**Approach.** Verifier determinism runs the verifier on one final state several
+times and requires identical rewards. For `llm_judged` environments it requires
+variance within the declared bound, reusing the Level 2 variance path. Portability
+reruns the verifier with networking denied, as the RFC's hermeticity rule for #40
+requires. Replayability re-executes a recorded trajectory and compares rewards. The
+leakage check scans observations and agent-visible files for oracle artifacts such
+as solution text, expected outputs and test files.
+
+**Evidence.** Replay transcripts and reward vectors, the network mode of each
+verifier run, and every leakage match with its location.
+
+**Cost and owner.** M. Validation core.
+
+### 5. Static graders and complete reports
+
+Reports now name every unimplemented static check as a SKIP, so a static run reads
+WARN rather than PASS after one check.
+
+**Remaining.** Add the nine graders in two batches. The first batch reads the
+manifest and lockfiles and needs no build: `static.dependency_pinning`,
+`static.task_distribution_pinning`, `static.resource_declaration` and
+`static.timeout_ceiling`. The second batch inspects built images:
+`static.oci_labels`, `static.sbom`, `static.image_hygiene`, `static.layout` and
+`static.reproducible_build`, which builds twice and compares digests.
+
+**Evidence.** The files and image metadata each grader read.
+
+**Cost and owner.** S for the first batch and M for the image batch. Any
+contributor; these make good first issues.
+
+### 6. Cloud execution on HF Sandbox
+
+The HF Sandbox provider refuses every launch because it has no immutable image
+binding. Runtime and semantic checks therefore cannot run in the cloud, or on GPUs,
+through the RFC provider interface. Cloud audits work around this with their own
+runner.
+
+**Approach.** Resolve a Space revision to the registry digest of the image it
+builds, verify it, and launch by digest. Qualify public Space images first, then add
+an authenticated pull path for private ones. Once launches work, the cloud audit can
+call the RFC graders instead of its own runner.
+
+**Evidence.** The Space revision, the resolved digest, the pull receipt and the same
+cleanup evidence Docker-local records.
+
+**Cost and owner.** M. Hub pipeline team.
+
+### 7. Statistical reference procedure
+
+The learnable tests (#13 to #16) and the gameability gap (#27) are reserved ids with
+no reference procedure. The RFC also defers the shared metric namespace.
+
+**Approach.** Write the metric-namespace RFC, with definitions, sample sizes and
+confidence intervals for `pass@k`, headroom and variance bound. Then publish a
+reference operator procedure, outside the core package, that maps target-model
+attempts onto the reserved ids:
+
+- `statistical.reward_reachability`: some attempt reaches maximum reward;
+- `statistical.difficulty_separation`: pass rates differ across model tiers;
+- `statistical.headroom`: the target model's pass rate leaves room to improve;
+- `statistical.variance_bound`: reward noise across repeats stays within a bound.
+
+The cloud audit already measures target-model pass rates with confidence intervals.
+It needs to emit them under these ids in hub-lane reports.
+
+**Evidence.** Model identity and provider, attempt counts, per-attempt rewards and
+intervals.
+
+**Cost and owner.** L, since every task needs inference. Hub pipeline team, with the
+metric-namespace RFC from validation core.
+
+### 8. Wider task shapes
+
+The Harbor binding refuses multi-step tasks, task-authored Docker Compose files and
+tasks that need credentials. The cloud audit also hands off only individual output
+files. The RFC defers credential delivery explicitly.
+
+**Approach.** Add a workspace or git-patch handoff so repository tasks are graded on
+the agent's full change. Treat multi-container support as a provider capability, as
+the RFC does for GPUs, so providers without it SKIP with the capability named. Write
+the credential-delivery amendment the RFC asks for, covering delivery, access
+isolation, lifetime, network access and evidence redaction.
+
+**Evidence.** Handoff manifests with hashes, and provider capability reports.
+
+**Cost and owner.** M for workspace handoff and L for multi-container and
+credentials. The Hub pipeline team builds the adapters, and validation core writes
+the amendment.
+
+### 9. Training signal
+
+`statistical.training_signal` (#17, advisory) asks whether training on an environment
+improves a model. Nothing measures it yet.
+
+**Approach.** Use the evaluation harness proposed in RFC 013 to run short training
+jobs on a sample of validated environments, then report the change in held-out
+success. The training experiments in #1582 are a first measurement. Keep the result
+advisory, as the RFC's severity policy says, because a short run cannot show that an
+environment is unlearnable.
+
+**Evidence.** Training configuration, seeds, and held-out success before and after
+training.
+
+**Cost and owner.** L: GPU time. Validation experiments, with the RFC 013 authors.
+
+## Sequencing
+
+The verifier sanity graders came first because canaries, verifier determinism and
+replay all reuse oracle replay. The order of the remaining work follows from its
+dependencies:
+
+- Workstream 4 builds on the oracle replay the Harbor binding now runs.
+- Workstream 5 is independent and can start now.
+- Workstream 6 moves the Harbor and served checks to the cloud and is needed before
+ workstream 7 runs at scale.
+- The served half of workstream 1 waits on the oracle execution amendment.
+- Workstreams 8 and 9 widen coverage after the core lands.
+
+## Open questions for maintainers
+
+These questions change the scope of a workstream, so they need an answer before its
+first PR:
+
+1. How should an oracle drive a served WebSocket environment, and what is its
+ do-nothing episode?
+2. Should `semantic.canary_floor` also accept canaries the validator generates, or
+ only canaries shipped with the package?
+3. Should the adversarial panel be an RFC 008 amendment or its own RFC?
+4. Where should operator reference procedures live: `examples/` or a separate
+ repository?
+5. Should the runtime applicability table for one-shot task packages be normative
+ in the RFC?
+
+## Sources
+
+- `rfcs/008-environment-auto-validation.md`, including the Harbor task binding
+ section
+- `docs/plans/rfc008-level2-implementation.md`
+- `tests/fixtures/validation/runtime/cases.json`
+- `src/openenv/validation/runner.py`, `harbor.py`, `graders/semantic/` and
+ `providers/hf_sandbox.py`
+- Issues #778, #898 and #1177; PRs #1256 and #1582
diff --git a/docs/source/reference/cli.md b/docs/source/reference/cli.md
--- a/docs/source/reference/cli.md
+++ b/docs/source/reference/cli.md
@@ -60,7 +60,7 @@
openenv validate ./my_env --level runtime --local --output report.json-Runtime validation requires Docker and a validation.execution declaration in
+For served packages, runtime validation requires Docker and a validation.execution declaration in
openenv.yaml. The declaration points to a bounded JSON replay plan containing
one reset and a sequence of actions. The validator builds an immutable image,
collects the original episode, repeats the plan in a fresh WebSocket session and
@@ -69,6 +69,18 @@
during the fresh-container replay. Both use the same immutable image and are
removed even if a check fails.
+Harbor task.toml directories are also recognized. Install the harbor extra
+and add a validation/openenv.json contract that declares the reward key, range,
+oracle tolerance, floor margin and resources. The harbor-semantic-v1 binding runs
+fresh Harbor trials in Docker: a do-nothing agent, the oracle (solution/solve.sh),
+an oracle replay and any canaries declared in the contract. A canary is a known-wrong
+solve.sh replacement. The results report under runtime.startup,
+runtime.reward_well_formed, runtime.episode_determinism, semantic.oracle_max,
+semantic.floor_gap and semantic.canary_floor, and harbor-evidence.json keeps
+the raw trial evidence in the report bundle. Other checks are named SKIPs, so a
+passing task still reports WARN. This validates task packages, not the outer
+harbor_env server.
+
The 16 implemented runtime checks cover startup, rewards, observation schemas,
state continuity, network policy, host containment, resource bounds, episode isolation,
oracle containment, seed forwarding, determinism, emitted trajectory records, tool
@@ -77,7 +89,8 @@
the result WARN, which exits zero and does not mean Level 2 is complete.
FAIL exits 1, unsupported package formats exit 2, and internal or policy errors
exit 3. --level semantic
-includes the available lower-level checks but does not claim semantic execution.
+runs the semantic checks for Harbor tasks. Served packages have no oracle execution
+yet, so their semantic checks are named SKIPs.
--skip-build runs declaration checks and skips runtime execution entirely.
The Docker provider supports CPU workloads in public and no-network modes;
allowlist and GPU requirements skip runtime before building. A no-network
diff --git a/envs/grid_world_env/uv.lock b/envs/grid_world_env/uv.lock
--- a/envs/grid_world_env/uv.lock
+++ b/envs/grid_world_env/uv.lock
@@ -1343,7 +1343,7 @@
{ name = "fastapi", specifier = ">=0.116.0" },
{ name = "fastmcp", specifier = ">=3.0.0,<4.0.0" },
{ name = "gradio", specifier = ">=4.0.0" },
- { name = "harbor", extras = ["e2b", "modal", "daytona", "gke", "ec2", "runloop", "novita", "blaxel", "beam", "islo", "opensandbox", "cwsandbox", "use-computer", "cua"], marker = "python_full_version >= '3.12' and extra == 'harbor'", specifier = ">=0.22.0" },
- { name = "harbor", extras = ["e2b", "modal", "daytona", "gke", "ec2", "runloop", "novita", "blaxel", "beam", "islo", "opensandbox", "cwsandbox", "use-computer", "cua"], marker = "python_full_version >= '3.12' and extra == 'harbor'", specifier = "==0.25.0" },
{ name = "httpx", specifier = ">=0.28.1" },
{ name = "huggingface-hub", specifier = ">=1.29.0" },
{ name = "inspect-ai", marker = "extra == 'inspect'", specifier = ">=0.3.0" },
diff --git a/pyproject.toml b/pyproject.toml
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -72,7 +72,7 @@
# (websockets>=15) and tensorlake (websockets>=13,<14) together.
#
# Harbor requires Python >= 3.12, so this extra is excluded from the 3.11 CI leg.
- "harbor[e2b,modal,daytona,gke,ec2,runloop,novita,blaxel,beam,islo,opensandbox,cwsandbox,use-computer,cua]>=0.22.0; python_version >= '3.12'",
- "harbor[e2b,modal,daytona,gke,ec2,runloop,novita,blaxel,beam,islo,opensandbox,cwsandbox,use-computer,cua]==0.25.0; python_version >= '3.12'",
]
[project.scripts]
diff --git a/rfcs/008-environment-auto-validation.md b/rfcs/008-environment-auto-validation.md
--- a/rfcs/008-environment-auto-validation.md
+++ b/rfcs/008-environment-auto-validation.md
@@ -718,6 +718,66 @@
operator certification remain outside this feature; the test workflow only proves
the implementation's own acceptance cases.
+### Harbor task binding
+
+task.toml packages opt into the harbor-semantic-v1 profile through an
+operator-supplied validation/openenv.json contract. The contract declares the
+reward key, range, oracle tolerance, floor margin, finite resource budgets and,
+optionally, a directory of canaries. Harbor metadata and observed rewards never
+supply these declarations. The normalized schema-2 execution kind is harbor_task,
+with a process agent boundary. Parsing uses Harbor's data-only Task loader and
+never creates a Trial or executes package code.
+
+The profile supports single-step Linux tasks with one Dockerfile, one main Docker
+service, a shared verifier and public networking. Accelerators, external MCP
+services, task environment variables, credentials and Harbor extensions such as
+healthchecks or artifacts are refused with the reason named.
+
+Execution runs fresh Harbor Trials in a fixed order: a do-nothing (NOP) trial, the
+oracle (solution/solve.sh), an oracle replay, and one trial per declared canary.
+The NOP trial builds the image. The validator inspects the actual agent container
+at AGENT_START, retains its immutable image under a run-owned tag, and pins every
+later trial to that image and its effective container configuration. Each trial
+runs on a private copy of the task whose digest must equal the validated source
+digest before any overlay. A canary trial replaces solution/solve.sh in its copy
+with the canary script and records the script's digest. Trials run with
+delete=False and keep_containers=True, so the validator removes the exact
+run-owned Compose projects, including anonymous volumes registered from the
+containers' actual mounts, and verifies their absence. The oracle's exit code is
+read at AGENT_END, before sandbox log synchronization can overwrite it, and a
+later change fails the run.
+
+Results map to RFC check ids:
+
+| Check | Evidence |
+|---|---|
+| runtime.startup | Every trial ran from the pinned image in a fresh container, and cleanup was verified. |
+| runtime.reward_well_formed | Every raw reward map matches Harbor's result and is finite, nonboolean and in range. |
+| runtime.episode_determinism | The oracle replay reproduces every reward and the exit code. |
+| semantic.oracle_max | Both oracle runs exit 0 and earn at least max - oracle_tolerance. A task without solution/solve.sh fails. |
+| semantic.floor_gap | max - floor >= floor_margin, and oracle - floor >= floor_margin because the oracle may sit up to the tolerance below the maximum. |
+| semantic.canary_floor | Every declared canary earns no more than the measured floor. Selected only when canaries are declared. |
+
+The semantic ids depend on runtime.episode_determinism, so a diverging replay
+SKIPs them with the dependency named. Reward comparisons use exact rationals parsed
+from the raw verifier output. Every other local check is a named SKIP for this
+binding. It produces no WebSocket exchange, containment or resource measurement,
+second verification of one final state, or leakage scan. A passing run is therefore
+WARN, and full RFC qualification still rejects these SKIPs.
+
+Canaries are operator-declared, known-wrong solve.sh replacements, at most eight
+per task. They give local validation the package-shipped form of test #28.
+Generating wrong answers, for example by mutating the oracle, belongs to the
+adversarial panel of test #26, which remains unspecified.
+
+harbor-evidence.json records the contract digest, the Harbor version, and each
+trial's role, image identity, private-copy digests, overlays, raw reward file hashes
+and cleanup outcome. It and the raw reward and exit files are checksummed bundle
+artifacts beside the report.
+
+Served openenv_ws packages have no oracle execution contract yet, so their
+semantic checks are named SKIPs.
+
Explicitly out of scope
No hub/runner/queue/coordinator; no submission or auth APIs; no statistical-level implementations
diff --git a/src/openenv/cli/commands/validate.py b/src/openenv/cli/commands/validate.py
--- a/src/openenv/cli/commands/validate.py
+++ b/src/openenv/cli/commands/validate.py
@@ -231,10 +231,15 @@
Validate a local package or a running OpenEnv server.
Local validation detects the package format by its well-known file (the
- formats this build can parse; currently
openenv.yaml), parses it into the
-
formats this build can parse:
openenv.yamlor Harbortask.toml), parses it into the
normalized manifest, runs the applicable graders up to the requested level,
applies the severity policy, and emits a report. -
Native Harbor tasks require the optional Harbor dependency and an explicit
-
validation/openenv.jsoncontract. Fresh Docker trials grade the oracle, the -
do-nothing floor and any declared canaries under the semantic check ids. Checks
-
without an implementation are named SKIPs, so the report is WARN at best.
-
Exit codes: 0 pass/warn · 1 fail · 2 unrecognized/unsupported package · 3
internal or policy error.
diff --git a/src/openenv/harbor/rollout.py b/src/openenv/harbor/rollout.py
--- a/src/openenv/harbor/rollout.py
+++ b/src/openenv/harbor/rollout.py
@@ -251,12 +251,17 @@
force_build: bool = False,
training: bool = False,
harness_profile: str | None = None,
-
environment_kwargs: dict[str, Any] | None = None,
) -> Any:
"""Assemble Harbor'sTrialConfigfor one rollout.The seam decides how this particular agent learns about the proxy — an env var, a constructor
kwarg, or a config file written by a subclass. That is the only per-agent knowledge involved;
everything downstream is identical. -
Args:
-
environment_kwargs (`dict[str, Any]`, *optional*): -
Backend kwargs passed to Harbor. `None` keeps Harbor's defaults."""
from harbor.models.trial.config import (
AgentConfig,
@@ -312,6 +317,9 @@
),
delete=not keep_sandbox,
force_build=force_build, -
**( -
{"kwargs": environment_kwargs} if environment_kwargs is not None else {} -
), ), verifier=VerifierConfig(), trial_name=trial_name,
@@ -344,6 +352,8 @@
eval_sampling: dict[str, Any] | None = None,
harness_profile: str | None = None,
on_session_created: Callable[[str], None] | None = None,
- on_trial_created: Callable[[Any], None] | None = None,
- environment_kwargs: dict[str, Any] | None = None,
) -> HarborRolloutResult:
"""Run one rollout end to end. Never raises.
@@ -386,6 +396,11 @@
on_session_created (Callable[[str], None], optional):
Receive this rollout's capture session ID before the agent starts. Live views must
follow this ID rather than discover unrelated sessions from the shared registry.
-
on_trial_created (`Callable[[Any], None]`, *optional*): -
Trusted infrastructure callback invoked with the native Harbor trial before it runs. -
Exceptions fail the rollout and prevent trial execution. Not exposed through MCP. -
environment_kwargs (`dict[str, Any]`, *optional*): -
Backend kwargs passed to Harbor. `None` keeps Harbor's defaults.Returns:
[HarborRolloutResult]: Reward, per-turn token ids and logprobs, and validation findings.
@@ -488,6 +503,7 @@
force_build=force_build,
training=sampling is not None,
harness_profile=harness_profile, -
environment_kwargs=environment_kwargs, ) # Held across construction because Harbor's wrappers read `os.environ` while building the
@@ -503,6 +519,8 @@
if not proc_env:
config = build_trial_config(**config_kwargs)
trial = await Trial.create(config)
-
if on_trial_created is not None: -
on_trial_created(trial) trial_result = await trial.run() else: # The override has to span `run()`, not just construction. Harbor's goose wrapper reads
@@ -535,6 +553,8 @@
with ctx_env_overlay(proc_env):
config = build_trial_config(**config_kwargs)
trial = await Trial.create(config)
-
if on_trial_created is not None: -
on_trial_created(trial) trial_result = await trial.run() else: # Fallback: mutate the real environment under a lock, one rollout at a time.
@@ -543,6 +563,8 @@
with process_env(seam.name, proc_env):
config = build_trial_config(**config_kwargs)
trial = await Trial.create(config)
-
if on_trial_created is not None: -
on_trial_created(trial) trial_result = await trial.run() finally: _PROC_ENV_LOCK.release()
diff --git a/src/openenv/harbor/runner.py b/src/openenv/harbor/runner.py
--- a/src/openenv/harbor/runner.py
+++ b/src/openenv/harbor/runner.py
@@ -11,6 +11,7 @@
import os
import secrets
from pathlib import Path
+from typing import Any, Callable
from openenv.core.harness.capture import CaptureServer
@@ -44,6 +45,10 @@
auth_header: str = "Authorization",
provider: str = "openai",
admin_key: str | None = None,
- agent_timeout_sec: float | None = None,
- agent_step_limit: int | None = None,
- on_trial_created: Callable[[Any], None] | None = None,
- environment_kwargs: dict[str, Any] | None = None,
) -> list[HarborRolloutResult]:
"""Runtask_indicesfromdatasetand print a per-rollout report.
@@ -64,6 +69,15 @@
Key the capture proxy's session-management routes require. Defaults to
$OPENENV_CAPTURE_ADMIN_KEY, else a random one minted for this batch. The proxy is
published on a public URL, so these routes are never left open.
-
agent_timeout_sec (`float`, *optional*): -
Override each agent's execution timeout in seconds. -
agent_step_limit (`int`, *optional*): -
Limit each rollout's agent steps and captured model calls. -
on_trial_created (`Callable[[Any], None]`, *optional*): -
Trusted infrastructure callback invoked with each native Harbor trial before it runs. -
Exceptions fail that rollout and prevent trial execution. Not exposed through MCP. -
environment_kwargs (`dict[str, Any]`, *optional*): -
Backend kwargs passed to Harbor. `None` keeps Harbor's defaults.Returns:
list[HarborRolloutResult]: One per index, in order.
@@ -163,9 +177,13 @@
# training path (run_rollout's default) still refuses to pick a signal.
require_reward=False,
keep_sandbox=keep_sandbox, -
agent_timeout_sec=agent_timeout_sec, -
agent_step_limit=agent_step_limit, force_build=force_build, capture_level=capture_level, inference=capture.inference, -
on_trial_created=on_trial_created, -
environment_kwargs=environment_kwargs, ) results.append(result) print(" " + _summarise(result))
diff --git a/src/openenv/validation/graders/init.py b/src/openenv/validation/graders/init.py
--- a/src/openenv/validation/graders/init.py
+++ b/src/openenv/validation/graders/init.py
@@ -13,7 +13,12 @@
from ..manifest import CapabilitiesSpec, NormalizedManifest
from ..providers import RunningSubject
from ..report import CheckResult
-from ..runtime.contracts import HttpProbe, IsolationEvidence, RuntimeEvidence
+from ..runtime.contracts import (
- HttpProbe,
- IsolationEvidence,
- OracleEvidence,
- RuntimeEvidence,
+)
from ..types import Level, ProviderCapability
ENTRY_POINT_GROUP = "openenv.validation.graders"
@@ -46,6 +51,8 @@
Same-session and fresh episodes on the running subject.
oracle_http (tuple[HttpProbe, ...]):
GET probes for the declared oracle path on the control endpoint.
-
oracle_evidence ([`~openenv.validation.runtime.contracts.OracleEvidence`], *optional*): -
Oracle, do-nothing and canary rewards for the Level 3 checks."""
root: Path
@@ -59,6 +66,7 @@
resource_evidence_json: str | None = None
isolation_evidence: IsolationEvidence | None = None
oracle_http: tuple[HttpProbe, ...] = () -
oracle_evidence: OracleEvidence | None = None
@runtime_checkable
diff --git a/src/openenv/validation/graders/semantic/init.py b/src/openenv/validation/graders/semantic/init.py
new file mode 100644
--- /dev/null
+++ b/src/openenv/validation/graders/semantic/init.py
@@ -1,0 +1,5 @@
+"""Semantic-level graders."""
+
+from .oracle import CanaryFloorGrader, FloorGapGrader, OracleMaxGrader
+
+all = ["CanaryFloorGrader", "FloorGapGrader", "OracleMaxGrader"]
diff --git a/src/openenv/validation/graders/semantic/oracle.py b/src/openenv/validation/graders/semantic/oracle.py
new file mode 100644
--- /dev/null
+++ b/src/openenv/validation/graders/semantic/oracle.py
@@ -1,0 +1,145 @@
+"""Verifier sanity over oracle, do-nothing and canary rewards (#778 tests #25, #28).
+
+The execution binding measures the rewards; these graders only compare them with the
+manifest's declared range, tolerance and margin. Comparisons use exact rationals so a
+tiny floor cannot round into the margin.
+"""
+
+import time
+from fractions import Fraction
+
+from ...report import CheckResult
+from ...types import CheckStatus, Level
+
+
+class _SemanticGrader:
- level = Level.SEMANTIC
- requires_capabilities = frozenset()
- requires_provider = frozenset()
-
RFC 008: episode determinism is a precondition of every semantic check.
- depends_on = ("runtime.episode_determinism",)
- remediation = ""
- def applies_to(self, manifest) -> bool:
-
return True - def run(self, subject) -> CheckResult:
-
started = time.monotonic() -
evidence = subject.oracle_evidence -
if evidence is None: -
return CheckResult( -
check_id=self.check_id, -
status=CheckStatus.SKIP, -
evidence=["oracle evidence is unavailable"], -
duration_s=0, -
) -
reward = subject.manifest.reward -
problems, measured = self.check( -
evidence, -
maximum=Fraction(str(reward.range[1])), -
tolerance=Fraction(str(reward.oracle_tolerance)), -
margin=Fraction(str(reward.floor_margin)), -
) -
return CheckResult( -
check_id=self.check_id, -
status=CheckStatus.FAIL if problems else CheckStatus.PASS, -
measured=measured, -
evidence=problems or [self.passed], -
remediation=self.remediation if problems else None, -
duration_s=time.monotonic() - started, -
)
+class OracleMaxGrader(_SemanticGrader):
- """Every oracle run exits cleanly and earns at least
max - oracle_tolerance.""" - check_id = "semantic.oracle_max"
- passed = "every oracle run reached the declared maximum within tolerance"
- remediation = (
-
"Fix the oracle or the verifier so a correct solution exits cleanly and " -
"earns the declared maximum reward." - )
- def check(self, evidence, *, maximum, tolerance, margin):
-
threshold = maximum - tolerance -
problems = [] -
for run, (reward, code) in enumerate( -
zip(evidence.oracle, evidence.oracle_exit_codes, strict=True), start=1 -
): -
if code != 0: -
problems.append(f"oracle run {run} exited with code {code}") -
if reward < threshold: -
problems.append( -
f"oracle run {run} earned {float(reward)}, below {float(threshold)}" -
) -
return problems, { -
"oracle_rewards": [float(reward) for reward in evidence.oracle], -
"oracle_exit_codes": list(evidence.oracle_exit_codes), -
"declared_max": float(maximum), -
"oracle_tolerance": float(tolerance), -
}
+class FloorGapGrader(_SemanticGrader):
- """
- The do-nothing reward sits at least
floor_marginbelow the maximum and below - every oracle run. The oracle may sit up to the tolerance below the maximum, so
- both gaps are required.
- """
- check_id = "semantic.floor_gap"
- passed = "the do-nothing reward leaves the declared margin"
- remediation = (
-
"The verifier rewards doing nothing. Check for vacuous success conditions, " -
"such as comparisons against empty output." - )
- def check(self, evidence, *, maximum, tolerance, margin):
-
floor = evidence.floor -
problems = [] -
if maximum - floor < margin: -
problems.append( -
f"do-nothing reward {float(floor)} is within {float(maximum - floor)} " -
f"of the maximum, below the margin {float(margin)}" -
) -
lowest = min(evidence.oracle) -
if lowest - floor < margin: -
problems.append( -
f"oracle reward {float(lowest)} is within {float(lowest - floor)} " -
f"of the do-nothing reward, below the margin {float(margin)}" -
) -
return problems, { -
"measured_floor": float(floor), -
"oracle_reward": float(lowest), -
"declared_max": float(maximum), -
"declared_margin": float(margin), -
}
+class CanaryFloorGrader(_SemanticGrader):
- """Every declared canary, a known-wrong solution, earns no more than doing nothing."""
- check_id = "semantic.canary_floor"
- requires_capabilities = frozenset({"canaries"})
- passed = "every canary stayed at the do-nothing reward"
- remediation = (
-
"The verifier rewards a known-wrong solution. Tighten the tests so each " -
"canary earns no more than doing nothing." - )
- def check(self, evidence, *, maximum, tolerance, margin):
-
problems = [ -
f"canary {name} earned {float(reward)}, above the do-nothing reward " -
f"{float(evidence.floor)}" -
for name, reward in evidence.canaries -
if reward > evidence.floor -
] -
# An empty run list must not pass vacuously. -
if not evidence.canaries: -
problems.append("no canary run was recorded") -
return problems, {
... diff truncated: showing 800 of 3513 lines
</details>
<sub>You can send follow-ups to the cloud agent <a href="https://cursor.com/agents/bc-9da5b933-e0af-4d31-a008-a047135a73cd">here</a>.</sub>
</details>
The contract reader parsed JSON numbers as floats, so a floor_margin of 0.100000000000000001 became 0.1. A measured gap of exactly 0.1 then passed semantic.floor_gap although it is below the declared margin. The reader now parses reward and resource numbers as exact decimals and rejects any whose float value differs from the declared token. Every value the graders compare is therefore the operator's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keeps the levels_run fix from abf12d0 and reverts its other two changes: - Pinning the harbor extra to ==0.25.0 would hold every Harbor-based feature in OpenEnv to that version. The Harbor task parser already refuses other versions with a named error. - Ignoring temp-directory cleanup errors guards against root-owned trial files. The binding runs Harbor with auto_mounting disabled, so trial outputs are downloaded by the host and owned by it. Ignoring errors would also leave private task copies on disk without notice. Also rename the version error, which still called the binding a pilot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fast reference suite runs tests/test_validation in its locked project and fails on any skip. Without Harbor installed, the Harbor task binding's tests skipped. Adding harbor==0.25.0 runs them. Harbor's dependencies cap websockets below 16, so the lock moves websockets from 17.1 to 15.0.1, which is within OpenEnv's websockets>=15.0.1. The fast suite (1,711 tests) and the protocol suite (138 tests) pass with no skips under the new lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Harbor deps pollute Docker fixture image
- Moved Harbor into the runtime project's dev dependency group so host tests install it while the Docker fixture's --no-dev export excludes it and its transitive dependencies.
Preview (156e411eea)
diff --git a/docs/plans/rfc008-validation-completion.md b/docs/plans/rfc008-validation-completion.md
new file mode 100644
--- /dev/null
+++ b/docs/plans/rfc008-validation-completion.md
@@ -1,0 +1,302 @@
+# Completing RFC 008 validation
+
+Planning baseline: 2026-11-10, `openenv/OpenEnv` main at
+`fb36d7d62ff9548ebd44f3323386cbea908e8706`.
+
+Trackers: [RFC 008 #778](https://github.com/openenv/OpenEnv/issues/778),
+[Level 2 umbrella #1177](https://github.com/openenv/OpenEnv/issues/1177) and
+[task packages #898](https://github.com/openenv/OpenEnv/issues/898).
+
+RFC 008 promises that `openenv validate` grades an environment against the 44
+acceptance tests of #778. At the baseline, main ran every Level 2 runtime check for
+served environments on Docker, but no Level 3 semantic check. That includes the
+check the RFC opens with: a correct solution must earn maximum reward, and doing
+nothing must not. The PR that adds this document delivers that check for Harbor
+task packages and makes every report name the checks it did not run. This document
+records where validation stands after that PR, defines what "complete" means, and
+ranks the remaining work as PR-sized workstreams.
+
+## Where validation stands
+
+Each RFC check has a stable id, a level and a lane. The lane says who can run the
+check. `local` checks run inside `openenv validate`. `hub` and `statistical` ids are
+reserved for operators, the teams that run validation hubs or lab intake. The table
+counts the local ids in the committed severity policy and the ids each execution
+binding grades.
+
+| Level | Local ids | Served packages (`openenv_ws`) | Harbor tasks (`harbor_task`) |
+|---|---|---|---|
+| 1 Static | 10 | 1 (`static.manifest`) | 1 (`static.manifest`) |
+| 2 Runtime | 16 | 16, on Docker-local | 3: startup, reward well-formedness, episode determinism |
+| 3 Semantic | 8 | 0 | 3: oracle reaches max, floor gap, canary floor |
+| 4 Statistical | 0 (7 reserved) | Operator-side by design | Operator-side by design |
+| Hub | 0 (5 reserved) | Operator-side by design | Operator-side by design |
+
+Every requested local check that does not run is a named SKIP with its reason, at
+every level. A report therefore reads WARN until all requested checks pass, and a
+PASS on a subset cannot pass for full coverage.
+
+Three limits apply across bindings:
+
+- **Parsers.** `openenv.yaml` and Harbor `task.toml` ship. PostTrain `task.md`
+ packages are refused as unrecognized.
+- **Providers.** Docker-local enforces `public` and `no-network` networking and
+ refuses allowlists and GPUs. The HF Sandbox provider refuses every launch with
+ `missing provider capability: immutable_image`, because nothing yet binds a Space
+ revision to an image digest.
+- **Served oracles.** RFC 008 does not yet say how to execute an oracle against a
+ served WebSocket environment, so served packages report their semantic checks as
+ named SKIPs.
+
+Work outside main measures more than the RFC's local checks. A cloud audit on HF
+Jobs adds generated reference solutions, wrong-answer controls, repeat runs,
+leakage probes and target-model pass rates.
+[#1582](https://github.com/openenv/OpenEnv/pull/1582) adds training and validator
+performance experiments. These results report under their own fields rather than
+RFC check ids, so they do not count toward RFC coverage yet.
+
+## What complete means
+
+The RFC splits responsibility between the repo and operators, so completion has two
+targets. The repo can reach the first one alone. The second one needs an operator
+procedure that the repo documents but does not run.
+
+**Local completeness.** Every local-lane id has a grader for each shipped execution
+binding where it applies, and reports name every inapplicable or unimplemented id
+with its reason. For served packages this means nine static graders, an oracle
+execution contract and eight semantic graders. For Harbor tasks it means the same
+static graders, the remaining semantic graders, and a decision on which runtime ids
+apply to one-shot tasks.
+
+**Operator completeness.** Each reserved `statistical.*` id has at least one
+documented reference procedure that emits results in the shared report schema. The
+repo still ships no hub, as the RFC requires, so the procedure lives in an example or
+an operator repository. Comparable operator results also need the metric namespace
+(`pass@k`, headroom, variance bound) that the RFC defers to a future RFC.
+
+Validation still certifies that an environment can be trained on, not that it should
+be. Measuring training signal comes last, and it belongs with the evaluation work in
+[RFC 013 (#1256)](https://github.com/openenv/OpenEnv/pull/1256).
+
+## Proposed workstreams
+
+The workstreams are ranked by dependency and by how much each one improves
+environment selection. Each one is one to three PRs. Costs are relative: S needs no
+build or inference, M needs Docker builds and replays, and L needs model inference
+or GPU time. Owners are suggestions for discussion.
+
+| Rank | Workstream | #778 tests | Status | Cost | Suggested owner |
+|---|---|---|---|---|---|
+| 1 | Verifier sanity graders | #11, #25 | Harbor done; envelope and served open | M | Validation core |
+| 2 | Harbor binding on main | Harbor parser slice | Done | M | Validation core |
+| 3 | Verifier strength | #28, #26 procedure | Harbor canaries done; panel open | M, then L | Validation core and Hub pipeline |
+| 4 | Determinism, replay and leakage graders | #30, #39, #40, #44 | Open | M | Validation core |
+| 5 | Static graders and complete reports | #1, #3, #4, #8, #9, #10, #12, #41, #42 | Reports done; graders open | S to M | Any contributor |
+| 6 | Cloud execution on HF Sandbox | Provider qualification | Open | M | Hub pipeline |
+| 7 | Statistical reference procedure | #13 to #16, #27 | Open | L | Hub pipeline |
+| 8 | Wider task shapes | Workspace handoff, multi-container, credentials | Open | M to L | Hub pipeline and validation core |
+| 9 | Training signal | #17 | Open | L | Validation experiments and RFC 013 |
+
+### 1. Verifier sanity graders
+
+`semantic.oracle_max` and `semantic.floor_gap` now grade Harbor tasks. The graders
+read a format-neutral `OracleEvidence`, so another binding only has to measure the
+rewards.
+
+**Remaining.** Served packages need an oracle execution contract first: the RFC
+names injected state and executable scripts, but not how either drives a served
+WebSocket session or what doing nothing means there. That needs an RFC amendment.
+`semantic.resource_envelope` also remains: record peak CPU, memory and disk during
+the oracle run and compare them with the declared resources.
+
+**Evidence.** Oracle and do-nothing rewards, the declared margin and tolerance, the
+image digest and the cleanup outcome.
+
+**Cost and owner.** M: Docker builds and short episodes, with no inference.
+Validation core.
+
+### 2. Harbor binding on main
+
+Done. Harbor `task.toml` packages validate through the `harbor-semantic-v1`
+profile and an operator-supplied `validation/openenv.json` contract. Fresh Docker
+trials run a do-nothing agent, the oracle, an oracle replay and declared canaries,
+all pinned to one immutable image.
+
+**Remaining.** Decide which runtime ids apply to one-shot tasks. Containment,
+network policy, resource bounds and oracle containment could apply. Observation
+schema, state contract and tool declaration accuracy do not.
+
+**Cost and owner.** M. Validation core.
+
+### 3. Verifier strength: canaries and an adversarial panel
+
+A verifier can pass the oracle and floor checks and still reward wrong answers.
+`semantic.canary_floor` now runs operator-declared canaries on Harbor tasks: each
+known-wrong `solve.sh` replacement must earn no more than doing nothing.
+
+**Remaining.** The reserved `statistical.adversarial_floor` (#26) still has no
+procedure; the RFC notes that the panel "is not yet specified anywhere". Write an
+RFC 008 amendment that specifies:
+
+- where wrong answers come from: mutations of the reference solution, known-fault
+ corpora, empty and partial outputs, and plausible solutions from a weaker model
+ that an independent check confirms are wrong;
+- how many are needed for a verdict;
+- the pass rule, which is that every wrong answer scores at the floor.
+
+The cloud audit's wrong-answer controls are a working prototype of this panel.
+
+**Evidence.** Each wrong answer's provenance and reward, plus the verifier version.
+
+**Cost and owner.** L for generated panels, which need inference. Validation core
+writes the amendment, and the Hub pipeline team implements the operator panel.
+
+### 4. Determinism, replay and leakage graders
+
+`semantic.verifier_determinism`, `semantic.verifier_portability`,
+`semantic.replayability` and `semantic.no_solution_leakage` have no graders.
+Harbor tasks now establish `runtime.episode_determinism` with an oracle replay in a
+fresh container, which these checks depend on.
+
+**Approach.** Verifier determinism runs the verifier on one final state several
+times and requires identical rewards. For `llm_judged` environments it requires
+variance within the declared bound, reusing the Level 2 variance path. Portability
+reruns the verifier with networking denied, as the RFC's hermeticity rule for #40
+requires. Replayability re-executes a recorded trajectory and compares rewards. The
+leakage check scans observations and agent-visible files for oracle artifacts such
+as solution text, expected outputs and test files.
+
+**Evidence.** Replay transcripts and reward vectors, the network mode of each
+verifier run, and every leakage match with its location.
+
+**Cost and owner.** M. Validation core.
+
+### 5. Static graders and complete reports
+
+Reports now name every unimplemented static check as a SKIP, so a static run reads
+WARN rather than PASS after one check.
+
+**Remaining.** Add the nine graders in two batches. The first batch reads the
+manifest and lockfiles and needs no build: `static.dependency_pinning`,
+`static.task_distribution_pinning`, `static.resource_declaration` and
+`static.timeout_ceiling`. The second batch inspects built images:
+`static.oci_labels`, `static.sbom`, `static.image_hygiene`, `static.layout` and
+`static.reproducible_build`, which builds twice and compares digests.
+
+**Evidence.** The files and image metadata each grader read.
+
+**Cost and owner.** S for the first batch and M for the image batch. Any
+contributor; these make good first issues.
+
+### 6. Cloud execution on HF Sandbox
+
+The HF Sandbox provider refuses every launch because it has no immutable image
+binding. Runtime and semantic checks therefore cannot run in the cloud, or on GPUs,
+through the RFC provider interface. Cloud audits work around this with their own
+runner.
+
+**Approach.** Resolve a Space revision to the registry digest of the image it
+builds, verify it, and launch by digest. Qualify public Space images first, then add
+an authenticated pull path for private ones. Once launches work, the cloud audit can
+call the RFC graders instead of its own runner.
+
+**Evidence.** The Space revision, the resolved digest, the pull receipt and the same
+cleanup evidence Docker-local records.
+
+**Cost and owner.** M. Hub pipeline team.
+
+### 7. Statistical reference procedure
+
+The learnable tests (#13 to #16) and the gameability gap (#27) are reserved ids with
+no reference procedure. The RFC also defers the shared metric namespace.
+
+**Approach.** Write the metric-namespace RFC, with definitions, sample sizes and
+confidence intervals for `pass@k`, headroom and variance bound. Then publish a
+reference operator procedure, outside the core package, that maps target-model
+attempts onto the reserved ids:
+
+- `statistical.reward_reachability`: some attempt reaches maximum reward;
+- `statistical.difficulty_separation`: pass rates differ across model tiers;
+- `statistical.headroom`: the target model's pass rate leaves room to improve;
+- `statistical.variance_bound`: reward noise across repeats stays within a bound.
+
+The cloud audit already measures target-model pass rates with confidence intervals.
+It needs to emit them under these ids in hub-lane reports.
+
+**Evidence.** Model identity and provider, attempt counts, per-attempt rewards and
+intervals.
+
+**Cost and owner.** L, since every task needs inference. Hub pipeline team, with the
+metric-namespace RFC from validation core.
+
+### 8. Wider task shapes
+
+The Harbor binding refuses multi-step tasks, task-authored Docker Compose files and
+tasks that need credentials. The cloud audit also hands off only individual output
+files. The RFC defers credential delivery explicitly.
+
+**Approach.** Add a workspace or git-patch handoff so repository tasks are graded on
+the agent's full change. Treat multi-container support as a provider capability, as
+the RFC does for GPUs, so providers without it SKIP with the capability named. Write
+the credential-delivery amendment the RFC asks for, covering delivery, access
+isolation, lifetime, network access and evidence redaction.
+
+**Evidence.** Handoff manifests with hashes, and provider capability reports.
+
+**Cost and owner.** M for workspace handoff and L for multi-container and
+credentials. The Hub pipeline team builds the adapters, and validation core writes
+the amendment.
+
+### 9. Training signal
+
+`statistical.training_signal` (#17, advisory) asks whether training on an environment
+improves a model. Nothing measures it yet.
+
+**Approach.** Use the evaluation harness proposed in RFC 013 to run short training
+jobs on a sample of validated environments, then report the change in held-out
+success. The training experiments in #1582 are a first measurement. Keep the result
+advisory, as the RFC's severity policy says, because a short run cannot show that an
+environment is unlearnable.
+
+**Evidence.** Training configuration, seeds, and held-out success before and after
+training.
+
+**Cost and owner.** L: GPU time. Validation experiments, with the RFC 013 authors.
+
+## Sequencing
+
+The verifier sanity graders came first because canaries, verifier determinism and
+replay all reuse oracle replay. The order of the remaining work follows from its
+dependencies:
+
+- Workstream 4 builds on the oracle replay the Harbor binding now runs.
+- Workstream 5 is independent and can start now.
+- Workstream 6 moves the Harbor and served checks to the cloud and is needed before
+ workstream 7 runs at scale.
+- The served half of workstream 1 waits on the oracle execution amendment.
+- Workstreams 8 and 9 widen coverage after the core lands.
+
+## Open questions for maintainers
+
+These questions change the scope of a workstream, so they need an answer before its
+first PR:
+
+1. How should an oracle drive a served WebSocket environment, and what is its
+ do-nothing episode?
+2. Should `semantic.canary_floor` also accept canaries the validator generates, or
+ only canaries shipped with the package?
+3. Should the adversarial panel be an RFC 008 amendment or its own RFC?
+4. Where should operator reference procedures live: `examples/` or a separate
+ repository?
+5. Should the runtime applicability table for one-shot task packages be normative
+ in the RFC?
+
+## Sources
+
+- `rfcs/008-environment-auto-validation.md`, including the Harbor task binding
+ section
+- `docs/plans/rfc008-level2-implementation.md`
+- `tests/fixtures/validation/runtime/cases.json`
+- `src/openenv/validation/runner.py`, `harbor.py`, `graders/semantic/` and
+ `providers/hf_sandbox.py`
+- Issues #778, #898 and #1177; PRs #1256 and #1582
diff --git a/docs/source/reference/cli.md b/docs/source/reference/cli.md
--- a/docs/source/reference/cli.md
+++ b/docs/source/reference/cli.md
@@ -60,7 +60,7 @@
openenv validate ./my_env --level runtime --local --output report.json-Runtime validation requires Docker and a validation.execution declaration in
+For served packages, runtime validation requires Docker and a validation.execution declaration in
openenv.yaml. The declaration points to a bounded JSON replay plan containing
one reset and a sequence of actions. The validator builds an immutable image,
collects the original episode, repeats the plan in a fresh WebSocket session and
@@ -69,6 +69,18 @@
during the fresh-container replay. Both use the same immutable image and are
removed even if a check fails.
+Harbor task.toml directories are also recognized. Install the harbor extra
+and add a validation/openenv.json contract that declares the reward key, range,
+oracle tolerance, floor margin and resources. The harbor-semantic-v1 binding runs
+fresh Harbor trials in Docker: a do-nothing agent, the oracle (solution/solve.sh),
+an oracle replay and any canaries declared in the contract. A canary is a known-wrong
+solve.sh replacement. The results report under runtime.startup,
+runtime.reward_well_formed, runtime.episode_determinism, semantic.oracle_max,
+semantic.floor_gap and semantic.canary_floor, and harbor-evidence.json keeps
+the raw trial evidence in the report bundle. Other checks are named SKIPs, so a
+passing task still reports WARN. This validates task packages, not the outer
+harbor_env server.
+
The 16 implemented runtime checks cover startup, rewards, observation schemas,
state continuity, network policy, host containment, resource bounds, episode isolation,
oracle containment, seed forwarding, determinism, emitted trajectory records, tool
@@ -77,7 +89,8 @@
the result WARN, which exits zero and does not mean Level 2 is complete.
FAIL exits 1, unsupported package formats exit 2, and internal or policy errors
exit 3. --level semantic
-includes the available lower-level checks but does not claim semantic execution.
+runs the semantic checks for Harbor tasks. Served packages have no oracle execution
+yet, so their semantic checks are named SKIPs.
--skip-build runs declaration checks and skips runtime execution entirely.
The Docker provider supports CPU workloads in public and no-network modes;
allowlist and GPU requirements skip runtime before building. A no-network
diff --git a/rfcs/008-environment-auto-validation.md b/rfcs/008-environment-auto-validation.md
--- a/rfcs/008-environment-auto-validation.md
+++ b/rfcs/008-environment-auto-validation.md
@@ -718,6 +718,66 @@
operator certification remain outside this feature; the test workflow only proves
the implementation's own acceptance cases.
+### Harbor task binding
+
+task.toml packages opt into the harbor-semantic-v1 profile through an
+operator-supplied validation/openenv.json contract. The contract declares the
+reward key, range, oracle tolerance, floor margin, finite resource budgets and,
+optionally, a directory of canaries. Harbor metadata and observed rewards never
+supply these declarations. The normalized schema-2 execution kind is harbor_task,
+with a process agent boundary. Parsing uses Harbor's data-only Task loader and
+never creates a Trial or executes package code.
+
+The profile supports single-step Linux tasks with one Dockerfile, one main Docker
+service, a shared verifier and public networking. Accelerators, external MCP
+services, task environment variables, credentials and Harbor extensions such as
+healthchecks or artifacts are refused with the reason named.
+
+Execution runs fresh Harbor Trials in a fixed order: a do-nothing (NOP) trial, the
+oracle (solution/solve.sh), an oracle replay, and one trial per declared canary.
+The NOP trial builds the image. The validator inspects the actual agent container
+at AGENT_START, retains its immutable image under a run-owned tag, and pins every
+later trial to that image and its effective container configuration. Each trial
+runs on a private copy of the task whose digest must equal the validated source
+digest before any overlay. A canary trial replaces solution/solve.sh in its copy
+with the canary script and records the script's digest. Trials run with
+delete=False and keep_containers=True, so the validator removes the exact
+run-owned Compose projects, including anonymous volumes registered from the
+containers' actual mounts, and verifies their absence. The oracle's exit code is
+read at AGENT_END, before sandbox log synchronization can overwrite it, and a
+later change fails the run.
+
+Results map to RFC check ids:
+
+| Check | Evidence |
+|---|---|
+| runtime.startup | Every trial ran from the pinned image in a fresh container, and cleanup was verified. |
+| runtime.reward_well_formed | Every raw reward map matches Harbor's result and is finite, nonboolean and in range. |
+| runtime.episode_determinism | The oracle replay reproduces every reward and the exit code. |
+| semantic.oracle_max | Both oracle runs exit 0 and earn at least max - oracle_tolerance. A task without solution/solve.sh fails. |
+| semantic.floor_gap | max - floor >= floor_margin, and oracle - floor >= floor_margin because the oracle may sit up to the tolerance below the maximum. |
+| semantic.canary_floor | Every declared canary earns no more than the measured floor. Selected only when canaries are declared. |
+
+The semantic ids depend on runtime.episode_determinism, so a diverging replay
+SKIPs them with the dependency named. Reward comparisons use exact rationals parsed
+from the raw verifier output. Every other local check is a named SKIP for this
+binding. It produces no WebSocket exchange, containment or resource measurement,
+second verification of one final state, or leakage scan. A passing run is therefore
+WARN, and full RFC qualification still rejects these SKIPs.
+
+Canaries are operator-declared, known-wrong solve.sh replacements, at most eight
+per task. They give local validation the package-shipped form of test #28.
+Generating wrong answers, for example by mutating the oracle, belongs to the
+adversarial panel of test #26, which remains unspecified.
+
+harbor-evidence.json records the contract digest, the Harbor version, and each
+trial's role, image identity, private-copy digests, overlays, raw reward file hashes
+and cleanup outcome. It and the raw reward and exit files are checksummed bundle
+artifacts beside the report.
+
+Served openenv_ws packages have no oracle execution contract yet, so their
+semantic checks are named SKIPs.
+
Explicitly out of scope
No hub/runner/queue/coordinator; no submission or auth APIs; no statistical-level implementations
diff --git a/src/openenv/cli/commands/validate.py b/src/openenv/cli/commands/validate.py
--- a/src/openenv/cli/commands/validate.py
+++ b/src/openenv/cli/commands/validate.py
@@ -231,10 +231,15 @@
Validate a local package or a running OpenEnv server.
Local validation detects the package format by its well-known file (the
- formats this build can parse; currently
openenv.yaml), parses it into the
-
formats this build can parse:
openenv.yamlor Harbortask.toml), parses it into the
normalized manifest, runs the applicable graders up to the requested level,
applies the severity policy, and emits a report. -
Native Harbor tasks require the optional Harbor dependency and an explicit
-
validation/openenv.jsoncontract. Fresh Docker trials grade the oracle, the -
do-nothing floor and any declared canaries under the semantic check ids. Checks
-
without an implementation are named SKIPs, so the report is WARN at best.
-
Exit codes: 0 pass/warn · 1 fail · 2 unrecognized/unsupported package · 3
internal or policy error.
diff --git a/src/openenv/harbor/rollout.py b/src/openenv/harbor/rollout.py
--- a/src/openenv/harbor/rollout.py
+++ b/src/openenv/harbor/rollout.py
@@ -251,12 +251,17 @@
force_build: bool = False,
training: bool = False,
harness_profile: str | None = None,
-
environment_kwargs: dict[str, Any] | None = None,
) -> Any:
"""Assemble Harbor'sTrialConfigfor one rollout.The seam decides how this particular agent learns about the proxy — an env var, a constructor
kwarg, or a config file written by a subclass. That is the only per-agent knowledge involved;
everything downstream is identical. -
Args:
-
environment_kwargs (`dict[str, Any]`, *optional*): -
Backend kwargs passed to Harbor. `None` keeps Harbor's defaults."""
from harbor.models.trial.config import (
AgentConfig,
@@ -312,6 +317,9 @@
),
delete=not keep_sandbox,
force_build=force_build, -
**( -
{"kwargs": environment_kwargs} if environment_kwargs is not None else {} -
), ), verifier=VerifierConfig(), trial_name=trial_name,
@@ -344,6 +352,8 @@
eval_sampling: dict[str, Any] | None = None,
harness_profile: str | None = None,
on_session_created: Callable[[str], None] | None = None,
- on_trial_created: Callable[[Any], None] | None = None,
- environment_kwargs: dict[str, Any] | None = None,
) -> HarborRolloutResult:
"""Run one rollout end to end. Never raises.
@@ -386,6 +396,11 @@
on_session_created (Callable[[str], None], optional):
Receive this rollout's capture session ID before the agent starts. Live views must
follow this ID rather than discover unrelated sessions from the shared registry.
-
on_trial_created (`Callable[[Any], None]`, *optional*): -
Trusted infrastructure callback invoked with the native Harbor trial before it runs. -
Exceptions fail the rollout and prevent trial execution. Not exposed through MCP. -
environment_kwargs (`dict[str, Any]`, *optional*): -
Backend kwargs passed to Harbor. `None` keeps Harbor's defaults.Returns:
[HarborRolloutResult]: Reward, per-turn token ids and logprobs, and validation findings.
@@ -488,6 +503,7 @@
force_build=force_build,
training=sampling is not None,
harness_profile=harness_profile, -
environment_kwargs=environment_kwargs, ) # Held across construction because Harbor's wrappers read `os.environ` while building the
@@ -503,6 +519,8 @@
if not proc_env:
config = build_trial_config(**config_kwargs)
trial = await Trial.create(config)
-
if on_trial_created is not None: -
on_trial_created(trial) trial_result = await trial.run() else: # The override has to span `run()`, not just construction. Harbor's goose wrapper reads
@@ -535,6 +553,8 @@
with ctx_env_overlay(proc_env):
config = build_trial_config(**config_kwargs)
trial = await Trial.create(config)
-
if on_trial_created is not None: -
on_trial_created(trial) trial_result = await trial.run() else: # Fallback: mutate the real environment under a lock, one rollout at a time.
@@ -543,6 +563,8 @@
with process_env(seam.name, proc_env):
config = build_trial_config(**config_kwargs)
trial = await Trial.create(config)
-
if on_trial_created is not None: -
on_trial_created(trial) trial_result = await trial.run() finally: _PROC_ENV_LOCK.release()
diff --git a/src/openenv/harbor/runner.py b/src/openenv/harbor/runner.py
--- a/src/openenv/harbor/runner.py
+++ b/src/openenv/harbor/runner.py
@@ -11,6 +11,7 @@
import os
import secrets
from pathlib import Path
+from typing import Any, Callable
from openenv.core.harness.capture import CaptureServer
@@ -44,6 +45,10 @@
auth_header: str = "Authorization",
provider: str = "openai",
admin_key: str | None = None,
- agent_timeout_sec: float | None = None,
- agent_step_limit: int | None = None,
- on_trial_created: Callable[[Any], None] | None = None,
- environment_kwargs: dict[str, Any] | None = None,
) -> list[HarborRolloutResult]:
"""Runtask_indicesfromdatasetand print a per-rollout report.
@@ -64,6 +69,15 @@
Key the capture proxy's session-management routes require. Defaults to
$OPENENV_CAPTURE_ADMIN_KEY, else a random one minted for this batch. The proxy is
published on a public URL, so these routes are never left open.
-
agent_timeout_sec (`float`, *optional*): -
Override each agent's execution timeout in seconds. -
agent_step_limit (`int`, *optional*): -
Limit each rollout's agent steps and captured model calls. -
on_trial_created (`Callable[[Any], None]`, *optional*): -
Trusted infrastructure callback invoked with each native Harbor trial before it runs. -
Exceptions fail that rollout and prevent trial execution. Not exposed through MCP. -
environment_kwargs (`dict[str, Any]`, *optional*): -
Backend kwargs passed to Harbor. `None` keeps Harbor's defaults.Returns:
list[HarborRolloutResult]: One per index, in order.
@@ -163,9 +177,13 @@
# training path (run_rollout's default) still refuses to pick a signal.
require_reward=False,
keep_sandbox=keep_sandbox, -
agent_timeout_sec=agent_timeout_sec, -
agent_step_limit=agent_step_limit, force_build=force_build, capture_level=capture_level, inference=capture.inference, -
on_trial_created=on_trial_created, -
environment_kwargs=environment_kwargs, ) results.append(result) print(" " + _summarise(result))
diff --git a/src/openenv/validation/graders/init.py b/src/openenv/validation/graders/init.py
--- a/src/openenv/validation/graders/init.py
+++ b/src/openenv/validation/graders/init.py
@@ -13,7 +13,12 @@
from ..manifest import CapabilitiesSpec, NormalizedManifest
from ..providers import RunningSubject
from ..report import CheckResult
-from ..runtime.contracts import HttpProbe, IsolationEvidence, RuntimeEvidence
+from ..runtime.contracts import (
- HttpProbe,
- IsolationEvidence,
- OracleEvidence,
- RuntimeEvidence,
+)
from ..types import Level, ProviderCapability
ENTRY_POINT_GROUP = "openenv.validation.graders"
@@ -46,6 +51,8 @@
Same-session and fresh episodes on the running subject.
oracle_http (tuple[HttpProbe, ...]):
GET probes for the declared oracle path on the control endpoint.
-
oracle_evidence ([`~openenv.validation.runtime.contracts.OracleEvidence`], *optional*): -
Oracle, do-nothing and canary rewards for the Level 3 checks."""
root: Path
@@ -59,6 +66,7 @@
resource_evidence_json: str | None = None
isolation_evidence: IsolationEvidence | None = None
oracle_http: tuple[HttpProbe, ...] = () -
oracle_evidence: OracleEvidence | None = None
@runtime_checkable
diff --git a/src/openenv/validation/graders/semantic/init.py b/src/openenv/validation/graders/semantic/init.py
new file mode 100644
--- /dev/null
+++ b/src/openenv/validation/graders/semantic/init.py
@@ -1,0 +1,5 @@
+"""Semantic-level graders."""
+
+from .oracle import CanaryFloorGrader, FloorGapGrader, OracleMaxGrader
+
+all = ["CanaryFloorGrader", "FloorGapGrader", "OracleMaxGrader"]
diff --git a/src/openenv/validation/graders/semantic/oracle.py b/src/openenv/validation/graders/semantic/oracle.py
new file mode 100644
--- /dev/null
+++ b/src/openenv/validation/graders/semantic/oracle.py
@@ -1,0 +1,145 @@
+"""Verifier sanity over oracle, do-nothing and canary rewards (#778 tests #25, #28).
+
+The execution binding measures the rewards; these graders only compare them with the
+manifest's declared range, tolerance and margin. Comparisons use exact rationals so a
+tiny floor cannot round into the margin.
+"""
+
+import time
+from fractions import Fraction
+
+from ...report import CheckResult
+from ...types import CheckStatus, Level
+
+
+class _SemanticGrader:
- level = Level.SEMANTIC
- requires_capabilities = frozenset()
- requires_provider = frozenset()
-
RFC 008: episode determinism is a precondition of every semantic check.
- depends_on = ("runtime.episode_determinism",)
- remediation = ""
- def applies_to(self, manifest) -> bool:
-
return True - def run(self, subject) -> CheckResult:
-
started = time.monotonic() -
evidence = subject.oracle_evidence -
if evidence is None: -
return CheckResult( -
check_id=self.check_id, -
status=CheckStatus.SKIP, -
evidence=["oracle evidence is unavailable"], -
duration_s=0, -
) -
reward = subject.manifest.reward -
problems, measured = self.check( -
evidence, -
maximum=Fraction(str(reward.range[1])), -
tolerance=Fraction(str(reward.oracle_tolerance)), -
margin=Fraction(str(reward.floor_margin)), -
) -
return CheckResult( -
check_id=self.check_id, -
status=CheckStatus.FAIL if problems else CheckStatus.PASS, -
measured=measured, -
evidence=problems or [self.passed], -
remediation=self.remediation if problems else None, -
duration_s=time.monotonic() - started, -
)
+class OracleMaxGrader(_SemanticGrader):
- """Every oracle run exits cleanly and earns at least
max - oracle_tolerance.""" - check_id = "semantic.oracle_max"
- passed = "every oracle run reached the declared maximum within tolerance"
- remediation = (
-
"Fix the oracle or the verifier so a correct solution exits cleanly and " -
"earns the declared maximum reward." - )
- def check(self, evidence, *, maximum, tolerance, margin):
-
threshold = maximum - tolerance -
problems = [] -
for run, (reward, code) in enumerate( -
zip(evidence.oracle, evidence.oracle_exit_codes, strict=True), start=1 -
): -
if code != 0: -
problems.append(f"oracle run {run} exited with code {code}") -
if reward < threshold: -
problems.append( -
f"oracle run {run} earned {float(reward)}, below {float(threshold)}" -
) -
return problems, { -
"oracle_rewards": [float(reward) for reward in evidence.oracle], -
"oracle_exit_codes": list(evidence.oracle_exit_codes), -
"declared_max": float(maximum), -
"oracle_tolerance": float(tolerance), -
}
+class FloorGapGrader(_SemanticGrader):
- """
- The do-nothing reward sits at least
floor_marginbelow the maximum and below - every oracle run. The oracle may sit up to the tolerance below the maximum, so
- both gaps are required.
- """
- check_id = "semantic.floor_gap"
- passed = "the do-nothing reward leaves the declared margin"
- remediation = (
-
"The verifier rewards doing nothing. Check for vacuous success conditions, " -
"such as comparisons against empty output." - )
- def check(self, evidence, *, maximum, tolerance, margin):
-
floor = evidence.floor -
problems = [] -
if maximum - floor < margin: -
problems.append( -
f"do-nothing reward {float(floor)} is within {float(maximum - floor)} " -
f"of the maximum, below the margin {float(margin)}" -
) -
lowest = min(evidence.oracle) -
if lowest - floor < margin: -
problems.append( -
f"oracle reward {float(lowest)} is within {float(lowest - floor)} " -
f"of the do-nothing reward, below the margin {float(margin)}" -
) -
return problems, { -
"measured_floor": float(floor), -
"oracle_reward": float(lowest), -
"declared_max": float(maximum), -
"declared_margin": float(margin), -
}
+class CanaryFloorGrader(_SemanticGrader):
- """Every declared canary, a known-wrong solution, earns no more than doing nothing."""
- check_id = "semantic.canary_floor"
- requires_capabilities = frozenset({"canaries"})
- passed = "every canary stayed at the do-nothing reward"
- remediation = (
-
"The verifier rewards a known-wrong solution. Tighten the tests so each " -
"canary earns no more than doing nothing." - )
- def check(self, evidence, *, maximum, tolerance, margin):
-
problems = [ -
f"canary {name} earned {float(reward)}, above the do-nothing reward " -
f"{float(evidence.floor)}" -
for name, reward in evidence.canaries -
if reward > evidence.floor -
] -
# An empty run list must not pass vacuously. -
if not evidence.canaries: -
problems.append("no canary run was recorded") -
return problems, { -
"measured_floor": float(evidence.floor), -
"canary_rewards": { -
name: float(reward) for name, reward in evidence.canaries -
}, -
}
diff --git a/src/openenv/validation/harbor.py b/src/openenv/validation/harbor.py
new file mode 100644
--- /dev/null
+++ b/src/openenv/validation/harbor.py
@@ -1,0 +1,931 @@
+"""Native Harbor execution: fresh Docker trials graded through RFC 008 check ids.
+
+No served-protocol evidence is synthesized. A do-nothing trial builds the image, and
+the oracle, its replay and any canaries run pinned to that image.
+"""
+
+import asyncio
+import hashlib
+import json
+import os
+import re
+import shutil
+import stat
+import subprocess
+import tempfile
... diff truncated: showing 800 of 4391 lines
</details>
<sub>You can send follow-ups to the cloud agent <a href="https://cursor.com/agents/bc-c181c873-0478-43d0-ba3e-a487f5bbe6d4">here</a>.</sub>
</details>
…suite Every report now names the nine static checks without a grader as SKIPs. A served run where every runtime check passes therefore reads WARN until those graders land. The acceptance tests derive the expected verdict from a STATIC_PENDING set, check that every report names those SKIPs with their reason, and include them in coverage.json's incomplete list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Python skips assert statements under -O, so the side effect would vanish and the later count check would change meaning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Require the NOP trial to build the reviewed Dockerfile. Refuse environment.docker_image before any trial can retain and reuse a registry image.
Grade a missing solution whenever semantic validation is requested and static checks pass, including skipped builds and unsupported providers.
Treat Harbor reward-file and parse exceptions as reward failures, preserving raw output. Report Dockerfile failures and subject timeouts as phase-specific startup FAILs while keeping validator and cleanup failures as ERRORs.
Grade the verified NOP, oracle and replay prefix even if a later canary fails. Keep incomplete canary collections ungraded and preserve the startup cleanup error.
Classify retained-image drift and failed image verification as host failures, even when the same trial has malformed rewards.
Forward report_target from the runner to the native binding so Space reports and their bundles identify the resolved Space revision.
Reuse the Docker validator build budget for NOP build/start and resume the episode deadline afterward. Kill and reap Compose process groups on cancellation, including spawn races. Keep daemon completion explicitly unverified for cancelled allocating commands and record their project and operation before cleanup can claim success.
Exercise image, label, retained-tag and configuration checks with fake Docker responses. Cover symlink, FIFO, bounded-read and size guards with temporary local files, and verify guard-removal mutants fail.
Keep pre-allocation inventory checks inside trial evidence collection without cleaning projects whose ownership was not verified. Preserve trusted NativeEvidenceError reasons, remove stale pilot wording, and import the JSON object hook once.
* Add bounded operator coding audits with HF Sandboxes * Parse candidate Bash fences and retain raw response hashes * fix(coding-audit): reject linked artifacts while snapshotting Read original task files through the descriptor-based reader before writing the frozen copy, so copying cannot erase hard-link evidence. Cover context, instruction, verifier and reference artifacts through the runner with offline SDK fakes. * fix(coding-audit): reject empty tool-call completions Require a nonempty tool-call list when the provider finishes with tool_calls. Exercise null, missing and empty calls through collection and scoring after a valid terminal action, and verify failed targets are never graded. * fix(coding-audit): correct offline test import path Include the repository root in PYTHONPATH so pytest can import the examples package. Verified the documented command with all 417 offline tests passing.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Episode timeout starves verifier
- Removed the duplicate whole-trial timeout so Harbor's independently bounded setup, agent, and verifier phases can each complete.
Preview (1a165f3597)
diff --git a/docs/plans/rfc008-validation-completion.md b/docs/plans/rfc008-validation-completion.md
new file mode 100644
--- /dev/null
+++ b/docs/plans/rfc008-validation-completion.md
@@ -1,0 +1,302 @@
+# Completing RFC 008 validation
+
+Planning baseline: 2026-11-10, `openenv/OpenEnv` main at
+`fb36d7d62ff9548ebd44f3323386cbea908e8706`.
+
+Trackers: [RFC 008 #778](https://github.com/openenv/OpenEnv/issues/778),
+[Level 2 umbrella #1177](https://github.com/openenv/OpenEnv/issues/1177) and
+[task packages #898](https://github.com/openenv/OpenEnv/issues/898).
+
+RFC 008 promises that `openenv validate` grades an environment against the 44
+acceptance tests of #778. At the baseline, main ran every Level 2 runtime check for
+served environments on Docker, but no Level 3 semantic check. That includes the
+check the RFC opens with: a correct solution must earn maximum reward, and doing
+nothing must not. The PR that adds this document delivers that check for Harbor
+task packages and makes every report name the checks it did not run. This document
+records where validation stands after that PR, defines what "complete" means, and
+ranks the remaining work as PR-sized workstreams.
+
+## Where validation stands
+
+Each RFC check has a stable id, a level and a lane. The lane says who can run the
+check. `local` checks run inside `openenv validate`. `hub` and `statistical` ids are
+reserved for operators, the teams that run validation hubs or lab intake. The table
+counts the local ids in the committed severity policy and the ids each execution
+binding grades.
+
+| Level | Local ids | Served packages (`openenv_ws`) | Harbor tasks (`harbor_task`) |
+|---|---|---|---|
+| 1 Static | 10 | 1 (`static.manifest`) | 1 (`static.manifest`) |
+| 2 Runtime | 16 | 16, on Docker-local | 3: startup, reward well-formedness, episode determinism |
+| 3 Semantic | 8 | 0 | 3: oracle reaches max, floor gap, canary floor |
+| 4 Statistical | 0 (7 reserved) | Operator-side by design | Operator-side by design |
+| Hub | 0 (5 reserved) | Operator-side by design | Operator-side by design |
+
+Every requested local check that does not run is a named SKIP with its reason, at
+every level. A report therefore reads WARN until all requested checks pass, and a
+PASS on a subset cannot pass for full coverage.
+
+Three limits apply across bindings:
+
+- **Parsers.** `openenv.yaml` and Harbor `task.toml` ship. PostTrain `task.md`
+ packages are refused as unrecognized.
+- **Providers.** Docker-local enforces `public` and `no-network` networking and
+ refuses allowlists and GPUs. The HF Sandbox provider refuses every launch with
+ `missing provider capability: immutable_image`, because nothing yet binds a Space
+ revision to an image digest.
+- **Served oracles.** RFC 008 does not yet say how to execute an oracle against a
+ served WebSocket environment, so served packages report their semantic checks as
+ named SKIPs.
+
+Work outside main measures more than the RFC's local checks. A cloud audit on HF
+Jobs adds generated reference solutions, wrong-answer controls, repeat runs,
+leakage probes and target-model pass rates.
+[#1582](https://github.com/openenv/OpenEnv/pull/1582) adds training and validator
+performance experiments. These results report under their own fields rather than
+RFC check ids, so they do not count toward RFC coverage yet.
+
+## What complete means
+
+The RFC splits responsibility between the repo and operators, so completion has two
+targets. The repo can reach the first one alone. The second one needs an operator
+procedure that the repo documents but does not run.
+
+**Local completeness.** Every local-lane id has a grader for each shipped execution
+binding where it applies, and reports name every inapplicable or unimplemented id
+with its reason. For served packages this means nine static graders, an oracle
+execution contract and eight semantic graders. For Harbor tasks it means the same
+static graders, the remaining semantic graders, and a decision on which runtime ids
+apply to one-shot tasks.
+
+**Operator completeness.** Each reserved `statistical.*` id has at least one
+documented reference procedure that emits results in the shared report schema. The
+repo still ships no hub, as the RFC requires, so the procedure lives in an example or
+an operator repository. Comparable operator results also need the metric namespace
+(`pass@k`, headroom, variance bound) that the RFC defers to a future RFC.
+
+Validation still certifies that an environment can be trained on, not that it should
+be. Measuring training signal comes last, and it belongs with the evaluation work in
+[RFC 013 (#1256)](https://github.com/openenv/OpenEnv/pull/1256).
+
+## Proposed workstreams
+
+The workstreams are ranked by dependency and by how much each one improves
+environment selection. Each one is one to three PRs. Costs are relative: S needs no
+build or inference, M needs Docker builds and replays, and L needs model inference
+or GPU time. Owners are suggestions for discussion.
+
+| Rank | Workstream | #778 tests | Status | Cost | Suggested owner |
+|---|---|---|---|---|---|
+| 1 | Verifier sanity graders | #11, #25 | Harbor done; envelope and served open | M | Validation core |
+| 2 | Harbor binding on main | Harbor parser slice | Done | M | Validation core |
+| 3 | Verifier strength | #28, #26 procedure | Harbor canaries done; panel open | M, then L | Validation core and Hub pipeline |
+| 4 | Determinism, replay and leakage graders | #30, #39, #40, #44 | Open | M | Validation core |
+| 5 | Static graders and complete reports | #1, #3, #4, #8, #9, #10, #12, #41, #42 | Reports done; graders open | S to M | Any contributor |
+| 6 | Cloud execution on HF Sandbox | Provider qualification | Open | M | Hub pipeline |
+| 7 | Statistical reference procedure | #13 to #16, #27 | Open | L | Hub pipeline |
+| 8 | Wider task shapes | Workspace handoff, multi-container, credentials | Open | M to L | Hub pipeline and validation core |
+| 9 | Training signal | #17 | Open | L | Validation experiments and RFC 013 |
+
+### 1. Verifier sanity graders
+
+`semantic.oracle_max` and `semantic.floor_gap` now grade Harbor tasks. The graders
+read a format-neutral `OracleEvidence`, so another binding only has to measure the
+rewards.
+
+**Remaining.** Served packages need an oracle execution contract first: the RFC
+names injected state and executable scripts, but not how either drives a served
+WebSocket session or what doing nothing means there. That needs an RFC amendment.
+`semantic.resource_envelope` also remains: record peak CPU, memory and disk during
+the oracle run and compare them with the declared resources.
+
+**Evidence.** Oracle and do-nothing rewards, the declared margin and tolerance, the
+image digest and the cleanup outcome.
+
+**Cost and owner.** M: Docker builds and short episodes, with no inference.
+Validation core.
+
+### 2. Harbor binding on main
+
+Done. Harbor `task.toml` packages validate through the `harbor-semantic-v1`
+profile and an operator-supplied `validation/openenv.json` contract. Fresh Docker
+trials run a do-nothing agent, the oracle, an oracle replay and declared canaries,
+all pinned to one immutable image.
+
+**Remaining.** Decide which runtime ids apply to one-shot tasks. Containment,
+network policy, resource bounds and oracle containment could apply. Observation
+schema, state contract and tool declaration accuracy do not.
+
+**Cost and owner.** M. Validation core.
+
+### 3. Verifier strength: canaries and an adversarial panel
+
+A verifier can pass the oracle and floor checks and still reward wrong answers.
+`semantic.canary_floor` now runs operator-declared canaries on Harbor tasks: each
+known-wrong `solve.sh` replacement must earn no more than doing nothing.
+
+**Remaining.** The reserved `statistical.adversarial_floor` (#26) still has no
+procedure; the RFC notes that the panel "is not yet specified anywhere". Write an
+RFC 008 amendment that specifies:
+
+- where wrong answers come from: mutations of the reference solution, known-fault
+ corpora, empty and partial outputs, and plausible solutions from a weaker model
+ that an independent check confirms are wrong;
+- how many are needed for a verdict;
+- the pass rule, which is that every wrong answer scores at the floor.
+
+The cloud audit's wrong-answer controls are a working prototype of this panel.
+
+**Evidence.** Each wrong answer's provenance and reward, plus the verifier version.
+
+**Cost and owner.** L for generated panels, which need inference. Validation core
+writes the amendment, and the Hub pipeline team implements the operator panel.
+
+### 4. Determinism, replay and leakage graders
+
+`semantic.verifier_determinism`, `semantic.verifier_portability`,
+`semantic.replayability` and `semantic.no_solution_leakage` have no graders.
+Harbor tasks now establish `runtime.episode_determinism` with an oracle replay in a
+fresh container, which these checks depend on.
+
+**Approach.** Verifier determinism runs the verifier on one final state several
+times and requires identical rewards. For `llm_judged` environments it requires
+variance within the declared bound, reusing the Level 2 variance path. Portability
+reruns the verifier with networking denied, as the RFC's hermeticity rule for #40
+requires. Replayability re-executes a recorded trajectory and compares rewards. The
+leakage check scans observations and agent-visible files for oracle artifacts such
+as solution text, expected outputs and test files.
+
+**Evidence.** Replay transcripts and reward vectors, the network mode of each
+verifier run, and every leakage match with its location.
+
+**Cost and owner.** M. Validation core.
+
+### 5. Static graders and complete reports
+
+Reports now name every unimplemented static check as a SKIP, so a static run reads
+WARN rather than PASS after one check.
+
+**Remaining.** Add the nine graders in two batches. The first batch reads the
+manifest and lockfiles and needs no build: `static.dependency_pinning`,
+`static.task_distribution_pinning`, `static.resource_declaration` and
+`static.timeout_ceiling`. The second batch inspects built images:
+`static.oci_labels`, `static.sbom`, `static.image_hygiene`, `static.layout` and
+`static.reproducible_build`, which builds twice and compares digests.
+
+**Evidence.** The files and image metadata each grader read.
+
+**Cost and owner.** S for the first batch and M for the image batch. Any
+contributor; these make good first issues.
+
+### 6. Cloud execution on HF Sandbox
+
+The HF Sandbox provider refuses every launch because it has no immutable image
+binding. Runtime and semantic checks therefore cannot run in the cloud, or on GPUs,
+through the RFC provider interface. Cloud audits work around this with their own
+runner.
+
+**Approach.** Resolve a Space revision to the registry digest of the image it
+builds, verify it, and launch by digest. Qualify public Space images first, then add
+an authenticated pull path for private ones. Once launches work, the cloud audit can
+call the RFC graders instead of its own runner.
+
+**Evidence.** The Space revision, the resolved digest, the pull receipt and the same
+cleanup evidence Docker-local records.
+
+**Cost and owner.** M. Hub pipeline team.
+
+### 7. Statistical reference procedure
+
+The learnable tests (#13 to #16) and the gameability gap (#27) are reserved ids with
+no reference procedure. The RFC also defers the shared metric namespace.
+
+**Approach.** Write the metric-namespace RFC, with definitions, sample sizes and
+confidence intervals for `pass@k`, headroom and variance bound. Then publish a
+reference operator procedure, outside the core package, that maps target-model
+attempts onto the reserved ids:
+
+- `statistical.reward_reachability`: some attempt reaches maximum reward;
+- `statistical.difficulty_separation`: pass rates differ across model tiers;
+- `statistical.headroom`: the target model's pass rate leaves room to improve;
+- `statistical.variance_bound`: reward noise across repeats stays within a bound.
+
+The cloud audit already measures target-model pass rates with confidence intervals.
+It needs to emit them under these ids in hub-lane reports.
+
+**Evidence.** Model identity and provider, attempt counts, per-attempt rewards and
+intervals.
+
+**Cost and owner.** L, since every task needs inference. Hub pipeline team, with the
+metric-namespace RFC from validation core.
+
+### 8. Wider task shapes
+
+The Harbor binding refuses multi-step tasks, task-authored Docker Compose files and
+tasks that need credentials. The cloud audit also hands off only individual output
+files. The RFC defers credential delivery explicitly.
+
+**Approach.** Add a workspace or git-patch handoff so repository tasks are graded on
+the agent's full change. Treat multi-container support as a provider capability, as
+the RFC does for GPUs, so providers without it SKIP with the capability named. Write
+the credential-delivery amendment the RFC asks for, covering delivery, access
+isolation, lifetime, network access and evidence redaction.
+
+**Evidence.** Handoff manifests with hashes, and provider capability reports.
+
+**Cost and owner.** M for workspace handoff and L for multi-container and
+credentials. The Hub pipeline team builds the adapters, and validation core writes
+the amendment.
+
+### 9. Training signal
+
+`statistical.training_signal` (#17, advisory) asks whether training on an environment
+improves a model. Nothing measures it yet.
+
+**Approach.** Use the evaluation harness proposed in RFC 013 to run short training
+jobs on a sample of validated environments, then report the change in held-out
+success. The training experiments in #1582 are a first measurement. Keep the result
+advisory, as the RFC's severity policy says, because a short run cannot show that an
+environment is unlearnable.
+
+**Evidence.** Training configuration, seeds, and held-out success before and after
+training.
+
+**Cost and owner.** L: GPU time. Validation experiments, with the RFC 013 authors.
+
+## Sequencing
+
+The verifier sanity graders came first because canaries, verifier determinism and
+replay all reuse oracle replay. The order of the remaining work follows from its
+dependencies:
+
+- Workstream 4 builds on the oracle replay the Harbor binding now runs.
+- Workstream 5 is independent and can start now.
+- Workstream 6 moves the Harbor and served checks to the cloud and is needed before
+ workstream 7 runs at scale.
+- The served half of workstream 1 waits on the oracle execution amendment.
+- Workstreams 8 and 9 widen coverage after the core lands.
+
+## Open questions for maintainers
+
+These questions change the scope of a workstream, so they need an answer before its
+first PR:
+
+1. How should an oracle drive a served WebSocket environment, and what is its
+ do-nothing episode?
+2. Should `semantic.canary_floor` also accept canaries the validator generates, or
+ only canaries shipped with the package?
+3. Should the adversarial panel be an RFC 008 amendment or its own RFC?
+4. Where should operator reference procedures live: `examples/` or a separate
+ repository?
+5. Should the runtime applicability table for one-shot task packages be normative
+ in the RFC?
+
+## Sources
+
+- `rfcs/008-environment-auto-validation.md`, including the Harbor task binding
+ section
+- `docs/plans/rfc008-level2-implementation.md`
+- `tests/fixtures/validation/runtime/cases.json`
+- `src/openenv/validation/runner.py`, `harbor.py`, `graders/semantic/` and
+ `providers/hf_sandbox.py`
+- Issues #778, #898 and #1177; PRs #1256 and #1582
diff --git a/docs/source/reference/cli.md b/docs/source/reference/cli.md
--- a/docs/source/reference/cli.md
+++ b/docs/source/reference/cli.md
@@ -60,7 +60,7 @@
openenv validate ./my_env --level runtime --local --output report.json-Runtime validation requires Docker and a validation.execution declaration in
+For served packages, runtime validation requires Docker and a validation.execution declaration in
openenv.yaml. The declaration points to a bounded JSON replay plan containing
one reset and a sequence of actions. The validator builds an immutable image,
collects the original episode, repeats the plan in a fresh WebSocket session and
@@ -69,6 +69,18 @@
during the fresh-container replay. Both use the same immutable image and are
removed even if a check fails.
+Harbor task.toml directories are also recognized. Install the harbor extra
+and add a validation/openenv.json contract that declares the reward key, range,
+oracle tolerance, floor margin and resources. The harbor-semantic-v1 binding runs
+fresh Harbor trials in Docker: a do-nothing agent, the oracle (solution/solve.sh),
+an oracle replay and any canaries declared in the contract. A canary is a known-wrong
+solve.sh replacement. The results report under runtime.startup,
+runtime.reward_well_formed, runtime.episode_determinism, semantic.oracle_max,
+semantic.floor_gap and semantic.canary_floor, and harbor-evidence.json keeps
+the raw trial evidence in the report bundle. Other checks are named SKIPs, so a
+passing task still reports WARN. This validates task packages, not the outer
+harbor_env server.
+
The 16 implemented runtime checks cover startup, rewards, observation schemas,
state continuity, network policy, host containment, resource bounds, episode isolation,
oracle containment, seed forwarding, determinism, emitted trajectory records, tool
@@ -77,7 +89,8 @@
the result WARN, which exits zero and does not mean Level 2 is complete.
FAIL exits 1, unsupported package formats exit 2, and internal or policy errors
exit 3. --level semantic
-includes the available lower-level checks but does not claim semantic execution.
+runs the semantic checks for Harbor tasks. Served packages have no oracle execution
+yet, so their semantic checks are named SKIPs.
--skip-build runs declaration checks and skips runtime execution entirely.
The Docker provider supports CPU workloads in public and no-network modes;
allowlist and GPU requirements skip runtime before building. A no-network
diff --git a/examples/coding_audit/README.md b/examples/coding_audit/README.md
new file mode 100644
--- /dev/null
+++ b/examples/coding_audit/README.md
@@ -1,0 +1,66 @@
+# Operator coding audit
+
+This example collects and scores bounded coding-task evidence. It belongs to the
+operator workflow, separate from the installed OpenEnv validator and its RFC 008
+checks. Acceptance here does not establish full RFC 008 conformance, containment,
+served-model revision attestation, or training benefit.
+
+From a repository checkout, install the tested SDK and run the example:
+
+```bash
+uv sync --group dev
+uv pip install 'huggingface_hub==1.33.0'
+PYTHONPATH=src:envs uv run --no-sync python -m examples.coding_audit \
- --task /path/to/task \
- --config examples/coding_audit/config.json \
- --output /path/to/new-audit-directory \
- --json
+```
+The output directory is required and must be new. It retains report.json,
+evidence.json, config.json, raw evidence artifacts, and SHA256SUMS. Exit codes
+are 0 for accepted evidence, 1 for rejected or incomplete evidence, and 2 for
+configuration or input failure. --json prints the report as well.
+
+scoring.py provides pure assess_coding_audit and validate_audit_config
+functions. runner.py provides normalize_coding_config, coding_budget,
+validate_coding_task, and run_coding_validation. hf.py contains
+HFTaskBackend, the bounded Sandbox and Inference Providers implementation.
+The schemas bind task, verifier, immutable image references and independent execution identities.
+
+Models may declare an explicit boolean enable_thinking, forwarded through the
+HF client's extra_body.chat_template_kwargs. It is omitted by default. A
+candidate-only override leaves the target protocol unchanged and is recorded
+alongside request parameters. Support depends on the selected provider/model.
+Candidate generation streams bounded content and reasoning; only a complete
+stop response with validated usage can produce an executable candidate. A
+length response remains rejected even when its terminal usage is retained.
+Recognized Bash/sh Markdown fences are parsed before execution; the raw response
+and both raw/script hashes are retained. Other language fences are rejected.
+
+Candidate generation can opt into public task API context by adding
+"context_paths": ["environment/source/package/api.py"] to the generator
+model configuration. Only those declared regular UTF-8 files are read; paths must
+be unique, normalized, and below environment/source/. Up to 32 files and 65,536
+combined bytes are allowed. Linked files or parents, hidden or private paths,
+NUL bytes, invalid UTF-8 and oversized context are rejected before inference.
+The existing message and token budgets still apply. No source files are added by
+default, and target.context_paths is rejected.
+
+The candidate prompt labels each source path and its runtime filename relative to
+the configured workdir, with environment/source/ removed. Each generation record
+retains candidate_context entries containing path, runtime_path, sha256,
+and size, bound to the frozen task and checksummed evidence. This option does
+not enumerate task directories or include verifier, reference or specification
+files.
+
+The HF backend launches paid services when invoked without a supplied test
+backend. Use reviewed tasks and distinct execution and inference credentials.
+The Hub pipeline's cloud worker packages these production example modules with
+its frozen source and task bundle; its tests stay outside the bundle.
+
+Run the offline suite with:
+
+bash +PYTHONPATH=.:src:envs uv run pytest examples/coding_audit -q +
diff --git a/examples/coding_audit/init.py b/examples/coding_audit/init.py
new file mode 100644
--- /dev/null
+++ b/examples/coding_audit/init.py
@@ -1,0 +1 @@
+"""Operator-owned coding audits, separate from core RFC 008 validation."""
diff --git a/examples/coding_audit/main.py b/examples/coding_audit/main.py
new file mode 100644
--- /dev/null
+++ b/examples/coding_audit/main.py
@@ -1,0 +1,63 @@
+"""Run a bounded operator coding audit and retain its evidence bundle."""
+
+import argparse
+import signal
+import sys
+import threading
+from contextlib import contextmanager
+from pathlib import Path
+
+from openenv.validation.runtime.json_values import dumps, loads
+
+from .runner import run_coding_validation
+
+
+@contextmanager
+def _sigterm_as_interrupt():
- if threading.current_thread() is not threading.main_thread():
-
yield -
return - def interrupt(signum, frame):
-
raise KeyboardInterrupt - previous = signal.signal(signal.SIGTERM, interrupt)
- try:
-
yield - finally:
-
signal.signal(signal.SIGTERM, previous)
+def main(argv=None):
- parser = argparse.ArgumentParser(description=doc)
- parser.add_argument("--task", type=Path, required=True)
- parser.add_argument("--config", type=Path, required=True)
- parser.add_argument(
-
"--output", -
type=Path, -
required=True, -
help="New directory for the report, evidence and checksummed artifacts", - )
- parser.add_argument("--json", action="store_true", dest="json_output")
- args = parser.parse_args(argv)
- try:
-
config = loads(args.config.read_text()) -
with _sigterm_as_interrupt(): -
report = run_coding_validation(args.task, config, args.output) - except (OSError, ValueError) as error:
-
print( -
f"Coding audit configuration/input failure ({type(error).__name__})", -
file=sys.stderr, -
) -
return 2 - if args.json_output:
-
print(dumps(report)) - else:
-
print(f"Coding audit: {report['verdict'].upper()} ({report['profile']})") -
for name, check in report["checks"].items(): -
print(f" {check['status'].upper():5s} {name}") - return 0 if report["accepted"] else 1
+if name == "main":
- raise SystemExit(main())
diff --git a/examples/coding_audit/config.json b/examples/coding_audit/config.json
new file mode 100644
--- /dev/null
+++ b/examples/coding_audit/config.json
@@ -1,0 +1,83 @@
+{
- "schema_version": 1,
- "profile": "harbor-coding-audit-v1",
- "target": {
- "model_id": "Qwen/Qwen3.8-27B",
- "revision": "37561816e66b5b9153b1b64c10968c18c1e6f4a0",
- "provider": "deepinfra",
- "supports_seed": false,
- "supports_tools": true,
- "temperature": 0.7,
- "max_steps": 16,
- "max_tokens": 2048,
- "max_total_tokens": 16384
- },
- "execution": {
- "agent_image": "docker.io/library/python@sha256:48d884813cf44e927cf64c5529dd630dd3b18338c6dc727991eef9b71e934a20",
- "verifier_image": "docker.io/library/python@sha256:48d884813cf44e927cf64c5529dd630dd3b18338c6dc727991eef9b71e934a20",
- "flavor": "cpu-basic",
- "timeout_seconds": 120,
- "workdir": "/app",
- "submission_paths": [
-
"/app/answer.txt" - ],
- "private_paths": [
-
"/tests", -
"/solution" - ],
- "verifier_command": [
-
"/bin/bash", -
"/tests/test.sh" - ],
- "reward_path": "/logs/verifier/reward.txt",
- "reward_key": "reward"
- },
- "validation": {
- "reference_repeats": 3,
- "target_attempts": 8,
- "reward_min": 0,
- "reward_max": 1,
- "oracle_tolerance": 0,
- "floor_margin": 0.1,
- "wrong_answer_max": 0,
- "required_negative_cases": [
-
"wrong-value", -
"empty-answer", -
"near-miss" - ]
- },
- "generation": {
- "candidate_count": 2
- },
- "negative_cases": [
- {
-
"id": "wrong-value", -
"script": "mkdir -p /app; printf '0\\n' > /app/answer.txt" - },
- {
-
"id": "empty-answer", -
"script": "mkdir -p /app; : > /app/answer.txt" - },
- {
-
"id": "near-miss", -
"script": "mkdir -p /app; printf '41\\n' > /app/answer.txt" - }
- ],
- "seeds": [
- 1001,
- 1002,
- 1003,
- 1004,
- 1005,
- 1006,
- 1007,
- 1008
- ],
- "cloud": {
- "results_repo": "example-org/private-audit-results",
- "flavor": "cpu-basic",
- "timeout_seconds": 1800,
- "max_sandboxes": 64,
- "max_inference_calls": 160
- }
+}
diff --git a/examples/coding_audit/hf.py b/examples/coding_audit/hf.py
new file mode 100644
--- /dev/null
+++ b/examples/coding_audit/hf.py
@@ -1,0 +1,1290 @@
+"""Bounded HF Jobs coding trials for reviewed tasks, with controller-side inference.
+
+Image receipts attest only the requested immutable control-plane reference. The
+Sandbox RPC bearer reaches a workload-controlled server; this is not a hostile
+code security boundary or a served-model weight revision guarantee.
+"""
+
+import hashlib
+import json
+import math
+import os
+import re
+import signal
+import stat
+import threading
+import time
+import uuid
+from contextlib import closing, contextmanager
+from decimal import Decimal
+from importlib.metadata import version
+from pathlib import Path, PurePosixPath
+
+from huggingface_hub import HfApi, InferenceClient, Sandbox
+from huggingface_hub.errors import SandboxError
+
+from .runner import _candidate_context_paths
+
+_IMAGE = re.compile(r"[^\s@]+@sha256:[0-9a-f]{64}\Z")
+_REVISION = re.compile(r"[0-9a-f]{40}\Z")
+_TERMINAL = {"COMPLETED", "CANCELED", "ERROR", "DELETED"}
+_LIMITATIONS = [
- "Immutable images are control-plane references, not measured runtime digests.",
- "Sandbox RPC exposes the execution bearer to the workload server; hostile-code credential isolation is not qualified.",
- "The verifier executes submitted code as the same user; verifier integrity against hostile submissions is not qualified.",
- "Inference Providers do not pin served weights to the recorded model source revision.",
+]
+_TOOL = { - "type": "function",
- "function": {
-
"name": "terminal", -
"description": "Run a shell command in the task workspace.", -
"parameters": { -
"type": "object", -
"properties": {"command": {"type": "string"}}, -
"required": ["command"], -
"additionalProperties": False, -
}, - },
+}
+class HFBackendError(ValueError):
- """A credential-free controller diagnostic."""
+class _DeadlineExceeded(HFBackendError):
- pass
+@contextmanager
+def _deadline(seconds):
-
SDK control-plane calls have no timeout argument. A controller must run on
-
the main Unix thread so a blocked synchronous SDK call can be interrupted.
- if threading.current_thread() is not threading.main_thread():
-
raise HFBackendError("HF coding controller requires the main thread") - previous_handler = signal.getsignal(signal.SIGALRM)
- previous_timer = signal.getitimer(signal.ITIMER_REAL)
- started = time.monotonic()
- def expired(signum, frame):
-
raise _DeadlineExceeded("controller deadline exceeded") - signal.signal(signal.SIGALRM, expired)
- signal.setitimer(
-
signal.ITIMER_REAL, -
min(seconds, previous_timer[0]) if previous_timer[0] else seconds, -
0.1, - )
- try:
-
yield - finally:
-
signal.setitimer(signal.ITIMER_REAL, 0) -
signal.signal(signal.SIGALRM, previous_handler) -
if previous_timer[0]: -
signal.setitimer( -
signal.ITIMER_REAL, -
max(0.000001, previous_timer[0] - (time.monotonic() - started)), -
previous_timer[1], -
)
+def _path(value):
- if not isinstance(value, str) or "\x00" in value:
-
raise HFBackendError("paths must be absolute normalized file paths") - path = PurePosixPath(value)
- if (
-
not path.is_absolute() -
or str(path) != value -
or ".." in path.parts -
or value == "/" -
or value.startswith("//") - ):
-
raise HFBackendError("paths must be absolute normalized file paths") - return value
+def _under(path, parent):
- return path == parent or path.startswith(parent + "/")
+def _candidate_script(raw):
- lines = raw.lstrip().splitlines(keepends=True)
- opening = lines[0].strip()
- if not opening.startswith("```"):
-
return raw, None - if opening not in {"
bash", "sh", "```shell"}: -
raise HFBackendError("candidate code fence must declare Bash or sh") - end = len(lines)
- while end > 1 and not lines[end - 1].strip():
-
end -= 1 - closed = end > 1 and lines[end - 1].strip() == "```"
- script = "".join(lines[1 : end - 1] if closed else lines[1:])
- if not script.strip():
-
raise HFBackendError("candidate response must contain a shell script") - return script, {
-
"wrapper": "markdown_bash", -
"closing_fence_present": closed, -
"raw_sha256": hashlib.sha256(raw.encode()).hexdigest(), -
"script_sha256": hashlib.sha256(script.encode()).hexdigest(), - }
+def submission_digest(files):
- """Hash sorted relative paths, NUL, content, NUL as source_digest does."""
- digest = hashlib.sha256()
- for name, content in sorted(files.items()):
-
digest.update(name.lstrip("/").encode()) -
digest.update(b"\0") -
digest.update(content) -
digest.update(b"\0") - return digest.hexdigest()
+def _local_file(path, limit):
-
Walk descriptors so neither a final link nor a linked parent is followed.
- path = Path(path).absolute()
- fd = os.open("/", os.O_RDONLY | os.O_DIRECTORY)
- try:
-
for component in path.parts[1:-1]: -
next_fd = os.open( -
component, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, dir_fd=fd -
) -
os.close(fd) -
fd = next_fd -
file_fd = os.open( -
path.name, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=fd -
) -
with os.fdopen(file_fd, "rb") as stream: -
info = os.fstat(stream.fileno()) -
if not stat.S_ISREG(info.st_mode) or info.st_nlink != 1: -
raise HFBackendError("task artifacts must be regular unlinked files") -
if info.st_size > limit: -
raise HFBackendError("task artifact exceeds byte limit") -
content = stream.read(limit + 1) -
if len(content) > limit: -
raise HFBackendError("task artifact exceeds byte limit") -
return content, format(stat.S_IMODE(info.st_mode), "o") - finally:
-
os.close(fd)
+def _plain(value):
-
HF generated output types derive from dict and preserve extra response fields.
- return json.loads(json.dumps(value))
+def _unique_pairs(pairs):
- result = {}
- for key, value in pairs:
-
if key in result: -
raise HFBackendError("duplicate JSON key") -
result[key] = value - return result
+class HFTaskBackend:
- """Execute each solve and grade in a fresh, dedicated HF Jobs Sandbox."""
- def init(
-
self, -
task_root: Path, -
execution: dict, -
*, -
token=None, -
api=None, -
sandbox_factory=None, -
inference_factory=None, -
canary: str | None = None, -
inference_token=None, - ):
-
if version("huggingface_hub") != "1.33.0": -
raise HFBackendError( -
"this execution adapter requires huggingface_hub==1.33.0" -
) -
self.task_root = Path(task_root) -
self.execution = dict(execution) -
self.token = token if token is not None else os.environ.get("HF_SANDBOX_TOKEN") -
self.inference_token = (
... diff truncated: showing 800 of 10424 lines
</details>
<sub>You can send follow-ups to the cloud agent <a href="https://cursor.com/agents/bc-859c23c0-b072-4faa-97da-bc511b5acf7b">here</a>.</sub>
</details>
Keep trial creation and trial.run outside a shared deadline so Harbor preserves its setup, agent and verifier phase limits. Bound the initial image build/start with the separate supervisor budget, retain phase classification, and keep cancelled Compose clients killed, reaped and marked cleanup-unverified. Resolve the merged test expectations by counting the independent build timeout and modelling the agent phase limit in the fake Harbor trial. Preserve all binding regression tests and the verifier-timeout autofix test.
Check every reward field and numeric resource field before coercion. Preserve string GPU types and cover the precision bypass.
The binding fixes already classify all six teardown, cleanup and retained-image failure labels as host failures. Add regressions for teardown errors, unverified cleanup, reward artifact drift and unsafe artifacts overriding an earlier malformed reward. Removing the host classifications makes all four tests fail; restoring them passes these and both retained-image regressions.
Track the run-owned image binding before tagging so failures during capture can release it. Remove only that tag after any FAIL or ERROR, verify its image identity before removal, and disable parent-image pruning. A completion guard also releases the tag if grading or report writing is interrupted or raises. Publish retained_image_tag in runtime.startup.measured only for completed runs without failed or errored checks. Cover retention, semantic failure, startup error, interruption, reporting failure, partial capture and ownership refusal with fake Docker responses. The report schema already allows measured keys.
Validate served execution declarations as openenv_ws before normalization. Report a static.manifest failure instead of entering Harbor execution.
Convert numeric overflow and excessive JSON recursion into ManifestError. Keep malformed declarations on the package-failure path.
Check the installed Harbor version outside package parsing. Exit through the existing internal-error path with the required and installed versions.
State the Python and Harbor version requirements. Load the CLI reference JSON example in the contract-reader tests.
Scale oracle tolerance and both floor gaps by the declared width using exact rationals. Document fractional policy bounds and test non-unit ranges and exact boundaries.
Share the hook lifecycle across credential paths and close Harbor log handlers on rejection. Remove the empty-canary fallback, fail missing evidence closed, and clean up contract and hook documentation.
Keep both floor gaps, scope reward replay determinism to one-shot Harbor tasks, and specify conditional image retention. Refuse task-declared images and preserve the Dockerfile build requirement.
Regenerates the RewardDeclaration description in the derived manifest and report schemas with scripts/sync_validation_schemas.py --fix after the range-normalized margin wording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inventory the native main-service tag before allocation and during cleanup. Verify its image identity and remove it with --no-prune after removing containers, including on rejected runs.
Keep the original reporting exception when retained-image removal also fails. Mark cleanup incomplete and attach a safe cleanup diagnostic to the original exception.
Name the effective validator build/start budget in startup errors. Keep Harbor setup, agent and verifier timeouts as phase-specific package failures, and treat unexpected timeouts as validator errors.
Run startup through Harbor 0.25.0 Trial._start_agent_environment and catch regressions to the NOP build budget or RuntimeError phase gate. Read and record the existing Harbor timeout before overriding it so a missing binding fails closed.
Document unsupported task shapes and providers, validator and phase budgets, contract input restrictions, and the pinned Harbor requirement. Record the prebuilt-only task follow-up and image cleanup behavior.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Readiness wait ignores trial deadline
- The readiness loop now continues polling SCHEDULING jobs until the caller's existing deadline interrupts it, with regression coverage for waits beyond twenty polls.
Preview (46301f8a24)
diff --git a/docs/plans/rfc008-validation-completion.md b/docs/plans/rfc008-validation-completion.md
new file mode 100644
--- /dev/null
+++ b/docs/plans/rfc008-validation-completion.md
@@ -1,0 +1,316 @@
+# Completing RFC 008 validation
+
+Planning baseline: 2026-11-10, `openenv/OpenEnv` main at
+`fb36d7d62ff9548ebd44f3323386cbea908e8706`.
+
+Trackers: [RFC 008 #778](https://github.com/openenv/OpenEnv/issues/778),
+[Level 2 umbrella #1177](https://github.com/openenv/OpenEnv/issues/1177) and
+[task packages #898](https://github.com/openenv/OpenEnv/issues/898).
+
+RFC 008 promises that `openenv validate` grades an environment against the 44
+acceptance tests of #778. At the baseline, main ran every Level 2 runtime check for
+served environments on Docker, but no Level 3 semantic check. That includes the
+check the RFC opens with: a correct solution must earn maximum reward, and doing
+nothing must not. The PR that adds this document delivers that check for Harbor
+task packages and makes every report name the checks it did not run. This document
+records where validation stands after that PR, defines what "complete" means, and
+ranks the remaining work as PR-sized workstreams.
+
+## Where validation stands
+
+Each RFC check has a stable id, a level and a lane. The lane says who can run the
+check. `local` checks run inside `openenv validate`. `hub` and `statistical` ids are
+reserved for operators, the teams that run validation hubs or lab intake. The table
+counts the local ids in the committed severity policy and the ids each execution
+binding grades.
+
+| Level | Local ids | Served packages (`openenv_ws`) | Harbor tasks (`harbor_task`) |
+|---|---|---|---|
+| 1 Static | 10 | 1 (`static.manifest`) | 1 (`static.manifest`) |
+| 2 Runtime | 16 | 16, on Docker-local | 3: startup, reward well-formedness, episode determinism |
+| 3 Semantic | 8 | 0 | 3: oracle reaches max, floor gap, canary floor |
+| 4 Statistical | 0 (7 reserved) | Operator-side by design | Operator-side by design |
+| Hub | 0 (5 reserved) | Operator-side by design | Operator-side by design |
+
+Every requested local check that does not run is a named SKIP with its reason, at
+every level. A report therefore reads WARN until all requested checks pass, and a
+PASS on a subset cannot pass for full coverage.
+
+Three limits apply across bindings:
+
+- **Parsers.** `openenv.yaml` and Harbor `task.toml` ship. PostTrain `task.md`
+ packages are refused as unrecognized.
+- **Providers.** Docker-local enforces `public` and `no-network` networking and
+ refuses allowlists and GPUs. The HF Sandbox provider refuses every launch with
+ `missing provider capability: immutable_image`, because nothing yet binds a Space
+ revision to an image digest.
+- **Served oracles.** RFC 008 does not yet say how to execute an oracle against a
+ served WebSocket environment, so served packages report their semantic checks as
+ named SKIPs.
+
+Work outside main measures more than the RFC's local checks. A cloud audit on HF
+Jobs adds generated reference solutions, wrong-answer controls, repeat runs,
+leakage probes and target-model pass rates.
+[#1582](https://github.com/openenv/OpenEnv/pull/1582) adds training and validator
+performance experiments. These results report under their own fields rather than
+RFC check ids, so they do not count toward RFC coverage yet.
+
+## What complete means
+
+The RFC splits responsibility between the repo and operators, so completion has two
+targets. The repo can reach the first one alone. The second one needs an operator
+procedure that the repo documents but does not run.
+
+**Local completeness.** Every local-lane id has a grader for each shipped execution
+binding where it applies, and reports name every inapplicable or unimplemented id
+with its reason. For served packages this means nine static graders, an oracle
+execution contract and eight semantic graders. For Harbor tasks it means the same
+static graders, the remaining semantic graders, and a decision on which runtime ids
+apply to one-shot tasks.
+
+**Operator completeness.** Each reserved `statistical.*` id has at least one
+documented reference procedure that emits results in the shared report schema. The
+repo still ships no hub, as the RFC requires, so the procedure lives in an example or
+an operator repository. Comparable operator results also need the metric namespace
+(`pass@k`, headroom, variance bound) that the RFC defers to a future RFC.
+
+Validation still certifies that an environment can be trained on, not that it should
+be. Measuring training signal comes last, and it belongs with the evaluation work in
+[RFC 013 (#1256)](https://github.com/openenv/OpenEnv/pull/1256).
+
+## Proposed workstreams
+
+The workstreams are ranked by dependency and by how much each one improves
+environment selection. Each one is one to three PRs. Costs are relative: S needs no
+build or inference, M needs Docker builds and replays, and L needs model inference
+or GPU time. Owners are suggestions for discussion.
+
+| Rank | Workstream | #778 tests | Status | Cost | Suggested owner |
+|---|---|---|---|---|---|
+| 1 | Verifier sanity graders | #11, #25 | Harbor done; envelope and served open | M | Validation core |
+| 2 | Harbor binding on main | Harbor parser slice | Done | M | Validation core |
+| 3 | Verifier strength | #28, #26 procedure | Harbor canaries done; panel open | M, then L | Validation core and Hub pipeline |
+| 4 | Determinism, replay and leakage graders | #30, #39, #40, #44 | Open | M | Validation core |
+| 5 | Static graders and complete reports | #1, #3, #4, #8, #9, #10, #12, #41, #42 | Reports done; graders open | S to M | Any contributor |
+| 6 | Cloud execution on HF Sandbox | Provider qualification | Open | M | Hub pipeline |
+| 7 | Statistical reference procedure | #13 to #16, #27 | Open | L | Hub pipeline |
+| 8 | Wider task shapes | Workspace handoff, multi-container, credentials | Open | M to L | Hub pipeline and validation core |
+| 9 | Training signal | #17 | Open | L | Validation experiments and RFC 013 |
+
+### 1. Verifier sanity graders
+
+`semantic.oracle_max` and `semantic.floor_gap` now grade Harbor tasks. The graders
+read a format-neutral `OracleEvidence`, so another binding only has to measure the
+rewards. Oracle tolerance and floor margin are fractions of the declared reward
+range. Both the maximum and every oracle reward must exceed the measured floor
+by at least the declared margin times the range width.
+
+**Remaining.** Served packages need an oracle execution contract first: the RFC
+names injected state and executable scripts, but not how either drives a served
+WebSocket session or what doing nothing means there. That needs an RFC amendment.
+`semantic.resource_envelope` also remains: record peak CPU, memory and disk during
+the oracle run and compare them with the declared resources.
+
+**Evidence.** Oracle and do-nothing rewards, the declared margin and tolerance, the
+image digest and the cleanup outcome.
+
+**Cost and owner.** M: Docker builds and short episodes, with no inference.
+Validation core.
+
+### 2. Harbor binding on main
+
+Done. Harbor `task.toml` packages validate through the `harbor-semantic-v1`
+profile and an operator-supplied `validation/openenv.json` contract. Fresh Docker
+trials run a do-nothing agent, the oracle, an oracle replay and declared canaries,
+all pinned to one immutable image. The binding requires Python 3.12 or newer and
+`harbor==0.25.0` alongside `openenv[harbor]`. It refuses task-declared `docker_image`
+values, so the NOP trial builds from the task's Dockerfile. The validator keeps
+`openenv-validation:<run id>` only when the run has no FAIL or ERROR and was not
+interrupted. It records the tag in `runtime.startup.measured` as
+`retained_image_tag`. Otherwise it removes the tag.
+
+**Follow-up.** Tasks that ship only a prebuilt image cannot run under
+`harbor-semantic-v1`. Some Terminal Bench tasks are affected. Supporting them needs
+a reviewable Dockerfile path or digest-pinned image support.
+
+**Remaining.** Decide which runtime ids apply to one-shot tasks. Containment,
+network policy, resource bounds and oracle containment could apply. Observation
+schema, state contract and tool declaration accuracy do not.
+
+**Cost and owner.** M. Validation core.
+
+### 3. Verifier strength: canaries and an adversarial panel
+
+A verifier can pass the oracle and floor checks and still reward wrong answers.
+`semantic.canary_floor` now runs operator-declared canaries on Harbor tasks: each
+known-wrong `solve.sh` replacement must earn no more than doing nothing.
+
+**Remaining.** The reserved `statistical.adversarial_floor` (#26) still has no
+procedure; the RFC notes that the panel "is not yet specified anywhere". Write an
+RFC 008 amendment that specifies:
+
+- where wrong answers come from: mutations of the reference solution, known-fault
+ corpora, empty and partial outputs, and plausible solutions from a weaker model
+ that an independent check confirms are wrong;
+- how many are needed for a verdict;
+- the pass rule, which is that every wrong answer scores at the floor.
+
+The cloud audit's wrong-answer controls are a working prototype of this panel.
+
+**Evidence.** Each wrong answer's provenance and reward, plus the verifier version.
+
+**Cost and owner.** L for generated panels, which need inference. Validation core
+writes the amendment, and the Hub pipeline team implements the operator panel.
+
+### 4. Determinism, replay and leakage graders
+
+`semantic.verifier_determinism`, `semantic.verifier_portability`,
+`semantic.replayability` and `semantic.no_solution_leakage` have no graders.
+Harbor tasks now establish `runtime.episode_determinism` when an oracle replay in a
+fresh container reproduces every reward and the oracle exit code. Each trial is a
+one-shot episode with no per-step observations. Served environments retain the
+general comparison of observation, reward, done, tool output and state. The semantic
+checks depend on the determinism finding.
+
+**Approach.** Verifier determinism runs the verifier on one final state several
+times and requires identical rewards. For `llm_judged` environments it requires
+variance within the declared bound, reusing the Level 2 variance path. Portability
+reruns the verifier with networking denied, as the RFC's hermeticity rule for #40
+requires. Replayability re-executes a recorded trajectory and compares rewards. The
+leakage check scans observations and agent-visible files for oracle artifacts such
+as solution text, expected outputs and test files.
+
+**Evidence.** Replay transcripts and reward vectors, the network mode of each
+verifier run, and every leakage match with its location.
+
+**Cost and owner.** M. Validation core.
+
+### 5. Static graders and complete reports
+
+Reports now name every unimplemented static check as a SKIP, so a static run reads
+WARN rather than PASS after one check.
+
+**Remaining.** Add the nine graders in two batches. The first batch reads the
+manifest and lockfiles and needs no build: `static.dependency_pinning`,
+`static.task_distribution_pinning`, `static.resource_declaration` and
+`static.timeout_ceiling`. The second batch inspects built images:
+`static.oci_labels`, `static.sbom`, `static.image_hygiene`, `static.layout` and
+`static.reproducible_build`, which builds twice and compares digests.
+
+**Evidence.** The files and image metadata each grader read.
+
+**Cost and owner.** S for the first batch and M for the image batch. Any
+contributor; these make good first issues.
+
+### 6. Cloud execution on HF Sandbox
+
+The HF Sandbox provider refuses every launch because it has no immutable image
+binding. Runtime and semantic checks therefore cannot run in the cloud, or on GPUs,
+through the RFC provider interface. Cloud audits work around this with their own
+runner.
+
+**Approach.** Resolve a Space revision to the registry digest of the image it
+builds, verify it, and launch by digest. Qualify public Space images first, then add
+an authenticated pull path for private ones. Once launches work, the cloud audit can
+call the RFC graders instead of its own runner.
+
+**Evidence.** The Space revision, the resolved digest, the pull receipt and the same
+cleanup evidence Docker-local records.
+
+**Cost and owner.** M. Hub pipeline team.
+
+### 7. Statistical reference procedure
+
+The learnable tests (#13 to #16) and the gameability gap (#27) are reserved ids with
+no reference procedure. The RFC also defers the shared metric namespace.
+
+**Approach.** Write the metric-namespace RFC, with definitions, sample sizes and
+confidence intervals for `pass@k`, headroom and variance bound. Then publish a
+reference operator procedure, outside the core package, that maps target-model
+attempts onto the reserved ids:
+
+- `statistical.reward_reachability`: some attempt reaches maximum reward;
+- `statistical.difficulty_separation`: pass rates differ across model tiers;
+- `statistical.headroom`: the target model's pass rate leaves room to improve;
+- `statistical.variance_bound`: reward noise across repeats stays within a bound.
+
+The cloud audit already measures target-model pass rates with confidence intervals.
+It needs to emit them under these ids in hub-lane reports.
+
+**Evidence.** Model identity and provider, attempt counts, per-attempt rewards and
+intervals.
+
+**Cost and owner.** L, since every task needs inference. Hub pipeline team, with the
+metric-namespace RFC from validation core.
+
+### 8. Wider task shapes
+
+The Harbor binding refuses multi-step tasks, task-authored Docker Compose files and
+tasks that need credentials. The cloud audit also hands off only individual output
+files. The RFC defers credential delivery explicitly.
+
+**Approach.** Add a workspace or git-patch handoff so repository tasks are graded on
+the agent's full change. Treat multi-container support as a provider capability, as
+the RFC does for GPUs, so providers without it SKIP with the capability named. Write
+the credential-delivery amendment the RFC asks for, covering delivery, access
+isolation, lifetime, network access and evidence redaction.
+
+**Evidence.** Handoff manifests with hashes, and provider capability reports.
+
+**Cost and owner.** M for workspace handoff and L for multi-container and
+credentials. The Hub pipeline team builds the adapters, and validation core writes
+the amendment.
+
+### 9. Training signal
+
+`statistical.training_signal` (#17, advisory) asks whether training on an environment
+improves a model. Nothing measures it yet.
+
+**Approach.** Use the evaluation harness proposed in RFC 013 to run short training
+jobs on a sample of validated environments, then report the change in held-out
+success. The training experiments in #1582 are a first measurement. Keep the result
+advisory, as the RFC's severity policy says, because a short run cannot show that an
+environment is unlearnable.
+
+**Evidence.** Training configuration, seeds, and held-out success before and after
+training.
+
+**Cost and owner.** L: GPU time. Validation experiments, with the RFC 013 authors.
+
+## Sequencing
+
+The verifier sanity graders came first because canaries, verifier determinism and
+replay all reuse oracle replay. The order of the remaining work follows from its
+dependencies:
+
+- Workstream 4 builds on the oracle replay the Harbor binding now runs.
+- Workstream 5 is independent and can start now.
+- Workstream 6 moves the Harbor and served checks to the cloud and is needed before
+ workstream 7 runs at scale.
+- The served half of workstream 1 waits on the oracle execution amendment.
+- Workstreams 8 and 9 widen coverage after the core lands.
+
+## Open questions for maintainers
+
+These questions change the scope of a workstream, so they need an answer before its
+first PR:
+
+1. How should an oracle drive a served WebSocket environment, and what is its
+ do-nothing episode?
+2. Should `semantic.canary_floor` also accept canaries the validator generates, or
+ only canaries shipped with the package?
+3. Should the adversarial panel be an RFC 008 amendment or its own RFC?
+4. Where should operator reference procedures live: `examples/` or a separate
+ repository?
+5. Should the runtime applicability table for one-shot task packages be normative
+ in the RFC?
+
+## Sources
+
+- `rfcs/008-environment-auto-validation.md`, including the Harbor task binding
+ section
+- `docs/plans/rfc008-level2-implementation.md`
+- `tests/fixtures/validation/runtime/cases.json`
+- `src/openenv/validation/runner.py`, `harbor.py`, `graders/semantic/` and
+ `providers/hf_sandbox.py`
+- Issues #778, #898 and #1177; PRs #1256 and #1582
diff --git a/docs/source/reference/cli.md b/docs/source/reference/cli.md
--- a/docs/source/reference/cli.md
+++ b/docs/source/reference/cli.md
@@ -60,7 +60,7 @@
openenv validate ./my_env --level runtime --local --output report.json-Runtime validation requires Docker and a validation.execution declaration in
+For served packages, runtime validation requires Docker and a validation.execution declaration in
openenv.yaml. The declaration points to a bounded JSON replay plan containing
one reset and a sequence of actions. The validator builds an immutable image,
collects the original episode, repeats the plan in a fresh WebSocket session and
@@ -69,6 +69,78 @@
during the fresh-container replay. Both use the same immutable image and are
removed even if a check fails.
+Harbor task.toml directories are also recognized. Validation requires Python 3.12 or newer
+and harbor==0.25.0 installed alongside openenv[harbor]:
+
+bash +pip install "openenv[harbor]" "harbor==0.25.0" +
+
+A missing or mismatched Harbor installation exits 3 and names harbor==0.25.0.
+
+Add a validation/openenv.json contract with profile: "harbor-semantic-v1",
+reward_key, reward, resources and types. The reward range, oracle tolerance
+and floor margin must be explicit. Oracle tolerance and floor margin are fractions
+of the declared reward range. Numeric strings in numeric fields, oversized integers
+and excessive JSON nesting fail static.manifest. Reward and resource numbers must
+retain their exact value when stored as floats. resources.gpu_types remains a list
+of strings. This is a minimal complete contract:
+
+```json
+{
- "profile": "harbor-semantic-v1",
- "reward_key": "reward",
- "reward": {
- "range": [0, 1],
- "oracle_tolerance": 0,
- "floor_margin": 0.1
- },
- "resources": {
- "cpu": 1,
- "memory_mb": 512,
- "disk_mb": 1024,
- "episode_timeout_s": 300
- },
- "types": {"tags": ["swe"]}
+}
+```
+The harbor-semantic-v1 binding runs
+fresh Harbor trials in Docker: a do-nothing agent, the oracle (solution/solve.sh),
+an oracle replay and any canaries declared in the contract. A canary is a known-wrong
+solve.sh replacement. The results report under runtime.startup,
+runtime.reward_well_formed, runtime.episode_determinism, semantic.oracle_max,
+semantic.floor_gap and semantic.canary_floor, and harbor-evidence.json keeps
+the raw trial evidence in the report bundle. Other checks are named SKIPs, so a
+passing task still reports WARN. This validates task packages, not the outer
+harbor_env server.
+
+Supported tasks use Linux, a single step, one Dockerfile, a shared verifier and
+public networking. Unsupported shapes include multi-step or non-Linux tasks,
+task-declared environment.docker_image, missing Dockerfiles, task-authored Compose
+files and separate verifier environments. GPU/TPU or external MCP requirements,
+credentials, non-public or differing phase networks, task environment variables,
+healthchecks, skills, verifier collection and artifact extensions are also
+unsupported. A supplied provider, including --space, is unsupported. Each case
+gives runtime.startup SKIP with a named reason and runs no trials. With valid
+static declarations and an oracle, the verdict is WARN and the exit code is 0.
+A missing solution/solve.sh still FAILs semantic.oracle_max at the semantic level,
+including with --skip-build or --space.
+
+The initial image build and environment start share a separate 600-second validator
+budget. Exceeding it gives runtime.startup ERROR. Later environment starts use the
+episode budget. There is no shared outer trial deadline. Harbor bounds setup, agent
+and verifier separately with the declared resources.episode_timeout_s. A timeout
+in one of these phases is a package FAIL that names the phase.
+
+The do-nothing trial builds from the task's Dockerfile. The validator keeps
+openenv-validation:<run id> only when the run has no FAIL or ERROR and was not
+interrupted. It records the tag in runtime.startup.measured as
+retained_image_tag. Otherwise it removes the tag. The run-owned Compose build tag
+is removed on every run. Tag removal verifies image identity and uses --no-prune.
+A removal failure marks cleanup incomplete and gives a startup ERROR. During an
+abort, cleanup preserves the original exception and records the removal failure.
+
The 16 implemented runtime checks cover startup, rewards, observation schemas,
state continuity, network policy, host containment, resource bounds, episode isolation,
oracle containment, seed forwarding, determinism, emitted trajectory records, tool
@@ -77,7 +149,8 @@
the result WARN, which exits zero and does not mean Level 2 is complete.
FAIL exits 1, unsupported package formats exit 2, and internal or policy errors
exit 3. --level semantic
-includes the available lower-level checks but does not claim semantic execution.
+runs the semantic checks for Harbor tasks. Served packages have no oracle execution
+yet, so their semantic checks are named SKIPs.
--skip-build runs declaration checks and skips runtime execution entirely.
The Docker provider supports CPU workloads in public and no-network modes;
allowlist and GPU requirements skip runtime before building. A no-network
diff --git a/examples/coding_audit/README.md b/examples/coding_audit/README.md
new file mode 100644
--- /dev/null
+++ b/examples/coding_audit/README.md
@@ -1,0 +1,66 @@
+# Operator coding audit
+
+This example collects and scores bounded coding-task evidence. It belongs to the
+operator workflow, separate from the installed OpenEnv validator and its RFC 008
+checks. Acceptance here does not establish full RFC 008 conformance, containment,
+served-model revision attestation, or training benefit.
+
+From a repository checkout, install the tested SDK and run the example:
+
+```bash
+uv sync --group dev
+uv pip install 'huggingface_hub==1.33.0'
+PYTHONPATH=src:envs uv run --no-sync python -m examples.coding_audit \
- --task /path/to/task \
- --config examples/coding_audit/config.json \
- --output /path/to/new-audit-directory \
- --json
+```
+The output directory is required and must be new. It retains report.json,
+evidence.json, config.json, raw evidence artifacts, and SHA256SUMS. Exit codes
+are 0 for accepted evidence, 1 for rejected or incomplete evidence, and 2 for
+configuration or input failure. --json prints the report as well.
+
+scoring.py provides pure assess_coding_audit and validate_audit_config
+functions. runner.py provides normalize_coding_config, coding_budget,
+validate_coding_task, and run_coding_validation. hf.py contains
+HFTaskBackend, the bounded Sandbox and Inference Providers implementation.
+The schemas bind task, verifier, immutable image references and independent execution identities.
+
+Models may declare an explicit boolean enable_thinking, forwarded through the
+HF client's extra_body.chat_template_kwargs. It is omitted by default. A
+candidate-only override leaves the target protocol unchanged and is recorded
+alongside request parameters. Support depends on the selected provider/model.
+Candidate generation streams bounded content and reasoning; only a complete
+stop response with validated usage can produce an executable candidate. A
+length response remains rejected even when its terminal usage is retained.
+Recognized Bash/sh Markdown fences are parsed before execution; the raw response
+and both raw/script hashes are retained. Other language fences are rejected.
+
+Candidate generation can opt into public task API context by adding
+"context_paths": ["environment/source/package/api.py"] to the generator
+model configuration. Only those declared regular UTF-8 files are read; paths must
+be unique, normalized, and below environment/source/. Up to 32 files and 65,536
+combined bytes are allowed. Linked files or parents, hidden or private paths,
+NUL bytes, invalid UTF-8 and oversized context are rejected before inference.
+The existing message and token budgets still apply. No source files are added by
+default, and target.context_paths is rejected.
+
+The candidate prompt labels each source path and its runtime filename relative to
+the configured workdir, with environment/source/ removed. Each generation record
+retains candidate_context entries containing path, runtime_path, sha256,
+and size, bound to the frozen task and checksummed evidence. This option does
+not enumerate task directories or include verifier, reference or specification
+files.
+
+The HF backend launches paid services when invoked without a supplied test
+backend. Use reviewed tasks and distinct execution and inference credentials.
+The Hub pipeline's cloud worker packages these production example modules with
+its frozen source and task bundle; its tests stay outside the bundle.
+
+Run the offline suite with:
+
+bash +PYTHONPATH=.:src:envs uv run pytest examples/coding_audit -q +
diff --git a/examples/coding_audit/init.py b/examples/coding_audit/init.py
new file mode 100644
--- /dev/null
+++ b/examples/coding_audit/init.py
@@ -1,0 +1 @@
+"""Operator-owned coding audits, separate from core RFC 008 validation."""
diff --git a/examples/coding_audit/main.py b/examples/coding_audit/main.py
new file mode 100644
--- /dev/null
+++ b/examples/coding_audit/main.py
@@ -1,0 +1,63 @@
+"""Run a bounded operator coding audit and retain its evidence bundle."""
+
+import argparse
+import signal
+import sys
+import threading
+from contextlib import contextmanager
+from pathlib import Path
+
+from openenv.validation.runtime.json_values import dumps, loads
+
+from .runner import run_coding_validation
+
+
+@contextmanager
+def _sigterm_as_interrupt():
- if threading.current_thread() is not threading.main_thread():
-
yield -
return - def interrupt(signum, frame):
-
raise KeyboardInterrupt - previous = signal.signal(signal.SIGTERM, interrupt)
- try:
-
yield - finally:
-
signal.signal(signal.SIGTERM, previous)
+def main(argv=None):
- parser = argparse.ArgumentParser(description=doc)
- parser.add_argument("--task", type=Path, required=True)
- parser.add_argument("--config", type=Path, required=True)
- parser.add_argument(
-
"--output", -
type=Path, -
required=True, -
help="New directory for the report, evidence and checksummed artifacts", - )
- parser.add_argument("--json", action="store_true", dest="json_output")
- args = parser.parse_args(argv)
- try:
-
config = loads(args.config.read_text()) -
with _sigterm_as_interrupt(): -
report = run_coding_validation(args.task, config, args.output) - except (OSError, ValueError) as error:
-
print( -
f"Coding audit configuration/input failure ({type(error).__name__})", -
file=sys.stderr, -
) -
return 2 - if args.json_output:
-
print(dumps(report)) - else:
-
print(f"Coding audit: {report['verdict'].upper()} ({report['profile']})") -
for name, check in report["checks"].items(): -
print(f" {check['status'].upper():5s} {name}") - return 0 if report["accepted"] else 1
+if name == "main":
- raise SystemExit(main())
diff --git a/examples/coding_audit/config.json b/examples/coding_audit/config.json
new file mode 100644
--- /dev/null
+++ b/examples/coding_audit/config.json
@@ -1,0 +1,83 @@
+{
- "schema_version": 1,
- "profile": "harbor-coding-audit-v1",
- "target": {
- "model_id": "Qwen/Qwen3.8-27B",
- "revision": "37561816e66b5b9153b1b64c10968c18c1e6f4a0",
- "provider": "deepinfra",
- "supports_seed": false,
- "supports_tools": true,
- "temperature": 0.7,
- "max_steps": 16,
- "max_tokens": 2048,
- "max_total_tokens": 16384
- },
- "execution": {
- "agent_image": "docker.io/library/python@sha256:48d884813cf44e927cf64c5529dd630dd3b18338c6dc727991eef9b71e934a20",
- "verifier_image": "docker.io/library/python@sha256:48d884813cf44e927cf64c5529dd630dd3b18338c6dc727991eef9b71e934a20",
- "flavor": "cpu-basic",
- "timeout_seconds": 120,
- "workdir": "/app",
- "submission_paths": [
-
"/app/answer.txt" - ],
- "private_paths": [
-
"/tests", -
"/solution" - ],
- "verifier_command": [
-
"/bin/bash", -
"/tests/test.sh" - ],
- "reward_path": "/logs/verifier/reward.txt",
- "reward_key": "reward"
- },
- "validation": {
- "reference_repeats": 3,
- "target_attempts": 8,
- "reward_min": 0,
- "reward_max": 1,
- "oracle_tolerance": 0,
- "floor_margin": 0.1,
- "wrong_answer_max": 0,
- "required_negative_cases": [
-
"wrong-value", -
"empty-answer", -
"near-miss" - ]
- },
- "generation": {
- "candidate_count": 2
- },
- "negative_cases": [
- {
-
"id": "wrong-value", -
"script": "mkdir -p /app; printf '0\\n' > /app/answer.txt" - },
- {
-
"id": "empty-answer", -
"script": "mkdir -p /app; : > /app/answer.txt" - },
- {
-
"id": "near-miss", -
"script": "mkdir -p /app; printf '41\\n' > /app/answer.txt" - }
- ],
- "seeds": [
- 1001,
- 1002,
- 1003,
- 1004,
- 1005,
- 1006,
- 1007,
- 1008
- ],
- "cloud": {
- "results_repo": "example-org/private-audit-results",
- "flavor": "cpu-basic",
- "timeout_seconds": 1800,
- "max_sandboxes": 64,
- "max_inference_calls": 160
- }
+}
diff --git a/examples/coding_audit/hf.py b/examples/coding_audit/hf.py
new file mode 100644
--- /dev/null
+++ b/examples/coding_audit/hf.py
@@ -1,0 +1,1287 @@
+"""Bounded HF Jobs coding trials for reviewed tasks, with controller-side inference.
+
+Image receipts attest only the requested immutable control-plane reference. The
+Sandbox RPC bearer reaches a workload-controlled server; this is not a hostile
+code security boundary or a served-model weight revision guarantee.
+"""
+
+import hashlib
+import json
+import math
+import os
+import re
+import signal
+import stat
+import threading
+import time
+import uuid
+from contextlib import closing, contextmanager
+from decimal import Decimal
+from importlib.metadata import version
+from pathlib import Path, PurePosixPath
+
+from huggingface_hub import HfApi, InferenceClient, Sandbox
+from huggingface_hub.errors import SandboxError
+
+from .runner import _candidate_context_paths
+
+_IMAGE = re.compile(r"[^\s@]+@sha256:[0-9a-f]{64}\Z")
+_REVISION = re.compile(r"[0-9a-f]{40}\Z")
+_TERMINAL = {"COMPLETED", "CANCELED", "ERROR", "DELETED"}
+_LIMITATIONS = [
- "Immutable images are control-plane references, not measured runtime digests.",
- "Sandbox RPC exposes the execution bearer to the workload server; hostile-code credential isolation is not qualified.",
- "The verifier executes submitted code as the same user; verifier integrity against hostile submissions is not qualified.",
- "Inference Providers do not pin served weights to the recorded model source revision.",
+]
+_TOOL = { - "type": "function",
- "function": {
-
"name": "terminal", -
"description": "Run a shell command in the task workspace.", -
"parameters": { -
"type": "object", -
"properties": {"command": {"type": "string"}}, -
"required": ["command"], -
"additionalProperties": False, -
}, - },
+}
+class HFBackendError(ValueError):
- """A credential-free controller diagnostic."""
+class _DeadlineExceeded(HFBackendError):
- pass
+@contextmanager
+def _deadline(seconds):
-
SDK control-plane calls have no timeout argument. A controller must run on
-
the main Unix thread so a blocked synchronous SDK call can be interrupted.
- if threading.current_thread() is not threading.main_thread():
-
raise HFBackendError("HF coding controller requires the main thread") - previous_handler = signal.getsignal(signal.SIGALRM)
- previous_timer = signal.getitimer(signal.ITIMER_REAL)
- started = time.monotonic()
- def expired(signum, frame):
-
raise _DeadlineExceeded("controller deadline exceeded") - signal.signal(signal.SIGALRM, expired)
- signal.setitimer(
-
signal.ITIMER_REAL, -
min(seconds, previous_timer[0]) if previous_timer[0] else seconds, -
0.1, - )
- try:
-
yield - finally:
-
signal.setitimer(signal.ITIMER_REAL, 0) -
signal.signal(signal.SIGALRM, previous_handler) -
if previous_timer[0]: -
signal.setitimer( -
signal.ITIMER_REAL, -
max(0.000001, previous_timer[0] - (time.monotonic() - started)), -
previous_timer[1], -
)
+def _path(value):
- if not isinstance(value, str) or "\x00" in value:
-
raise HFBackendError("paths must be absolute normalized file paths") - path = PurePosixPath(value)
- if (
-
not path.is_absolute() -
or str(path) != value -
or ".." in path.parts -
or value == "/" -
or value.startswith("//") - ):
-
raise HFBackendError("paths must be absolute normalized file paths") - return value
+def _under(path, parent):
- return path == parent or path.startswith(parent + "/")
+def _candidate_script(raw):
- lines = raw.lstrip().splitlines(keepends=True)
- opening = lines[0].strip()
- if not opening.startswith("```"):
-
return raw, None - if opening not in {"
bash", "sh", "```shell"}: -
raise HFBackendError("candidate code fence must declare Bash or sh") - end = len(lines)
- while end > 1 and not lines[end - 1].strip():
-
end -= 1 - closed = end > 1 and lines[end - 1].strip() == "```"
- script = "".join(lines[1 : end - 1] if closed else lines[1:])
- if not script.strip():
-
raise HFBackendError("candidate response must contain a shell script") - return script, {
-
"wrapper": "markdown_bash", -
"closing_fence_present": closed, -
"raw_sha256": hashlib.sha256(raw.encode()).hexdigest(), -
"script_sha256": hashlib.sha256(script.encode()).hexdigest(), - }
+def submission_digest(files):
- """Hash sorted relative paths, NUL, content, NUL as source_digest does."""
... diff truncated: showing 800 of 12310 lines
</details>
<sub>You can send follow-ups to the cloud agent <a href="https://cursor.com/agents/bc-701df0d9-f26f-4b4d-b3bd-c5cad0e44a8e">here</a>.</sub>
<!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_END -->
<sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 082b2f254da22aea25f2198ffbd87eb855c2dc65. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>

Summary
Part of #778 (RFC 008). Implements the Harbor task lane from #898 and follows the Level 2 work in #1177.
RFC 008 promises that
openenv validatechecks a verifier mechanically: a correct solution must earn the maximum reward, and doing nothing must not. Main ran no Level 3 check. This PR runs those checks for Harbor task packages and makes every report name the checks that did not run.task.tomlbinding from the Hub validation pipeline branch, with its tests. That covers the parser, Docker trials, image pinning and owned-resource cleanup.semantic.oracle_max,semantic.floor_gapandsemantic.canary_floornow grade Harbor tasks. An oracle replay in a fresh container establishesruntime.episode_determinism, which the RFC makes a precondition of every semantic check. The graders read a format-neutralOracleEvidence, so a future served binding can reuse them.solve.shreplacements invalidation/canaries/. Each one must earn no more than doing nothing. This catches a verifier that passes the oracle and floor checks but rewards wrong answers.run_rolloutandrun_batchaccept a trustedon_trial_createdcallback and forwardenvironment_kwargs, so a caller can verify a trial's image before it runs.harbor-semantic-v1binding and updates the CLI reference. It also addsdocs/plans/rfc008-validation-completion.md, which ranks the remaining RFC 008 work.Coding-audit examples, from #1592
#1592 was merged into this branch, so this PR also adds
examples/coding_auditand its standalone CLI,python -m examples.coding_audit. It measures tasks on the operator side of RFC 008 and adds no installed validation API. For a reviewed task it:The HF backend keeps inference in the controller, sends only declared submission files to fresh graders, and verifies cleanup. It uses
HfApi,SandboxandInferenceClientwith explicit image, flavor and timeout settings, followinghf jobsconventions. Running the live backend launches paid services. Sandbox RPC exposes its execution bearer to code the workload controls, so use it only with reviewed tasks. The full Hub workflow's Git-pinned Harbor install also depends on #1304, the FastMCP 4 migration.Behaviour changes for review
STATIC_PENDINGset, so they expect PASS again once those graders land.width = max - min,semantic.floor_gaprequires bothmax - floor >= floor_margin * widthandoracle - floor >= floor_margin * widthfor every oracle run. Without the second condition, a verifier whose oracle scores no better than doing nothing could pass. The maintainer kept the stricter rule, and RFC 008's central contract now states both conditions.max - minwith exact rational arithmetic, so a[0, 100]verifier cannot pay 99.9 for doing nothing. The policy bounds are at most 0.1 and at least 0.1 of the range.harbor-semantic-v1, a fresh oracle replay that reproduces every reward and the oracle exit code establishesruntime.episode_determinism. A Harbor trial is a one-shot episode with no per-step observations. Served environments keep the general comparison of observation, reward, done, tool output and state.harbor-semantic-v1. The pilot profile was never on main.openenv-validation:<run id>only when it has no FAIL or ERROR and was not interrupted, and records the tag inruntime.startup.measuredasretained_image_tag. Otherwise it removes that run-owned tag after checking the image identity, with--no-prune. Harbor's Compose build tag (<project>-main) is removed on every run, so a rejected run leaves no image tag behind. A removal failure marks cleanup incomplete and gives a startup ERROR. During an abort, the original exception is kept. A caller can then run a model on the exact validated image.[environment] docker_imageis refused. The NOP trial builds from the task's Dockerfile, so validation never grades a registry image the reviewer did not see.solution/solve.shfailssemantic.oracle_maxwhenever the static checks pass, including with--skip-buildor a provider.floor_marginof0.100000000000000001would otherwise become0.1and admit a gap below the declared margin.resources.gpu_typesstays a list of strings. Oversized integers and excessive JSON nesting arestatic.manifestFAILs.openenv.yamlthat declaresvalidation.execution.kind: harbor_taskis astatic.manifestFAIL with exit 1.harbor==0.25.0. It is no longer reported as a package FAIL.--spacereports keepspaces/<id>@<sha>as the target.1a165f35removes the outer timeout around each Harbor trial, so one shared budget cannot cut off a verifier that is still within its own limit. The review fixes give the initial environment build and start their own 600-second supervisor budget, so a slow build cannot spend the episode budget. A cancelled Compose client is killed and reaped. When the daemon's completion cannot be proven, cleanup is reported as unverified. Exceeding the build/start budget, or any unexpected timeout, is a startup ERROR. Harbor's setup, agent and verifier timeouts remain package FAILs that name the phase. A Harbor build without the expected build-timeout attribute fails closed.abf12d02is partly kept. Itslevels_runfix stays. Its pin of the wholeharborextra to==0.25.0, and its suppression of temp-directory cleanup errors, are reverted ine328ab3d, which gives the reasons.Testing
pytest tests/test_validation tests/test_cli tests/envs/test_harbor_validated_trial_hook.py: 2,069 passed and 66 skipped on Python 3.13 with theharborextra. After examples: add bounded coding audits with HF Sandboxes #1592 merged in,pytest tests/test_validation examples/coding_auditgives 2,259 passed and 66 skipped. The newtest_semantic_graders.pyneeds no Harbor, so it also runs on Python 3.11.semantic.floor_gapfails: the do-nothing reward is 1.0.semantic.canary_floorfails: the wrong answer earns 1.0.solution/solve.shsemantic.oracle_maxfails before any trial starts.No containers, networks or volumes were left behind, and every bundle checksum verified.
usortandruffare clean on the changed paths.The runtime reference project installs
harbor==0.25.0in itsdevgroup, so the fast suite runs the Harbor tests rather than skipping them, since a skip fails--require-complete. Cursor Agent's156e411emoved it there from the main dependencies, which keeps Harbor out of the--no-devexport behind the Docker test image. Harbor's dependencies capwebsocketsbelow 16, so that lock moves it from 17.1 to 15.0.1, which is within OpenEnv'swebsockets>=15.0.1. Run locally before the review fixes,reproduce.py --suite fast --require-completepassed 1,711 tests and--suite protocol --require-completepassed 138, with no skips.After the review fixes, the same selection at
7df6474egives 2,200 passed and 66 skipped on Python 3.13 with Harbor 0.25.0. An earlier run at7a5af24chad onetests/test_cli/test_serve.pycase miss its health deadline under local load; that file andserve.pymatch main. Changed Python files passusortandruff, andscripts/sync_validation_schemas.py --checkpasses.The head
d1e748b0merges main's 0.13.0 release (Release OpenEnv 0.12.1 #1577), which changes only version strings. The one shared file,tests/validation_runtime/uv.lock, merged cleanly and passesuv lock --check. All 15 CI checks pass ond1e748b0, including Linux Docker.tests/envs/test_harbor_capture_server.py::test_start_refuses_a_port_held_by_a_non_capture_listenerfails locally on macOS. It exercises only the capture server, which this PR does not change.The installed validation command adds no Hub-facing behaviour. The coding-audit example adds HF Sandbox and inference calls that follow the SDK conventions described above.
🤖 Generated with Claude Code