Skip to content

examples: discover, validate and evaluate pinned Hub environments - #1593

Merged
burtenshaw merged 31 commits into
mainfrom
ben/hub-validation-examples
Nov 12, 2026
Merged

burtenshaw merged 31 commits into
mainfrom
ben/hub-validation-examples

Conversation

@burtenshaw

@burtenshaw burtenshaw commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

HF exact-tag search + Harbor Hub packages
  → pinned native task acquisition
  → OpenEnv semantic validation
  → exclude failing or incomplete tasks
  → guarded model handoff / frozen HF Job audit plans

Adds the reproducible Hub pipeline, a tiny local control demo, explicit operator-supplied evaluation panels, immutable cloud bundles and cohort launch scripts. Native admission verifies standard semantic results and raw reward/replay/canary artifacts, scales tolerance and both floor gaps by the reward-range width, binds the startup-recorded retained image and accepts reviewed Dockerfile builds or digest-pinned prebuilt images without rewriting acquired tasks. Mutable image tags remain excluded. Missing oracles remain semantic failures for unsupported startup shapes.

Cloud plans bind acquired file hashes, consumed plan hashes, runtime versions and exact shared-spend allocations. Both preparation producers emit the relative plan path and exact hash consumed by the prepared-cohort launcher. The remote bootstrap performs frozen sync, installs and verifies its declared HF SDK pin before starting the worker. Acquired native assets under execution-excluded paths are explicitly unsupported rather than silently filtered. Expired deadlines prevent SDK dispatch; malformed provider metadata retains available integer billing counts before response normalization. Uncertain launches keep durable receipts and reservations for reconciliation.

Based on main after #1591 (which includes the operator coding audit from #1592) and #1607 (digest-pinned prebuilt/image-only Harbor tasks). The Git-pinned Harbor integration depends on #1304 (JianYan11's FastMCP 4 migration), which remains an external dependency rather than silently incorporating its commits.

HF comparison: discovery uses the conjunction library:harbor and rl-environment; acquisition resolves one full revision and uses normal SDK revision/path semantics. Uploads add files through ordinary Hub commits with optimistic parent checks, matching hf upload's additive behavior. They do not mirror-delete remote files. Jobs receive runtime secrets, immutable inputs and explicit flavor/timeout settings.

Evidence

  • Refreshed onto main d65475d3b6c978c19cf62e0c6ce0b731fe16cbf2. Public head baab8461c806c23d3832b978cc5d166e49dafb87; fast-forward push only. The previous exact-head merge authorization was not used after main advanced; this new head needs the user's merge decision.
  • 1,121 focused CPU tests passed, including the admission, native Harbor binding, semantic graders, coding audit and trial hook suites. The renamed mutable-image exclusion regression passes in the separate 85-test admission run.
  • On prior head 1c1405859, 2,895 broader CPU tests passed, 8 skipped and 58 deselected across validation, both examples and the trial hook. Live/network/Docker marks were excluded.
  • All 28 example Python files pass Ruff lint/format and usort. Runtime-project uv lock and uv lock --check pass; no manual lockfile resolution.
  • Two independent read-only reviews APPROVE the prior head 1c1405859: correctness (448 scoped tests) and spec/public boundary (442 scoped tests). A fresh independent final-head review APPROVE covers the six-file 1c1405859..9c2c0b3db quality-review delta (249 scoped tests). Neither review asserts live-provider, Docker, billing or training qualification.
  • All five review findings have baseline-red/green regressions, including actual producer → preparation → submission seams with injected Hub/Jobs APIs, generated bootstrap installation/version checks, source change-and-restore protection, deadline-crossing dispatch prevention and malformed-usage receipt retention. All 18 review threads are resolved, including the repeated intentional interrupt-handler warning. The later quality-review fixes preserve shared pytest fixture registration and intentional sanitized interrupt evidence; the worker tests now exercise KeyboardInterrupt and SystemExit as well as operational failures.

Merge Danger

Door: two-way

Blast Radius: examples

Explicit submission incurs cloud/provider costs. Reservations model configured workloads rather than enforcing an account billing cap. Catalog enumeration is metadata, not runtime qualification. Required checks and any FAIL/ERROR block admission; supplemental audit success does not establish full RFC compliance, hostile-code verifier integrity or learning benefit. Runtime-pinning changes require fresh preparation/review of older plans. This PR remains open for the user's merge decision.

Main-refresh compatibility: new source_image metadata is additive; the schema-2 trial overlays, retained-image handoff and inspected source/config/image/cleanup bindings are unchanged. Six independent offline compatibility cases passed. Example README and pipeline plan now describe pinned prebuilt/image-only support accurately; no admission/runtime relaxation was needed.

@burtenshaw burtenshaw added feature size: extra-large Extra-large pull request labels Nov 10, 2026 — with Cursor

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alignment Review Report

Automated Checks

  • Lint: PASS — ruff format --check, ruff check, and usort check all pass cleanly on every new/changed file under examples/hub_validation_pipeline/ (17 files). .claude/hooks/lint.sh itself exits non‑zero, but only on the same ~47 pre‑existing envs//src/ formatting drifts noted in AGENTS.md (e.g. tests/envs/test_grid_world.py, tests/envs/test_julia_env.py) — none touched by this diff, which doesn't modify src/, tests/, or envs/ at all.
  • Debug code: CLEAN — .claude/hooks/check-debug.sh only scans src/, so I also scanned the new files directly: no TODO/FIXME/HACK, no breakpoint()/pdb, and every print( is in an intentional CLI/demo entry point (cloud.py, spend.py, pipeline.py, demo.py, real_tasks.py, full_run.py), not leftover debugging.

Note: this PR is stacked on ben/coding-audit-operator, not main, so Tests/Lint/Package CI (which only trigger on PRs into main/release) won't run here — only build_pr_documentation and the base-filter-free Runtime validation / Linux Docker workflow do, and that workflow exercises tests/validation_runtime/, not this example. The finding below is therefore invisible on this PR's own checks and will only surface once the stack retargets main.

Open RFCs Context

  • RFC 008 — Environment Auto-Validation (rfcs/008-environment-auto-validation.md, Status: In Review, @zkwentz) is directly implicated: this PR's README.md/CODING_AUDIT.md explicitly position themselves against it ("Core native validation remains in OpenEnv... Neither corpus admission nor a supplemental coding-audit PASS establishes full RFC 008 qualification"). See Tier 2 below.
  • Other In‑Review/Draft RFCs (000, 001, 002, 003, 005, 006, 011 Draft, 012, 014) are not materially touched by this change; no conflicts found there.

Tier 1: Fixes Required

  • examples/hub_validation_pipeline/test_sources.py (test_harbor_native_cli_json_and_model_contract, test_harbor_native_acquisition_models) — these use pytest.importorskip("harbor") to gate on the Harbor-git-pin capability, but that guard doesn't actually detect the capability: harbor==0.25.0 is already a core dependency (pyproject.toml:75, >=0.22.0) that uv sync --all-extras installs from PyPI, so import harbor succeeds and the test runs instead of skipping. It then hits sources._harbor_api()'s RuntimeError("Harbor source capability mismatch...") and fails rather than skipping. Verified directly: running the README's own documented verification command out of the box —
    PYTHONPATH=.:src:envs uv run --no-sync python -m pytest examples/hub_validation_pipeline/ tests/test_validation/test_harbor_validation.py tests/envs/test_harbor_validated_trial_hook.py -q
    
    gives 2 failed, 414 passed in the standard repo setup from AGENTS.md (uv sync --all-extras, no manual pinned-Harbor install). Since this PR is stacked off-main, Tests CI won't catch this either. Fix: skip (not fail) when the installed Harbor lacks the pinned Git capability, e.g. wrap the _harbor_api() capability probe in the test and call pytest.skip(...) on RuntimeError, or check the installed revision directly before calling the real API in test_harbor_native_cli_json_and_model_contract/test_harbor_native_acquisition_models.

No other mechanical issues found: no uninitialized variables, no missing imports/syntax errors, and no credential-exposure issues — cloud.py/real_tasks.py/spend.py are unusually careful here (redaction in main()'s exception handler, _no_secrets() scanning every persisted artifact for hf_... tokens and private-key headers, distinct-secret-value checks in _runtime_secrets()).

Tier 2: Alignment Discussion

Principle Conflicts

ALIGNMENT FLAG: ~8,000 lines of new "example" infrastructure vs. the repo's leanness principle

  • Principle/RFC at stake: CLAUDE.md "Simplicity" ("OpenEnv values leanness and simplicity... Do not add layers of indirection... Do not add defensive code, fallback paths or configuration options 'just in case'... When in doubt, prefer less code.")
  • The concern: This PR adds 21 files / ~8,079 lines of implementation+test code under examples/hub_validation_pipeline/ (cloud.py alone is 1,064 lines) implementing a content-addressed bundle format, a custom ZIP-manifest validator, an offline shared-spend ledger with file locking and exact-rational accounting (spend.py), a census/cohort submission pipeline (full_run.py), and bounded-concurrency job polling (prepared_batch.py). This is architecturally far more than an illustrative "example" — it reads like production infrastructure that happens to live outside src/. Worth a maintainer gut-check on whether this scope belongs in examples/ as-is, or should be trimmed/simplified, independent of the RFC question below.
  • Suggested reviewer: @Darktex (author of PRINCIPLES.md/INVARIANTS.md/the Simplicity section)

RFC Conflicts

ALIGNMENT FLAG: Cloud submission/spend/queue machinery built on top of RFC 008 while RFC 008 explicitly says the repo ships none of that

  • Principle/RFC at stake: RFC 008 (In Review, @zkwentz), "The central architectural commitment: ship local validation and the contracts that let any team build a hub; do not ship a hub... The repo therefore ships no runner, queue, submission API, or gate policy — only the pieces every operator needs to agree on."
  • The concern: cloud.py (submit/stage_inputs/worker, HF Jobs launch specs, three-secret credential model, claim/receipt protocol), spend.py (a shared budget ledger with reservations/settlement — i.e. a gate policy), and full_run.py/prepared_batch.py (census-driven cohort submission with bounded concurrency — i.e. a queue/coordinator) are, functionally, exactly the "runner, queue, submission API, gate policy" RFC 008 says operators own and the repo does not ship. The PR's own docs are aware of the tension and thread the needle by keeping it under examples/ rather than src/openenv/validation/ ("Core native validation remains in OpenEnv... cloud orchestration... remain example code"), and by repeatedly disclaiming rfc008_complete: false/trainability_established: false. That mitigates but doesn't resolve the conflict: RFC 008's non-goal reads as scoped to "the repo" (i.e. this GitHub repository), not just src/, and a maintained, tested, documented HF-Jobs submission+budget system living in-tree is a de facto reference hub implementation — the exact outcome RFC 008 says to avoid ("a first-pass hub implementation in the shared repo would quietly make one operator's answers the standard"). Separately, per the rfc-check skill's own criteria ("New abstractions or design patterns" → RFC required, not restricted to src/core/), the bundle-manifest format, panel schema (openenv-native-evaluation-panel-v1), spend-ledger schema (openenv-shared-spend-v1), and cloud-plan schema (openenv-coding-cloud-v1) introduced here are all new, versioned wire contracts that arguably deserved their own RFC rather than landing directly as example code.
  • Suggested reviewer: @zkwentz (RFC 008 author, owns the no-hub commitment) — @Darktex for the "examples vs. core" line-drawing question.

Summary

  • 1 mechanical issue to fix (Tier 1: Harbor-capability test guard fails instead of skipping; invisible on this PR's own CI since it's stacked off main)
  • 1 alignment point for human review (scope/complexity of the new example vs. repo leanness principle)
  • 1 RFC conflict to discuss (cloud submission/spend/queue machinery vs. RFC 008's explicit "do not ship a hub" commitment)
Open in Web View Automation 

Sent by Cursor Automation: Pre-review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

There are 4 total unresolved issues (including 2 from previous reviews).

Autofix Details

Bugbot Autofix prepared fixes for both issues found in the latest run.

  • ✅ Fixed: Root task paths are rejected
    • Allowed the explicit root locator and mapped it to the Hub API's root-tree request while retaining path validation for all other task paths.
  • ✅ Fixed: Budget check uses exact not reserved
    • Changed the budget gate to compare the rounded-up reservation amount that the shared ledger actually reserves.

Create PR

Or push these changes by commenting:

@cursor push 7c9f901ebb
Preview (7c9f901ebb)
diff --git a/examples/hub_validation_pipeline/full_run.py b/examples/hub_validation_pipeline/full_run.py
--- a/examples/hub_validation_pipeline/full_run.py
+++ b/examples/hub_validation_pipeline/full_run.py
@@ -260,7 +260,7 @@
         manifest["stage"] = "budget"
         manifest["estimate"] = estimate(configs)
         manifest["within_budget"] = (
-            Fraction(manifest["estimate"]["reservation_usd_exact"]) <= cap
+            Fraction(manifest["estimate"]["reservation_usd_rounded_up"]) <= cap
         )
         manifest["status"] = "prepared"
         _write(manifest_path, manifest)

diff --git a/examples/hub_validation_pipeline/real_tasks.py b/examples/hub_validation_pipeline/real_tasks.py
--- a/examples/hub_validation_pipeline/real_tasks.py
+++ b/examples/hub_validation_pipeline/real_tasks.py
@@ -13,7 +13,7 @@
 import stat
 import tempfile
 import tomllib
-from pathlib import Path
+from pathlib import Path, PurePosixPath
 
 from huggingface_hub import hf_hub_download, HfApi, RepoFile, RepoFolder
 from openenv.validation.runner import source_digest
@@ -54,7 +54,12 @@
         if not isinstance(task, dict):
             raise ValueError("Panel tasks must be objects")
         name = cloud._safe_name(task.get("name", ""))
-        task_path = cloud._safe_name(task.get("task_path", ""))
+        raw_task_path = task.get("task_path", "")
+        task_path = (
+            PurePosixPath(".")
+            if raw_task_path == "."
+            else cloud._safe_name(raw_task_path)
+        )
         if (
             len(name.parts) != 1
             or name.as_posix().casefold() in names
@@ -103,7 +108,8 @@
     """Acquire only one pinned native tree, with size and Git-blob checks."""
     revision = sources._pin(source["revision"], "hf")
     repo_id, task_path = source["repo_id"], task["task_path"]
-    cloud._safe_name(task_path)
+    if task_path != ".":
+        cloud._safe_name(task_path)
     destination = Path(destination)
     if destination.exists() or destination.is_symlink():
         raise FileExistsError("Acquisition requires a fresh task directory")
@@ -117,7 +123,7 @@
     total = 0
     for item in api.list_repo_tree(
         repo_id,
-        path_in_repo=task_path,
+        path_in_repo=None if task_path == "." else task_path,
         repo_type="dataset",
         revision=revision,
         recursive=True,

You can send follow-ups to the cloud agent here.

burtenshaw and others added 7 commits November 10, 2026 19:20
Read original task files through the descriptor-based reader before writing the frozen copy, so copying cannot erase hard-link evidence. Cover context, instruction, verifier and reference artifacts through the runner with offline SDK fakes.
Require a nonempty tool-call list when the provider finishes with tool_calls. Exercise null, missing and empty calls through collection and scoring after a valid terminal action, and verify failed targets are never graded.
Include the repository root in PYTHONPATH so pytest can import the examples package. Verified the documented command with all 417 offline tests passing.
Remove the unguarded CLI and discovery submission paths. Bind reviewed plan hashes and cost allocations into shared reservations, and verify the active hold under the ledger lock before claiming or launching a Job.
Hash the frozen task tree rather than a restored live source. Require verified file inventories for remote identities and carry acquisition hashes through the discovery pipeline.
Reuse native file-count and byte caps before downloading selected tasks and shared assets. Verify pinned file inventories, stream Harbor downloads under a byte cap, and check bounded tar contents before materializing task files.
Takes the coding-audit fixes for linked artifacts, empty tool-call completions and the offline test path (c1fc62a..4278afa); no file overlaps this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

There are 6 total unresolved issues (including 4 from previous reviews).

Autofix Details

Bugbot Autofix prepared fixes for both issues found in the latest run.

  • ✅ Fixed: Deadline discards completed job status
    • Removed the post-inspection deadline override so a verified completed controller remains completed.
  • ✅ Fixed: Submission receipts skip durable fsync
    • Receipt writes now flush and fsync the temporary file before replacement and fsync the parent directory afterward.

Create PR

Or push these changes by commenting:

@cursor push 771b963ef7
Preview (771b963ef7)
diff --git a/examples/hub_validation_pipeline/cloud.py b/examples/hub_validation_pipeline/cloud.py
--- a/examples/hub_validation_pipeline/cloud.py
+++ b/examples/hub_validation_pipeline/cloud.py
@@ -769,8 +769,16 @@
 
 def _write_receipt(path, value):
     temporary = path.with_suffix(".tmp")
-    temporary.write_bytes(_json_bytes(value))
-    temporary.replace(path)
+    with temporary.open("wb") as stream:
+        stream.write(_json_bytes(value))
+        stream.flush()
+        os.fsync(stream.fileno())
+    os.replace(temporary, path)
+    directory = os.open(path.parent, os.O_RDONLY)
+    try:
+        os.fsync(directory)
+    finally:
+        os.close(directory)
 
 
 def stage_inputs(plan_path, *, api=None, secrets=None, expected_plan_sha256=None):

diff --git a/examples/hub_validation_pipeline/prepared_batch.py b/examples/hub_validation_pipeline/prepared_batch.py
--- a/examples/hub_validation_pipeline/prepared_batch.py
+++ b/examples/hub_validation_pipeline/prepared_batch.py
@@ -54,8 +54,6 @@
         or job.owner.name != owner
     ):
         raise ValueError("Controller identity mismatch")
-    if clock() >= deadline:
-        return "TIMEOUT"
     return job.status.stage if job.status.stage in STAGES else "UNKNOWN"

You can send follow-ups to the cloud agent here.

@burtenshaw
burtenshaw changed the base branch from ben/coding-audit-operator to ben/rfc008-validation-completion November 10, 2026 19:34

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Autofix Details

Bugbot Autofix prepared fixes for both issues found in the latest run.

  • ✅ Fixed: Batch launcher rejects prepared manifests
    • Preparation writers now persist relative plan_path values and bound plan_sha256 digests that prepared_batch can validate and launch.
  • ✅ Fixed: Worker omits required Hub pin
    • Remote bootstrap now installs huggingface_hub==1.33.0 into the synced worker environment before starting the worker.

Create PR

Or push these changes by commenting:

@cursor push 0aff254920

You can send follow-ups to the cloud agent here.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: Non-positive poll deadline disables timeout
    • Clamped alarm durations to one microsecond and protected timer setup with cleanup so expired poll budgets cannot disable or leak the controller timeout.

Create PR

Or push these changes by commenting:

@cursor push 3a695019ec
Preview (3a695019ec)
diff --git a/examples/coding_audit/hf.py b/examples/coding_audit/hf.py
--- a/examples/coding_audit/hf.py
+++ b/examples/coding_audit/hf.py
@@ -72,6 +72,7 @@
     # the main Unix thread so a blocked synchronous SDK call can be interrupted.
     if threading.current_thread() is not threading.main_thread():
         raise HFBackendError("HF coding controller requires the main thread")
+    seconds = max(0.000001, seconds)
     previous_handler = signal.getsignal(signal.SIGALRM)
     previous_timer = signal.getitimer(signal.ITIMER_REAL)
     started = time.monotonic()
@@ -79,13 +80,13 @@
     def expired(signum, frame):
         raise _DeadlineExceeded("controller deadline exceeded")
 
-    signal.signal(signal.SIGALRM, expired)
-    signal.setitimer(
-        signal.ITIMER_REAL,
-        min(seconds, previous_timer[0]) if previous_timer[0] else seconds,
-        0.1,
-    )
     try:
+        signal.signal(signal.SIGALRM, expired)
+        signal.setitimer(
+            signal.ITIMER_REAL,
+            min(seconds, previous_timer[0]) if previous_timer[0] else seconds,
+            0.1,
+        )
         yield
     finally:
         signal.setitimer(signal.ITIMER_REAL, 0)

You can send follow-ups to the cloud agent here.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

There are 5 total unresolved issues (including 3 from previous reviews).

Fix All in Cursor

Bugbot Autofix prepared fixes for both issues found in the latest run.

  • ✅ Fixed: Acquisition hashes disagree with bundle tree
    • Acquisition verification now applies the bundle's excluded-directory policy before comparing captured task hashes.
  • ✅ Fixed: Usage dropped before receipt fallback
    • Target responses now pass typed usage directly to the receipt fallback so billed token counts survive non-JSON provider details.

Create PR

Or push these changes by commenting:

@cursor push 4cb0068c7e
Preview (4cb0068c7e)
diff --git a/examples/coding_audit/hf.py b/examples/coding_audit/hf.py
--- a/examples/coding_audit/hf.py
+++ b/examples/coding_audit/hf.py
@@ -753,7 +753,7 @@
             parsed = client.chat_completion(**request)
             response = _plain(parsed)
             # Access the typed nested value: outer SDK dicts can omit extras.
-            response["usage"] = _plain(parsed.usage)
+            response["usage"] = parsed.usage
             choices = response.get("choices")
             receipt = {
                 "model": response.get("model"),

diff --git a/examples/coding_audit/test_target_budget.py b/examples/coding_audit/test_target_budget.py
--- a/examples/coding_audit/test_target_budget.py
+++ b/examples/coding_audit/test_target_budget.py
@@ -119,6 +119,22 @@
     assert receipt["response_validated"] and not receipt["discarded"]
 
 
+def test_non_json_target_usage_retains_billed_counts(harness):
+    h = harness
+    parsed = response("saved")
+    parsed.usage["provider_detail"] = object()
+    h.inference.responses = [parsed]
+    solved = h.backend.solve(model(), 0)
+    assert solved["error"] == "provider returned invalid usage details"
+    assert solved["usage"] == {"input_tokens": 10, "output_tokens": 5}
+    receipt = solved["model_metadata"]["responses"][0]
+    assert receipt["usage_counts"] == {
+        "prompt_tokens": 10,
+        "completion_tokens": 5,
+        "total_tokens": 15,
+    }
+
+
 @pytest.mark.parametrize(
     "problem", ["missing_total", "bad_total", "nonfinite_detail", "oversized_detail"]
 )

diff --git a/examples/hub_validation_pipeline/cloud.py b/examples/hub_validation_pipeline/cloud.py
--- a/examples/hub_validation_pipeline/cloud.py
+++ b/examples/hub_validation_pipeline/cloud.py
@@ -256,19 +256,23 @@
             raise ValueError("Remote source requires verified acquisition file hashes")
         return
     expected = source["task_file_hashes"]
-    if (
-        not isinstance(expected, dict)
-        or set(expected) != set(task_files)
-        or any(
-            not isinstance(value, str) or not re.fullmatch(r"[0-9a-f]{64}", value)
-            for value in expected.values()
-        )
-        or any(
-            hashlib.sha256(data).hexdigest() != expected[name]
-            for name, data in task_files.items()
-        )
+    if not isinstance(expected, dict) or any(
+        not isinstance(name, str)
+        or not isinstance(value, str)
+        or not re.fullmatch(r"[0-9a-f]{64}", value)
+        for name, value in expected.items()
     ):
         raise ValueError("Captured task files disagree with verified acquisition")
+    captured = {
+        name: value
+        for name, value in expected.items()
+        if not any(part in _EXCLUDED for part in PurePosixPath(name).parts)
+    }
+    if set(captured) != set(task_files) or any(
+        hashlib.sha256(data).hexdigest() != captured[name]
+        for name, data in task_files.items()
+    ):
+        raise ValueError("Captured task files disagree with verified acquisition")
 
 
 def _config(config):

diff --git a/examples/hub_validation_pipeline/test_cloud.py b/examples/hub_validation_pipeline/test_cloud.py
--- a/examples/hub_validation_pipeline/test_cloud.py
+++ b/examples/hub_validation_pipeline/test_cloud.py
@@ -669,12 +669,25 @@
 def test_task_exclusions_match_runner_source_identity(inputs, tmp_path):
     from openenv.validation.runner import source_digest
 
-    task = inputs[1]
+    repo, task, config = inputs
     for name in (".git", ".worktrees", ".venv", "__pycache__", "outputs"):
         (task / name).mkdir()
         (task / name / "binary-secret").write_bytes(b"\x00ignored")
     before = source_digest(task)
-    output, plan = prepared(inputs, tmp_path)
+    source = {
+        "kind": "hf",
+        "revision": SHA,
+        "task_path": ".",
+        "task_file_hashes": {
+            path.relative_to(task).as_posix(): hashlib.sha256(
+                path.read_bytes()
+            ).hexdigest()
+            for path in task.rglob("*")
+            if path.is_file()
+        },
+    }
+    output = tmp_path / "prepared"
+    plan = cloud.prepare(task, config, output, repo_root=repo, source=source)
     destination = tmp_path / "snapshot"
     cloud.extract_bundle(
         (output / "bundle.zip").read_bytes(), destination, plan["bundle_sha256"]

You can send follow-ups to the cloud agent here.

Reviewed by Cursor Bugbot for commit fe9363b. Configure here.

Comment thread examples/coding_audit/hf.py Resolved
@burtenshaw
burtenshaw changed the base branch from ben/rfc008-validation-completion to main November 12, 2026 09:17
@burtenshaw

Copy link
Copy Markdown
Collaborator Author

Independent read-only final-head review gate: both reviewers APPROVE 1c1405859fbf1ac0d33de22f4ab5e5e8bf4299aa, with no actionable findings. Correctness review ran 448 offline tests; spec/public-boundary review ran 442. All five new findings are closed; acquired excluded assets remain an explicit unsupported shape, not a filtered inventory. Parent verification: 1,115 focused and 2,895 broader CPU tests passed (8 skips, 58 deselected), lint/import/format and runtime lock checks passed. These are independent agent review verdicts, not a self-submitted GitHub APPROVED review or live qualification. No merge was performed.

@burtenshaw

Copy link
Copy Markdown
Collaborator Author

Independent final-head delta review APPROVE for 9c2c0b3db9700ec766cb7711db0168792749ef61: no actionable findings in the six-file quality-review delta over prior independently approved 1c1405859. Reviewer ran 249 offline tests and scoped lint/import/format checks. Parent focused run: 1,117 passed. All 17 threads resolved; runtime lock check passes. CI is still running. This is an independent agent review verdict, not a self-submitted GitHub approval. No merge performed.

@burtenshaw
burtenshaw merged commit f39a723 into main Nov 12, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature size: extra-large Extra-large pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant