Skip to content

fix(harnesses): run a Harbor task's prebuilt docker_image in docker-* harnesses - #873

Merged
antonwolfy merged 10 commits into
intel:mainfrom
vlad-perevezentsev:fix/harbor-docker-image
Nov 17, 2026
Merged

antonwolfy merged 10 commits into
intel:mainfrom
vlad-perevezentsev:fix/harbor-docker-image

Conversation

@vlad-perevezentsev

@vlad-perevezentsev vlad-perevezentsev commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A Harbor task whose task.toml sets [environment] docker_image and ships no environment/Dockerfile could not run in any docker-* harness: the harness always ran docker build, and the cell errored with open Dockerfile: no such file or directory. That is 10 of the 17 uxlfoundation/skills tasks at 6feb3f3 (every answer-quality task and sycl-device-discovery). CI's external-catalog smoke runs only onetbb-histogram-local-aggregation, which has a Dockerfile, so it never showed.

The docker-* harnesses now follow Harbor 0.24 (harbor/environments/definition.py):

  • Which image. should_use_prebuilt_docker_image: when docker_image is set, Harbor starts from it, also beside an environment/Dockerfile; only --force-build builds then, and dokimi has no such flag. So docker_image wins. A task without it builds dokimi/<task>:latest as before. Only the Harbor format reads the key: a staged SWE-bench task's Dockerfile starts FROM its docker_image, and it is still built.
  • Pull. The image is pulled only when the daemon lacks it (docker image inspect, then docker pull -q), as Harbor's docker compose up does, within the task's build_timeout_sec. A pull that fails or runs out of time errors the cell, like a build. A value that starts with - or holds a space is refused before any docker call: docker would read it as a flag. A docker_image that is not a string (docker_image = 5) fails the task's load, as Harbor's EnvironmentConfig (str | None) refuses it.
  • environment/ upload. should_upload_environment_dir: for a prebuilt-image task whose non-empty environment/ has no Dockerfile or docker-compose.yaml, Harbor copies its files into the workdir after the start (environment.workdir, else the container's pwd). A top-level environment/skills is left out, as it is left out of a build context.
  • Workdir. A prebuilt-image task that declares environment.workdir starts with docker run -w <workdir>: Harbor runs every exec there (DockerEnvironment.exec: cwd or task_env_config.workdir), and python:3.12-slim has no WORKDIR, so without it the oracle and the verifier ran in / and missed relative paths. docker run -w also creates the directory, so /app exists in python:3.12-slim before the copy and before docker-claude checks it. A task built from its Dockerfile keeps its image's WORKDIR, as on main (Harbor would override it; see below).
  • Network. The pull is the daemon's, like the FROM image of a docker build: --network build=none takes the network from a build's RUN steps and does not stop either. Checked live: docker build --network none of FROM alpine:3.19 pulled the missing image. To run offline, have the image on the daemon first.
  • Provenance. metrics.image records the image a cell ran on: {"source": "docker_image", "ref", "pulled", "id", "repo_digests"} for a prebuilt image (a tag moves, the id does not), {"source": "build", "ref"} for a built one. metrics is an open object in task_runs.v1, so the schema is unchanged.
  • eval inspect now lists docker-* for a Harbor task with a docker_image and a verifier. Before, it listed only harbor for exactly these tasks.

The new code is in harnesses/task_image.py. TaskSpec gets docker_image and environment_upload. docker_solver.py only calls the new module and gets shorter (893 → 889 lines; run() complexity 17 → 16; the growth baseline is lowered).

Changed output

On onetbb-bounded-image-flow-graph (uxl-skills 6feb3f3), tasks run --harnesses docker-oracle,docker-probe under the shared docker lock:

# before (main 96dfccf1)
docker-oracle  reward=None  RuntimeError: docker build failed (exit 1); see run.log
docker-probe   reward=None  RuntimeError: docker build failed (exit 1); see run.log
run.log: ERROR: failed to build: failed to solve: failed to read dockerfile: open Dockerfile: no such file or directory

# after (f6afa44d)
docker-oracle  reward=None  RuntimeError: verifier exited 0 but wrote no /logs/verifier/reward.txt; see run.log
docker-probe   reward=1.0   image={"source": "docker_image", "ref": "python:3.12-slim", "pulled": false,
                                  "id": "sha256:5e2bdb1b…", "repo_digests": ["python@sha256:5e2bdb1b…"]}
run.log: docker image inspect … python:3.12-slim / docker pull -q python:3.12-slim / docker run … python:3.12-slim sleep …
         docker exec -u root <c> mkdir -p /app / docker cp …/environment/. <c>:/app / solve.sh / test.sh

The task now starts, solves and verifies, and the probe passes on the image. The oracle cell is still errored, but for another reason: the uxl verifier writes only reward.json, and since #695 a verifier that exits 0 without reward.txt errors the cell. Answer-quality tasks stay errored, not scored, until T44 (reading reward.json, backlog bug 13). The probe shows pulled: false because the oracle cell had already pulled the image in the same run.

For a task that sets docker_image beside environment/Dockerfile, behaviour changes: it now runs on the image (as in Harbor), and run.log says [image] Dockerfile not built: [environment] docker_image <ref> wins, as in Harbor. With a tag no registry has, the cell errors with docker pull failed. The task.toml template in contributing-terminal-bench-task.md wrote such a pair (intel-hpc-bench/<task-name>:latest); that line is removed. To keep building the Dockerfile, remove docker_image. I found no such task in the uxl-skills checkout.

Evidence

  • tests/test_harbor_docker_image.py (25 cases) on fixtures shaped like the uxl answer-quality task:
    • A docker_image-only task pulls and runs that image with no build, under docker-oracle, docker-claude and docker-probe. An image the daemon has is not pulled.
    • environment/ is copied into /app after run and before the solve, without skills/, into the image WORKDIR or / when no workdir is declared, and not at all when environment/ is empty.
    • A Dockerfile-only task builds dokimi/<task>:latest with the same argv as on main. With both present, the image wins, there is no upload, and run.log says so.
    • Only the Harbor format reads the key. The pull happens under build=none. A pull timeout and a failed pull error the cell. A failed upload still removes the container. Bad references are refused.
  • On main the file fails 16 of 17 cases (before the review fixes). The one that passes pins the Dockerfile-only build argv. On the branch all pass. test_task_format_detect.py gets a case for the eval inspect consumers.
  • Mutations, each caught (12): always pull; any format reads docker_image; upload beside a Dockerfile; skills/ uploaded; no upload; no reference check; workdir fallback /app; upload outside the container's try; eval inspect ignoring docker_image; no run.log line for the skipped Dockerfile; no -w for the declared workdir; a non-string docker_image accepted.
  • Live, under the shared docker lock: the before/after above. A task with docker_image = "dokimi-test/no-such-image:none" errors with docker pull failed (exit 1) ("pull access denied … repository does not exist"). CI's smoke (onetbb-histogram-local-aggregation, Dockerfile) passes: docker-oracle 1/1, docker-probe 1/1, check_smoke_runs.py exit 0, metrics.image.source == "build".
  • Review round 1 on the PR: the container got no -w for the declared workdir (fixed, live: a solve.sh with cp /solution/answer.md answer.md printed pwd=/app), and a non-string docker_image was read as unset (now refused).
  • An independent review found two bugs, both fixed in separate commits. A failed upload left the task container running. eval inspect still said docker-* cannot run these tasks. It also asked for the run.log line and the changelog note for Dockerfile + docker_image tasks.
  • make ci-local --offline: 0 failed, 23 passed, 20 skipped (pytest 22615 passed, 79 skipped). check_growth.py, check_comment_blocks.py: clean.
  • Not run: sycl-device-discovery (its intel/oneapi image is many GB), and a paid docker-claude cell; the in-container launch on python:3.12-slim is covered by the passing probe, which runs the same workdir check.

Not in this PR

  • A built Harbor task ignores environment.workdir (left as on main): Harbor overrides the image's WORKDIR with it for every task (EnvironmentConfig.workdir, DockerEnvironment.exec). A test pins today's behaviour.

  • A non-string environment.workdir (workdir = 5) is ignored, while Harbor refuses it (str | None), as it now is for docker_image.

  • Reading reward.json (T44, backlog bug 13): until then the uxl answer-quality cells are errored.

  • A prebuilt image in other formats: a BenchFlow task.md sandbox image, a terminal-bench v1 compose service with image: and no build:.

  • The id of a built image in metrics.image (it would add a docker call to the build path).

  • Harbor's --force-build, and Harbor's image OS check (_validate_image_os).

  • Merge order: feat(tasks): docker-null checks the untouched start fails its tests #848 edits the same growth_baseline.json entries for docker_solver.py and the locals and imports of run(); feat(provenance): optional version and provenance on tasks and question-set items #800 edits tasks.py and contributing-terminal-bench-task.md. Whichever merges second rebases; the baseline lines come back with check_growth.py --update.

Checklist

  • Tests added/updated and pytest passes: tests/test_harbor_docker_image.py, tests/test_task_format_detect.py; 22615 passed
  • ruff, mypy, and schema checks pass: make ci-local --offline 0 failed; no schema change
  • Docs updated (or N/A): docs/coding-harnesses.md (new "Prebuilt images" section, the format table, the network table), docs/contributing-terminal-bench-task.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md: mutations, other sites of the bug class (eval inspect fixed, the rest listed above), bad inputs run live
  • Each new raise/exit and test names the doc, contract or issue it traces to: the reference check cites distribution/reference and the doc; the tests cite coding-harnesses.md or Harbor's functions
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header

A Harbor task whose task.toml sets [environment] docker_image and
ships no environment/Dockerfile failed every docker-* cell with
"open Dockerfile: no such file": the harness always ran docker build.
That is 10 of the 17 uxlfoundation/skills tasks (every answer-quality
task and sycl-device-discovery).

Follow Harbor 0.24 (environments/definition.py):
should_use_prebuilt_docker_image starts from docker_image whenever it
is set, also beside a Dockerfile (only --force-build builds then), and
should_upload_environment_dir copies a non-empty environment/ without
a Dockerfile or docker-compose.yaml into the workdir after the start;
that copy of environment/.keep is what creates /app in
python:3.12-slim for the uxl tasks.

The image is pulled only when the daemon lacks it (compose up's
policy), within build_timeout_sec; a pull timeout errors the cell like
a build timeout. --network build=none does not stop the pull, as it
does not stop a build's FROM. metrics.image records which image ran,
with the id and repo digests of a prebuilt one. Only the Harbor format
reads the key: a staged SWE-bench task builds its FROM-image
Dockerfile as before. The new code is in harnesses/task_image.py;
docker_solver.py shrinks (893 -> 889 lines, run() 17 -> 16).

The task.toml template in contributing-terminal-bench-task.md set a
local-only docker_image beside a required Dockerfile; that line goes,
as such a task would now pull a tag no registry has.

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
… fails

The copy of environment/ into a prebuilt-image container ran after
the container started but outside the try whose finally runs
`docker rm -f`, so a failed mkdir, inspect or cp left a container
sleeping for at least 1800 s. It now runs inside that try.

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
…image task

The Harbor probe counted a task as docker-* runnable only with an
environment/Dockerfile, so `eval inspect` left docker-* out for the
docker_image-only tasks the docker harnesses now run (e.g. uxl's
onetbb-bounded-image-flow-graph printed `-> harbor`). It now accepts
TaskSpec.docker_image as well; the verifier is still required.

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
A task that sets [environment] docker_image beside
environment/Dockerfile now runs on the image, as Harbor does, and no
longer builds the Dockerfile (the old task.toml template wrote such a
pair with a local-only tag). run.log now names the Dockerfile that was
not built, and the changelog fragment and the doc say to remove
docker_image to keep building it.

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
@vlad-perevezentsev
vlad-perevezentsev requested a review from a team as a code owner November 17, 2026 16:38
@vlad-perevezentsev vlad-perevezentsev self-assigned this Nov 17, 2026
@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s).

CI at `a33c200`: 1 failing, 25 passing, 3 skipped
Check Status
Tests (py3.11) failing at step Run tests
Review details
  • Commit reviewed: a33c200 against merge base a3049cf
  • Coverage: 2 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 126k tokens (99k from cache), 31s
  • Tool calls: 10 (6 file_read, 3 code_search, 1 local_check_results)
  • small change set: 2 files

@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.80%. Comparing base (ea458d2) to head (a33c200).
⚠️ Report is 3 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #873   +/-   ##
=======================================
  Coverage   95.79%   95.80%           
=======================================
  Files         429      430    +1     
  Lines       56821    56909   +88     
  Branches     9783     9796   +13     
=======================================
+ Hits        54433    54519   +86     
- Misses       1485     1487    +2     
  Partials      903      903           
Files with missing lines Coverage Δ
dokimi/dokimi/harnesses/docker_solver.py 97.97% <100.00%> (+<0.01%) ⬆️
dokimi/dokimi/harnesses/task_image.py 100.00% <100.00%> (ø)
dokimi/dokimi/harnesses/tasks.py 97.33% <100.00%> (+0.11%) ⬆️
dokimi/dokimi/task_formats/dirs.py 91.22% <100.00%> (+0.07%) ⬆️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not approved yet · reviewed ba13076

Next: @vlad-perevezentsev: answer 1 open thread from exai-bot and settle 1 open question (below).

Blocking

  • Still holds on this head: thread by exai-bot – With docker_image = "python:3.12-slim" and workdir = "/app", the oracle and verifier docker exec calls carry no -w (dokimi/dokimi/harnesses/docker_solver.py:383, :637), so bash -lc starts in / while the upload went to /app.
  • Open question for @vlad-perevezentsev: A docker_image that is not a string, such as docker_image = 5, is read as unset (dokimi/dokimi/harnesses/tasks.py:140). Refuse a non-string docker_image the way a bad reference is refused (dokimi/dokimi/harnesses/task_image.py:46), or state in dokimi/docs/coding-harnesses.md that a non-string is ignored.

Verified on ba13076: ✅ changed tests: 94 passed, fail on main · ✅ full suite with main merged in: 22568 passed · ✅ merges onto main · ⚠️ 12 broken variants of the changed code: 11 caught by the tests, 1 not · 2 mutations tried · 9 edge inputs tried

23 checks behind it, each with its evidence
Area What holds How
Dockerfile-only build A task with no docker_image builds dokimi/<task>:latest with the build argv the test pins, and nothing is pulled or uploaded tests on the head, edge input (none)
Prebuilt image runs A docker_image-only task pulls and runs that image with no build under docker-oracle, docker-claude and docker-probe, and metrics.image records the image id and repo digests tests on the head
Pull only when missing An image the daemon already has is not pulled, and a missing one is pulled with -q under the task's build_timeout_sec tests on the head
Pull timeout A pull that runs out of time raises docker pull timed out after 77s, the cell's reward is None, and run.log holds PULL TIMEOUT tests on the head, edge input (concurrency)
Pull failure A pull that exits 1 raises docker pull failed (exit 1) before any run is started tests on the head
Pull under build=none A build=none setting does not stop the pull, and metrics.image still records pulled: true tests on the head
Upload into workdir The files of environment/ are copied into /app after the start and before the solve, /app is made first with mkdir -p, and a top-level skills/ is left out tests on the head
Upload destination Without a declared workdir the copy goes to the image WORKDIR, else / tests on the head
Failed upload A failed docker cp raises docker cp failed (exit 1) and still runs rm -f on the task's container tests on the head
Image over dockerfile With both docker_image and environment/Dockerfile, the image is pulled, nothing is built or uploaded, and run.log names the Dockerfile as not built tests on the head
Staged swe-bench tasks A docker_image on a non-Harbor task is not read, so a staged SWE-bench task still builds its Dockerfile tests on the head
Image reference check A blank docker_image, one starting with ß, one ending in CRLF and one holding a space are each refused with ValueError, and no docker call runs ran it, edge input (text)
Flag-like reference --privileged as a docker_image is refused before any docker call tests on the head, edge input (argv-flag)
Non-numeric timeout A build_timeout_sec of "soon" falls back to the 600 s default for the pull ran it, edge input (numbers)
No environment dir A task with no environment/ directory has no upload: environment_upload is None ran it, edge input (paths)
Paths with spaces A task directory with a space and ß in its name reaches docker cp as one argument, unchanged ran it, edge input (paths)
Eval inspect HarborProbe.describe lists docker-* for a Harbor task with a docker_image and no Dockerfile, and not for a task with neither ran it, edge input (cli)
Missing or empty image key A task whose docker_image is absent or "" builds its Dockerfile and runs no pull ran it, edge input (structured)
Reference check pinned Negating the reference check in ready_image (dokimi/dokimi/harnesses/task_image.py:46) is caught by the changed tests broke the code, tests caught it
Build context skills pinned Dropping the copy of environment/ in _build_image (dokimi/dokimi/harnesses/docker_solver.py:324) is caught by tests/test_docker_task_md.py, which checks that the build context drops bundled skills broke the code, tests caught it
Changed tests The 94 cases in the two changed test files pass on the head, and 18 of them fail on main tests on the head
Full suite The full suite passes with main merged in full suite, main merged in
Merge The PR merges onto main with no conflict merge onto main

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
…orkdir

Harbor runs every exec of a task in environment.workdir
(DockerEnvironment.exec: cwd or task_env_config.workdir). The docker-*
harnesses gave the container no -w, so on an image without a WORKDIR
(python:3.12-slim) the oracle and the verifier ran in / while the
environment/ files had been copied to /app, and a relative path in
solve.sh or test.sh missed them. A prebuilt-image task with a declared
workdir now starts with `docker run -w <workdir>`; a built image keeps
its own WORKDIR, as before.

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
`docker_image = 5` (or a list, a boolean) was read as unset, so the
task was built from a Dockerfile it may not have, while Harbor 0.24
refuses it: EnvironmentConfig types the field `str | None`, and
pydantic rejects a non-string ("Input should be a valid string").
load_task now raises ValueError for such a value, naming the file.

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
The test named "without a declared workdir" also covered a built task
that does declare one. Split it: a prebuilt-image task without a
workdir gets no -w, and a built task keeps its image's WORKDIR despite
a declared one, which differs from Harbor (it overrides WORKDIR for
every task) and is left as on main on purpose. The workdir_args
docstring and the doc say so.

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
Since the container of a prebuilt-image task starts with
`docker run -w <workdir>`, docker creates /app itself, so the
environment/.keep copy no longer is what creates it: that claim goes
from the fragment and coding-harnesses.md. The fragment now says the
container runs in environment.workdir and that a non-string
docker_image fails the task's load.

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
@vlad-perevezentsev

Copy link
Copy Markdown
Contributor Author

🟡 Not approved yet · reviewed ba13076 Next: @vlad-perevezentsev: answer 1 open thread from exai-b…

@vchamarthi Both points are addressed. The workdir thread is fixed in 26b7e9b. A non-string docker_image is now refused (ccc5433): load_task raises task.toml [environment] docker_image must be a string, not 5, as Harbor 0.24 does (EnvironmentConfig types it str | None, and pydantic rejects 5 with "Input should be a valid string"); an empty string still means no image, as in Harbor. coding-harnesses.md says so, and a test covers 5, true and a list.

Comment thread dokimi/dokimi/harnesses/task_image.py Outdated
if pulled:
docker(["pull", "-q", image], log, timeout=_task_build_timeout(task))
found = docker(inspect, log)
return image, {"source": "docker_image", "ref": image, "pulled": pulled, **json.loads(found.stdout or "")}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — Impact: metrics.image.id and repo_digests can describe a different image from the one evaluated, breaking the documented provenance contract. Trigger: on a shared daemon, another process pulls or retags the configured tag after this inspection but before _start_container creates the container (sandbox setup also runs between those steps). Evidence: this returns the mutable reference for docker run, while the metrics retain the earlier inspected ID; no post-start reconciliation updates them. Return the inspected image ID for startup while retaining the configured reference in ref, or derive provenance from the created container's .Image. Add a fake-daemon regression where the tag changes between inspection and startup. Workaround: configure a digest-qualified image reference. Severity: medium because this causes bounded evaluation-record and reproducibility errors, not demonstrated compromise or data loss.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The image choice follows Harbor's rule, the pull happens only when the daemon lacks the image, the upload sits inside the try that removes the container, and the -w and non-string docker_image fixes from the earlier round hold. One nit inline.

What I ran, on a Linux review host at a3049cf:

  • make ci-local with docker: 39 passed, 0 failed, 14 skipped; 22,676 tests; changed-line coverage 100%
  • The external-catalog smoke passes through docker-oracle and docker-probe (1/1 each) on the Dockerfile task.
  • Every non-merge commit carries a Signed-off-by.

Checked, no issues

  • Scope: docker_image is read only for the Harbor format, so a staged SWE-bench task still builds its Dockerfile.
  • Cleanup: upload_environment runs inside the try whose finally removes the container.
  • Docs: the format table, the network table and the task.toml template match the code; no schema change is needed, as metrics is an open object.
  • Changed lines: 81 measured, 100% covered.

Comment thread dokimi/dokimi/harnesses/task_image.py Resolved
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I pushed a commit per nit, up to a33c200, which fix the nit from my review:

  • a33c200: dokimi/dokimi/harnesses/task_image.py:56: ready_image parses the inspect output once and returns the inspected image id to start from, keeping the configured tag in ref.

make ci-local at a33c200: 39 passed, 0 failed, 14 skipped.

@antonwolfy
antonwolfy merged commit 51968ca into intel:main Nov 17, 2026
37 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants