Repository navigation
fix(harnesses): run a Harbor task's prebuilt docker_image in docker-* harnesses - #873
Conversation
A Harbor task whose task.toml sets [environment] docker_image and ships no environment/Dockerfile failed every docker-* cell with "open Dockerfile: no such file": the harness always ran docker build. That is 10 of the 17 uxlfoundation/skills tasks (every answer-quality task and sycl-device-discovery). Follow Harbor 0.24 (environments/definition.py): should_use_prebuilt_docker_image starts from docker_image whenever it is set, also beside a Dockerfile (only --force-build builds then), and should_upload_environment_dir copies a non-empty environment/ without a Dockerfile or docker-compose.yaml into the workdir after the start; that copy of environment/.keep is what creates /app in python:3.12-slim for the uxl tasks. The image is pulled only when the daemon lacks it (compose up's policy), within build_timeout_sec; a pull timeout errors the cell like a build timeout. --network build=none does not stop the pull, as it does not stop a build's FROM. metrics.image records which image ran, with the id and repo digests of a prebuilt one. Only the Harbor format reads the key: a staged SWE-bench task builds its FROM-image Dockerfile as before. The new code is in harnesses/task_image.py; docker_solver.py shrinks (893 -> 889 lines, run() 17 -> 16). The task.toml template in contributing-terminal-bench-task.md set a local-only docker_image beside a required Dockerfile; that line goes, as such a task would now pull a tag no registry has. Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
… fails The copy of environment/ into a prebuilt-image container ran after the container started but outside the try whose finally runs `docker rm -f`, so a failed mkdir, inspect or cp left a container sleeping for at least 1800 s. It now runs inside that try. Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
…image task The Harbor probe counted a task as docker-* runnable only with an environment/Dockerfile, so `eval inspect` left docker-* out for the docker_image-only tasks the docker harnesses now run (e.g. uxl's onetbb-bounded-image-flow-graph printed `-> harbor`). It now accepts TaskSpec.docker_image as well; the verifier is still required. Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
A task that sets [environment] docker_image beside environment/Dockerfile now runs on the image, as Harbor does, and no longer builds the Dockerfile (the old task.toml template wrote such a pair with a local-only tag). run.log now names the Dockerfile that was not built, and the changelog fragment and the doc say to remove docker_image to keep building it. Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s). CI at `a33c200`: 1 failing, 25 passing, 3 skipped
Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #873 +/- ##
=======================================
Coverage 95.79% 95.80%
=======================================
Files 429 430 +1
Lines 56821 56909 +88
Branches 9783 9796 +13
=======================================
+ Hits 54433 54519 +86
- Misses 1485 1487 +2
Partials 903 903
🚀 New features to boost your workflow:
|
vchamarthi
left a comment
There was a problem hiding this comment.
🟡 Not approved yet · reviewed ba13076
Next: @vlad-perevezentsev: answer 1 open thread from exai-bot and settle 1 open question (below).
Blocking
- Still holds on this head: thread by exai-bot – With
docker_image = "python:3.12-slim"andworkdir = "/app", the oracle and verifierdocker execcalls carry no-w(dokimi/dokimi/harnesses/docker_solver.py:383,:637), sobash -lcstarts in/while the upload went to/app. - Open question for @vlad-perevezentsev: A
docker_imagethat is not a string, such asdocker_image = 5, is read as unset (dokimi/dokimi/harnesses/tasks.py:140). Refuse a non-stringdocker_imagethe way a bad reference is refused (dokimi/dokimi/harnesses/task_image.py:46), or state indokimi/docs/coding-harnesses.mdthat a non-string is ignored.
Verified on ba13076: ✅ changed tests: 94 passed, fail on main · ✅ full suite with main merged in: 22568 passed · ✅ merges onto main ·
23 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Dockerfile-only build | A task with no docker_image builds dokimi/<task>:latest with the build argv the test pins, and nothing is pulled or uploaded |
tests on the head, edge input (none) |
| Prebuilt image runs | A docker_image-only task pulls and runs that image with no build under docker-oracle, docker-claude and docker-probe, and metrics.image records the image id and repo digests |
tests on the head |
| Pull only when missing | An image the daemon already has is not pulled, and a missing one is pulled with -q under the task's build_timeout_sec |
tests on the head |
| Pull timeout | A pull that runs out of time raises docker pull timed out after 77s, the cell's reward is None, and run.log holds PULL TIMEOUT |
tests on the head, edge input (concurrency) |
| Pull failure | A pull that exits 1 raises docker pull failed (exit 1) before any run is started |
tests on the head |
| Pull under build=none | A build=none setting does not stop the pull, and metrics.image still records pulled: true |
tests on the head |
| Upload into workdir | The files of environment/ are copied into /app after the start and before the solve, /app is made first with mkdir -p, and a top-level skills/ is left out |
tests on the head |
| Upload destination | Without a declared workdir the copy goes to the image WORKDIR, else / |
tests on the head |
| Failed upload | A failed docker cp raises docker cp failed (exit 1) and still runs rm -f on the task's container |
tests on the head |
| Image over dockerfile | With both docker_image and environment/Dockerfile, the image is pulled, nothing is built or uploaded, and run.log names the Dockerfile as not built |
tests on the head |
| Staged swe-bench tasks | A docker_image on a non-Harbor task is not read, so a staged SWE-bench task still builds its Dockerfile |
tests on the head |
| Image reference check | A blank docker_image, one starting with ß, one ending in CRLF and one holding a space are each refused with ValueError, and no docker call runs |
ran it, edge input (text) |
| Flag-like reference | --privileged as a docker_image is refused before any docker call |
tests on the head, edge input (argv-flag) |
| Non-numeric timeout | A build_timeout_sec of "soon" falls back to the 600 s default for the pull |
ran it, edge input (numbers) |
| No environment dir | A task with no environment/ directory has no upload: environment_upload is None |
ran it, edge input (paths) |
| Paths with spaces | A task directory with a space and ß in its name reaches docker cp as one argument, unchanged |
ran it, edge input (paths) |
| Eval inspect | HarborProbe.describe lists docker-* for a Harbor task with a docker_image and no Dockerfile, and not for a task with neither |
ran it, edge input (cli) |
| Missing or empty image key | A task whose docker_image is absent or "" builds its Dockerfile and runs no pull |
ran it, edge input (structured) |
| Reference check pinned | Negating the reference check in ready_image (dokimi/dokimi/harnesses/task_image.py:46) is caught by the changed tests |
broke the code, tests caught it |
| Build context skills pinned | Dropping the copy of environment/ in _build_image (dokimi/dokimi/harnesses/docker_solver.py:324) is caught by tests/test_docker_task_md.py, which checks that the build context drops bundled skills |
broke the code, tests caught it |
| Changed tests | The 94 cases in the two changed test files pass on the head, and 18 of them fail on main | tests on the head |
| Full suite | The full suite passes with main merged in | full suite, main merged in |
| Merge | The PR merges onto main with no conflict | merge onto main |
Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
…orkdir Harbor runs every exec of a task in environment.workdir (DockerEnvironment.exec: cwd or task_env_config.workdir). The docker-* harnesses gave the container no -w, so on an image without a WORKDIR (python:3.12-slim) the oracle and the verifier ran in / while the environment/ files had been copied to /app, and a relative path in solve.sh or test.sh missed them. A prebuilt-image task with a declared workdir now starts with `docker run -w <workdir>`; a built image keeps its own WORKDIR, as before. Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
`docker_image = 5` (or a list, a boolean) was read as unset, so the
task was built from a Dockerfile it may not have, while Harbor 0.24
refuses it: EnvironmentConfig types the field `str | None`, and
pydantic rejects a non-string ("Input should be a valid string").
load_task now raises ValueError for such a value, naming the file.
Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
The test named "without a declared workdir" also covered a built task that does declare one. Split it: a prebuilt-image task without a workdir gets no -w, and a built task keeps its image's WORKDIR despite a declared one, which differs from Harbor (it overrides WORKDIR for every task) and is left as on main on purpose. The workdir_args docstring and the doc say so. Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
Since the container of a prebuilt-image task starts with `docker run -w <workdir>`, docker creates /app itself, so the environment/.keep copy no longer is what creates it: that claim goes from the fragment and coding-harnesses.md. The fragment now says the container runs in environment.workdir and that a non-string docker_image fails the task's load. Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
@vchamarthi Both points are addressed. The workdir thread is fixed in 26b7e9b. A non-string docker_image is now refused (ccc5433): load_task raises |
| if pulled: | ||
| docker(["pull", "-q", image], log, timeout=_task_build_timeout(task)) | ||
| found = docker(inspect, log) | ||
| return image, {"source": "docker_image", "ref": image, "pulled": pulled, **json.loads(found.stdout or "")} |
There was a problem hiding this comment.
Must-fix — Impact: metrics.image.id and repo_digests can describe a different image from the one evaluated, breaking the documented provenance contract. Trigger: on a shared daemon, another process pulls or retags the configured tag after this inspection but before _start_container creates the container (sandbox setup also runs between those steps). Evidence: this returns the mutable reference for docker run, while the metrics retain the earlier inspected ID; no post-start reconciliation updates them. Return the inspected image ID for startup while retaining the configured reference in ref, or derive provenance from the created container's .Image. Add a fake-daemon regression where the tag changes between inspection and startup. Workaround: configure a digest-qualified image reference. Severity: medium because this causes bounded evaluation-record and reproducibility errors, not demonstrated compromise or data loss.
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. The image choice follows Harbor's rule, the pull happens only when the daemon lacks the image, the upload sits inside the try that removes the container, and the -w and non-string docker_image fixes from the earlier round hold. One nit inline.
What I ran, on a Linux review host at a3049cf:
make ci-localwith docker: 39 passed, 0 failed, 14 skipped; 22,676 tests; changed-line coverage 100%- The external-catalog smoke passes through
docker-oracleanddocker-probe(1/1 each) on the Dockerfile task. - Every non-merge commit carries a
Signed-off-by.
Checked, no issues
- Scope:
docker_imageis read only for the Harbor format, so a staged SWE-bench task still builds its Dockerfile. - Cleanup:
upload_environmentruns inside thetrywhosefinallyremoves the container. - Docs: the format table, the network table and the
task.tomltemplate match the code; no schema change is needed, asmetricsis an open object. - Changed lines: 81 measured, 100% covered.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. I pushed a commit per nit, up to a33c200, which fix the nit from my review:
a33c200:dokimi/dokimi/harnesses/task_image.py:56:ready_imageparses the inspect output once and returns the inspected image id to start from, keeping the configured tag inref.
make ci-local at a33c200: 39 passed, 0 failed, 14 skipped.
Summary
A Harbor task whose
task.tomlsets[environment] docker_imageand ships noenvironment/Dockerfilecould not run in anydocker-*harness: the harness always randocker build, and the cell errored withopen Dockerfile: no such file or directory. That is 10 of the 17 uxlfoundation/skills tasks at 6feb3f3 (every answer-quality task andsycl-device-discovery). CI's external-catalog smoke runs onlyonetbb-histogram-local-aggregation, which has a Dockerfile, so it never showed.The
docker-*harnesses now follow Harbor 0.24 (harbor/environments/definition.py):should_use_prebuilt_docker_image: whendocker_imageis set, Harbor starts from it, also beside anenvironment/Dockerfile; only--force-buildbuilds then, and dokimi has no such flag. Sodocker_imagewins. A task without it buildsdokimi/<task>:latestas before. Only the Harbor format reads the key: a staged SWE-bench task's Dockerfile startsFROMitsdocker_image, and it is still built.docker image inspect, thendocker pull -q), as Harbor'sdocker compose updoes, within the task'sbuild_timeout_sec. A pull that fails or runs out of time errors the cell, like a build. A value that starts with-or holds a space is refused before any docker call: docker would read it as a flag. Adocker_imagethat is not a string (docker_image = 5) fails the task's load, as Harbor'sEnvironmentConfig(str | None) refuses it.environment/upload.should_upload_environment_dir: for a prebuilt-image task whose non-emptyenvironment/has noDockerfileordocker-compose.yaml, Harbor copies its files into the workdir after the start (environment.workdir, else the container'spwd). A top-levelenvironment/skillsis left out, as it is left out of a build context.environment.workdirstarts withdocker run -w <workdir>: Harbor runs every exec there (DockerEnvironment.exec:cwd or task_env_config.workdir), andpython:3.12-slimhas noWORKDIR, so without it the oracle and the verifier ran in/and missed relative paths.docker run -walso creates the directory, so/appexists inpython:3.12-slimbefore the copy and before docker-claude checks it. A task built from its Dockerfile keeps its image'sWORKDIR, as on main (Harbor would override it; see below).FROMimage of adocker build:--network build=nonetakes the network from a build'sRUNsteps and does not stop either. Checked live:docker build --network noneofFROM alpine:3.19pulled the missing image. To run offline, have the image on the daemon first.metrics.imagerecords the image a cell ran on:{"source": "docker_image", "ref", "pulled", "id", "repo_digests"}for a prebuilt image (a tag moves, the id does not),{"source": "build", "ref"}for a built one.metricsis an open object intask_runs.v1, so the schema is unchanged.eval inspectnow listsdocker-*for a Harbor task with adocker_imageand a verifier. Before, it listed onlyharborfor exactly these tasks.The new code is in
harnesses/task_image.py.TaskSpecgetsdocker_imageandenvironment_upload.docker_solver.pyonly calls the new module and gets shorter (893 → 889 lines;run()complexity 17 → 16; the growth baseline is lowered).Changed output
On onetbb-bounded-image-flow-graph (uxl-skills 6feb3f3),
tasks run --harnesses docker-oracle,docker-probeunder the shared docker lock:The task now starts, solves and verifies, and the probe passes on the image. The oracle cell is still errored, but for another reason: the uxl verifier writes only
reward.json, and since #695 a verifier that exits 0 withoutreward.txterrors the cell. Answer-quality tasks stay errored, not scored, until T44 (readingreward.json, backlog bug 13). The probe showspulled: falsebecause the oracle cell had already pulled the image in the same run.For a task that sets
docker_imagebesideenvironment/Dockerfile, behaviour changes: it now runs on the image (as in Harbor), andrun.logsays[image] Dockerfile not built: [environment] docker_image <ref> wins, as in Harbor. With a tag no registry has, the cell errors withdocker pull failed. Thetask.tomltemplate incontributing-terminal-bench-task.mdwrote such a pair (intel-hpc-bench/<task-name>:latest); that line is removed. To keep building the Dockerfile, removedocker_image. I found no such task in the uxl-skills checkout.Evidence
tests/test_harbor_docker_image.py(25 cases) on fixtures shaped like the uxl answer-quality task:docker_image-only task pulls and runs that image with nobuild, underdocker-oracle,docker-claudeanddocker-probe. An image the daemon has is not pulled.environment/is copied into/appafterrunand before the solve, withoutskills/, into the imageWORKDIRor/when no workdir is declared, and not at all whenenvironment/is empty.dokimi/<task>:latestwith the same argv as on main. With both present, the image wins, there is no upload, andrun.logsays so.build=none. A pull timeout and a failed pull error the cell. A failed upload still removes the container. Bad references are refused.test_task_format_detect.pygets a case for theeval inspectconsumers.docker_image; upload beside a Dockerfile;skills/uploaded; no upload; no reference check; workdir fallback/app; upload outside the container'stry;eval inspectignoringdocker_image; norun.logline for the skipped Dockerfile; no-wfor the declared workdir; a non-stringdocker_imageaccepted.docker_image = "dokimi-test/no-such-image:none"errors withdocker pull failed (exit 1)("pull access denied … repository does not exist"). CI's smoke (onetbb-histogram-local-aggregation, Dockerfile) passes: docker-oracle 1/1, docker-probe 1/1,check_smoke_runs.pyexit 0,metrics.image.source == "build".-wfor the declared workdir (fixed, live: asolve.shwithcp /solution/answer.md answer.mdprintedpwd=/app), and a non-stringdocker_imagewas read as unset (now refused).eval inspectstill saiddocker-*cannot run these tasks. It also asked for therun.logline and the changelog note for Dockerfile +docker_imagetasks.make ci-local --offline: 0 failed, 23 passed, 20 skipped (pytest 22615 passed, 79 skipped).check_growth.py,check_comment_blocks.py: clean.sycl-device-discovery(itsintel/oneapiimage is many GB), and a paiddocker-claudecell; the in-container launch onpython:3.12-slimis covered by the passing probe, which runs the same workdir check.Not in this PR
A built Harbor task ignores
environment.workdir(left as on main): Harbor overrides the image'sWORKDIRwith it for every task (EnvironmentConfig.workdir,DockerEnvironment.exec). A test pins today's behaviour.A non-string
environment.workdir(workdir = 5) is ignored, while Harbor refuses it (str | None), as it now is fordocker_image.Reading
reward.json(T44, backlog bug 13): until then the uxl answer-quality cells are errored.A prebuilt image in other formats: a BenchFlow
task.mdsandboximage, a terminal-bench v1 compose service withimage:and nobuild:.The id of a built image in
metrics.image(it would add a docker call to the build path).Harbor's
--force-build, and Harbor's image OS check (_validate_image_os).Merge order: feat(tasks): docker-null checks the untouched start fails its tests #848 edits the same
growth_baseline.jsonentries fordocker_solver.pyand the locals and imports ofrun(); feat(provenance): optional version and provenance on tasks and question-set items #800 editstasks.pyandcontributing-terminal-bench-task.md. Whichever merges second rebases; the baseline lines come back withcheck_growth.py --update.Checklist
pytestpasses:tests/test_harbor_docker_image.py,tests/test_task_format_detect.py; 22615 passedruff,mypy, and schema checks pass:make ci-local --offline0 failed; no schema changedocs/coding-harnesses.md(new "Prebuilt images" section, the format table, the network table),docs/contributing-terminal-bench-task.mdAGENTS.md: mutations, other sites of the bug class (eval inspectfixed, the rest listed above), bad inputs run liveraise/exit and test names the doc, contract or issue it traces to: the reference check cites distribution/reference and the doc; the tests citecoding-harnesses.mdor Harbor's functions