Repository navigation
fix(harnesses): run a Harbor task's prebuilt docker_image in docker-* harnesses #873
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
antonwolfy
merged 10 commits into
intel:main
from
vlad-perevezentsev:fix/harbor-docker-image
Nov 17, 2026
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
f6afa44
fix(harnesses): run a Harbor task on its prebuilt docker_image
vlad-perevezentsev a5131a7
fix(harnesses): remove the task container when the environment upload…
vlad-perevezentsev f517a49
fix(task-formats): eval inspect lists docker-* for a prebuilt docker_…
vlad-perevezentsev ba13076
fix(harnesses): say in run.log that docker_image skips the Dockerfile
vlad-perevezentsev 152d3d6
Merge intel/main into fix/harbor-docker-image
vlad-perevezentsev 26b7e9b
fix(harnesses): run a prebuilt-image task container in its declared w…
vlad-perevezentsev ccc5433
fix(tasks): refuse a Harbor docker_image that is not a string
vlad-perevezentsev 16a11b5
test(harnesses): pin the built-task workdir case under its own name
vlad-perevezentsev a3049cf
docs(harnesses): fragment and doc say what -w and the load check change
vlad-perevezentsev a33c200
fix(harnesses): start the container from the inspected image id
antonwolfy File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
23 changes: 23 additions & 0 deletions
23
dokimi/changelog.d/20261117_160000_vlad-perevezentsev_harbor-docker-image.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| <!-- | ||
| Copyright (C) 2026 Intel Corporation | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| --> | ||
|
|
||
| ### Fixed | ||
|
|
||
| - **`docker-*` harnesses run a Harbor task on its prebuilt `[environment] docker_image`** — a task | ||
| that sets one and ships no `environment/Dockerfile` (10 of the 17 uxlfoundation/skills tasks: | ||
| every answer-quality task and `sycl-device-discovery`) failed its cell with `docker build`'s | ||
| "open Dockerfile: no such file". As in Harbor 0.24, the image is used whenever it is set (also | ||
| beside a Dockerfile), pulled only when the daemon lacks it, within `build_timeout_sec`, and the | ||
| loose files of `environment/` are copied into the workdir after the start. The container runs | ||
| in the task's `environment.workdir` (`docker run -w`), so `solve.sh` and `test.sh` find relative | ||
| paths. A `docker_image` that is not a string (`docker_image = 5`) now fails the task's load, as | ||
| in Harbor. `--network build=none` does not stop the pull, as it does not stop a | ||
| build's `FROM`. `metrics.image` records the image a cell ran on: its id and repo digests for a | ||
| prebuilt one. A task without `docker_image` builds as before. A task that sets `docker_image` | ||
| beside `environment/Dockerfile` (as the old `task.toml` template in | ||
| `contributing-terminal-bench-task.md` did, with a local `intel-hpc-bench/...` tag) now runs on | ||
| that image and no longer builds the Dockerfile; `run.log` says so, and a tag no registry has | ||
| errors the cell with `docker pull failed`. Remove `docker_image` to keep building the | ||
| Dockerfile. See `docs/coding-harnesses.md`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| # Copyright (C) 2026 Intel Corporation | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """The image a ``docker-*`` task container starts from: built from the task, or Harbor's prebuilt one. | ||
|
|
||
| A Harbor task with ``[environment] docker_image`` (:attr:`.TaskSpec.docker_image`) runs on that | ||
| image, as ``harbor run`` does: the image is pulled only when the daemon lacks it (what Harbor's | ||
| ``docker compose up`` does), and once the container runs, the task's loose ``environment/`` files | ||
| are copied into its workdir (:attr:`.TaskSpec.environment_upload`). Any other task builds | ||
| ``dokimi/<task>:latest``. ``metrics.image`` records which ran, and a pulled image's id and digests. | ||
|
|
||
| The pull is the daemon's, like a ``docker build``'s base image: ``--network build=none``, which | ||
| takes the network away from a build's ``RUN`` steps, does not stop it. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import json | ||
| import re | ||
| import shutil | ||
| import tempfile | ||
| from pathlib import Path | ||
| from typing import Any, Callable | ||
|
|
||
| from dokimi.docker_cmd import Docker | ||
|
|
||
| from .tasks import TaskSpec, _task_build_timeout | ||
|
|
||
| # One JSON object per image; RepoDigests is empty for an image that came from no registry. | ||
| _RECORD_FORMAT = '{"id": {{json .Id}}, "repo_digests": {{json .RepoDigests}}}' | ||
| # A reference starts with a letter or digit (distribution/reference); "-..." would be a docker flag. | ||
| _REFERENCE_RE = re.compile(r"[A-Za-z0-9]\S*") | ||
|
|
||
|
|
||
| def ready_image(task: TaskSpec, log, docker: Docker, build: Callable[[str], None]) -> tuple[str, dict[str, Any]]: | ||
| """The image to start the task container from, and its ``metrics.image`` record. | ||
|
|
||
| *build* builds the task's Dockerfile under the tag it is given. The pull gets the task's | ||
| ``build_timeout_sec``: running out raises :class:`subprocess.TimeoutExpired`, as a build does. | ||
| """ | ||
| image = task.docker_image | ||
| if image is None: | ||
| tag = f"dokimi/{task.name}:latest" | ||
| build(tag) | ||
| return tag, {"source": "build", "ref": tag} | ||
| if not _REFERENCE_RE.fullmatch(image): | ||
| raise ValueError(f"{task.name}: [environment] docker_image {image!r} is not a docker image reference") | ||
| if (task.build_context / task.dockerfile).is_file(): | ||
| log.write(f"[image] {task.dockerfile} not built: [environment] docker_image {image} wins, as in Harbor\n") | ||
| inspect = ["image", "inspect", "-f", _RECORD_FORMAT, image] | ||
| found = docker(inspect, log, check=False) | ||
| pulled = found.returncode != 0 | ||
| if pulled: | ||
| docker(["pull", "-q", image], log, timeout=_task_build_timeout(task)) | ||
| found = docker(inspect, log) | ||
| record = json.loads(found.stdout or "") | ||
| return record["id"], {"source": "docker_image", "ref": image, "pulled": pulled, **record} | ||
|
|
||
|
|
||
| def workdir_args(task: TaskSpec) -> list[str]: | ||
| """``-w`` for a prebuilt-image task's ``environment.workdir``: Harbor runs every exec there | ||
| (``DockerEnvironment.exec``: ``cwd or task_env_config.workdir``). Harbor does so for a built task | ||
| too; a built task keeps its image's ``WORKDIR`` here on purpose, as on main.""" | ||
| workdir = task.environment.get("workdir") | ||
| return ["-w", workdir] if task.docker_image is not None and isinstance(workdir, str) and workdir else [] | ||
|
|
||
|
|
||
| def upload_environment(task: TaskSpec, container: str, log, docker: Docker) -> None: | ||
| """Copy :attr:`.TaskSpec.environment_upload` into the started container, as Harbor does. | ||
|
|
||
| Into ``environment.workdir``, else the image's ``WORKDIR``, else ``/`` (Harbor's | ||
| ``_upload_environment_dir_after_start`` runs ``pwd``). A top-level ``skills/`` stays out, as | ||
| it stays out of a build context. | ||
| """ | ||
| src = task.environment_upload | ||
| if src is None: | ||
| return | ||
| workdir = task.environment.get("workdir") | ||
| if not isinstance(workdir, str) or not workdir: | ||
| workdir = (docker(["inspect", "-f", "{{.Config.WorkingDir}}", container], log).stdout or "").strip() or "/" | ||
| with tempfile.TemporaryDirectory(prefix="dokimi-upload-") as tmp: | ||
| if task.skills_dir is not None: | ||
| staged = Path(tmp) / "environment" | ||
| shutil.copytree(src, staged, symlinks=True, ignore=ignore_top_level_skills(src)) | ||
| src = staged | ||
| docker(["exec", "-u", "root", container, "mkdir", "-p", workdir], log) | ||
| docker(["cp", f"{src}/.", f"{container}:{workdir}"], log) | ||
|
|
||
|
|
||
| def ignore_top_level_skills(env_dir: Path): | ||
| """``shutil.copytree`` ignore hook that drops only ``<env_dir>/skills``.""" | ||
| def ignore(directory: str, names: list[str]) -> list[str]: | ||
| return ["skills"] if Path(directory) == env_dir and "skills" in names else [] | ||
| return ignore |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.