Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Fixed

- **`docker-*` harnesses run a Harbor task on its prebuilt `[environment] docker_image`** — a task
that sets one and ships no `environment/Dockerfile` (10 of the 17 uxlfoundation/skills tasks:
every answer-quality task and `sycl-device-discovery`) failed its cell with `docker build`'s
"open Dockerfile: no such file". As in Harbor 0.24, the image is used whenever it is set (also
beside a Dockerfile), pulled only when the daemon lacks it, within `build_timeout_sec`, and the
loose files of `environment/` are copied into the workdir after the start. The container runs
in the task's `environment.workdir` (`docker run -w`), so `solve.sh` and `test.sh` find relative
paths. A `docker_image` that is not a string (`docker_image = 5`) now fails the task's load, as
in Harbor. `--network build=none` does not stop the pull, as it does not stop a
build's `FROM`. `metrics.image` records the image a cell ran on: its id and repo digests for a
prebuilt one. A task without `docker_image` builds as before. A task that sets `docker_image`
beside `environment/Dockerfile` (as the old `task.toml` template in
`contributing-terminal-bench-task.md` did, with a local `intel-hpc-bench/...` tag) now runs on
that image and no longer builds the Dockerfile; `run.log` says so, and a tag no registry has
errors the cell with `docker pull failed`. Remove `docker_image` to keep building the
Dockerfile. See `docs/coding-harnesses.md`.
39 changes: 37 additions & 2 deletions dokimi/docs/coding-harnesses.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ real files in each format, with their sources in its `MANIFEST.yaml`.

| Format | Recognised by | Kind | Consumed by |
|---|---|---|---|
| `harbor` | directory with `task.toml` | executable | Harbor-backed harnesses (unless Harbor would refuse it), `docker-*` (with `environment/Dockerfile`) |
| `harbor` | directory with `task.toml` | executable | Harbor-backed harnesses (unless Harbor would refuse it), `docker-*` (with `environment/Dockerfile` or a [prebuilt `docker_image`](#prebuilt-images-environment-docker_image)) |
| `task-md` | directory with a BenchFlow `task.md` | executable | `benchflow*`, `docker-*` (with `environment/Dockerfile`) |
| `terminal-bench-v1` | directory with `task.yaml` + `run-tests.sh` / `Dockerfile` / `solution.sh` | executable | `docker-*` run it as it is; `harbor`, `benchflow` through a cached Harbor export (not a `solution.yaml` task) |
| `swe-bench` | JSON/JSONL rows with `instance_id`, `repo`, `base_commit`, `problem_statement` (`FAIL_TO_PASS`) | executable | `docker-*` via `tasks run --swe-bench` (Python SWE-bench repos) |
Expand Down Expand Up @@ -241,6 +241,41 @@ split) rather than a converter here. An export renames `oracle/` and
as it works in `/app`, for the agent arms); the Harbor-backed harnesses refuse
it.

### Prebuilt images: `[environment] docker_image`

A Harbor task whose `task.toml` sets `[environment] docker_image` runs on that
image, as `harbor run` does, and a docker-* run starts from that image instead of
building: Harbor 0.24 uses it whenever it is set, also beside an
`environment/Dockerfile` (only Harbor's `--force-build` builds then;
`should_use_prebuilt_docker_image` in `harbor/environments/definition.py`;
`run.log` names the Dockerfile not built: remove `docker_image` to build it). A
task without one builds `dokimi/<task>:latest` from `environment/`, as before; so
does a staged SWE-bench task, whose Dockerfile starts `FROM` its image. Other
formats do not read the key.

- The image is pulled only when the daemon does not have it, as Harbor's
`docker compose up` does, within the task's `build_timeout_sec`; a pull that
runs out of time errors the cell, like a build. A value that is not an image
reference (it starts with `-`, or holds a space) is refused before any docker
call: docker would read it as a flag. A `docker_image` that is not a string
(`docker_image = 5`) fails the task's load, as Harbor's `EnvironmentConfig` refuses
it; an empty string is no image.
- The pull is the daemon's, like the `FROM` image of a `docker build`:
`--network build=none` takes the network from a build's `RUN` steps and does
not stop it. To run offline, have the image on the daemon first.
- The container runs in `environment.workdir` when the task declares one (`docker run -w`), as
Harbor runs every exec there, so the oracle and the verifier find relative paths; without one,
in the image's `WORKDIR`. A task built from its Dockerfile keeps the image's `WORKDIR` even
when it declares one (Harbor overrides it there too).
- After the container starts, the files of a non-empty `environment/` without a
`Dockerfile` or `docker-compose.yaml` are copied into `environment.workdir`,
else the image's `WORKDIR`, else `/` (Harbor's
`should_upload_environment_dir`). A top-level
`environment/skills` is left out, as it is left out of a build context.
- `metrics.image` says which image ran: `{"source": "docker_image", "ref",
"pulled", "id", "repo_digests"}` for a prebuilt one (the id and digests of what
ran, since a tag moves), `{"source": "build", "ref"}` for a built one.

## Supported Harnesses

Harbor-backed (delegate to an installed `harbor`):
Expand Down Expand Up @@ -620,7 +655,7 @@ be checked for the same network before their verdicts are compared:

| Phase | Default | What runs |
|---|---|---|
| `build` | `public` (host network, host proxy as build args) | `docker build` |
| `build` | `public` (host network, host proxy as build args) | `docker build` (a [prebuilt image](#prebuilt-images-environment-docker_image)'s pull ignores it) |
| `solve` | `none` (an in-container agent still reaches its LLM through the sidecar) | the oracle, or the agent |
| `verify` | `none` | `tests/test.sh` |

Expand Down
3 changes: 1 addition & 2 deletions dokimi/docs/contributing-terminal-bench-task.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,6 @@ timeout_sec = 300.0

[environment]
build_timeout_sec = 300.0
docker_image = "intel-hpc-bench/<task-name>:latest"
cpus = 4
memory_mb = 2048
storage_mb = 5120
Expand All @@ -115,7 +114,7 @@ Field by field, with the consumer that actually reads it:
| `agent.timeout_sec` | `TaskSpec.timeout_sec`, the solver harnesses | Wall-clock budget for the agent's attempt. Typically 300. |
| `verifier.timeout_sec` | Harbor, human reviewers | Verifier budget. Keep it small (≤ 60s where possible) so CI stays fast. |
| `environment.build_timeout_sec` | `docker-*` harnesses | Image-build budget; used instead of the default when set (oneAPI/pip-heavy images need > 10 min). |
| `environment.docker_image` | Harbor | Image tag Harbor expects to exist (`harbor run` does **not** build it for you). The in-repo `docker-*` harnesses build and tag `dokimi/<task-name>:latest` themselves. |
| `environment.docker_image` | Harbor, `docker-*` harnesses | A prebuilt image both run instead of building `environment/Dockerfile`, pulled when the daemon lacks it, so leave it out of a task that ships a Dockerfile. Without it, the `docker-*` harnesses build and tag `dokimi/<task-name>:latest`. See [Prebuilt images](./coding-harnesses.md#prebuilt-images-environment-docker_image). |
| `environment.cpus`, `memory_mb` | `docker-*` harnesses | Container limits (`--cpus`, `--memory`); default to 4 CPUs / 2048 MB when absent. A malformed value falls back to the default rather than failing the run. |
| `environment.network_mode` | `docker-*` harnesses, Harbor | `"no-network"` (absent means the same), `"allowlist"` with `allowed_hosts`, or `"public"` (NAT'd internet, bridge). It applies to the **verifier** and `docker-oracle`'s solve, not to an agent: an agent solves on an internal network whose only way out is the egress proxy (see `environment.egress`), unless `[agent] network_mode` says otherwise. `[agent]` and `[verifier]` take the same keys for their phase. Any other value, or `allowed_hosts` without `"allowlist"`, errors the run. The image *build* gets host network unless the operator passes `--network build=none`. See [Network policy](./coding-harnesses.md#network-policy). Prefer offline. |
| `environment.egress` | `docker-claude` | Hosts the agent's solve may reach through the egress proxy, e.g. `["pypi.org", "files.pythonhosted.org"]`. A bare host means `host:443`; `host:port` is accepted. No wildcards (`*.pythonhosted.org`), commas or leading/trailing dots: list each host. A plain-HTTP host is reached directly, so it must be in the operator's `NO_PROXY`. |
Expand Down
38 changes: 17 additions & 21 deletions dokimi/dokimi/harnesses/docker_solver.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@
resolve_recorded, skill_needs, start_network, redact as _redact)
from .skill_treatment import Treatment, build_plugin, check_skill, skill_listing, treatment_notice
from .solver_stream import record_solver, run_solver
from .task_image import ignore_top_level_skills, ready_image, upload_environment, workdir_args
from .task_env import _exec_env, unresolved_env # re-exported
from .tbench_v1 import reward as _pytest_reward
from .tasks import ( # noqa: F401 (timeouts re-exported)
Expand Down Expand Up @@ -208,7 +209,6 @@ def run(
log_path = output_dir / "run.log"
log = log_path.open("w", encoding="utf-8")
usage: Optional[UsageRecord] = None
image = f"dokimi/{task.name}:latest"
container = f"dokimi-run-{task.name}-{os.getpid()}-{os.urandom(3).hex()}" # two cells in one process
solve_timeout = timeout_sec or task.timeout_sec or _DEFAULT_SOLVE_TIMEOUT
start = time.time()
Expand All @@ -217,21 +217,14 @@ def run(
network = self._network_metrics(plan)
reward: Optional[float] = None
sandbox: Optional[ContainerSandbox] = None
image_record: dict[str, Any] = {}
try:
try:
self._build_image(task, image, log, plan.build)
except subprocess.TimeoutExpired as exc:
# The image never got built, so no agent ran and there is no
# verdict: raise so the cell is errored like a build that fails
# fast. Only a solve or verify timeout below is a reward-0 failure.
log.write(f"\nBUILD TIMEOUT: {_describe_timeout(exc)}\n")
raise RuntimeError(
f"docker build timed out after {exc.timeout}s; see run.log"
) from exc
image, image_record = self._ready_image(task, log, plan.build)
sandbox = self._open_sandbox(plan, container, log)
try:
bridged = self._start_container(task, image, container, log, sandbox, plan)
try:
upload_environment(task, container, log, self._docker)
Comment thread
vlad-perevezentsev marked this conversation as resolved.
Resolved
if self.mode == "oracle":
self._solve_oracle(task, container, log, plan.solve, timeout=solve_timeout)
elif self.mode == "claude":
Expand Down Expand Up @@ -270,7 +263,7 @@ def run(
(output_dir / "network.json").write_text(json.dumps(network, indent=1), encoding="utf-8")

elapsed = time.time() - start
metrics: dict[str, Any] = {"dry_run": False, "mode": self.mode, "network": network}
metrics: dict[str, Any] = {"dry_run": False, "mode": self.mode, "network": network, "image": image_record}
passed = reward is not None and reward >= 1.0
if usage is not None:
metrics.update(usage.as_metrics_dict())
Expand All @@ -297,6 +290,16 @@ def _docker(self, args: list[str], log, **kwargs: Any) -> process.Completed:

# ── pipeline steps ──────────────────────────────────────────────────────

def _ready_image(self, task: TaskSpec, log, net: PhasePolicy) -> tuple[str, dict[str, Any]]:
"""The task image, built or pulled (:mod:`.task_image`). Running out of time errors the cell:
no agent ran, so there is no verdict (a solve or verify timeout is a reward-0 failure)."""
step = "pull" if task.docker_image else "build"
try:
return ready_image(task, log, self._docker, lambda tag: self._build_image(task, tag, log, net))
except subprocess.TimeoutExpired as exc:
log.write(f"\n{step.upper()} TIMEOUT: {_describe_timeout(exc)}\n")
raise RuntimeError(f"docker {step} timed out after {exc.timeout}s; see run.log") from exc

def _build_image(self, task: TaskSpec, image: str, log, net: Optional[PhasePolicy] = None) -> None:
"""Build the task image under *net*: ``public`` passes the host proxy through.

Expand All @@ -319,7 +322,7 @@ def _build_image(self, task: TaskSpec, image: str, log, net: Optional[PhasePolic
# following them would pull host files into the image, and a
# dangling one would fail the copy.
shutil.copytree(env_dir, context, symlinks=True,
ignore=_ignore_top_level_skills(env_dir))
ignore=ignore_top_level_skills(env_dir))
# Left empty rather than absent: some Dockerfiles `COPY skills
# ...` (radar-vital-signs copies it into seven agent paths), and
# a missing source fails the build in every arm.
Expand Down Expand Up @@ -356,7 +359,7 @@ def _start_container(self, task: TaskSpec, image: str, container: str, log,
sandbox_flags = sandbox.run_args() if sandbox else []
self._docker(docker_cmd.run_spec(
task_container_policy(network=network, cpus=cpus, memory=memory),
[["-d"], ["--name", container], "init", "cap_drop", "security", "pids", "cap_add",
[["-d"], ["--name", container], workdir_args(task), "init", "cap_drop", "security", "pids", "cap_add",
"network", sandbox_flags, docker_cmd.env_args(task.container_env), "resource"],
image, ["sleep", str(lifetime)]), log)
return enter_phase(plan.solve if inside else DEFAULTS["solve"], container, self._docker, log,
Expand Down Expand Up @@ -835,13 +838,6 @@ def _user(step: ContainerStep) -> list[str]:
return ["-u", step.user] if step.user else []


def _ignore_top_level_skills(env_dir: Path):
"""``shutil.copytree`` ignore hook that drops only ``<env_dir>/skills``."""
def ignore(directory: str, names: list[str]) -> list[str]:
return ["skills"] if Path(directory) == env_dir and "skills" in names else []
return ignore


def _describe_timeout(exc: subprocess.TimeoutExpired) -> str:
"""``TimeoutExpired`` for run.log: its ``str()`` holds the raw argv."""
cmd = exc.cmd if isinstance(exc.cmd, (list, tuple)) else [str(exc.cmd)]
Expand Down
94 changes: 94 additions & 0 deletions dokimi/dokimi/harnesses/task_image.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Copyright (C) 2026 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

"""The image a ``docker-*`` task container starts from: built from the task, or Harbor's prebuilt one.

A Harbor task with ``[environment] docker_image`` (:attr:`.TaskSpec.docker_image`) runs on that
image, as ``harbor run`` does: the image is pulled only when the daemon lacks it (what Harbor's
``docker compose up`` does), and once the container runs, the task's loose ``environment/`` files
are copied into its workdir (:attr:`.TaskSpec.environment_upload`). Any other task builds
``dokimi/<task>:latest``. ``metrics.image`` records which ran, and a pulled image's id and digests.

The pull is the daemon's, like a ``docker build``'s base image: ``--network build=none``, which
takes the network away from a build's ``RUN`` steps, does not stop it.
"""

from __future__ import annotations

import json
import re
import shutil
import tempfile
from pathlib import Path
from typing import Any, Callable

from dokimi.docker_cmd import Docker

from .tasks import TaskSpec, _task_build_timeout

# One JSON object per image; RepoDigests is empty for an image that came from no registry.
_RECORD_FORMAT = '{"id": {{json .Id}}, "repo_digests": {{json .RepoDigests}}}'
# A reference starts with a letter or digit (distribution/reference); "-..." would be a docker flag.
_REFERENCE_RE = re.compile(r"[A-Za-z0-9]\S*")


def ready_image(task: TaskSpec, log, docker: Docker, build: Callable[[str], None]) -> tuple[str, dict[str, Any]]:
"""The image to start the task container from, and its ``metrics.image`` record.

*build* builds the task's Dockerfile under the tag it is given. The pull gets the task's
``build_timeout_sec``: running out raises :class:`subprocess.TimeoutExpired`, as a build does.
"""
image = task.docker_image
if image is None:
tag = f"dokimi/{task.name}:latest"
build(tag)
return tag, {"source": "build", "ref": tag}
if not _REFERENCE_RE.fullmatch(image):
raise ValueError(f"{task.name}: [environment] docker_image {image!r} is not a docker image reference")
if (task.build_context / task.dockerfile).is_file():
log.write(f"[image] {task.dockerfile} not built: [environment] docker_image {image} wins, as in Harbor\n")
inspect = ["image", "inspect", "-f", _RECORD_FORMAT, image]
found = docker(inspect, log, check=False)
pulled = found.returncode != 0
if pulled:
docker(["pull", "-q", image], log, timeout=_task_build_timeout(task))
found = docker(inspect, log)
record = json.loads(found.stdout or "")
return record["id"], {"source": "docker_image", "ref": image, "pulled": pulled, **record}


def workdir_args(task: TaskSpec) -> list[str]:
"""``-w`` for a prebuilt-image task's ``environment.workdir``: Harbor runs every exec there
(``DockerEnvironment.exec``: ``cwd or task_env_config.workdir``). Harbor does so for a built task
too; a built task keeps its image's ``WORKDIR`` here on purpose, as on main."""
workdir = task.environment.get("workdir")
return ["-w", workdir] if task.docker_image is not None and isinstance(workdir, str) and workdir else []


def upload_environment(task: TaskSpec, container: str, log, docker: Docker) -> None:
"""Copy :attr:`.TaskSpec.environment_upload` into the started container, as Harbor does.

Into ``environment.workdir``, else the image's ``WORKDIR``, else ``/`` (Harbor's
``_upload_environment_dir_after_start`` runs ``pwd``). A top-level ``skills/`` stays out, as
it stays out of a build context.
"""
src = task.environment_upload
if src is None:
return
workdir = task.environment.get("workdir")
if not isinstance(workdir, str) or not workdir:
workdir = (docker(["inspect", "-f", "{{.Config.WorkingDir}}", container], log).stdout or "").strip() or "/"
with tempfile.TemporaryDirectory(prefix="dokimi-upload-") as tmp:
if task.skills_dir is not None:
staged = Path(tmp) / "environment"
shutil.copytree(src, staged, symlinks=True, ignore=ignore_top_level_skills(src))
src = staged
docker(["exec", "-u", "root", container, "mkdir", "-p", workdir], log)
docker(["cp", f"{src}/.", f"{container}:{workdir}"], log)


def ignore_top_level_skills(env_dir: Path):
"""``shutil.copytree`` ignore hook that drops only ``<env_dir>/skills``."""
def ignore(directory: str, names: list[str]) -> list[str]:
return ["skills"] if Path(directory) == env_dir and "skills" in names else []
return ignore
Loading
Loading