Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
7f37022
feat(provenance): optional version and provenance on tasks and questi…
napetrov Nov 15, 2026
04f112e
Merge branch 'main' of https://github.com/intel/dokimi into pr800-r6
napetrov Nov 16, 2026
c100397
docs(provenance): put the provenance key under its [metadata] header
napetrov Nov 16, 2026
5444866
docs(provenance): say repo validate, not repo check
napetrov Nov 16, 2026
9d2d2af
fix(provenance): refuse a bad block without a type error, and read it…
napetrov Nov 16, 2026
d22f617
Merge branch 'main' of https://github.com/intel/dokimi into pr800-r6
napetrov Nov 16, 2026
b2a07f3
Merge branch 'main' of https://github.com/intel/dokimi into pr800-r6
napetrov Nov 16, 2026
f01b1b8
Address review: questions.v2 accepts old version and provenance values
napetrov Nov 16, 2026
b75ef4e
Address review: report reads a task's status from result.json
napetrov Nov 16, 2026
9f3a95c
Address review: questions.v2 says a malformed value is kept as written
napetrov Nov 16, 2026
5af24e5
Address review: the report test checks the rewritten copy, not a corr…
napetrov Nov 16, 2026
afcde91
Keep the result record unchanged for a task without provenance
napetrov Nov 16, 2026
6df5db9
Merge branch 'main' of https://github.com/intel/dokimi into feat/task…
napetrov Nov 17, 2026
a682ece
Address review: golden result-key claims are already fixed in afcde91345
napetrov Nov 17, 2026
dc380a9
Merge remote-tracking branch 'intel/main' into feat/task-question-pro…
napetrov Nov 17, 2026
e5ee638
Address review: name the question generator, not the command, in prov…
napetrov Nov 17, 2026
f418483
Address review: name the swebench exception to metadata.version
napetrov Nov 17, 2026
a6289fb
Address review: a report row whose cells disagree on status says mixed
napetrov Nov 17, 2026
3b9644d
Address review: regression test for the host writing task_status
napetrov Nov 17, 2026
5297ec8
test: regenerate the task-runs golden for the Status column after the…
napetrov Nov 17, 2026
23ed869
Address review: keep the task_status test in its own file
napetrov Nov 17, 2026
d39789a
Address review: lint-clean task_status test
napetrov Nov 17, 2026
dc2c209
docs: name the generated stamps the code writes
napetrov Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Added

- **Optional `version` and `provenance` on repository tasks (`task.json`), executable tasks (`metadata` in `task.toml`) and question-set items.** `provenance` is `{source, reviewed_by, status}`: `source` is `hand-written`, `generated:<tool>@<version>` or `imported:<format>`, and `status` is `candidate`, `validated` or `frozen`. Every key is optional and an absent one is `unknown`, never an error. `eval tasks generate` fills `source` as `generated:dokimi-tasks-generate@<dokimi version>`, and question generation fills `generated:dokimi-questions-persona@…` or, for chunk-grounded questions, `generated:dokimi-questions-chunk@…`. `repo report` shows each task's status in a `status` column (and in its JSON rows), and the executable task-run report shows it in a `Status` column. A malformed value fails `repo validate` for `task.json`, and is a warning for a `task.toml` and for a question-set item (the set still loads, and the value is kept as written). See `docs/repo-evaluation-contract.md` §3 and `docs/contributing-terminal-bench-task.md` §3 and §9.
10 changes: 10 additions & 0 deletions dokimi/docs/contributing-terminal-bench-task.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ Field by field, with the consumer that actually reads it:
| `metadata.difficulty` | `eval tasks list`, the task-runs report | `easy` / `medium` / `hard`; drives the difficulty rollup in the Markdown report. Missing renders as `?`. |
| `metadata.level` | [`difficulty-rubric.md`](./difficulty-rubric.md) | Workflow depth: `triage`, `diagnosis`, `end_to_end`. Optional but recommended for skill evaluation. |
| `metadata.difficulty_explanation`, `category`, `author_*` | humans | Provenance and review context; carried through in the run artifact. |
| `metadata.version`, `metadata.provenance` | `load_task` (a malformed value is a warning; the task still loads), the task-runs report | Optional. `version` is the task's revision; it is not the top-level `version`, which is the format version. A `swebench` task's `version` is the upstream package version (e.g. `"3.0"` for django), kept as written, not a revision. `provenance` is `{source, reviewed_by, status}` as in the [repository contract](./repo-evaluation-contract.md#3-taskjson): `status` is `candidate`, `validated` or `frozen`. The task-runs report shows each task's status in a `Status` column; an absent one is `unknown`. |
| `agent.timeout_sec` | `TaskSpec.timeout_sec`, the solver harnesses | Wall-clock budget for the agent's attempt. Typically 300. |
| `verifier.timeout_sec` | Harbor, human reviewers | Verifier budget. Keep it small (≤ 60s where possible) so CI stays fast. |
| `environment.build_timeout_sec` | `docker-*` harnesses | Image-build budget; used instead of the default when set (oneAPI/pip-heavy images need > 10 min). |
Expand Down Expand Up @@ -270,6 +271,15 @@ results are never reported as official upstream benchmark numbers. The content
repository is the right place for that record, kept next to the tasks it
describes; keep it current in the same change that adds the task.

Record the status in the task too, as a `provenance` key inside the existing `[metadata]` table (see §3). Keep the table header on its own line:

```toml
[metadata]
provenance = { source = "hand-written", reviewed_by = "...", status = "candidate" }
```

`candidate` is a task nobody has checked yet, `validated` one a reviewer has checked, `frozen` one that must not change. Bump `metadata.version` when you change a task that results were already reported on (a repo `dokimi:typecheck` task has no revision `version`: there it is the pinned mypy version, see the [repository contract](./repo-evaluation-contract.md#3-taskjson)). A task without these keys reads as `unknown`; that is not an error.

## 10 — CI

CI in this repository proves the *framework* against external content: the
Expand Down
2 changes: 2 additions & 0 deletions dokimi/docs/methodology.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,8 @@ Question records preserve source fields such as `source_type`, `question_source`

### Question record format

Each question may carry `version` (the author's revision, a string or an integer) and `provenance` (`source`, `reviewed_by`, `status` as `candidate`, `validated` or `frozen`). Every key is optional and an absent one is unknown. `eval questions generate` writes `source` as `generated:dokimi-questions-persona@<dokimi version>`, and the pipeline's chunk-grounded questions carry `generated:dokimi-questions-chunk@<dokimi version>`; a hand-written set is `hand-written` by convention. Both shapes are checked when the set is read: a malformed one is a warning, never a refusal, so files written before the key existed still load.

A question set is a JSON object with a `questions` list. The framework writes it as `questions.v2` (`dokimi/schemas/questions.v2.json`, where every field is documented); `questions.v1` files are still read.

- **v2 (written):** each record needs `id` and a non-empty `question`. A v2 record may not also carry `text`, so no record holds two question texts that readers could disagree on.
Expand Down
2 changes: 2 additions & 0 deletions dokimi/docs/repo-evaluation-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,8 @@ belongs to the repository's grader and is passed through unread, except `grader`
| `egress` | no | framework | extra hosts (DNS names, optional `:port`; `*` stands for one whole label and the last two labels are literal, so `*.pythonhosted.org` passes and `*` or `*.org` does not) this task's agent and entry points reach through the proxy (§2), for example `["pypi.org", "files.pythonhosted.org"]` |
| `expected_skill` | no | framework | the name (directory name) of the skill the agent should use; a `skill-catalog:` arm must install it, or the arm is refused before anything runs. On a `guidance:as-skill` catalog arm the generated `<repo>-guidance` skill is refused too (its usage is `guidance_skill_loaded`). `result.json` `skills` records whether the agent used it and whether it used it first |
| `useful` | no | framework | the definition of the run's first useful result, reported as `first_useful` in `result.json` (§4.1): `{"command": "<regex>"}` for a task whose grade runs the target tests, or `{"edit": "<glob>"}` (or a list of globs) for one that grades an edit to a target file. Omit it when no event is the useful result. A malformed value is refused by `repo validate`, `tasks check` and `repo run` |
| `version` | no | framework (reporting) | the task's revision, a non-empty string or an integer; absent is unknown. The grader does not read it, except `dokimi:typecheck`, which reads `version` as the image's mypy version (its row in the grader table) and so must not carry a revision there |
| `provenance` | no | framework (reporting) | `source` (`hand-written`, `generated:<tool>@<version>` or `imported:<format>`; `eval tasks generate` writes `generated:dokimi-tasks-generate@<dokimi version>`), `reviewed_by` (a non-empty string) and `status` (`candidate`, `validated` or `frozen`); every key optional. `repo report` shows each task's `status` in a column, `unknown` when absent. A malformed block fails `repo validate` (`dokimi/provenance.py`) |
| anything else | — | repository | for example `hidden_files`, `targets`, `regression_targets`, `manifest` |

`setup_patch` (a single string) is accepted as a synonym for a one-element
Expand Down
12 changes: 12 additions & 0 deletions dokimi/dokimi/harnesses/tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@

import yaml

from dokimi.provenance import ProvenanceError, parse_provenance, parse_version

from . import tbench_v1

# Executable tasks live in a separate evaluation-content checkout, so there is
Expand Down Expand Up @@ -340,6 +342,15 @@ def _read_tables(path: Path, fmt: str) -> tuple[dict[str, Any], str]:
return read_task_config(path), instruction_path.read_text(encoding="utf-8")


def _check_provenance(spec: TaskSpec) -> None:
"""Warn, never fail, on a malformed metadata ``version`` or ``provenance``: a repository may already use the key."""
try:
parse_version(spec.metadata.get("version"))
parse_provenance(spec.metadata.get("provenance"))
except ProvenanceError as exc:
_warn_once(f"task {spec.path}: metadata {exc}; its status reads as unknown")


def load_task(path_or_name: str | Path, root: Path | None = None) -> TaskSpec:
"""Load one task by directory path, or by task name relative to *root*.

Expand Down Expand Up @@ -390,6 +401,7 @@ def load_task(path_or_name: str | Path, root: Path | None = None) -> TaskSpec:
format=fmt,
schema_version=str(version) if version is not None else None,
)
_check_provenance(spec)
drift = legacy_drift(spec)
if drift:
_warn_once(
Expand Down
81 changes: 81 additions & 0 deletions dokimi/dokimi/provenance.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Copyright (C) 2026 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

"""The optional ``version`` and ``provenance`` of a task and of a question-set item.

``provenance`` says where the record came from and how far it has been checked:
``source`` is ``hand-written``, ``generated:<tool>@<dokimi version>`` or ``imported:<format>``;
``reviewed_by`` names the reviewer; ``status`` is ``candidate``, ``validated`` or ``frozen``.
Every key is optional and an absent one is *unknown*, never an error. ``questions.v2`` declares
the same shape, but accepts any value: a malformed one is a warning where it is read, not a refusal.
"""

from __future__ import annotations

import re
from collections.abc import Mapping
from typing import Any

from dokimi._version import __version__

UNKNOWN = "unknown"
STATUSES = ("candidate", "validated", "frozen")
KEYS = ("source", "reviewed_by", "status")
SOURCE_RE = re.compile(r"^(hand-written|generated:[A-Za-z0-9_.-]+@[A-Za-z0-9_.+-]+|imported:[A-Za-z0-9_.-]+)$")


class ProvenanceError(ValueError):
"""A ``version`` or ``provenance`` value of the wrong shape."""


def parse_version(raw: Any) -> str | int | None:
"""The ``version`` value (a non-empty string or an integer), or None when absent; :class:`ProvenanceError`
otherwise. Integers are allowed because task.json is open and some repositories already write ``"version": 2``."""
if raw is None:
return None
if isinstance(raw, int) and not isinstance(raw, bool):
return raw
if isinstance(raw, str) and raw.strip():
return raw
raise ProvenanceError(f"version must be a non-empty string or an integer, got {raw!r}")


def parse_provenance(raw: Any) -> dict[str, Any] | None:
"""The ``provenance`` block, checked, or None when absent; :class:`ProvenanceError` on a bad shape."""
if raw is None:
return None
if not isinstance(raw, dict):
raise ProvenanceError(f"provenance must be an object with optional keys {', '.join(KEYS)}")
unknown = sorted(set(raw) - set(KEYS), key=str)
if unknown:
raise ProvenanceError(f"provenance: unknown key(s) {', '.join(map(str, unknown))} "
f"(known: {', '.join(KEYS)})")
for key in KEYS:
if key in raw and raw[key] is None:
raise ProvenanceError(f"provenance.{key} is null: omit the key instead")
source = raw.get("source")
if source is not None and not (isinstance(source, str) and SOURCE_RE.fullmatch(source)):
raise ProvenanceError(f"provenance.source must be hand-written, generated:<tool>@<version> or "
f"imported:<format>, got {source!r}")
reviewer = raw.get("reviewed_by")
if reviewer is not None and not (isinstance(reviewer, str) and reviewer.strip()):
raise ProvenanceError(f"provenance.reviewed_by must be a non-empty string, got {reviewer!r}")
status = raw.get("status")
if status is not None and status not in STATUSES:
raise ProvenanceError(f"provenance.status must be one of {', '.join(STATUSES)}, got {status!r}")
return dict(raw)


def status_of(record: Mapping[str, Any]) -> str:
"""The record's ``provenance.status``, or :data:`UNKNOWN` when it has none, an invalid one or an invalid block."""
try:
block = parse_provenance(record.get("provenance"))
except ProvenanceError:
return UNKNOWN
status = block.get("status") if block else None
return status if status in STATUSES else UNKNOWN


def generated_by(tool: str) -> dict[str, Any]:
"""The ``provenance`` a generator stamps on what it writes: ``generated:dokimi-<tool>@<dokimi version>``."""
return {"source": f"generated:dokimi-{tool}@{__version__}"}
19 changes: 19 additions & 0 deletions dokimi/dokimi/questions/artifact.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@
schema_errors,
schema_version_for,
)
from dokimi.provenance import ProvenanceError, parse_provenance, parse_version
from dokimi.questions.normalizer import canonical_question, record_id
from dokimi.utils import question_id, read_json_input

Expand Down Expand Up @@ -213,6 +214,20 @@ def _explain(path: Path, data: dict, errors: list[str]) -> str:
f"dokimi/schemas/{SCHEMA_VERSIONS[KIND]}.json, or drop the record.")


def _warn_provenance(path: Path, records: list) -> None:
"""A malformed ``version`` or ``provenance`` on a record is a WARNING, never a refusal: the schema accepts any
value so that files written before the key existed still load, and the value is kept as written."""
for i, q in enumerate(records):
if not isinstance(q, dict):
continue
for key, check in (("version", parse_version), ("provenance", parse_provenance)):
try:
check(q.get(key))
except ProvenanceError as exc:
qid = question_id(q)
logger.warning("%s: question %s: %s; kept as written", path, qid if qid is not None else i, exc)


def load_questions(path: Path | str, *, validate: bool = True, flag: str | None = None) -> QuestionSet:
"""Read a question set in any shape this framework has read, and return it as questions.v2.

Expand All @@ -226,6 +241,9 @@ def load_questions(path: Path | str, *, validate: bool = True, flag: str | None
must a ``schema_version`` this reader does not know. A set that is not a list of questions at
all (e.g. ``{"questions": {...}}``) is refused either way.

A malformed ``version`` or ``provenance`` on a record is logged at WARNING and kept as written; it does not
refuse the set.

With ``flag`` (the option the file was given to) the file is read by
:func:`dokimi.utils.read_json_input`: an unreadable, non-UTF-8 or non-JSON file raises
:class:`~dokimi.utils.InputError` ``<flag> <path>: <reason>``. Without it the read is bare.
Expand All @@ -250,6 +268,7 @@ def load_questions(path: Path | str, *, validate: bool = True, flag: str | None
for qid in (question_id(q),) if isinstance(qid, (str, int))}
read = [_read_record(q, i, seen, taken) if isinstance(q, dict) else q
for i, q in enumerate(records)]
_warn_provenance(path, read)
artifact = upgrade_questions({**data, "questions": read})

source_version: str | None
Expand Down
12 changes: 10 additions & 2 deletions dokimi/dokimi/questions/llm_gen.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,14 @@
from dokimi.llm import LLM, ChatModel, Invoker, check_caller_provider
from dokimi.llm import LANGCHAIN_AVAILABLE
from dokimi.defaults import DEFAULT_GENERATOR_MODEL, DEFAULT_GENERATOR_PROVIDER
from dokimi.provenance import generated_by


def _stamped(questions: List[Dict[str, Any]], tool: str) -> List[Dict[str, Any]]:
"""Each generated question with the ``provenance`` of the generator ``tool`` (a set by hand is kept)."""
for q in questions:
q.setdefault("provenance", generated_by(tool))
return questions


QUESTION_GENERATION_PROMPT = """You are generating technical questions for documentation quality evaluation.
Expand Down Expand Up @@ -160,7 +168,7 @@ def generate_questions(
all_questions.append(q)

logger.info(f"Generated {len(all_questions)} total questions")
return all_questions
return _stamped(all_questions, "questions-persona")

def generate_hybrid(
self,
Expand Down Expand Up @@ -231,7 +239,7 @@ def generate_hybrid(
f"Hybrid result: {len(persona_qs)} persona + {len(chunk_qs)} chunk "
f"= {len(all_questions)} total"
)
return all_questions
return _stamped(all_questions, "questions-chunk")

def _generate_for_persona(
self,
Expand Down
8 changes: 5 additions & 3 deletions dokimi/dokimi/report/task_runs_report.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
from dokimi.eval.binary_stats import PASS_RATE_CI_CONFIDENCE
from dokimi.harnesses.runner import is_errored
from dokimi.measure import is_number
from dokimi.provenance import UNKNOWN, status_of
from dokimi.report import fmt


Expand Down Expand Up @@ -176,6 +177,7 @@ def render_task_runs_report(data: dict[str, Any]) -> str:
t["name"]: t.get("metadata", {}).get("difficulty", "?")
for t in data.get("tasks", [])
}
status_of_task = {t["name"]: status_of(t.get("metadata") or {}) for t in data.get("tasks", [])}
rows = data.get("results", [])
baseline = data.get("baseline_harness")
# Fall back to harnesses observed in the rows if the summary is empty.
Expand Down Expand Up @@ -310,18 +312,18 @@ def render_task_runs_report(data: dict[str, Any]) -> str:
# ── per-task pass + cost (one column pair per harness) ──
lines.append("## Per-task pass + cost")
lines.append("")
lines.append("| Task | Diff | " + " | ".join(
lines.append("| Task | Diff | Status | " + " | ".join(
f"`{h}` pass | `{h}` $" for h in harnesses
) + " |")
lines.append("| --- | :---: |" + "|".join(" :---: | ---: " for _ in harnesses) + "|")
lines.append("| --- | :---: | :---: |" + "|".join(" :---: | ---: " for _ in harnesses) + "|")
for t in tasks:
cols = []
for h in harnesses:
c = cell[(t, h)]
err_s = f" (+{c['errored']} err)" if c["errored"] else ""
pass_s = f"{c['passed']}/{c['n']}" if c["n"] else "—"
cols.append(f"{pass_s}{err_s} | {fmt.usd(c['cost'])}")
lines.append(f"| {t} | {diff_of.get(t, '?')} | " + " | ".join(cols) + " |")
lines.append(f"| {t} | {diff_of.get(t, '?')} | {status_of_task.get(t, UNKNOWN)} | " + " | ".join(cols) + " |")
lines.append("")

# ── per-cell answer detail ──
Expand Down
3 changes: 2 additions & 1 deletion dokimi/dokimi/repos/graph_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
from typing import Any, Callable

from dokimi.atomic_write import write_text_atomic
from dokimi.provenance import generated_by
from dokimi.repos.code_index import CodeIndex
from dokimi.repos.code_index.symbols import descriptor_names, descriptors
from dokimi.repos.environment.files import is_generated, is_test_file
Expand Down Expand Up @@ -469,7 +470,7 @@ def _task(c: Candidate, granularity: str, opts: Options,
"id": task_id(c), "family": FAMILY, "base": opts.base, "prompt": prompt, "grader": GRADER,
"gold": GOLD, "score": {"metric": "f1", "max": 1.0}, "max_turns": opts.max_turns,
"question": c.kind, "answer_key": key, "symbols": list(c.symbols), "ambiguous_name": c.ambiguous,
"code_index": code_index,
"code_index": code_index, "provenance": generated_by("tasks-generate"),
}
rule = {"files": "files with a reference to the symbol, outside the files that define it",
"functions": "functions whose range holds a reference to the symbol, outside the files that define it",
Expand Down
Loading
Loading