From 7f37022953d79d6a978ffbc2f7ea02ae57173d69 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Sat, 14 Nov 2026 21:53:06 -0800 Subject: [PATCH 01/18] feat(provenance): optional version and provenance on tasks and question-set items A repo task.json and a questions.v2 item may carry `version` and `provenance` ({source, reviewed_by, status}). Absent is unknown, never an error. `eval tasks generate` and the question generator stamp `generated:dokimi-@`. The repo report and the task-runs report show each task's status, so results can be split by it. Signed-off-by: Nikolay Petrov --- ...20000_napetrov_task-question-provenance.md | 8 + .../docs/contributing-terminal-bench-task.md | 3 + dokimi/docs/methodology.md | 2 + dokimi/docs/repo-evaluation-contract.md | 2 + dokimi/dokimi/harnesses/tasks.py | 12 ++ dokimi/dokimi/provenance.py | 78 +++++++++ dokimi/dokimi/questions/llm_gen.py | 12 +- dokimi/dokimi/report/task_runs_report.py | 8 +- dokimi/dokimi/repos/graph_tasks.py | 3 +- dokimi/dokimi/repos/manifest.py | 15 ++ dokimi/dokimi/repos/replay/authoring.py | 2 + dokimi/dokimi/repos/replay/build.py | 2 + dokimi/dokimi/repos/report.py | 19 +- dokimi/dokimi/schemas/questions.v2.json | 26 +++ .../tests/test_eval_tasks_generate_graph.py | 2 + dokimi/tests/test_import_architecture.py | 2 +- dokimi/tests/test_provenance.py | 163 ++++++++++++++++++ dokimi/tests/test_questions_llm_gen.py | 10 ++ dokimi/tests/test_task_runs_report.py | 8 +- 19 files changed, 364 insertions(+), 13 deletions(-) create mode 100644 dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md create mode 100644 dokimi/dokimi/provenance.py create mode 100644 dokimi/tests/test_provenance.py diff --git a/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md b/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md new file mode 100644 index 000000000..2356e491e --- /dev/null +++ b/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md @@ -0,0 +1,8 @@ + + +### Added + +- **Optional `version` and `provenance` on repository tasks (`task.json`), executable tasks (`metadata` in `task.toml`) and question-set items.** `provenance` is `{source, reviewed_by, status}`: `source` is `hand-written`, `generated:@` or `imported:`, and `status` is `candidate`, `validated` or `frozen`. Every key is optional and an absent one is `unknown`, never an error. `eval tasks generate` and `eval questions generate` fill `source` as `generated:dokimi-@`. `repo report` shows each task's status in a `status` column (and in its JSON rows), and the executable task-run report shows it in a `Status` column. A malformed value fails `repo check` for `task.json`, is refused by the `questions.v2` schema, and is a warning for a `task.toml`. See `docs/repo-evaluation-contract.md` §3 and `docs/contributing-terminal-bench-task.md` §3 and §9. diff --git a/dokimi/docs/contributing-terminal-bench-task.md b/dokimi/docs/contributing-terminal-bench-task.md index 46105f72f..ea09c2af2 100644 --- a/dokimi/docs/contributing-terminal-bench-task.md +++ b/dokimi/docs/contributing-terminal-bench-task.md @@ -112,6 +112,7 @@ Field by field, with the consumer that actually reads it: | `metadata.difficulty` | `eval tasks list`, the task-runs report | `easy` / `medium` / `hard`; drives the difficulty rollup in the Markdown report. Missing renders as `?`. | | `metadata.level` | [`difficulty-rubric.md`](./difficulty-rubric.md) | Workflow depth: `triage`, `diagnosis`, `end_to_end`. Optional but recommended for skill evaluation. | | `metadata.difficulty_explanation`, `category`, `author_*` | humans | Provenance and review context; carried through in the run artifact. | +| `metadata.version`, `metadata.provenance` | `load_task` (a malformed value is a warning; the task still loads), the task-runs report | Optional. `version` is the task's revision; it is not the top-level `version`, which is the format version. `provenance` is `{source, reviewed_by, status}` as in the [repository contract](./repo-evaluation-contract.md#3-taskjson): `status` is `candidate`, `validated` or `frozen`. The task-runs report shows each task's status in a `Status` column; an absent one is `unknown`. | | `agent.timeout_sec` | `TaskSpec.timeout_sec`, the solver harnesses | Wall-clock budget for the agent's attempt. Typically 300. | | `verifier.timeout_sec` | Harbor, human reviewers | Verifier budget. Keep it small (≤ 60s where possible) so CI stays fast. | | `environment.build_timeout_sec` | `docker-*` harnesses | Image-build budget; used instead of the default when set (oneAPI/pip-heavy images need > 10 min). | @@ -271,6 +272,8 @@ results are never reported as official upstream benchmark numbers. The content repository is the right place for that record, kept next to the tasks it describes; keep it current in the same change that adds the task. +Record the status in the task too: `[metadata] provenance = { source = "hand-written", reviewed_by = "...", status = "candidate" }` (see §3). `candidate` is a task nobody has checked yet, `validated` one a reviewer has checked, `frozen` one that must not change. Bump `metadata.version` when you change a task that results were already reported on (a repo `dokimi:typecheck` task has no revision `version`: there it is the pinned mypy version, see the [repository contract](./repo-evaluation-contract.md#3-taskjson)). A task without these keys reads as `unknown`; that is not an error. + ## 10 — CI CI in this repository proves the *framework* against external content: the diff --git a/dokimi/docs/methodology.md b/dokimi/docs/methodology.md index 4669fbdf7..5f400d598 100644 --- a/dokimi/docs/methodology.md +++ b/dokimi/docs/methodology.md @@ -103,6 +103,8 @@ Question records preserve source fields such as `source_type`, `question_source` ### Question record format +Each question may carry `version` (the author's revision, a string or an integer) and `provenance` (`source`, `reviewed_by`, `status` as `candidate`, `validated` or `frozen`). Every key is optional and an absent one is unknown. `eval questions generate` writes `source` as `generated:dokimi-questions-generate@`; a hand-written set is `hand-written` by convention. Both shapes are checked by the schema. + A question set is a JSON object with a `questions` list. The framework writes it as `questions.v2` (`dokimi/schemas/questions.v2.json`, where every field is documented); `questions.v1` files are still read. - **v2 (written):** each record needs `id` and a non-empty `question`. A v2 record may not also carry `text`, so no record holds two question texts that readers could disagree on. diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index ccf72154b..79b115e0e 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -333,6 +333,8 @@ belongs to the repository's grader and is passed through unread, except `grader` | `egress` | no | framework | extra hosts (DNS names, optional `:port`; `*` stands for one whole label and the last two labels are literal, so `*.pythonhosted.org` passes and `*` or `*.org` does not) this task's agent and entry points reach through the proxy (§2), for example `["pypi.org", "files.pythonhosted.org"]` | | `expected_skill` | no | framework | the name (directory name) of the skill the agent should use; a `skill-catalog:` arm must install it, or the arm is refused before anything runs. On a `guidance:as-skill` catalog arm the generated `-guidance` skill is refused too (its usage is `guidance_skill_loaded`). `result.json` `skills` records whether the agent used it and whether it used it first | | `useful` | no | framework | the definition of the run's first useful result, reported as `first_useful` in `result.json` (§4.1): `{"command": ""}` for a task whose grade runs the target tests, or `{"edit": ""}` (or a list of globs) for one that grades an edit to a target file. Omit it when no event is the useful result. A malformed value is refused by `repo validate`, `tasks check` and `repo run` | +| `version` | no | framework (reporting) | the task's revision, a non-empty string or an integer; absent is unknown. The grader does not read it, except `dokimi:typecheck`, which reads `version` as the image's mypy version (its row in the grader table) and so must not carry a revision there | +| `provenance` | no | framework (reporting) | `source` (`hand-written`, `generated:@` or `imported:`; `eval tasks generate` writes `generated:dokimi-tasks-generate@`), `reviewed_by` (a non-empty string) and `status` (`candidate`, `validated` or `frozen`); every key optional. `repo report` shows each task's `status` in a column, `unknown` when absent. A malformed block fails `repo check` (`dokimi/provenance.py`) | | anything else | — | repository | for example `hidden_files`, `targets`, `regression_targets`, `manifest` | `setup_patch` (a single string) is accepted as a synonym for a one-element diff --git a/dokimi/dokimi/harnesses/tasks.py b/dokimi/dokimi/harnesses/tasks.py index adecabf5c..8774f43d1 100644 --- a/dokimi/dokimi/harnesses/tasks.py +++ b/dokimi/dokimi/harnesses/tasks.py @@ -41,6 +41,8 @@ import yaml +from dokimi.provenance import ProvenanceError, parse_provenance, parse_version + from . import tbench_v1 # Executable tasks live in a separate evaluation-content checkout, so there is @@ -316,6 +318,15 @@ def _read_tables(path: Path, fmt: str) -> tuple[dict[str, Any], str]: return read_task_config(path), instruction_path.read_text(encoding="utf-8") +def _check_provenance(spec: TaskSpec) -> None: + """Warn, never fail, on a malformed metadata ``version`` or ``provenance``: a repository may already use the key.""" + try: + parse_version(spec.metadata.get("version")) + parse_provenance(spec.metadata.get("provenance")) + except ProvenanceError as exc: + _warn_once(f"task {spec.path}: metadata {exc}; its status reads as unknown") + + def load_task(path_or_name: str | Path, root: Path | None = None) -> TaskSpec: """Load one task by directory path, or by task name relative to *root*. @@ -365,6 +376,7 @@ def load_task(path_or_name: str | Path, root: Path | None = None) -> TaskSpec: format=fmt, schema_version=str(version) if version is not None else None, ) + _check_provenance(spec) drift = legacy_drift(spec) if drift: _warn_once( diff --git a/dokimi/dokimi/provenance.py b/dokimi/dokimi/provenance.py new file mode 100644 index 000000000..9c2a804c6 --- /dev/null +++ b/dokimi/dokimi/provenance.py @@ -0,0 +1,78 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""The optional ``version`` and ``provenance`` of a task and of a question-set item. + +``provenance`` says where the record came from and how far it has been checked: +``source`` is ``hand-written``, ``generated:@`` or ``imported:``; +``reviewed_by`` names the reviewer; ``status`` is ``candidate``, ``validated`` or ``frozen``. +Every key is optional and an absent one is *unknown*, never an error. ``questions.v2`` declares +the same shape in ``$defs.provenance``; ``tests/test_provenance.py`` keeps the two in agreement. +""" + +from __future__ import annotations + +import re +from collections.abc import Mapping +from typing import Any + +from dokimi._version import __version__ + +UNKNOWN = "unknown" +STATUSES = ("candidate", "validated", "frozen") +KEYS = ("source", "reviewed_by", "status") +SOURCE_RE = re.compile(r"^(hand-written|generated:[A-Za-z0-9_.-]+@[A-Za-z0-9_.+-]+|imported:[A-Za-z0-9_.-]+)$") + + +class ProvenanceError(ValueError): + """A ``version`` or ``provenance`` value of the wrong shape.""" + + +def parse_version(raw: Any) -> str | int | None: + """The ``version`` value (a non-empty string or an integer), or None when absent; :class:`ProvenanceError` + otherwise. Integers are allowed because task.json is open and some repositories already write ``"version": 2``.""" + if raw is None: + return None + if isinstance(raw, int) and not isinstance(raw, bool): + return raw + if isinstance(raw, str) and raw.strip(): + return raw + raise ProvenanceError(f"version must be a non-empty string or an integer, got {raw!r}") + + +def parse_provenance(raw: Any) -> dict[str, Any] | None: + """The ``provenance`` block, checked, or None when absent; :class:`ProvenanceError` on a bad shape.""" + if raw is None: + return None + if not isinstance(raw, dict): + raise ProvenanceError(f"provenance must be an object with optional keys {', '.join(KEYS)}") + unknown = sorted(set(raw) - set(KEYS)) + if unknown: + raise ProvenanceError(f"provenance: unknown key(s) {', '.join(map(str, unknown))} " + f"(known: {', '.join(KEYS)})") + for key in KEYS: + if key in raw and raw[key] is None: + raise ProvenanceError(f"provenance.{key} is null: omit the key instead") + source = raw.get("source") + if source is not None and not (isinstance(source, str) and SOURCE_RE.fullmatch(source)): + raise ProvenanceError(f"provenance.source must be hand-written, generated:@ or " + f"imported:, got {source!r}") + reviewer = raw.get("reviewed_by") + if reviewer is not None and not (isinstance(reviewer, str) and reviewer.strip()): + raise ProvenanceError(f"provenance.reviewed_by must be a non-empty string, got {reviewer!r}") + status = raw.get("status") + if status is not None and status not in STATUSES: + raise ProvenanceError(f"provenance.status must be one of {', '.join(STATUSES)}, got {status!r}") + return dict(raw) + + +def status_of(record: Mapping[str, Any]) -> str: + """The record's ``provenance.status``, or :data:`UNKNOWN` when it has none or an invalid one.""" + block = record.get("provenance") + status = block.get("status") if isinstance(block, Mapping) else None + return status if status in STATUSES else UNKNOWN + + +def generated_by(tool: str) -> dict[str, Any]: + """The ``provenance`` a generator stamps on what it writes: ``generated:dokimi-@``.""" + return {"source": f"generated:dokimi-{tool}@{__version__}"} diff --git a/dokimi/dokimi/questions/llm_gen.py b/dokimi/dokimi/questions/llm_gen.py index bb0dfe340..a0cfca0fa 100644 --- a/dokimi/dokimi/questions/llm_gen.py +++ b/dokimi/dokimi/questions/llm_gen.py @@ -12,6 +12,14 @@ from dokimi.llm import LLM, ChatModel, Invoker, check_caller_provider from dokimi.llm import LANGCHAIN_AVAILABLE from dokimi.defaults import DEFAULT_GENERATOR_MODEL, DEFAULT_GENERATOR_PROVIDER +from dokimi.provenance import generated_by + + +def _stamped(questions: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + """Each generated question with the ``provenance`` the generator filled (a set by hand is kept).""" + for q in questions: + q.setdefault("provenance", generated_by("questions-generate")) + return questions QUESTION_GENERATION_PROMPT = """You are generating technical questions for documentation quality evaluation. @@ -160,7 +168,7 @@ def generate_questions( all_questions.append(q) logger.info(f"Generated {len(all_questions)} total questions") - return all_questions + return _stamped(all_questions) def generate_hybrid( self, @@ -231,7 +239,7 @@ def generate_hybrid( f"Hybrid result: {len(persona_qs)} persona + {len(chunk_qs)} chunk " f"= {len(all_questions)} total" ) - return all_questions + return _stamped(all_questions) def _generate_for_persona( self, diff --git a/dokimi/dokimi/report/task_runs_report.py b/dokimi/dokimi/report/task_runs_report.py index 9432d3fec..a8deeed4a 100644 --- a/dokimi/dokimi/report/task_runs_report.py +++ b/dokimi/dokimi/report/task_runs_report.py @@ -28,6 +28,7 @@ from dokimi.eval.binary_stats import PASS_RATE_CI_CONFIDENCE from dokimi.harnesses.runner import is_errored +from dokimi.provenance import UNKNOWN, status_of def _or_dash(value: object) -> object: @@ -198,6 +199,7 @@ def render_task_runs_report(data: dict[str, Any]) -> str: t["name"]: t.get("metadata", {}).get("difficulty", "?") for t in data.get("tasks", []) } + status_of_task = {t["name"]: status_of(t.get("metadata") or {}) for t in data.get("tasks", [])} rows = data.get("results", []) baseline = data.get("baseline_harness") # Fall back to harnesses observed in the rows if the summary is empty. @@ -341,10 +343,10 @@ def render_task_runs_report(data: dict[str, Any]) -> str: # ── per-task pass + cost (one column pair per harness) ── lines.append("## Per-task pass + cost") lines.append("") - lines.append("| Task | Diff | " + " | ".join( + lines.append("| Task | Diff | Status | " + " | ".join( f"`{h}` pass | `{h}` $" for h in harnesses ) + " |") - lines.append("| --- | :---: |" + "|".join(" :---: | ---: " for _ in harnesses) + "|") + lines.append("| --- | :---: | :---: |" + "|".join(" :---: | ---: " for _ in harnesses) + "|") for t in tasks: cols = [] for h in harnesses: @@ -352,7 +354,7 @@ def render_task_runs_report(data: dict[str, Any]) -> str: err_s = f" (+{c['errored']} err)" if c["errored"] else "" pass_s = f"{c['passed']}/{c['n']}" if c["n"] else "—" cols.append(f"{pass_s}{err_s} | {_fmt_cost(c['cost'])}") - lines.append(f"| {t} | {diff_of.get(t, '?')} | " + " | ".join(cols) + " |") + lines.append(f"| {t} | {diff_of.get(t, '?')} | {status_of_task.get(t, UNKNOWN)} | " + " | ".join(cols) + " |") lines.append("") # ── per-cell answer detail ── diff --git a/dokimi/dokimi/repos/graph_tasks.py b/dokimi/dokimi/repos/graph_tasks.py index e4c888e3a..45a0f537d 100644 --- a/dokimi/dokimi/repos/graph_tasks.py +++ b/dokimi/dokimi/repos/graph_tasks.py @@ -28,6 +28,7 @@ from typing import Any, Callable from dokimi.atomic_write import write_text_atomic +from dokimi.provenance import generated_by from dokimi.repos.code_index import CodeIndex from dokimi.repos.code_index.symbols import descriptor_names, descriptors from dokimi.repos.environment.files import is_generated, is_test_file @@ -468,7 +469,7 @@ def _task(c: Candidate, granularity: str, opts: Options, "id": task_id(c), "family": FAMILY, "base": opts.base, "prompt": prompt, "grader": GRADER, "gold": GOLD, "score": {"metric": "f1", "max": 1.0}, "max_turns": opts.max_turns, "question": c.kind, "answer_key": key, "symbols": list(c.symbols), "ambiguous_name": c.ambiguous, - "code_index": code_index, + "code_index": code_index, "provenance": generated_by("tasks-generate"), } rule = {"files": "files with a reference to the symbol, outside the files that define it", "functions": "functions whose range holds a reference to the symbol, outside the files that define it", diff --git a/dokimi/dokimi/repos/manifest.py b/dokimi/dokimi/repos/manifest.py index 9dc481491..373770c5a 100644 --- a/dokimi/dokimi/repos/manifest.py +++ b/dokimi/dokimi/repos/manifest.py @@ -30,6 +30,7 @@ from dokimi.artifacts import schema_errors from dokimi.measure import SCORE_BOUND, as_float, is_finite +from dokimi.provenance import ProvenanceError, parse_provenance, parse_version from .graders import DATA_FILES, NAMES as BUILTIN_NAMES, PREFIX as BUILTIN_PREFIX, TASK_PROBLEMS from .graders.hidden_tests import HIDDEN_DIR, hidden_files @@ -579,6 +580,7 @@ def _check_task(tree: RepoTree, task_dir: str, report: RepoCheckReport) -> None: except HostSpecError as exc: problems.append(str(exc).split("task.json ", 1)[-1]) problems += _useful_problems(raw) + problems += _provenance_problems(raw) report.errors.extend(f"{where}: {p}" for p in problems) if not problems: report.tasks.append(TaskSummary( @@ -620,6 +622,19 @@ def _useful_problems(raw: dict[str, Any]) -> list[str]: return [] +def _provenance_problems(raw: dict[str, Any]) -> list[str]: + """The ``version`` and ``provenance`` keys' shape problems, if any; both are optional (dokimi.provenance).""" + for key in ("version", "provenance"): + if key in raw and raw[key] is None: + return [f"{key} is null: omit the key instead"] + try: + parse_version(raw.get("version")) + parse_provenance(raw.get("provenance")) + except ProvenanceError as exc: + return [str(exc)] + return [] + + def _hidden_file_problem(tree: RepoTree, task_dir: str, rel: str) -> str | None: """Why ``hidden/`` is not a file the grade can copy: it is missing, or a symbolic link (contract §4.5).""" path = f"{task_dir}/{HIDDEN_DIR}/{rel}" diff --git a/dokimi/dokimi/repos/replay/authoring.py b/dokimi/dokimi/repos/replay/authoring.py index 35ead8159..8c2ae1470 100644 --- a/dokimi/dokimi/repos/replay/authoring.py +++ b/dokimi/dokimi/repos/replay/authoring.py @@ -23,6 +23,7 @@ from typing import Any from dokimi.atomic_write import write_text_atomic +from ...provenance import generated_by from ..graders.hidden_tests import HIDDEN_DIR from ..graders.typecheck import REFERENCE_DIR from ..manifest import RepoManifest @@ -137,6 +138,7 @@ def _make(h: History, manifest: RepoManifest, change: Change, "hidden_tests": tests, "hidden_files": hidden, **change.fields, "patch_files": sorted({p for _, p in rest}), # the paths the change makes, tests too; as replay's "oracle_lines": n_lines, "prompt_dropped": prompt.dropped, + "provenance": generated_by("tasks-generate"), } deleted = [p for s, p in code if s == "D"] if deleted: diff --git a/dokimi/dokimi/repos/replay/build.py b/dokimi/dokimi/repos/replay/build.py index 425bc302c..618b23685 100644 --- a/dokimi/dokimi/repos/replay/build.py +++ b/dokimi/dokimi/repos/replay/build.py @@ -29,6 +29,7 @@ from typing import Any from dokimi.atomic_write import write_bytes_atomic, write_text_atomic +from ...provenance import generated_by from ..manifest import RepoManifest from .history import History, addressed, carry from .merged import Skip, base_of, check_merged, left_out, need_utf8_paths @@ -217,6 +218,7 @@ def _task(pr: PullRequest, base: str, rev: str, leave_out: list[str], opts: Opti "id": task_id(pr), "family": "review", "base": base, "grader": "dokimi:review_replay", "score": {"metric": "recall", "max": 1.0}, "review_patch": PATCH, "gold": GOLD, "window": opts.window, "source_pr": pr.number, "reviewed_commit": rev, "merge_commit": pr.merge, "prompt": opts.prompt, + "provenance": generated_by("tasks-generate"), } if pr.url: task["source_url"] = pr.url diff --git a/dokimi/dokimi/repos/report.py b/dokimi/dokimi/repos/report.py index e9901dfec..a2e01bfb3 100644 --- a/dokimi/dokimi/repos/report.py +++ b/dokimi/dokimi/repos/report.py @@ -60,6 +60,7 @@ import math from dokimi.measure import as_float, is_number +from dokimi.provenance import UNKNOWN, status_of from dokimi.utils import same_model from .grading import SCORE_BOUND, read_grade_json @@ -198,6 +199,18 @@ def load_cells(batch_dir: Path) -> tuple[list[dict[str, Any]], list[str]]: return finished, sorted((planned | started) - done - unsupported) +def _task_status(cell_dir: Path) -> str: + """The task's ``provenance.status`` from the cell's copy of its task.json; unknown without one or with an + unreadable one, so a label never fails the report.""" + path = cell_dir / "task.json" + if not path.is_file(): + return UNKNOWN + try: + return status_of(_read_object(path)) + except CellFileError: + return UNKNOWN + + def load_unsupported(batch_dir: Path) -> list[dict[str, Any]]: """Cells the runner did not run on a host that missed their task's ``host`` requirements.""" out = [] @@ -228,6 +241,7 @@ def summarize(batch_dir: Path) -> dict[str, Any]: failures = sum(r["grade"]["status"] == "grader_failure" for r in recs) score_only = any(r["score_task"] for r in recs) row: dict[str, Any] = {"task": key[0], "arm": key[1], "model": key[2], "n": n, + "status": _task_status(batch_dir / recs[0]["cell"]), "grader_failures": failures, "agent_timeouts": sum(bool(r["agent"]["timed_out"]) for r in recs), "agent_errors": sum(bool(r["agent"]["is_error"]) for r in recs), @@ -277,7 +291,7 @@ def format_report(summary: dict[str, Any]) -> str: arm_w = max([len(r["arm"]) for r in rows] + [3]) model_w = max([len(r["model"]) for r in rows] + [5]) head = (f"{'task':<{width}} {'arm':<{arm_w}} {'model':<{model_w}} {'n':>3} {'result':<22} " - f"{'grader_fail':>11} {'missing':>7} {'cost':>8} {'turns':>6}") + f"{'grader_fail':>11} {'missing':>7} {'cost':>8} {'turns':>6} {'status':<9}") lines = [f"{summary['batch']}: {summary['cells']} finished cells, {len(summary['missing'])} missing, " f"${summary['total_cost_usd']:.4f}", head] for r in rows: @@ -289,7 +303,8 @@ def format_report(summary: dict[str, Any]) -> str: cost = "-" if r["cost_usd"] is None else f"{r['cost_usd']:.4f}" turns = "-" if r["turns"] is None else f"{r['turns']:.1f}" lines.append(f"{r['task']:<{width}} {r['arm']:<{arm_w}} {r['model']:<{model_w}} {r['n']:>3} " - f"{result:<22} {r['grader_failures']:>11} {r['missing']:>7} {cost:>8} {turns:>6}") + f"{result:<22} {r['grader_failures']:>11} {r['missing']:>7} {cost:>8} {turns:>6} " + f"{r['status']:<9}") notes = _row_notes(r) if notes: lines.append(" " * (width + 2) + " " + ", ".join(notes)) diff --git a/dokimi/dokimi/schemas/questions.v2.json b/dokimi/dokimi/schemas/questions.v2.json index 11c766e03..a81559063 100644 --- a/dokimi/dokimi/schemas/questions.v2.json +++ b/dokimi/dokimi/schemas/questions.v2.json @@ -236,6 +236,32 @@ "source_text": { "description": "The question text before `eval triggers generate --rewrite` turned it into the user-voice `question`; absent when it was not rewritten.", "type": "string" + }, + "version": { + "description": "The author's revision of this question: a non-empty string (e.g. `2.1`) or an integer. Absent means unknown; no generator writes it.", + "type": ["string", "integer"], + "pattern": "\\S" + }, + "provenance": { + "description": "Where the question came from and how far it has been checked: `source` (hand-written, generated:@, imported:), `reviewed_by`, `status` (candidate, validated, frozen). Every key is optional; absent is unknown. Filled by the question generators (dokimi.provenance).", + "$ref": "#/$defs/provenance" + } + } + }, + "provenance": { + "type": "object", + "additionalProperties": false, + "properties": { + "source": { + "type": "string", + "pattern": "^(hand-written|generated:[A-Za-z0-9_.-]+@[A-Za-z0-9_.+-]+|imported:[A-Za-z0-9_.-]+)(?![\\s\\S])" + }, + "reviewed_by": { + "type": "string", + "pattern": "\\S" + }, + "status": { + "enum": ["candidate", "validated", "frozen"] } } } diff --git a/dokimi/tests/test_eval_tasks_generate_graph.py b/dokimi/tests/test_eval_tasks_generate_graph.py index c67bdcfa8..7fa871983 100644 --- a/dokimi/tests/test_eval_tasks_generate_graph.py +++ b/dokimi/tests/test_eval_tasks_generate_graph.py @@ -13,6 +13,7 @@ import pytest +from dokimi._version import __version__ from dokimi.cli import build_parser from dokimi.repos import graph_tasks from dokimi.repos.code_index import CodeIndex @@ -182,6 +183,7 @@ def test_write_keeps_an_existing_task_unless_forced(tmp_path): task = json.loads((opts.out / tid / "task.json").read_text()) gold = json.loads((opts.out / tid / "gold.json").read_text()) assert task["grader"] == "dokimi:graph_enum" and task["base"] == "a" * 40 and task["id"] == tid + assert task["provenance"] == {"source": f"generated:dokimi-tasks-generate@{__version__}"} assert task["score"] == {"metric": "f1", "max": 1.0} and task["answer_key"] == "files" assert task["symbols"] == [RUN] and task["family"] == "navigate" and tid.startswith("graph_references_run_") assert "2 symbols named `run`" in task["prompt"] and '{"files": [' in task["prompt"] diff --git a/dokimi/tests/test_import_architecture.py b/dokimi/tests/test_import_architecture.py index bcc20cc63..2fc47f188 100644 --- a/dokimi/tests/test_import_architecture.py +++ b/dokimi/tests/test_import_architecture.py @@ -77,7 +77,7 @@ "dokimi.container_policy", "dokimi.defaults", "dokimi.docker_cmd", "dokimi.intents", "dokimi.judged", "dokimi.llm", "dokimi.llm_cli", "dokimi.llm_json", "dokimi.llm_pricing", "dokimi.measure", "dokimi.process", "dokimi.process_io", - "dokimi.atomic_write", "dokimi.records", "dokimi.registry", "dokimi.utils", "dokimi.write_probe", + "dokimi.atomic_write", "dokimi.provenance", "dokimi.records", "dokimi.registry", "dokimi.utils", "dokimi.write_probe", "dokimi.judge_rule", # Token/cost record the LLM chokepoint fills; it lives under metrics/ but # depends on nothing and is read by every LLM caller. diff --git a/dokimi/tests/test_provenance.py b/dokimi/tests/test_provenance.py new file mode 100644 index 000000000..e89b8d1cc --- /dev/null +++ b/dokimi/tests/test_provenance.py @@ -0,0 +1,163 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""The optional ``version`` and ``provenance`` of tasks and question-set items (dokimi.provenance). + +Absent is unknown, never an error. A malformed value is refused where the artifact is checked (task.json in +the repo check, questions.v2 by its schema) and warned about in a terminal-bench task.toml, which may already +carry the key. The repo report shows the status of each task (docs/repo-evaluation-contract.md §3). +""" + +from __future__ import annotations + +import json + +import pytest + +from dokimi import provenance +from dokimi._version import __version__ +from dokimi.artifacts import schema_errors +from dokimi.cli import build_parser +from dokimi.harnesses.tasks import load_task +from dokimi.repos.manifest import _provenance_problems + +GOOD_BLOCKS = [ + {"source": "hand-written"}, + {"source": "generated:dokimi-tasks-generate@0.2.0", "status": "candidate"}, + {"source": "imported:inspect-ai", "reviewed_by": "alice", "status": "frozen"}, + {"status": "validated"}, +] +BAD_BLOCKS = [ + "validated", + {"sauce": "hand-written"}, + {"source": "made-by-hand"}, + {"source": "generated:dokimi-tasks-generate"}, + {"source": "hand-written\n"}, + {"status": "done"}, + {"reviewed_by": " "}, + {"status": None}, +] + + +@pytest.mark.parametrize("record", [{}, {"provenance": {}}, {"provenance": None}, {"provenance": "validated"}]) +def test_absent_or_unusable_provenance_reads_as_unknown(record): + assert provenance.status_of(record) == provenance.UNKNOWN + + +@pytest.mark.parametrize("status", provenance.STATUSES) +def test_each_status_reads_back(status): + assert provenance.status_of({"provenance": {"status": status}}) == status + + +def test_an_unknown_status_value_reads_as_unknown(): + assert provenance.status_of({"provenance": {"status": "done"}}) == provenance.UNKNOWN + + +@pytest.mark.parametrize("block", GOOD_BLOCKS) +def test_a_well_formed_block_parses_to_itself(block): + assert provenance.parse_provenance(block) == block + + +@pytest.mark.parametrize("block", BAD_BLOCKS) +def test_a_malformed_block_is_refused(block): + with pytest.raises(provenance.ProvenanceError): + provenance.parse_provenance(block) + + +@pytest.mark.parametrize("value", ["2", "2.1", 3, 0]) +def test_a_version_is_a_non_empty_string_or_an_integer(value): + assert provenance.parse_version(value) == value + + +@pytest.mark.parametrize("value", ["", " ", True, 1.5, []]) +def test_a_malformed_version_is_refused(value): + with pytest.raises(provenance.ProvenanceError): + provenance.parse_version(value) + + +def test_a_generator_stamp_names_the_tool_and_the_dokimi_version(): + stamp = provenance.generated_by("tasks-generate") + assert stamp == {"source": f"generated:dokimi-tasks-generate@{__version__}"} + assert provenance.parse_provenance(stamp) == stamp + + +@pytest.mark.parametrize("block", GOOD_BLOCKS) +def test_questions_schema_accepts_what_the_parser_accepts(block): + item = {"id": "q1", "question": "How?", "provenance": block} + assert schema_errors("questions.v2", {"questions": [item]}) == [] + + +@pytest.mark.parametrize("block", BAD_BLOCKS) +def test_questions_schema_refuses_what_the_parser_refuses(block): + item = {"id": "q1", "question": "How?", "provenance": block} + assert schema_errors("questions.v2", {"questions": [item]}) + + +@pytest.mark.parametrize("value", ["2", 3]) +def test_questions_schema_accepts_a_version(value): + assert schema_errors("questions.v2", {"questions": [{"id": "q1", "question": "How?", "version": value}]}) == [] + + +@pytest.mark.parametrize("value", [" ", True, None]) +def test_questions_schema_refuses_a_malformed_version(value): + assert schema_errors("questions.v2", {"questions": [{"id": "q1", "question": "How?", "version": value}]}) + + +@pytest.mark.parametrize("block", GOOD_BLOCKS) +def test_task_json_with_a_good_block_has_no_problem(block): + assert _provenance_problems({"provenance": block, "version": "1"}) == [] + + +def test_task_json_without_the_keys_has_no_problem(): + assert _provenance_problems({"id": "t"}) == [] + + +@pytest.mark.parametrize("record", [{"version": None}, {"provenance": None}]) +def test_task_json_with_a_null_key_is_refused(record): + assert _provenance_problems(record) + + +def test_task_json_with_a_bad_status_names_the_key(): + (problem,) = _provenance_problems({"provenance": {"status": "done"}}) + assert "provenance.status" in problem + + +def test_a_bad_task_toml_metadata_warns_and_still_loads(tmp_path): + task = tmp_path / "t1" + task.mkdir() + (task / "task.toml").write_text('version = "1.0"\n\n[metadata]\nprovenance = { status = "done" }\n') + (task / "instruction.md").write_text("Do it.\n") + with pytest.warns(UserWarning, match="metadata provenance.status must be one of"): + spec = load_task(task) + assert spec.metadata["provenance"] == {"status": "done"} + + +def _cell(tmp_path, name: str, files: dict[str, object]) -> None: + for rel, content in files.items(): + path = tmp_path / "b1" / name / rel + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(content), encoding="utf-8") + + +_RESULT = {"grade": {"status": "ok", "score_only": False, "success": True}, "arm": "a", "model": "m", + "agent": {"timed_out": False, "is_error": False, "over_budget": False, "cost_usd": 0.1, "num_turns": 2}} + + +def test_repo_report_shows_each_tasks_status_and_unknown_without_one(tmp_path, capsys): + _cell(tmp_path, "t1__a__m__r1", {"result.json": {**_RESULT, "task": "t1"}, + "task.json": {"id": "t1", "provenance": {"status": "validated"}}}) + _cell(tmp_path, "t2__a__m__r1", {"result.json": {**_RESULT, "task": "t2"}}) + parser = build_parser() + args = parser.parse_args(["repo", "report", str(tmp_path / "b1"), "--json"]) + args.func(args) + rows = {r["task"]: r["status"] for r in json.loads(capsys.readouterr().out)["data"]["rows"]} + assert rows == {"t1": "validated", "t2": provenance.UNKNOWN} + + +def test_a_corrupt_task_copy_reads_as_unknown_and_keeps_the_report(tmp_path, capsys): + _cell(tmp_path, "t3__a__m__r1", {"result.json": {**_RESULT, "task": "t3"}, "task.json": "not an object"}) + parser = build_parser() + args = parser.parse_args(["repo", "report", str(tmp_path / "b1"), "--json"]) + args.func(args) + rows = {r["task"]: r["status"] for r in json.loads(capsys.readouterr().out)["data"]["rows"]} + assert rows == {"t3": provenance.UNKNOWN} diff --git a/dokimi/tests/test_questions_llm_gen.py b/dokimi/tests/test_questions_llm_gen.py index 8e9ecd77e..32137f3e5 100644 --- a/dokimi/tests/test_questions_llm_gen.py +++ b/dokimi/tests/test_questions_llm_gen.py @@ -17,6 +17,7 @@ mock_mod.ChatAnthropic = Mock sys.modules[mod_name] = mock_mod +from dokimi._version import __version__ from dokimi.questions.llm_gen import QuestionGenerator, QUESTION_GENERATION_PROMPT @@ -94,6 +95,15 @@ def test_returns_questions_for_every_persona_and_topic(self, sample_personas, sa for p in sample_personas } assert topics_by_persona == {p["id"]: sorted(sample_topics * 2) for p in sample_personas} + + def test_every_generated_question_carries_the_generator_provenance(self, sample_personas, sample_topics, + mock_llm): + mock_cls = Mock(return_value=mock_llm) + with patch('dokimi.questions.llm_gen.Invoker', mock_cls), \ + patch('dokimi.questions.llm_gen.LANGCHAIN_AVAILABLE', True): + gen = QuestionGenerator() + questions = gen.generate_questions("oneTBB", sample_personas, sample_topics, questions_per_topic=2) + assert {q["provenance"]["source"] for q in questions} == {f"generated:dokimi-questions-generate@{__version__}"} def test_assigns_unique_ids(self, sample_personas, sample_topics, mock_llm): mock_cls = Mock(return_value=mock_llm) diff --git a/dokimi/tests/test_task_runs_report.py b/dokimi/tests/test_task_runs_report.py index 86b002b3f..f5d806d85 100644 --- a/dokimi/tests/test_task_runs_report.py +++ b/dokimi/tests/test_task_runs_report.py @@ -113,7 +113,7 @@ def test_rollups_keep_distinct_harnesses(): # both harness ids appear as distinct columns; per-task row shows 0/1 and 1/1 assert "`claude-code` pass" in md assert "`codex` pass" in md - assert "| t1 | easy | 0/1 |" in md # claude-code column = 0/1, not merged with codex + assert "| t1 | easy | unknown | 0/1 |" in md # claude-code column = 0/1, not merged with codex def test_per_task_cost_unknown_renders_dash(): @@ -135,7 +135,7 @@ def test_per_task_cost_unknown_renders_dash(): }, "comparisons": {}}, } md = render_task_runs_report(data) - assert "| t1 | easy | 1/1 | — |" in md + assert "| t1 | easy | unknown | 1/1 | — |" in md assert "$0.0000" not in md @@ -165,7 +165,7 @@ def row(task, harness, passed, error=None): md = render_task_runs_report(data) assert "| `skill` | 100.00% (1/1) | 1 |" in md # headline: scored cells, errored column assert "missing data, not a 0" in md - assert "| t2 | easy | 1/1 | — | — (+1 err) | — |" in md + assert "| t2 | easy | unknown | 1/1 | — | — (+1 err) | — |" in md assert "— (arms lost different cells)" in md # unpaired deltas withheld assert "| `skill` | 1 | null | ERR |" in md assert "(harness error)" in md @@ -198,7 +198,7 @@ def row(task, harness, **extra): } md = render_task_runs_report(data) assert "| `base` | 100.00% (1/1) | 1 |" in md # headline agrees: 1 errored - assert "| t1 | easy | — (+1 err) | — | 1/1 | — |" in md + assert "| t1 | easy | unknown | — (+1 err) | — | 1/1 | — |" in md assert "| `base` | 1 | 1 | ERR |" in md From c100397232e9d255d079977c70ea13ae6a7958f7 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Mon, 16 Nov 2026 10:20:13 -0800 Subject: [PATCH 02/18] docs(provenance): put the provenance key under its [metadata] header Thread: PRRT_kwDOVBg7g863y56l Signed-off-by: Nikolay Petrov --- dokimi/docs/contributing-terminal-bench-task.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/dokimi/docs/contributing-terminal-bench-task.md b/dokimi/docs/contributing-terminal-bench-task.md index ea09c2af2..1351bd81b 100644 --- a/dokimi/docs/contributing-terminal-bench-task.md +++ b/dokimi/docs/contributing-terminal-bench-task.md @@ -272,7 +272,14 @@ results are never reported as official upstream benchmark numbers. The content repository is the right place for that record, kept next to the tasks it describes; keep it current in the same change that adds the task. -Record the status in the task too: `[metadata] provenance = { source = "hand-written", reviewed_by = "...", status = "candidate" }` (see §3). `candidate` is a task nobody has checked yet, `validated` one a reviewer has checked, `frozen` one that must not change. Bump `metadata.version` when you change a task that results were already reported on (a repo `dokimi:typecheck` task has no revision `version`: there it is the pinned mypy version, see the [repository contract](./repo-evaluation-contract.md#3-taskjson)). A task without these keys reads as `unknown`; that is not an error. +Record the status in the task too, as a `provenance` key inside the existing `[metadata]` table (see §3). Keep the table header on its own line: + +```toml +[metadata] +provenance = { source = "hand-written", reviewed_by = "...", status = "candidate" } +``` + +`candidate` is a task nobody has checked yet, `validated` one a reviewer has checked, `frozen` one that must not change. Bump `metadata.version` when you change a task that results were already reported on (a repo `dokimi:typecheck` task has no revision `version`: there it is the pinned mypy version, see the [repository contract](./repo-evaluation-contract.md#3-taskjson)). A task without these keys reads as `unknown`; that is not an error. ## 10 — CI From 5444866da266a95d4d7866445256764187c3d9ed Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Mon, 16 Nov 2026 10:20:13 -0800 Subject: [PATCH 03/18] docs(provenance): say repo validate, not repo check Thread: PRRT_kwDOVBg7g863y56q Signed-off-by: Nikolay Petrov --- .../20261114_120000_napetrov_task-question-provenance.md | 2 +- dokimi/docs/repo-evaluation-contract.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md b/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md index 2356e491e..34581706b 100644 --- a/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md +++ b/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md @@ -5,4 +5,4 @@ SPDX-License-Identifier: Apache-2.0 ### Added -- **Optional `version` and `provenance` on repository tasks (`task.json`), executable tasks (`metadata` in `task.toml`) and question-set items.** `provenance` is `{source, reviewed_by, status}`: `source` is `hand-written`, `generated:@` or `imported:`, and `status` is `candidate`, `validated` or `frozen`. Every key is optional and an absent one is `unknown`, never an error. `eval tasks generate` and `eval questions generate` fill `source` as `generated:dokimi-@`. `repo report` shows each task's status in a `status` column (and in its JSON rows), and the executable task-run report shows it in a `Status` column. A malformed value fails `repo check` for `task.json`, is refused by the `questions.v2` schema, and is a warning for a `task.toml`. See `docs/repo-evaluation-contract.md` §3 and `docs/contributing-terminal-bench-task.md` §3 and §9. +- **Optional `version` and `provenance` on repository tasks (`task.json`), executable tasks (`metadata` in `task.toml`) and question-set items.** `provenance` is `{source, reviewed_by, status}`: `source` is `hand-written`, `generated:@` or `imported:`, and `status` is `candidate`, `validated` or `frozen`. Every key is optional and an absent one is `unknown`, never an error. `eval tasks generate` and `eval questions generate` fill `source` as `generated:dokimi-@`. `repo report` shows each task's status in a `status` column (and in its JSON rows), and the executable task-run report shows it in a `Status` column. A malformed value fails `repo validate` for `task.json`, is refused by the `questions.v2` schema, and is a warning for a `task.toml`. See `docs/repo-evaluation-contract.md` §3 and `docs/contributing-terminal-bench-task.md` §3 and §9. diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index 2bdcd3be6..d4e1eafd7 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -335,7 +335,7 @@ belongs to the repository's grader and is passed through unread, except `grader` | `expected_skill` | no | framework | the name (directory name) of the skill the agent should use; a `skill-catalog:` arm must install it, or the arm is refused before anything runs. On a `guidance:as-skill` catalog arm the generated `-guidance` skill is refused too (its usage is `guidance_skill_loaded`). `result.json` `skills` records whether the agent used it and whether it used it first | | `useful` | no | framework | the definition of the run's first useful result, reported as `first_useful` in `result.json` (§4.1): `{"command": ""}` for a task whose grade runs the target tests, or `{"edit": ""}` (or a list of globs) for one that grades an edit to a target file. Omit it when no event is the useful result. A malformed value is refused by `repo validate`, `tasks check` and `repo run` | | `version` | no | framework (reporting) | the task's revision, a non-empty string or an integer; absent is unknown. The grader does not read it, except `dokimi:typecheck`, which reads `version` as the image's mypy version (its row in the grader table) and so must not carry a revision there | -| `provenance` | no | framework (reporting) | `source` (`hand-written`, `generated:@` or `imported:`; `eval tasks generate` writes `generated:dokimi-tasks-generate@`), `reviewed_by` (a non-empty string) and `status` (`candidate`, `validated` or `frozen`); every key optional. `repo report` shows each task's `status` in a column, `unknown` when absent. A malformed block fails `repo check` (`dokimi/provenance.py`) | +| `provenance` | no | framework (reporting) | `source` (`hand-written`, `generated:@` or `imported:`; `eval tasks generate` writes `generated:dokimi-tasks-generate@`), `reviewed_by` (a non-empty string) and `status` (`candidate`, `validated` or `frozen`); every key optional. `repo report` shows each task's `status` in a column, `unknown` when absent. A malformed block fails `repo validate` (`dokimi/provenance.py`) | | anything else | — | repository | for example `hidden_files`, `targets`, `regression_targets`, `manifest` | `setup_patch` (a single string) is accepted as a synonym for a one-element From 9d2d2affa832ea084b2603f9ceaa413c1cbf9112 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Mon, 16 Nov 2026 10:20:13 -0800 Subject: [PATCH 04/18] fix(provenance): refuse a bad block without a type error, and read it as unknown Thread: PRRT_kwDOVBg7g863y56x Thread: PRRT_kwDOVBg7g863y567 Signed-off-by: Nikolay Petrov --- dokimi/dokimi/provenance.py | 11 +++++++---- dokimi/tests/test_provenance.py | 10 ++++++++++ 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/dokimi/dokimi/provenance.py b/dokimi/dokimi/provenance.py index 9c2a804c6..72f229d9a 100644 --- a/dokimi/dokimi/provenance.py +++ b/dokimi/dokimi/provenance.py @@ -46,7 +46,7 @@ def parse_provenance(raw: Any) -> dict[str, Any] | None: return None if not isinstance(raw, dict): raise ProvenanceError(f"provenance must be an object with optional keys {', '.join(KEYS)}") - unknown = sorted(set(raw) - set(KEYS)) + unknown = sorted(set(raw) - set(KEYS), key=str) if unknown: raise ProvenanceError(f"provenance: unknown key(s) {', '.join(map(str, unknown))} " f"(known: {', '.join(KEYS)})") @@ -67,9 +67,12 @@ def parse_provenance(raw: Any) -> dict[str, Any] | None: def status_of(record: Mapping[str, Any]) -> str: - """The record's ``provenance.status``, or :data:`UNKNOWN` when it has none or an invalid one.""" - block = record.get("provenance") - status = block.get("status") if isinstance(block, Mapping) else None + """The record's ``provenance.status``, or :data:`UNKNOWN` when it has none, an invalid one or an invalid block.""" + try: + block = parse_provenance(record.get("provenance")) + except ProvenanceError: + return UNKNOWN + status = block.get("status") if block else None return status if status in STATUSES else UNKNOWN diff --git a/dokimi/tests/test_provenance.py b/dokimi/tests/test_provenance.py index e89b8d1cc..160f4f3f2 100644 --- a/dokimi/tests/test_provenance.py +++ b/dokimi/tests/test_provenance.py @@ -64,6 +64,16 @@ def test_a_malformed_block_is_refused(block): provenance.parse_provenance(block) +def test_a_block_with_a_non_string_key_is_refused_not_a_type_error(): + with pytest.raises(provenance.ProvenanceError): + provenance.parse_provenance({1: "legacy", "extra": "value"}) + + +def test_status_of_an_invalid_block_reads_as_unknown_even_with_a_valid_status(): + record = {"provenance": {"source": "made-by-hand", "status": "validated"}} + assert provenance.status_of(record) == provenance.UNKNOWN + + @pytest.mark.parametrize("value", ["2", "2.1", 3, 0]) def test_a_version_is_a_non_empty_string_or_an_integer(value): assert provenance.parse_version(value) == value From f01b1b87bf5c78f73456c7d9bec160ee8a2d8254 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Mon, 16 Nov 2026 11:55:57 -0800 Subject: [PATCH 05/18] Address review: questions.v2 accepts old version and provenance values A question written before the keys existed loaded fine, and the schema refused it once it carried version 1.5 or a string provenance. The schema now accepts any value; the parser logs a WARNING for a malformed one and keeps it as written, so one value no longer refuses the whole set. Thread: PRRT_kwDOVBg7g8630Azk Thread: PRR_kwDOVBg7g88AAAABYKCnEQ Signed-off-by: Nikolay Petrov --- ...20000_napetrov_task-question-provenance.md | 2 +- dokimi/docs/methodology.md | 2 +- dokimi/dokimi/provenance.py | 2 +- dokimi/dokimi/questions/artifact.py | 19 ++++++++++ dokimi/dokimi/schemas/questions.v2.json | 24 +----------- dokimi/tests/test_provenance.py | 37 ++++++++++++++----- 6 files changed, 52 insertions(+), 34 deletions(-) diff --git a/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md b/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md index 34581706b..a51ba3a63 100644 --- a/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md +++ b/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md @@ -5,4 +5,4 @@ SPDX-License-Identifier: Apache-2.0 ### Added -- **Optional `version` and `provenance` on repository tasks (`task.json`), executable tasks (`metadata` in `task.toml`) and question-set items.** `provenance` is `{source, reviewed_by, status}`: `source` is `hand-written`, `generated:@` or `imported:`, and `status` is `candidate`, `validated` or `frozen`. Every key is optional and an absent one is `unknown`, never an error. `eval tasks generate` and `eval questions generate` fill `source` as `generated:dokimi-@`. `repo report` shows each task's status in a `status` column (and in its JSON rows), and the executable task-run report shows it in a `Status` column. A malformed value fails `repo validate` for `task.json`, is refused by the `questions.v2` schema, and is a warning for a `task.toml`. See `docs/repo-evaluation-contract.md` §3 and `docs/contributing-terminal-bench-task.md` §3 and §9. +- **Optional `version` and `provenance` on repository tasks (`task.json`), executable tasks (`metadata` in `task.toml`) and question-set items.** `provenance` is `{source, reviewed_by, status}`: `source` is `hand-written`, `generated:@` or `imported:`, and `status` is `candidate`, `validated` or `frozen`. Every key is optional and an absent one is `unknown`, never an error. `eval tasks generate` and `eval questions generate` fill `source` as `generated:dokimi-@`. `repo report` shows each task's status in a `status` column (and in its JSON rows), and the executable task-run report shows it in a `Status` column. A malformed value fails `repo validate` for `task.json`, and is a warning for a `task.toml` and for a question-set item (the set still loads, and the value is kept as written). See `docs/repo-evaluation-contract.md` §3 and `docs/contributing-terminal-bench-task.md` §3 and §9. diff --git a/dokimi/docs/methodology.md b/dokimi/docs/methodology.md index 5f400d598..97d7ebc70 100644 --- a/dokimi/docs/methodology.md +++ b/dokimi/docs/methodology.md @@ -103,7 +103,7 @@ Question records preserve source fields such as `source_type`, `question_source` ### Question record format -Each question may carry `version` (the author's revision, a string or an integer) and `provenance` (`source`, `reviewed_by`, `status` as `candidate`, `validated` or `frozen`). Every key is optional and an absent one is unknown. `eval questions generate` writes `source` as `generated:dokimi-questions-generate@`; a hand-written set is `hand-written` by convention. Both shapes are checked by the schema. +Each question may carry `version` (the author's revision, a string or an integer) and `provenance` (`source`, `reviewed_by`, `status` as `candidate`, `validated` or `frozen`). Every key is optional and an absent one is unknown. `eval questions generate` writes `source` as `generated:dokimi-questions-generate@`; a hand-written set is `hand-written` by convention. Both shapes are checked when the set is read: a malformed one is a warning, never a refusal, so files written before the key existed still load. A question set is a JSON object with a `questions` list. The framework writes it as `questions.v2` (`dokimi/schemas/questions.v2.json`, where every field is documented); `questions.v1` files are still read. diff --git a/dokimi/dokimi/provenance.py b/dokimi/dokimi/provenance.py index 72f229d9a..580d9225c 100644 --- a/dokimi/dokimi/provenance.py +++ b/dokimi/dokimi/provenance.py @@ -7,7 +7,7 @@ ``source`` is ``hand-written``, ``generated:@`` or ``imported:``; ``reviewed_by`` names the reviewer; ``status`` is ``candidate``, ``validated`` or ``frozen``. Every key is optional and an absent one is *unknown*, never an error. ``questions.v2`` declares -the same shape in ``$defs.provenance``; ``tests/test_provenance.py`` keeps the two in agreement. +the same shape, but accepts any value: a malformed one is a warning where it is read, not a refusal. """ from __future__ import annotations diff --git a/dokimi/dokimi/questions/artifact.py b/dokimi/dokimi/questions/artifact.py index 01747bc1e..aee3e8cf0 100644 --- a/dokimi/dokimi/questions/artifact.py +++ b/dokimi/dokimi/questions/artifact.py @@ -43,6 +43,7 @@ schema_errors, schema_version_for, ) +from dokimi.provenance import ProvenanceError, parse_provenance, parse_version from dokimi.questions.normalizer import canonical_question, record_id from dokimi.utils import question_id, read_json_input @@ -213,6 +214,20 @@ def _explain(path: Path, data: dict, errors: list[str]) -> str: f"dokimi/schemas/{SCHEMA_VERSIONS[KIND]}.json, or drop the record.") +def _warn_provenance(path: Path, records: list) -> None: + """A malformed ``version`` or ``provenance`` on a record is a WARNING, never a refusal: the schema accepts any + value so that files written before the key existed still load, and the value is kept as written.""" + for i, q in enumerate(records): + if not isinstance(q, dict): + continue + for key, check in (("version", parse_version), ("provenance", parse_provenance)): + try: + check(q.get(key)) + except ProvenanceError as exc: + qid = question_id(q) + logger.warning("%s: question %s: %s; kept as written", path, qid if qid is not None else i, exc) + + def load_questions(path: Path | str, *, validate: bool = True, flag: str | None = None) -> QuestionSet: """Read a question set in any shape this framework has read, and return it as questions.v2. @@ -226,6 +241,9 @@ def load_questions(path: Path | str, *, validate: bool = True, flag: str | None must a ``schema_version`` this reader does not know. A set that is not a list of questions at all (e.g. ``{"questions": {...}}``) is refused either way. + A malformed ``version`` or ``provenance`` on a record is logged at WARNING and kept as written; it does not + refuse the set. + With ``flag`` (the option the file was given to) the file is read by :func:`dokimi.utils.read_json_input`: an unreadable, non-UTF-8 or non-JSON file raises :class:`~dokimi.utils.InputError` `` : ``. Without it the read is bare. @@ -250,6 +268,7 @@ def load_questions(path: Path | str, *, validate: bool = True, flag: str | None for qid in (question_id(q),) if isinstance(qid, (str, int))} read = [_read_record(q, i, seen, taken) if isinstance(q, dict) else q for i, q in enumerate(records)] + _warn_provenance(path, read) artifact = upgrade_questions({**data, "questions": read}) source_version: str | None diff --git a/dokimi/dokimi/schemas/questions.v2.json b/dokimi/dokimi/schemas/questions.v2.json index a81559063..935f38956 100644 --- a/dokimi/dokimi/schemas/questions.v2.json +++ b/dokimi/dokimi/schemas/questions.v2.json @@ -238,30 +238,10 @@ "type": "string" }, "version": { - "description": "The author's revision of this question: a non-empty string (e.g. `2.1`) or an integer. Absent means unknown; no generator writes it.", - "type": ["string", "integer"], - "pattern": "\\S" + "description": "The author's revision of this question: a string (e.g. `2.1`) or an integer; absent means unknown. Any value loads: a malformed one is a warning at load (dokimi.provenance), and no generator writes it." }, "provenance": { - "description": "Where the question came from and how far it has been checked: `source` (hand-written, generated:@, imported:), `reviewed_by`, `status` (candidate, validated, frozen). Every key is optional; absent is unknown. Filled by the question generators (dokimi.provenance).", - "$ref": "#/$defs/provenance" - } - } - }, - "provenance": { - "type": "object", - "additionalProperties": false, - "properties": { - "source": { - "type": "string", - "pattern": "^(hand-written|generated:[A-Za-z0-9_.-]+@[A-Za-z0-9_.+-]+|imported:[A-Za-z0-9_.-]+)(?![\\s\\S])" - }, - "reviewed_by": { - "type": "string", - "pattern": "\\S" - }, - "status": { - "enum": ["candidate", "validated", "frozen"] + "description": "Where the question came from and how far it has been checked: `source` (hand-written, generated:@, imported:), `reviewed_by`, `status` (candidate, validated, frozen). Every key is optional; absent is unknown. Any value loads: a malformed one is a warning at load and reads as unknown (dokimi.provenance)." } } } diff --git a/dokimi/tests/test_provenance.py b/dokimi/tests/test_provenance.py index 160f4f3f2..54aea8972 100644 --- a/dokimi/tests/test_provenance.py +++ b/dokimi/tests/test_provenance.py @@ -4,13 +4,15 @@ """The optional ``version`` and ``provenance`` of tasks and question-set items (dokimi.provenance). Absent is unknown, never an error. A malformed value is refused where the artifact is checked (task.json in -the repo check, questions.v2 by its schema) and warned about in a terminal-bench task.toml, which may already -carry the key. The repo report shows the status of each task (docs/repo-evaluation-contract.md §3). +the repo check) and warned about where it is read (a terminal-bench task.toml, which may already carry the key, +and a questions.v2 item, whose schema accepts any value so that old files keep loading). The repo report shows +the status of each task as the cell's result.json recorded it (docs/repo-evaluation-contract.md §3). """ from __future__ import annotations import json +import logging import pytest @@ -19,6 +21,7 @@ from dokimi.artifacts import schema_errors from dokimi.cli import build_parser from dokimi.harnesses.tasks import load_task +from dokimi.questions.artifact import load_questions from dokimi.repos.manifest import _provenance_problems GOOD_BLOCKS = [ @@ -98,19 +101,35 @@ def test_questions_schema_accepts_what_the_parser_accepts(block): @pytest.mark.parametrize("block", BAD_BLOCKS) -def test_questions_schema_refuses_what_the_parser_refuses(block): +def test_questions_schema_accepts_a_malformed_provenance_the_parser_warns_about(block): item = {"id": "q1", "question": "How?", "provenance": block} - assert schema_errors("questions.v2", {"questions": [item]}) + assert schema_errors("questions.v2", {"questions": [item]}) == [] -@pytest.mark.parametrize("value", ["2", 3]) -def test_questions_schema_accepts_a_version(value): +@pytest.mark.parametrize("value", ["2", 3, " ", True, 1.5, None]) +def test_questions_schema_accepts_any_version_so_old_files_still_load(value): assert schema_errors("questions.v2", {"questions": [{"id": "q1", "question": "How?", "version": value}]}) == [] -@pytest.mark.parametrize("value", [" ", True, None]) -def test_questions_schema_refuses_a_malformed_version(value): - assert schema_errors("questions.v2", {"questions": [{"id": "q1", "question": "How?", "version": value}]}) +@pytest.mark.parametrize(("version", "block"), [(1.5, "hand-written"), ("2", {"status": "done"}), (True, None)]) +def test_a_malformed_question_version_or_provenance_warns_and_the_set_still_loads(tmp_path, caplog, version, block): + path = tmp_path / "questions.json" + question = {"id": "q1", "question": "How?", "version": version, "provenance": block} + path.write_text(json.dumps({"schema_version": "questions.v2", "questions": [question]}), encoding="utf-8") + with caplog.at_level(logging.WARNING, logger="dokimi.questions.artifact"): + loaded = load_questions(path) + assert loaded.artifact["questions"][0]["version"] == version + assert loaded.artifact["questions"][0]["provenance"] == block + assert "q1" in caplog.text + + +def test_a_well_formed_question_version_and_provenance_do_not_warn(tmp_path, caplog): + path = tmp_path / "questions.json" + question = {"id": "q1", "question": "How?", "version": "2.1", "provenance": {"source": "hand-written"}} + path.write_text(json.dumps({"schema_version": "questions.v2", "questions": [question]}), encoding="utf-8") + with caplog.at_level(logging.WARNING, logger="dokimi.questions.artifact"): + load_questions(path) + assert "provenance" not in caplog.text and "version" not in caplog.text @pytest.mark.parametrize("block", GOOD_BLOCKS) From b75ef4e41da4ec611b8711f0fd294e82f49fec21 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Mon, 16 Nov 2026 11:56:01 -0800 Subject: [PATCH 06/18] Address review: report reads a task's status from result.json The status came from the cell's task.json copy in the run dir, which the grader had read-write and could rewrite to any status. The runner now records the task's provenance status in result.json, which the host writes, and `repo report` reads it there. Thread: PRRT_kwDOVBg7g864E-ZT Signed-off-by: Nikolay Petrov --- dokimi/dokimi/repos/report.py | 19 +++++++------------ dokimi/dokimi/repos/runner.py | 3 +++ dokimi/tests/test_provenance.py | 14 ++++++++++++-- 3 files changed, 22 insertions(+), 14 deletions(-) diff --git a/dokimi/dokimi/repos/report.py b/dokimi/dokimi/repos/report.py index e26685b30..a310f0b9b 100644 --- a/dokimi/dokimi/repos/report.py +++ b/dokimi/dokimi/repos/report.py @@ -60,7 +60,7 @@ import math from dokimi.measure import as_float, is_number -from dokimi.provenance import UNKNOWN, status_of +from dokimi.provenance import STATUSES, UNKNOWN from dokimi.utils import same_model from .grading import SCORE_BOUND, read_grade_json @@ -202,16 +202,11 @@ def load_cells(batch_dir: Path) -> tuple[list[dict[str, Any]], list[str]]: return finished, sorted((planned | started) - done - unsupported) -def _task_status(cell_dir: Path) -> str: - """The task's ``provenance.status`` from the cell's copy of its task.json; unknown without one or with an - unreadable one, so a label never fails the report.""" - path = cell_dir / "task.json" - if not path.is_file(): - return UNKNOWN - try: - return status_of(_read_object(path)) - except CellFileError: - return UNKNOWN +def _task_status(rec: dict[str, Any]) -> str: + """The task's ``provenance.status`` as the host wrote it into the cell's result.json when the cell ran; unknown + without one. Not read from the run dir's task.json: the grader had that dir read-write and could rewrite it.""" + status = rec.get("task_status") + return status if status in STATUSES else UNKNOWN def refuse_mixed_executions(cells: list[dict[str, Any]], where: str) -> None: @@ -260,7 +255,7 @@ def summarize(batch_dir: Path) -> dict[str, Any]: failures = sum(r["grade"]["status"] == "grader_failure" for r in recs) score_only = any(r["score_task"] for r in recs) row: dict[str, Any] = {"task": key[0], "arm": key[1], "model": key[2], "n": n, - "status": _task_status(batch_dir / recs[0]["cell"]), + "status": _task_status(recs[0]), "grader_failures": failures, "agent_timeouts": sum(bool(r["agent"]["timed_out"]) for r in recs), "agent_errors": sum(bool(r["agent"]["is_error"]) for r in recs), diff --git a/dokimi/dokimi/repos/runner.py b/dokimi/dokimi/repos/runner.py index a08eefaa4..d1cc7eaa9 100644 --- a/dokimi/dokimi/repos/runner.py +++ b/dokimi/dokimi/repos/runner.py @@ -48,6 +48,8 @@ from pathlib import Path from typing import Any +from dokimi.provenance import status_of + from .container import ( HARNESS_DIR, RUN_DIR, @@ -304,6 +306,7 @@ def run_cell(ctx: EvalContext, task: Task, arm: str, rep: int, batch_dir: Path, "model_resolved": meta.get("model_resolved"), "rep": rep, "harness": opts.profile.name, "harness_version": meta["harness_version"], "base_date": meta["base_date"], "execution": execution, "score_task": bool(task.score), + "task_status": status_of(task.raw), "score_max": task.score["max"] if task.score else None, "agent": agent, "process": process, "first_useful": first_useful(task.useful, useful_events), "grade": grade} _add_arm_use(record, meta, task, skills, uptake_of, events) diff --git a/dokimi/tests/test_provenance.py b/dokimi/tests/test_provenance.py index 54aea8972..8342375e1 100644 --- a/dokimi/tests/test_provenance.py +++ b/dokimi/tests/test_provenance.py @@ -173,8 +173,7 @@ def _cell(tmp_path, name: str, files: dict[str, object]) -> None: def test_repo_report_shows_each_tasks_status_and_unknown_without_one(tmp_path, capsys): - _cell(tmp_path, "t1__a__m__r1", {"result.json": {**_RESULT, "task": "t1"}, - "task.json": {"id": "t1", "provenance": {"status": "validated"}}}) + _cell(tmp_path, "t1__a__m__r1", {"result.json": {**_RESULT, "task": "t1", "task_status": "validated"}}) _cell(tmp_path, "t2__a__m__r1", {"result.json": {**_RESULT, "task": "t2"}}) parser = build_parser() args = parser.parse_args(["repo", "report", str(tmp_path / "b1"), "--json"]) @@ -183,6 +182,17 @@ def test_repo_report_shows_each_tasks_status_and_unknown_without_one(tmp_path, c assert rows == {"t1": "validated", "t2": provenance.UNKNOWN} +def test_a_task_copy_the_grader_rewrote_does_not_change_the_status(tmp_path, capsys): + # the run dir is read-write for the grader, so its task.json copy is not evidence of the status + _cell(tmp_path, "t4__a__m__r1", {"result.json": {**_RESULT, "task": "t4", "task_status": "candidate"}, + "task.json": {"id": "t4", "provenance": {"status": "frozen"}}}) + parser = build_parser() + args = parser.parse_args(["repo", "report", str(tmp_path / "b1"), "--json"]) + args.func(args) + rows = {r["task"]: r["status"] for r in json.loads(capsys.readouterr().out)["data"]["rows"]} + assert rows == {"t4": "candidate"} + + def test_a_corrupt_task_copy_reads_as_unknown_and_keeps_the_report(tmp_path, capsys): _cell(tmp_path, "t3__a__m__r1", {"result.json": {**_RESULT, "task": "t3"}, "task.json": "not an object"}) parser = build_parser() From 9f3a95c14f13cc92f5755655229bbcd57824b99e Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Mon, 16 Nov 2026 11:58:02 -0800 Subject: [PATCH 07/18] Address review: questions.v2 says a malformed value is kept as written The question's provenance is not read as a status anywhere, so the schema description no longer says it reads as unknown. Thread: PRRT_kwDOVBg7g8630Azk Thread: PRR_kwDOVBg7g88AAAABYKCnEQ Signed-off-by: Nikolay Petrov --- dokimi/dokimi/schemas/questions.v2.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dokimi/dokimi/schemas/questions.v2.json b/dokimi/dokimi/schemas/questions.v2.json index 935f38956..a942f670f 100644 --- a/dokimi/dokimi/schemas/questions.v2.json +++ b/dokimi/dokimi/schemas/questions.v2.json @@ -238,10 +238,10 @@ "type": "string" }, "version": { - "description": "The author's revision of this question: a string (e.g. `2.1`) or an integer; absent means unknown. Any value loads: a malformed one is a warning at load (dokimi.provenance), and no generator writes it." + "description": "The author's revision of this question: a string (e.g. `2.1`) or an integer; absent means unknown. Any value loads: a malformed one is a warning at load and is kept as written (dokimi.provenance); no generator writes it." }, "provenance": { - "description": "Where the question came from and how far it has been checked: `source` (hand-written, generated:@, imported:), `reviewed_by`, `status` (candidate, validated, frozen). Every key is optional; absent is unknown. Any value loads: a malformed one is a warning at load and reads as unknown (dokimi.provenance)." + "description": "Where the question came from and how far it has been checked: `source` (hand-written, generated:@, imported:), `reviewed_by`, `status` (candidate, validated, frozen). Every key is optional; absent is unknown. Any value loads: a malformed one is a warning at load and is kept as written (dokimi.provenance)." } } } From 5af24e58e6ef175a631ae387b20bdcb5f0d7aec4 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Mon, 16 Nov 2026 11:58:02 -0800 Subject: [PATCH 08/18] Address review: the report test checks the rewritten copy, not a corrupt one The report no longer opens the run dir's task.json, so the test of a corrupt copy did not test anything. The no-warning test now asserts no WARNING record. Thread: PRRT_kwDOVBg7g864E-ZT Signed-off-by: Nikolay Petrov --- dokimi/tests/test_provenance.py | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/dokimi/tests/test_provenance.py b/dokimi/tests/test_provenance.py index 8342375e1..08533e7a6 100644 --- a/dokimi/tests/test_provenance.py +++ b/dokimi/tests/test_provenance.py @@ -129,7 +129,7 @@ def test_a_well_formed_question_version_and_provenance_do_not_warn(tmp_path, cap path.write_text(json.dumps({"schema_version": "questions.v2", "questions": [question]}), encoding="utf-8") with caplog.at_level(logging.WARNING, logger="dokimi.questions.artifact"): load_questions(path) - assert "provenance" not in caplog.text and "version" not in caplog.text + assert not [r for r in caplog.records if r.levelno >= logging.WARNING] @pytest.mark.parametrize("block", GOOD_BLOCKS) @@ -191,12 +191,3 @@ def test_a_task_copy_the_grader_rewrote_does_not_change_the_status(tmp_path, cap args.func(args) rows = {r["task"]: r["status"] for r in json.loads(capsys.readouterr().out)["data"]["rows"]} assert rows == {"t4": "candidate"} - - -def test_a_corrupt_task_copy_reads_as_unknown_and_keeps_the_report(tmp_path, capsys): - _cell(tmp_path, "t3__a__m__r1", {"result.json": {**_RESULT, "task": "t3"}, "task.json": "not an object"}) - parser = build_parser() - args = parser.parse_args(["repo", "report", str(tmp_path / "b1"), "--json"]) - args.func(args) - rows = {r["task"]: r["status"] for r in json.loads(capsys.readouterr().out)["data"]["rows"]} - assert rows == {"t3": provenance.UNKNOWN} From afcde9134530f5df1010d7de5fb66d4b1251f408 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Mon, 16 Nov 2026 12:42:19 -0800 Subject: [PATCH 09/18] Keep the result record unchanged for a task without provenance The batch golden pins every key of result.json. task_status is written only when the task carries a status, so a task without provenance writes the same record as before; the report reads an absent key as unknown. Thread: PRRT_kwDOVBg7g864E-ZT Signed-off-by: Nikolay Petrov --- dokimi/dokimi/repos/runner.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/dokimi/dokimi/repos/runner.py b/dokimi/dokimi/repos/runner.py index d1cc7eaa9..30d5988ae 100644 --- a/dokimi/dokimi/repos/runner.py +++ b/dokimi/dokimi/repos/runner.py @@ -48,7 +48,7 @@ from pathlib import Path from typing import Any -from dokimi.provenance import status_of +from dokimi.provenance import UNKNOWN, status_of from .container import ( HARNESS_DIR, @@ -306,9 +306,10 @@ def run_cell(ctx: EvalContext, task: Task, arm: str, rep: int, batch_dir: Path, "model_resolved": meta.get("model_resolved"), "rep": rep, "harness": opts.profile.name, "harness_version": meta["harness_version"], "base_date": meta["base_date"], "execution": execution, "score_task": bool(task.score), - "task_status": status_of(task.raw), "score_max": task.score["max"] if task.score else None, "agent": agent, "process": process, "first_useful": first_useful(task.useful, useful_events), "grade": grade} + if status_of(task.raw) != UNKNOWN: # absent means unknown to the report: a task without one writes no key + record["task_status"] = status_of(task.raw) _add_arm_use(record, meta, task, skills, uptake_of, events) try: _own_dir(rd) # the grader had the run dir read-write, and may have taken write access away From a682ece9ec4c835298f4082eb97f954ac53b4a91 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Mon, 16 Nov 2026 22:57:05 -0800 Subject: [PATCH 10/18] Address review: golden result-key claims are already fixed in afcde91345 Thread: PRRT_kwDOVBg7g864IAxy: already fixed in afcde91345 Thread: PRRT_kwDOVBg7g864IAyA: already fixed in afcde91345 Thread: PRRT_kwDOVBg7g864IAyI: already fixed in afcde91345 Thread: PRRT_kwDOVBg7g864IAyP: already fixed in afcde91345 Thread: PRRT_kwDOVBg7g864IAyW: already fixed in afcde91345 Thread: PRRT_kwDOVBg7g864IAyb: already fixed in afcde91345 Signed-off-by: Nikolay Petrov From e5ee638ddd309aaea031185457b4647f0ce79eaf Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Tue, 17 Nov 2026 13:45:11 -0800 Subject: [PATCH 11/18] Address review: name the question generator, not the command, in provenance Thread: PRRT_kwDOVBg7g864ZXoB Signed-off-by: Nikolay Petrov --- dokimi/dokimi/questions/llm_gen.py | 10 +++++----- dokimi/tests/test_questions_llm_gen.py | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/dokimi/dokimi/questions/llm_gen.py b/dokimi/dokimi/questions/llm_gen.py index a0cfca0fa..df1f2d2fe 100644 --- a/dokimi/dokimi/questions/llm_gen.py +++ b/dokimi/dokimi/questions/llm_gen.py @@ -15,10 +15,10 @@ from dokimi.provenance import generated_by -def _stamped(questions: List[Dict[str, Any]]) -> List[Dict[str, Any]]: - """Each generated question with the ``provenance`` the generator filled (a set by hand is kept).""" +def _stamped(questions: List[Dict[str, Any]], tool: str) -> List[Dict[str, Any]]: + """Each generated question with the ``provenance`` of the generator ``tool`` (a set by hand is kept).""" for q in questions: - q.setdefault("provenance", generated_by("questions-generate")) + q.setdefault("provenance", generated_by(tool)) return questions @@ -168,7 +168,7 @@ def generate_questions( all_questions.append(q) logger.info(f"Generated {len(all_questions)} total questions") - return _stamped(all_questions) + return _stamped(all_questions, "questions-persona") def generate_hybrid( self, @@ -239,7 +239,7 @@ def generate_hybrid( f"Hybrid result: {len(persona_qs)} persona + {len(chunk_qs)} chunk " f"= {len(all_questions)} total" ) - return _stamped(all_questions) + return _stamped(all_questions, "questions-chunk") def _generate_for_persona( self, diff --git a/dokimi/tests/test_questions_llm_gen.py b/dokimi/tests/test_questions_llm_gen.py index 32137f3e5..76f9078ed 100644 --- a/dokimi/tests/test_questions_llm_gen.py +++ b/dokimi/tests/test_questions_llm_gen.py @@ -103,7 +103,7 @@ def test_every_generated_question_carries_the_generator_provenance(self, sample_ patch('dokimi.questions.llm_gen.LANGCHAIN_AVAILABLE', True): gen = QuestionGenerator() questions = gen.generate_questions("oneTBB", sample_personas, sample_topics, questions_per_topic=2) - assert {q["provenance"]["source"] for q in questions} == {f"generated:dokimi-questions-generate@{__version__}"} + assert {q["provenance"]["source"] for q in questions} == {f"generated:dokimi-questions-persona@{__version__}"} def test_assigns_unique_ids(self, sample_personas, sample_topics, mock_llm): mock_cls = Mock(return_value=mock_llm) From f418483bfe23034a0b4d2d7f9fb05429b1735f2f Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Tue, 17 Nov 2026 13:45:16 -0800 Subject: [PATCH 12/18] Address review: name the swebench exception to metadata.version Thread: PRRT_kwDOVBg7g864ZXoM Signed-off-by: Nikolay Petrov --- dokimi/docs/contributing-terminal-bench-task.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dokimi/docs/contributing-terminal-bench-task.md b/dokimi/docs/contributing-terminal-bench-task.md index 499376ac3..cb61da537 100644 --- a/dokimi/docs/contributing-terminal-bench-task.md +++ b/dokimi/docs/contributing-terminal-bench-task.md @@ -111,7 +111,7 @@ Field by field, with the consumer that actually reads it: | `metadata.difficulty` | `eval tasks list`, the task-runs report | `easy` / `medium` / `hard`; drives the difficulty rollup in the Markdown report. Missing renders as `?`. | | `metadata.level` | [`difficulty-rubric.md`](./difficulty-rubric.md) | Workflow depth: `triage`, `diagnosis`, `end_to_end`. Optional but recommended for skill evaluation. | | `metadata.difficulty_explanation`, `category`, `author_*` | humans | Provenance and review context; carried through in the run artifact. | -| `metadata.version`, `metadata.provenance` | `load_task` (a malformed value is a warning; the task still loads), the task-runs report | Optional. `version` is the task's revision; it is not the top-level `version`, which is the format version. `provenance` is `{source, reviewed_by, status}` as in the [repository contract](./repo-evaluation-contract.md#3-taskjson): `status` is `candidate`, `validated` or `frozen`. The task-runs report shows each task's status in a `Status` column; an absent one is `unknown`. | +| `metadata.version`, `metadata.provenance` | `load_task` (a malformed value is a warning; the task still loads), the task-runs report | Optional. `version` is the task's revision; it is not the top-level `version`, which is the format version. A `swebench` task's `version` is the upstream package version (e.g. `"3.0"` for django), kept as written, not a revision. `provenance` is `{source, reviewed_by, status}` as in the [repository contract](./repo-evaluation-contract.md#3-taskjson): `status` is `candidate`, `validated` or `frozen`. The task-runs report shows each task's status in a `Status` column; an absent one is `unknown`. | | `agent.timeout_sec` | `TaskSpec.timeout_sec`, the solver harnesses | Wall-clock budget for the agent's attempt. Typically 300. | | `verifier.timeout_sec` | Harbor, human reviewers | Verifier budget. Keep it small (≤ 60s where possible) so CI stays fast. | | `environment.build_timeout_sec` | `docker-*` harnesses | Image-build budget; used instead of the default when set (oneAPI/pip-heavy images need > 10 min). | From a6289fb452b080bbe21ef011ffde96e5b0992952 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Tue, 17 Nov 2026 13:45:40 -0800 Subject: [PATCH 13/18] Address review: a report row whose cells disagree on status says mixed Thread: PRRT_kwDOVBg7g864ZXoN Signed-off-by: Nikolay Petrov --- dokimi/dokimi/repos/report.py | 8 +++++++- dokimi/tests/test_provenance.py | 11 +++++++++++ 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/dokimi/dokimi/repos/report.py b/dokimi/dokimi/repos/report.py index a310f0b9b..288b464f3 100644 --- a/dokimi/dokimi/repos/report.py +++ b/dokimi/dokimi/repos/report.py @@ -209,6 +209,12 @@ def _task_status(rec: dict[str, Any]) -> str: return status if status in STATUSES else UNKNOWN +def _row_status(recs: list[dict[str, Any]]) -> str: + """The one status the row's cells recorded, or ``mixed`` when a resumed batch recorded more than one.""" + statuses = {_task_status(r) for r in recs} + return statuses.pop() if len(statuses) == 1 else "mixed" + + def refuse_mixed_executions(cells: list[dict[str, Any]], where: str) -> None: """One (batch, arm, model) ran with one execution: a ``--force`` run that stopped part way, or a harness binary replaced mid-batch, leaves two, and pooling them would blend two setups in one row.""" @@ -255,7 +261,7 @@ def summarize(batch_dir: Path) -> dict[str, Any]: failures = sum(r["grade"]["status"] == "grader_failure" for r in recs) score_only = any(r["score_task"] for r in recs) row: dict[str, Any] = {"task": key[0], "arm": key[1], "model": key[2], "n": n, - "status": _task_status(recs[0]), + "status": _row_status(recs), "grader_failures": failures, "agent_timeouts": sum(bool(r["agent"]["timed_out"]) for r in recs), "agent_errors": sum(bool(r["agent"]["is_error"]) for r in recs), diff --git a/dokimi/tests/test_provenance.py b/dokimi/tests/test_provenance.py index 08533e7a6..92674d26b 100644 --- a/dokimi/tests/test_provenance.py +++ b/dokimi/tests/test_provenance.py @@ -191,3 +191,14 @@ def test_a_task_copy_the_grader_rewrote_does_not_change_the_status(tmp_path, cap args.func(args) rows = {r["task"]: r["status"] for r in json.loads(capsys.readouterr().out)["data"]["rows"]} assert rows == {"t4": "candidate"} + + +def test_a_row_whose_cells_recorded_different_statuses_says_mixed(tmp_path, capsys): + # a batch resumed after the task went from candidate to validated: its cells disagree, the row must say so + _cell(tmp_path, "t5__a__m__r1", {"result.json": {**_RESULT, "task": "t5", "task_status": "candidate"}}) + _cell(tmp_path, "t5__a__m__r2", {"result.json": {**_RESULT, "task": "t5", "task_status": "validated"}}) + parser = build_parser() + args = parser.parse_args(["repo", "report", str(tmp_path / "b1"), "--json"]) + args.func(args) + rows = {r["task"]: r["status"] for r in json.loads(capsys.readouterr().out)["data"]["rows"]} + assert rows == {"t5": "mixed"} From 3b9644d4fad3d2b529f25db42065cea4c2dc9fdb Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Tue, 17 Nov 2026 13:46:22 -0800 Subject: [PATCH 14/18] Address review: regression test for the host writing task_status Verified against a mutant that drops the write: the test fails. Thread: PRRT_kwDOVBg7g864ZXn1 Signed-off-by: Nikolay Petrov --- dokimi/tests/test_repo_run.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/dokimi/tests/test_repo_run.py b/dokimi/tests/test_repo_run.py index f816d6b01..ce5458671 100644 --- a/dokimi/tests/test_repo_run.py +++ b/dokimi/tests/test_repo_run.py @@ -998,3 +998,14 @@ def bad_recall(run_dir): # the grader of one cell writes `bad` as its score met args = build_parser().parse_args(["repo", "compare", str(batch), "--baseline", "guidance:none"]) args.func(args) assert "review_add" in capsys.readouterr().out + + +def test_the_host_writes_the_tasks_status_into_result_json(setup): + # the report reads the status from this key only: a run that drops the write shows every task as unknown + task_json = setup["root"] / ".agent-eval/tasks/fix_add/task.json" + task = json.loads(task_json.read_text()) + task["provenance"] = {"source": "hand-written", "status": "validated"} + task_json.write_text(json.dumps(task)) + _run(setup, "--arm", "guidance:raw", source=str(setup["root"])) + rec = json.loads((setup["out"] / "b1/fix_add__guidance-raw__m-1__r1/result.json").read_text()) + assert rec["task_status"] == "validated" From 5297ec814363c4f1d03767ea4ada6fdacd067f31 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Tue, 17 Nov 2026 13:56:03 -0800 Subject: [PATCH 15/18] test: regenerate the task-runs golden for the Status column after the merge Signed-off-by: Nikolay Petrov --- dokimi/tests/fixtures/report_fmt.json | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/dokimi/tests/fixtures/report_fmt.json b/dokimi/tests/fixtures/report_fmt.json index de9142385..0ae81a709 100644 --- a/dokimi/tests/fixtures/report_fmt.json +++ b/dokimi/tests/fixtures/report_fmt.json @@ -333,10 +333,10 @@ "", "## Per-task pass + cost", "", - "| Task | Diff | `docker-a` pass | `docker-a` $ | `docker-b` pass | `docker-b` $ | `docker-b-skill-discover` pass | `docker-b-skill-discover` $ |", - "| --- | :---: | :---: | ---: | :---: | ---: | :---: | ---: |", - "| t1 | easy | 1/1 | <$0.0001 | 1/1 | $0.1235 | 0/1 | $0.0001 |", - "| t2 | ? | 0/1 | — | 1/1 | $0.0000 | 0/1 | — |", + "| Task | Diff | Status | `docker-a` pass | `docker-a` $ | `docker-b` pass | `docker-b` $ | `docker-b-skill-discover` pass | `docker-b-skill-discover` $ |", + "| --- | :---: | :---: | :---: | ---: | :---: | ---: | :---: | ---: |", + "| t1 | easy | unknown | 1/1 | <$0.0001 | 1/1 | $0.1235 | 0/1 | $0.0001 |", + "| t2 | ? | unknown | 0/1 | — | 1/1 | $0.0000 | 0/1 | — |", "", "## Per-cell answer detail", "", From 23ed869a513d8264235c472c78b3bb472a79b725 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Tue, 17 Nov 2026 14:12:31 -0800 Subject: [PATCH 16/18] Address review: keep the task_status test in its own file test_repo_run.py is at the 1000-line cap (scripts/check_growth.py); the test moves to test_repo_run_task_status.py. Thread: PRRT_kwDOVBg7g864ZXn1 Signed-off-by: Nikolay Petrov Item: PRRT_kwDOVBg7g864ZXn1.a Review: PRR_kwDOVBg7g88AAAABYePnSw --- dokimi/tests/test_repo_run.py | 11 ----------- dokimi/tests/test_repo_run_task_status.py | 21 +++++++++++++++++++++ 2 files changed, 21 insertions(+), 11 deletions(-) create mode 100644 dokimi/tests/test_repo_run_task_status.py diff --git a/dokimi/tests/test_repo_run.py b/dokimi/tests/test_repo_run.py index ce5458671..f816d6b01 100644 --- a/dokimi/tests/test_repo_run.py +++ b/dokimi/tests/test_repo_run.py @@ -998,14 +998,3 @@ def bad_recall(run_dir): # the grader of one cell writes `bad` as its score met args = build_parser().parse_args(["repo", "compare", str(batch), "--baseline", "guidance:none"]) args.func(args) assert "review_add" in capsys.readouterr().out - - -def test_the_host_writes_the_tasks_status_into_result_json(setup): - # the report reads the status from this key only: a run that drops the write shows every task as unknown - task_json = setup["root"] / ".agent-eval/tasks/fix_add/task.json" - task = json.loads(task_json.read_text()) - task["provenance"] = {"source": "hand-written", "status": "validated"} - task_json.write_text(json.dumps(task)) - _run(setup, "--arm", "guidance:raw", source=str(setup["root"])) - rec = json.loads((setup["out"] / "b1/fix_add__guidance-raw__m-1__r1/result.json").read_text()) - assert rec["task_status"] == "validated" diff --git a/dokimi/tests/test_repo_run_task_status.py b/dokimi/tests/test_repo_run_task_status.py new file mode 100644 index 000000000..d40d9ee3d --- /dev/null +++ b/dokimi/tests/test_repo_run_task_status.py @@ -0,0 +1,21 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""The host writes a task's status into result.json, the key the report reads.""" + +from __future__ import annotations + +import json + +from tests.test_repo_run import _run, setup # noqa: F401 # the fixture and the runner of test_repo_run + +def test_the_host_writes_the_tasks_status_into_result_json(setup): + # the report reads the status from this key only: a run that drops the write shows every task as unknown + task_json = setup["root"] / ".agent-eval/tasks/fix_add/task.json" + task = json.loads(task_json.read_text()) + task["provenance"] = {"source": "hand-written", "status": "validated"} + task_json.write_text(json.dumps(task)) + _run(setup, "--arm", "guidance:raw", source=str(setup["root"])) + rec = json.loads((setup["out"] / "b1/fix_add__guidance-raw__m-1__r1/result.json").read_text()) + assert rec["task_status"] == "validated" + From d39789aba147960b74a94a2137fb4f3ae9d5d94d Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Tue, 17 Nov 2026 14:18:01 -0800 Subject: [PATCH 17/18] Address review: lint-clean task_status test The test is the same as before; the fixture import is marked for F811. Thread: PRRT_kwDOVBg7g864ZXn1 Review: PRR_kwDOVBg7g88AAAABYePnSw Signed-off-by: Nikolay Petrov --- dokimi/tests/test_repo_run_task_status.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/dokimi/tests/test_repo_run_task_status.py b/dokimi/tests/test_repo_run_task_status.py index d40d9ee3d..6d84aedf7 100644 --- a/dokimi/tests/test_repo_run_task_status.py +++ b/dokimi/tests/test_repo_run_task_status.py @@ -9,7 +9,8 @@ from tests.test_repo_run import _run, setup # noqa: F401 # the fixture and the runner of test_repo_run -def test_the_host_writes_the_tasks_status_into_result_json(setup): + +def test_the_host_writes_the_tasks_status_into_result_json(setup): # noqa: F811 # the report reads the status from this key only: a run that drops the write shows every task as unknown task_json = setup["root"] / ".agent-eval/tasks/fix_add/task.json" task = json.loads(task_json.read_text()) From dc2c209bcd0dfa5bee331c9f5af8c8dd71a29ce8 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Tue, 17 Nov 2026 15:12:26 -0800 Subject: [PATCH 18/18] docs: name the generated stamps the code writes The methodology page and the changelog entry named one source stamp for every generated question. The code writes generated:dokimi-questions-persona@ for persona questions and generated:dokimi-questions-chunk@ for chunk-grounded ones. Thread: PRRT_kwDOVBg7g864ZXoB Signed-off-by: Nikolay Petrov --- .../20261114_120000_napetrov_task-question-provenance.md | 2 +- dokimi/docs/methodology.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md b/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md index a51ba3a63..33f1c93ea 100644 --- a/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md +++ b/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md @@ -5,4 +5,4 @@ SPDX-License-Identifier: Apache-2.0 ### Added -- **Optional `version` and `provenance` on repository tasks (`task.json`), executable tasks (`metadata` in `task.toml`) and question-set items.** `provenance` is `{source, reviewed_by, status}`: `source` is `hand-written`, `generated:@` or `imported:`, and `status` is `candidate`, `validated` or `frozen`. Every key is optional and an absent one is `unknown`, never an error. `eval tasks generate` and `eval questions generate` fill `source` as `generated:dokimi-@`. `repo report` shows each task's status in a `status` column (and in its JSON rows), and the executable task-run report shows it in a `Status` column. A malformed value fails `repo validate` for `task.json`, and is a warning for a `task.toml` and for a question-set item (the set still loads, and the value is kept as written). See `docs/repo-evaluation-contract.md` §3 and `docs/contributing-terminal-bench-task.md` §3 and §9. +- **Optional `version` and `provenance` on repository tasks (`task.json`), executable tasks (`metadata` in `task.toml`) and question-set items.** `provenance` is `{source, reviewed_by, status}`: `source` is `hand-written`, `generated:@` or `imported:`, and `status` is `candidate`, `validated` or `frozen`. Every key is optional and an absent one is `unknown`, never an error. `eval tasks generate` fills `source` as `generated:dokimi-tasks-generate@`, and question generation fills `generated:dokimi-questions-persona@…` or, for chunk-grounded questions, `generated:dokimi-questions-chunk@…`. `repo report` shows each task's status in a `status` column (and in its JSON rows), and the executable task-run report shows it in a `Status` column. A malformed value fails `repo validate` for `task.json`, and is a warning for a `task.toml` and for a question-set item (the set still loads, and the value is kept as written). See `docs/repo-evaluation-contract.md` §3 and `docs/contributing-terminal-bench-task.md` §3 and §9. diff --git a/dokimi/docs/methodology.md b/dokimi/docs/methodology.md index 97d7ebc70..42330a350 100644 --- a/dokimi/docs/methodology.md +++ b/dokimi/docs/methodology.md @@ -103,7 +103,7 @@ Question records preserve source fields such as `source_type`, `question_source` ### Question record format -Each question may carry `version` (the author's revision, a string or an integer) and `provenance` (`source`, `reviewed_by`, `status` as `candidate`, `validated` or `frozen`). Every key is optional and an absent one is unknown. `eval questions generate` writes `source` as `generated:dokimi-questions-generate@`; a hand-written set is `hand-written` by convention. Both shapes are checked when the set is read: a malformed one is a warning, never a refusal, so files written before the key existed still load. +Each question may carry `version` (the author's revision, a string or an integer) and `provenance` (`source`, `reviewed_by`, `status` as `candidate`, `validated` or `frozen`). Every key is optional and an absent one is unknown. `eval questions generate` writes `source` as `generated:dokimi-questions-persona@`, and the pipeline's chunk-grounded questions carry `generated:dokimi-questions-chunk@`; a hand-written set is `hand-written` by convention. Both shapes are checked when the set is read: a malformed one is a warning, never a refusal, so files written before the key existed still load. A question set is a JSON object with a `questions` list. The framework writes it as `questions.v2` (`dokimi/schemas/questions.v2.json`, where every field is documented); `questions.v1` files are still read.