diff --git a/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md b/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md new file mode 100644 index 000000000..33f1c93ea --- /dev/null +++ b/dokimi/changelog.d/20261114_120000_napetrov_task-question-provenance.md @@ -0,0 +1,8 @@ + + +### Added + +- **Optional `version` and `provenance` on repository tasks (`task.json`), executable tasks (`metadata` in `task.toml`) and question-set items.** `provenance` is `{source, reviewed_by, status}`: `source` is `hand-written`, `generated:@` or `imported:`, and `status` is `candidate`, `validated` or `frozen`. Every key is optional and an absent one is `unknown`, never an error. `eval tasks generate` fills `source` as `generated:dokimi-tasks-generate@`, and question generation fills `generated:dokimi-questions-persona@…` or, for chunk-grounded questions, `generated:dokimi-questions-chunk@…`. `repo report` shows each task's status in a `status` column (and in its JSON rows), and the executable task-run report shows it in a `Status` column. A malformed value fails `repo validate` for `task.json`, and is a warning for a `task.toml` and for a question-set item (the set still loads, and the value is kept as written). See `docs/repo-evaluation-contract.md` §3 and `docs/contributing-terminal-bench-task.md` §3 and §9. diff --git a/dokimi/docs/contributing-terminal-bench-task.md b/dokimi/docs/contributing-terminal-bench-task.md index 401efdd82..cb61da537 100644 --- a/dokimi/docs/contributing-terminal-bench-task.md +++ b/dokimi/docs/contributing-terminal-bench-task.md @@ -111,6 +111,7 @@ Field by field, with the consumer that actually reads it: | `metadata.difficulty` | `eval tasks list`, the task-runs report | `easy` / `medium` / `hard`; drives the difficulty rollup in the Markdown report. Missing renders as `?`. | | `metadata.level` | [`difficulty-rubric.md`](./difficulty-rubric.md) | Workflow depth: `triage`, `diagnosis`, `end_to_end`. Optional but recommended for skill evaluation. | | `metadata.difficulty_explanation`, `category`, `author_*` | humans | Provenance and review context; carried through in the run artifact. | +| `metadata.version`, `metadata.provenance` | `load_task` (a malformed value is a warning; the task still loads), the task-runs report | Optional. `version` is the task's revision; it is not the top-level `version`, which is the format version. A `swebench` task's `version` is the upstream package version (e.g. `"3.0"` for django), kept as written, not a revision. `provenance` is `{source, reviewed_by, status}` as in the [repository contract](./repo-evaluation-contract.md#3-taskjson): `status` is `candidate`, `validated` or `frozen`. The task-runs report shows each task's status in a `Status` column; an absent one is `unknown`. | | `agent.timeout_sec` | `TaskSpec.timeout_sec`, the solver harnesses | Wall-clock budget for the agent's attempt. Typically 300. | | `verifier.timeout_sec` | Harbor, human reviewers | Verifier budget. Keep it small (≤ 60s where possible) so CI stays fast. | | `environment.build_timeout_sec` | `docker-*` harnesses | Image-build budget; used instead of the default when set (oneAPI/pip-heavy images need > 10 min). | @@ -270,6 +271,15 @@ results are never reported as official upstream benchmark numbers. The content repository is the right place for that record, kept next to the tasks it describes; keep it current in the same change that adds the task. +Record the status in the task too, as a `provenance` key inside the existing `[metadata]` table (see §3). Keep the table header on its own line: + +```toml +[metadata] +provenance = { source = "hand-written", reviewed_by = "...", status = "candidate" } +``` + +`candidate` is a task nobody has checked yet, `validated` one a reviewer has checked, `frozen` one that must not change. Bump `metadata.version` when you change a task that results were already reported on (a repo `dokimi:typecheck` task has no revision `version`: there it is the pinned mypy version, see the [repository contract](./repo-evaluation-contract.md#3-taskjson)). A task without these keys reads as `unknown`; that is not an error. + ## 10 — CI CI in this repository proves the *framework* against external content: the diff --git a/dokimi/docs/methodology.md b/dokimi/docs/methodology.md index 4669fbdf7..42330a350 100644 --- a/dokimi/docs/methodology.md +++ b/dokimi/docs/methodology.md @@ -103,6 +103,8 @@ Question records preserve source fields such as `source_type`, `question_source` ### Question record format +Each question may carry `version` (the author's revision, a string or an integer) and `provenance` (`source`, `reviewed_by`, `status` as `candidate`, `validated` or `frozen`). Every key is optional and an absent one is unknown. `eval questions generate` writes `source` as `generated:dokimi-questions-persona@`, and the pipeline's chunk-grounded questions carry `generated:dokimi-questions-chunk@`; a hand-written set is `hand-written` by convention. Both shapes are checked when the set is read: a malformed one is a warning, never a refusal, so files written before the key existed still load. + A question set is a JSON object with a `questions` list. The framework writes it as `questions.v2` (`dokimi/schemas/questions.v2.json`, where every field is documented); `questions.v1` files are still read. - **v2 (written):** each record needs `id` and a non-empty `question`. A v2 record may not also carry `text`, so no record holds two question texts that readers could disagree on. diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index e0b1be55b..949f7d8f6 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -334,6 +334,8 @@ belongs to the repository's grader and is passed through unread, except `grader` | `egress` | no | framework | extra hosts (DNS names, optional `:port`; `*` stands for one whole label and the last two labels are literal, so `*.pythonhosted.org` passes and `*` or `*.org` does not) this task's agent and entry points reach through the proxy (§2), for example `["pypi.org", "files.pythonhosted.org"]` | | `expected_skill` | no | framework | the name (directory name) of the skill the agent should use; a `skill-catalog:` arm must install it, or the arm is refused before anything runs. On a `guidance:as-skill` catalog arm the generated `-guidance` skill is refused too (its usage is `guidance_skill_loaded`). `result.json` `skills` records whether the agent used it and whether it used it first | | `useful` | no | framework | the definition of the run's first useful result, reported as `first_useful` in `result.json` (§4.1): `{"command": ""}` for a task whose grade runs the target tests, or `{"edit": ""}` (or a list of globs) for one that grades an edit to a target file. Omit it when no event is the useful result. A malformed value is refused by `repo validate`, `tasks check` and `repo run` | +| `version` | no | framework (reporting) | the task's revision, a non-empty string or an integer; absent is unknown. The grader does not read it, except `dokimi:typecheck`, which reads `version` as the image's mypy version (its row in the grader table) and so must not carry a revision there | +| `provenance` | no | framework (reporting) | `source` (`hand-written`, `generated:@` or `imported:`; `eval tasks generate` writes `generated:dokimi-tasks-generate@`), `reviewed_by` (a non-empty string) and `status` (`candidate`, `validated` or `frozen`); every key optional. `repo report` shows each task's `status` in a column, `unknown` when absent. A malformed block fails `repo validate` (`dokimi/provenance.py`) | | anything else | — | repository | for example `hidden_files`, `targets`, `regression_targets`, `manifest` | `setup_patch` (a single string) is accepted as a synonym for a one-element diff --git a/dokimi/dokimi/harnesses/tasks.py b/dokimi/dokimi/harnesses/tasks.py index 9210c5954..f0cb97c46 100644 --- a/dokimi/dokimi/harnesses/tasks.py +++ b/dokimi/dokimi/harnesses/tasks.py @@ -41,6 +41,8 @@ import yaml +from dokimi.provenance import ProvenanceError, parse_provenance, parse_version + from . import tbench_v1 # Executable tasks live in a separate evaluation-content checkout, so there is @@ -340,6 +342,15 @@ def _read_tables(path: Path, fmt: str) -> tuple[dict[str, Any], str]: return read_task_config(path), instruction_path.read_text(encoding="utf-8") +def _check_provenance(spec: TaskSpec) -> None: + """Warn, never fail, on a malformed metadata ``version`` or ``provenance``: a repository may already use the key.""" + try: + parse_version(spec.metadata.get("version")) + parse_provenance(spec.metadata.get("provenance")) + except ProvenanceError as exc: + _warn_once(f"task {spec.path}: metadata {exc}; its status reads as unknown") + + def load_task(path_or_name: str | Path, root: Path | None = None) -> TaskSpec: """Load one task by directory path, or by task name relative to *root*. @@ -390,6 +401,7 @@ def load_task(path_or_name: str | Path, root: Path | None = None) -> TaskSpec: format=fmt, schema_version=str(version) if version is not None else None, ) + _check_provenance(spec) drift = legacy_drift(spec) if drift: _warn_once( diff --git a/dokimi/dokimi/provenance.py b/dokimi/dokimi/provenance.py new file mode 100644 index 000000000..580d9225c --- /dev/null +++ b/dokimi/dokimi/provenance.py @@ -0,0 +1,81 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""The optional ``version`` and ``provenance`` of a task and of a question-set item. + +``provenance`` says where the record came from and how far it has been checked: +``source`` is ``hand-written``, ``generated:@`` or ``imported:``; +``reviewed_by`` names the reviewer; ``status`` is ``candidate``, ``validated`` or ``frozen``. +Every key is optional and an absent one is *unknown*, never an error. ``questions.v2`` declares +the same shape, but accepts any value: a malformed one is a warning where it is read, not a refusal. +""" + +from __future__ import annotations + +import re +from collections.abc import Mapping +from typing import Any + +from dokimi._version import __version__ + +UNKNOWN = "unknown" +STATUSES = ("candidate", "validated", "frozen") +KEYS = ("source", "reviewed_by", "status") +SOURCE_RE = re.compile(r"^(hand-written|generated:[A-Za-z0-9_.-]+@[A-Za-z0-9_.+-]+|imported:[A-Za-z0-9_.-]+)$") + + +class ProvenanceError(ValueError): + """A ``version`` or ``provenance`` value of the wrong shape.""" + + +def parse_version(raw: Any) -> str | int | None: + """The ``version`` value (a non-empty string or an integer), or None when absent; :class:`ProvenanceError` + otherwise. Integers are allowed because task.json is open and some repositories already write ``"version": 2``.""" + if raw is None: + return None + if isinstance(raw, int) and not isinstance(raw, bool): + return raw + if isinstance(raw, str) and raw.strip(): + return raw + raise ProvenanceError(f"version must be a non-empty string or an integer, got {raw!r}") + + +def parse_provenance(raw: Any) -> dict[str, Any] | None: + """The ``provenance`` block, checked, or None when absent; :class:`ProvenanceError` on a bad shape.""" + if raw is None: + return None + if not isinstance(raw, dict): + raise ProvenanceError(f"provenance must be an object with optional keys {', '.join(KEYS)}") + unknown = sorted(set(raw) - set(KEYS), key=str) + if unknown: + raise ProvenanceError(f"provenance: unknown key(s) {', '.join(map(str, unknown))} " + f"(known: {', '.join(KEYS)})") + for key in KEYS: + if key in raw and raw[key] is None: + raise ProvenanceError(f"provenance.{key} is null: omit the key instead") + source = raw.get("source") + if source is not None and not (isinstance(source, str) and SOURCE_RE.fullmatch(source)): + raise ProvenanceError(f"provenance.source must be hand-written, generated:@ or " + f"imported:, got {source!r}") + reviewer = raw.get("reviewed_by") + if reviewer is not None and not (isinstance(reviewer, str) and reviewer.strip()): + raise ProvenanceError(f"provenance.reviewed_by must be a non-empty string, got {reviewer!r}") + status = raw.get("status") + if status is not None and status not in STATUSES: + raise ProvenanceError(f"provenance.status must be one of {', '.join(STATUSES)}, got {status!r}") + return dict(raw) + + +def status_of(record: Mapping[str, Any]) -> str: + """The record's ``provenance.status``, or :data:`UNKNOWN` when it has none, an invalid one or an invalid block.""" + try: + block = parse_provenance(record.get("provenance")) + except ProvenanceError: + return UNKNOWN + status = block.get("status") if block else None + return status if status in STATUSES else UNKNOWN + + +def generated_by(tool: str) -> dict[str, Any]: + """The ``provenance`` a generator stamps on what it writes: ``generated:dokimi-@``.""" + return {"source": f"generated:dokimi-{tool}@{__version__}"} diff --git a/dokimi/dokimi/questions/artifact.py b/dokimi/dokimi/questions/artifact.py index 01747bc1e..aee3e8cf0 100644 --- a/dokimi/dokimi/questions/artifact.py +++ b/dokimi/dokimi/questions/artifact.py @@ -43,6 +43,7 @@ schema_errors, schema_version_for, ) +from dokimi.provenance import ProvenanceError, parse_provenance, parse_version from dokimi.questions.normalizer import canonical_question, record_id from dokimi.utils import question_id, read_json_input @@ -213,6 +214,20 @@ def _explain(path: Path, data: dict, errors: list[str]) -> str: f"dokimi/schemas/{SCHEMA_VERSIONS[KIND]}.json, or drop the record.") +def _warn_provenance(path: Path, records: list) -> None: + """A malformed ``version`` or ``provenance`` on a record is a WARNING, never a refusal: the schema accepts any + value so that files written before the key existed still load, and the value is kept as written.""" + for i, q in enumerate(records): + if not isinstance(q, dict): + continue + for key, check in (("version", parse_version), ("provenance", parse_provenance)): + try: + check(q.get(key)) + except ProvenanceError as exc: + qid = question_id(q) + logger.warning("%s: question %s: %s; kept as written", path, qid if qid is not None else i, exc) + + def load_questions(path: Path | str, *, validate: bool = True, flag: str | None = None) -> QuestionSet: """Read a question set in any shape this framework has read, and return it as questions.v2. @@ -226,6 +241,9 @@ def load_questions(path: Path | str, *, validate: bool = True, flag: str | None must a ``schema_version`` this reader does not know. A set that is not a list of questions at all (e.g. ``{"questions": {...}}``) is refused either way. + A malformed ``version`` or ``provenance`` on a record is logged at WARNING and kept as written; it does not + refuse the set. + With ``flag`` (the option the file was given to) the file is read by :func:`dokimi.utils.read_json_input`: an unreadable, non-UTF-8 or non-JSON file raises :class:`~dokimi.utils.InputError` `` : ``. Without it the read is bare. @@ -250,6 +268,7 @@ def load_questions(path: Path | str, *, validate: bool = True, flag: str | None for qid in (question_id(q),) if isinstance(qid, (str, int))} read = [_read_record(q, i, seen, taken) if isinstance(q, dict) else q for i, q in enumerate(records)] + _warn_provenance(path, read) artifact = upgrade_questions({**data, "questions": read}) source_version: str | None diff --git a/dokimi/dokimi/questions/llm_gen.py b/dokimi/dokimi/questions/llm_gen.py index bb0dfe340..df1f2d2fe 100644 --- a/dokimi/dokimi/questions/llm_gen.py +++ b/dokimi/dokimi/questions/llm_gen.py @@ -12,6 +12,14 @@ from dokimi.llm import LLM, ChatModel, Invoker, check_caller_provider from dokimi.llm import LANGCHAIN_AVAILABLE from dokimi.defaults import DEFAULT_GENERATOR_MODEL, DEFAULT_GENERATOR_PROVIDER +from dokimi.provenance import generated_by + + +def _stamped(questions: List[Dict[str, Any]], tool: str) -> List[Dict[str, Any]]: + """Each generated question with the ``provenance`` of the generator ``tool`` (a set by hand is kept).""" + for q in questions: + q.setdefault("provenance", generated_by(tool)) + return questions QUESTION_GENERATION_PROMPT = """You are generating technical questions for documentation quality evaluation. @@ -160,7 +168,7 @@ def generate_questions( all_questions.append(q) logger.info(f"Generated {len(all_questions)} total questions") - return all_questions + return _stamped(all_questions, "questions-persona") def generate_hybrid( self, @@ -231,7 +239,7 @@ def generate_hybrid( f"Hybrid result: {len(persona_qs)} persona + {len(chunk_qs)} chunk " f"= {len(all_questions)} total" ) - return all_questions + return _stamped(all_questions, "questions-chunk") def _generate_for_persona( self, diff --git a/dokimi/dokimi/report/task_runs_report.py b/dokimi/dokimi/report/task_runs_report.py index 6e3bd2339..20873740e 100644 --- a/dokimi/dokimi/report/task_runs_report.py +++ b/dokimi/dokimi/report/task_runs_report.py @@ -29,6 +29,7 @@ from dokimi.eval.binary_stats import PASS_RATE_CI_CONFIDENCE from dokimi.harnesses.runner import is_errored from dokimi.measure import is_number +from dokimi.provenance import UNKNOWN, status_of from dokimi.report import fmt @@ -176,6 +177,7 @@ def render_task_runs_report(data: dict[str, Any]) -> str: t["name"]: t.get("metadata", {}).get("difficulty", "?") for t in data.get("tasks", []) } + status_of_task = {t["name"]: status_of(t.get("metadata") or {}) for t in data.get("tasks", [])} rows = data.get("results", []) baseline = data.get("baseline_harness") # Fall back to harnesses observed in the rows if the summary is empty. @@ -310,10 +312,10 @@ def render_task_runs_report(data: dict[str, Any]) -> str: # ── per-task pass + cost (one column pair per harness) ── lines.append("## Per-task pass + cost") lines.append("") - lines.append("| Task | Diff | " + " | ".join( + lines.append("| Task | Diff | Status | " + " | ".join( f"`{h}` pass | `{h}` $" for h in harnesses ) + " |") - lines.append("| --- | :---: |" + "|".join(" :---: | ---: " for _ in harnesses) + "|") + lines.append("| --- | :---: | :---: |" + "|".join(" :---: | ---: " for _ in harnesses) + "|") for t in tasks: cols = [] for h in harnesses: @@ -321,7 +323,7 @@ def render_task_runs_report(data: dict[str, Any]) -> str: err_s = f" (+{c['errored']} err)" if c["errored"] else "" pass_s = f"{c['passed']}/{c['n']}" if c["n"] else "—" cols.append(f"{pass_s}{err_s} | {fmt.usd(c['cost'])}") - lines.append(f"| {t} | {diff_of.get(t, '?')} | " + " | ".join(cols) + " |") + lines.append(f"| {t} | {diff_of.get(t, '?')} | {status_of_task.get(t, UNKNOWN)} | " + " | ".join(cols) + " |") lines.append("") # ── per-cell answer detail ── diff --git a/dokimi/dokimi/repos/graph_tasks.py b/dokimi/dokimi/repos/graph_tasks.py index 1da1f9dd6..7c5ebd1dd 100644 --- a/dokimi/dokimi/repos/graph_tasks.py +++ b/dokimi/dokimi/repos/graph_tasks.py @@ -28,6 +28,7 @@ from typing import Any, Callable from dokimi.atomic_write import write_text_atomic +from dokimi.provenance import generated_by from dokimi.repos.code_index import CodeIndex from dokimi.repos.code_index.symbols import descriptor_names, descriptors from dokimi.repos.environment.files import is_generated, is_test_file @@ -469,7 +470,7 @@ def _task(c: Candidate, granularity: str, opts: Options, "id": task_id(c), "family": FAMILY, "base": opts.base, "prompt": prompt, "grader": GRADER, "gold": GOLD, "score": {"metric": "f1", "max": 1.0}, "max_turns": opts.max_turns, "question": c.kind, "answer_key": key, "symbols": list(c.symbols), "ambiguous_name": c.ambiguous, - "code_index": code_index, + "code_index": code_index, "provenance": generated_by("tasks-generate"), } rule = {"files": "files with a reference to the symbol, outside the files that define it", "functions": "functions whose range holds a reference to the symbol, outside the files that define it", diff --git a/dokimi/dokimi/repos/manifest.py b/dokimi/dokimi/repos/manifest.py index a101f3873..1d9bb6f23 100644 --- a/dokimi/dokimi/repos/manifest.py +++ b/dokimi/dokimi/repos/manifest.py @@ -30,6 +30,7 @@ from dokimi.artifacts import schema_errors from dokimi.measure import SCORE_BOUND, as_float, is_finite +from dokimi.provenance import ProvenanceError, parse_provenance, parse_version from .graders import DATA_FILES, NAMES as BUILTIN_NAMES, PREFIX as BUILTIN_PREFIX, TASK_PROBLEMS from .graders.hidden_tests import HIDDEN_DIR, hidden_files @@ -579,6 +580,7 @@ def _check_task(tree: RepoTree, task_dir: str, report: RepoCheckReport) -> None: except HostSpecError as exc: problems.append(str(exc).split("task.json ", 1)[-1]) problems += _useful_problems(raw) + problems += _provenance_problems(raw) report.errors.extend(f"{where}: {p}" for p in problems) if not problems: report.tasks.append(TaskSummary( @@ -620,6 +622,19 @@ def _useful_problems(raw: dict[str, Any]) -> list[str]: return [] +def _provenance_problems(raw: dict[str, Any]) -> list[str]: + """The ``version`` and ``provenance`` keys' shape problems, if any; both are optional (dokimi.provenance).""" + for key in ("version", "provenance"): + if key in raw and raw[key] is None: + return [f"{key} is null: omit the key instead"] + try: + parse_version(raw.get("version")) + parse_provenance(raw.get("provenance")) + except ProvenanceError as exc: + return [str(exc)] + return [] + + def _hidden_file_problem(tree: RepoTree, task_dir: str, rel: str) -> str | None: """Why ``hidden/`` is not a file the grade can copy: it is missing, or a symbolic link (contract §4.5).""" path = f"{task_dir}/{HIDDEN_DIR}/{rel}" diff --git a/dokimi/dokimi/repos/replay/authoring.py b/dokimi/dokimi/repos/replay/authoring.py index 35ead8159..8c2ae1470 100644 --- a/dokimi/dokimi/repos/replay/authoring.py +++ b/dokimi/dokimi/repos/replay/authoring.py @@ -23,6 +23,7 @@ from typing import Any from dokimi.atomic_write import write_text_atomic +from ...provenance import generated_by from ..graders.hidden_tests import HIDDEN_DIR from ..graders.typecheck import REFERENCE_DIR from ..manifest import RepoManifest @@ -137,6 +138,7 @@ def _make(h: History, manifest: RepoManifest, change: Change, "hidden_tests": tests, "hidden_files": hidden, **change.fields, "patch_files": sorted({p for _, p in rest}), # the paths the change makes, tests too; as replay's "oracle_lines": n_lines, "prompt_dropped": prompt.dropped, + "provenance": generated_by("tasks-generate"), } deleted = [p for s, p in code if s == "D"] if deleted: diff --git a/dokimi/dokimi/repos/replay/build.py b/dokimi/dokimi/repos/replay/build.py index 425bc302c..618b23685 100644 --- a/dokimi/dokimi/repos/replay/build.py +++ b/dokimi/dokimi/repos/replay/build.py @@ -29,6 +29,7 @@ from typing import Any from dokimi.atomic_write import write_bytes_atomic, write_text_atomic +from ...provenance import generated_by from ..manifest import RepoManifest from .history import History, addressed, carry from .merged import Skip, base_of, check_merged, left_out, need_utf8_paths @@ -217,6 +218,7 @@ def _task(pr: PullRequest, base: str, rev: str, leave_out: list[str], opts: Opti "id": task_id(pr), "family": "review", "base": base, "grader": "dokimi:review_replay", "score": {"metric": "recall", "max": 1.0}, "review_patch": PATCH, "gold": GOLD, "window": opts.window, "source_pr": pr.number, "reviewed_commit": rev, "merge_commit": pr.merge, "prompt": opts.prompt, + "provenance": generated_by("tasks-generate"), } if pr.url: task["source_url"] = pr.url diff --git a/dokimi/dokimi/repos/report.py b/dokimi/dokimi/repos/report.py index 28c9eaa79..288b464f3 100644 --- a/dokimi/dokimi/repos/report.py +++ b/dokimi/dokimi/repos/report.py @@ -60,6 +60,7 @@ import math from dokimi.measure import as_float, is_number +from dokimi.provenance import STATUSES, UNKNOWN from dokimi.utils import same_model from .grading import SCORE_BOUND, read_grade_json @@ -201,6 +202,19 @@ def load_cells(batch_dir: Path) -> tuple[list[dict[str, Any]], list[str]]: return finished, sorted((planned | started) - done - unsupported) +def _task_status(rec: dict[str, Any]) -> str: + """The task's ``provenance.status`` as the host wrote it into the cell's result.json when the cell ran; unknown + without one. Not read from the run dir's task.json: the grader had that dir read-write and could rewrite it.""" + status = rec.get("task_status") + return status if status in STATUSES else UNKNOWN + + +def _row_status(recs: list[dict[str, Any]]) -> str: + """The one status the row's cells recorded, or ``mixed`` when a resumed batch recorded more than one.""" + statuses = {_task_status(r) for r in recs} + return statuses.pop() if len(statuses) == 1 else "mixed" + + def refuse_mixed_executions(cells: list[dict[str, Any]], where: str) -> None: """One (batch, arm, model) ran with one execution: a ``--force`` run that stopped part way, or a harness binary replaced mid-batch, leaves two, and pooling them would blend two setups in one row.""" @@ -247,6 +261,7 @@ def summarize(batch_dir: Path) -> dict[str, Any]: failures = sum(r["grade"]["status"] == "grader_failure" for r in recs) score_only = any(r["score_task"] for r in recs) row: dict[str, Any] = {"task": key[0], "arm": key[1], "model": key[2], "n": n, + "status": _row_status(recs), "grader_failures": failures, "agent_timeouts": sum(bool(r["agent"]["timed_out"]) for r in recs), "agent_errors": sum(bool(r["agent"]["is_error"]) for r in recs), @@ -296,7 +311,7 @@ def format_report(summary: dict[str, Any]) -> str: arm_w = max([len(r["arm"]) for r in rows] + [3]) model_w = max([len(r["model"]) for r in rows] + [5]) head = (f"{'task':<{width}} {'arm':<{arm_w}} {'model':<{model_w}} {'n':>3} {'result':<22} " - f"{'grader_fail':>11} {'missing':>7} {'cost':>8} {'turns':>6}") + f"{'grader_fail':>11} {'missing':>7} {'cost':>8} {'turns':>6} {'status':<9}") lines = [f"{summary['batch']}: {summary['cells']} finished cells, {len(summary['missing'])} missing, " f"${summary['total_cost_usd']:.4f}", head] for r in rows: @@ -308,7 +323,8 @@ def format_report(summary: dict[str, Any]) -> str: cost = "-" if r["cost_usd"] is None else f"{r['cost_usd']:.4f}" turns = "-" if r["turns"] is None else f"{r['turns']:.1f}" lines.append(f"{r['task']:<{width}} {r['arm']:<{arm_w}} {r['model']:<{model_w}} {r['n']:>3} " - f"{result:<22} {r['grader_failures']:>11} {r['missing']:>7} {cost:>8} {turns:>6}") + f"{result:<22} {r['grader_failures']:>11} {r['missing']:>7} {cost:>8} {turns:>6} " + f"{r['status']:<9}") notes = _row_notes(r) if notes: lines.append(" " * (width + 2) + " " + ", ".join(notes)) diff --git a/dokimi/dokimi/repos/runner.py b/dokimi/dokimi/repos/runner.py index a08eefaa4..30d5988ae 100644 --- a/dokimi/dokimi/repos/runner.py +++ b/dokimi/dokimi/repos/runner.py @@ -48,6 +48,8 @@ from pathlib import Path from typing import Any +from dokimi.provenance import UNKNOWN, status_of + from .container import ( HARNESS_DIR, RUN_DIR, @@ -306,6 +308,8 @@ def run_cell(ctx: EvalContext, task: Task, arm: str, rep: int, batch_dir: Path, "base_date": meta["base_date"], "execution": execution, "score_task": bool(task.score), "score_max": task.score["max"] if task.score else None, "agent": agent, "process": process, "first_useful": first_useful(task.useful, useful_events), "grade": grade} + if status_of(task.raw) != UNKNOWN: # absent means unknown to the report: a task without one writes no key + record["task_status"] = status_of(task.raw) _add_arm_use(record, meta, task, skills, uptake_of, events) try: _own_dir(rd) # the grader had the run dir read-write, and may have taken write access away diff --git a/dokimi/dokimi/schemas/questions.v2.json b/dokimi/dokimi/schemas/questions.v2.json index 11c766e03..a942f670f 100644 --- a/dokimi/dokimi/schemas/questions.v2.json +++ b/dokimi/dokimi/schemas/questions.v2.json @@ -236,6 +236,12 @@ "source_text": { "description": "The question text before `eval triggers generate --rewrite` turned it into the user-voice `question`; absent when it was not rewritten.", "type": "string" + }, + "version": { + "description": "The author's revision of this question: a string (e.g. `2.1`) or an integer; absent means unknown. Any value loads: a malformed one is a warning at load and is kept as written (dokimi.provenance); no generator writes it." + }, + "provenance": { + "description": "Where the question came from and how far it has been checked: `source` (hand-written, generated:@, imported:), `reviewed_by`, `status` (candidate, validated, frozen). Every key is optional; absent is unknown. Any value loads: a malformed one is a warning at load and is kept as written (dokimi.provenance)." } } } diff --git a/dokimi/tests/fixtures/report_fmt.json b/dokimi/tests/fixtures/report_fmt.json index de9142385..0ae81a709 100644 --- a/dokimi/tests/fixtures/report_fmt.json +++ b/dokimi/tests/fixtures/report_fmt.json @@ -333,10 +333,10 @@ "", "## Per-task pass + cost", "", - "| Task | Diff | `docker-a` pass | `docker-a` $ | `docker-b` pass | `docker-b` $ | `docker-b-skill-discover` pass | `docker-b-skill-discover` $ |", - "| --- | :---: | :---: | ---: | :---: | ---: | :---: | ---: |", - "| t1 | easy | 1/1 | <$0.0001 | 1/1 | $0.1235 | 0/1 | $0.0001 |", - "| t2 | ? | 0/1 | — | 1/1 | $0.0000 | 0/1 | — |", + "| Task | Diff | Status | `docker-a` pass | `docker-a` $ | `docker-b` pass | `docker-b` $ | `docker-b-skill-discover` pass | `docker-b-skill-discover` $ |", + "| --- | :---: | :---: | :---: | ---: | :---: | ---: | :---: | ---: |", + "| t1 | easy | unknown | 1/1 | <$0.0001 | 1/1 | $0.1235 | 0/1 | $0.0001 |", + "| t2 | ? | unknown | 0/1 | — | 1/1 | $0.0000 | 0/1 | — |", "", "## Per-cell answer detail", "", diff --git a/dokimi/tests/test_eval_tasks_generate_graph.py b/dokimi/tests/test_eval_tasks_generate_graph.py index c67bdcfa8..7fa871983 100644 --- a/dokimi/tests/test_eval_tasks_generate_graph.py +++ b/dokimi/tests/test_eval_tasks_generate_graph.py @@ -13,6 +13,7 @@ import pytest +from dokimi._version import __version__ from dokimi.cli import build_parser from dokimi.repos import graph_tasks from dokimi.repos.code_index import CodeIndex @@ -182,6 +183,7 @@ def test_write_keeps_an_existing_task_unless_forced(tmp_path): task = json.loads((opts.out / tid / "task.json").read_text()) gold = json.loads((opts.out / tid / "gold.json").read_text()) assert task["grader"] == "dokimi:graph_enum" and task["base"] == "a" * 40 and task["id"] == tid + assert task["provenance"] == {"source": f"generated:dokimi-tasks-generate@{__version__}"} assert task["score"] == {"metric": "f1", "max": 1.0} and task["answer_key"] == "files" assert task["symbols"] == [RUN] and task["family"] == "navigate" and tid.startswith("graph_references_run_") assert "2 symbols named `run`" in task["prompt"] and '{"files": [' in task["prompt"] diff --git a/dokimi/tests/test_import_architecture.py b/dokimi/tests/test_import_architecture.py index 0ab4ed847..24ac690b6 100644 --- a/dokimi/tests/test_import_architecture.py +++ b/dokimi/tests/test_import_architecture.py @@ -78,7 +78,7 @@ "dokimi.container_policy", "dokimi.defaults", "dokimi.docker_cmd", "dokimi.intents", "dokimi.judged", "dokimi.llm", "dokimi.llm_cli", "dokimi.llm_json", "dokimi.llm_pricing", "dokimi.measure", "dokimi.process", "dokimi.process_io", - "dokimi.atomic_write", "dokimi.records", "dokimi.registry", "dokimi.utils", "dokimi.write_probe", + "dokimi.atomic_write", "dokimi.provenance", "dokimi.records", "dokimi.registry", "dokimi.utils", "dokimi.write_probe", "dokimi.judge_rule", # Token/cost record the LLM chokepoint fills; it lives under metrics/ but # depends on nothing and is read by every LLM caller. diff --git a/dokimi/tests/test_provenance.py b/dokimi/tests/test_provenance.py new file mode 100644 index 000000000..92674d26b --- /dev/null +++ b/dokimi/tests/test_provenance.py @@ -0,0 +1,204 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""The optional ``version`` and ``provenance`` of tasks and question-set items (dokimi.provenance). + +Absent is unknown, never an error. A malformed value is refused where the artifact is checked (task.json in +the repo check) and warned about where it is read (a terminal-bench task.toml, which may already carry the key, +and a questions.v2 item, whose schema accepts any value so that old files keep loading). The repo report shows +the status of each task as the cell's result.json recorded it (docs/repo-evaluation-contract.md §3). +""" + +from __future__ import annotations + +import json +import logging + +import pytest + +from dokimi import provenance +from dokimi._version import __version__ +from dokimi.artifacts import schema_errors +from dokimi.cli import build_parser +from dokimi.harnesses.tasks import load_task +from dokimi.questions.artifact import load_questions +from dokimi.repos.manifest import _provenance_problems + +GOOD_BLOCKS = [ + {"source": "hand-written"}, + {"source": "generated:dokimi-tasks-generate@0.2.0", "status": "candidate"}, + {"source": "imported:inspect-ai", "reviewed_by": "alice", "status": "frozen"}, + {"status": "validated"}, +] +BAD_BLOCKS = [ + "validated", + {"sauce": "hand-written"}, + {"source": "made-by-hand"}, + {"source": "generated:dokimi-tasks-generate"}, + {"source": "hand-written\n"}, + {"status": "done"}, + {"reviewed_by": " "}, + {"status": None}, +] + + +@pytest.mark.parametrize("record", [{}, {"provenance": {}}, {"provenance": None}, {"provenance": "validated"}]) +def test_absent_or_unusable_provenance_reads_as_unknown(record): + assert provenance.status_of(record) == provenance.UNKNOWN + + +@pytest.mark.parametrize("status", provenance.STATUSES) +def test_each_status_reads_back(status): + assert provenance.status_of({"provenance": {"status": status}}) == status + + +def test_an_unknown_status_value_reads_as_unknown(): + assert provenance.status_of({"provenance": {"status": "done"}}) == provenance.UNKNOWN + + +@pytest.mark.parametrize("block", GOOD_BLOCKS) +def test_a_well_formed_block_parses_to_itself(block): + assert provenance.parse_provenance(block) == block + + +@pytest.mark.parametrize("block", BAD_BLOCKS) +def test_a_malformed_block_is_refused(block): + with pytest.raises(provenance.ProvenanceError): + provenance.parse_provenance(block) + + +def test_a_block_with_a_non_string_key_is_refused_not_a_type_error(): + with pytest.raises(provenance.ProvenanceError): + provenance.parse_provenance({1: "legacy", "extra": "value"}) + + +def test_status_of_an_invalid_block_reads_as_unknown_even_with_a_valid_status(): + record = {"provenance": {"source": "made-by-hand", "status": "validated"}} + assert provenance.status_of(record) == provenance.UNKNOWN + + +@pytest.mark.parametrize("value", ["2", "2.1", 3, 0]) +def test_a_version_is_a_non_empty_string_or_an_integer(value): + assert provenance.parse_version(value) == value + + +@pytest.mark.parametrize("value", ["", " ", True, 1.5, []]) +def test_a_malformed_version_is_refused(value): + with pytest.raises(provenance.ProvenanceError): + provenance.parse_version(value) + + +def test_a_generator_stamp_names_the_tool_and_the_dokimi_version(): + stamp = provenance.generated_by("tasks-generate") + assert stamp == {"source": f"generated:dokimi-tasks-generate@{__version__}"} + assert provenance.parse_provenance(stamp) == stamp + + +@pytest.mark.parametrize("block", GOOD_BLOCKS) +def test_questions_schema_accepts_what_the_parser_accepts(block): + item = {"id": "q1", "question": "How?", "provenance": block} + assert schema_errors("questions.v2", {"questions": [item]}) == [] + + +@pytest.mark.parametrize("block", BAD_BLOCKS) +def test_questions_schema_accepts_a_malformed_provenance_the_parser_warns_about(block): + item = {"id": "q1", "question": "How?", "provenance": block} + assert schema_errors("questions.v2", {"questions": [item]}) == [] + + +@pytest.mark.parametrize("value", ["2", 3, " ", True, 1.5, None]) +def test_questions_schema_accepts_any_version_so_old_files_still_load(value): + assert schema_errors("questions.v2", {"questions": [{"id": "q1", "question": "How?", "version": value}]}) == [] + + +@pytest.mark.parametrize(("version", "block"), [(1.5, "hand-written"), ("2", {"status": "done"}), (True, None)]) +def test_a_malformed_question_version_or_provenance_warns_and_the_set_still_loads(tmp_path, caplog, version, block): + path = tmp_path / "questions.json" + question = {"id": "q1", "question": "How?", "version": version, "provenance": block} + path.write_text(json.dumps({"schema_version": "questions.v2", "questions": [question]}), encoding="utf-8") + with caplog.at_level(logging.WARNING, logger="dokimi.questions.artifact"): + loaded = load_questions(path) + assert loaded.artifact["questions"][0]["version"] == version + assert loaded.artifact["questions"][0]["provenance"] == block + assert "q1" in caplog.text + + +def test_a_well_formed_question_version_and_provenance_do_not_warn(tmp_path, caplog): + path = tmp_path / "questions.json" + question = {"id": "q1", "question": "How?", "version": "2.1", "provenance": {"source": "hand-written"}} + path.write_text(json.dumps({"schema_version": "questions.v2", "questions": [question]}), encoding="utf-8") + with caplog.at_level(logging.WARNING, logger="dokimi.questions.artifact"): + load_questions(path) + assert not [r for r in caplog.records if r.levelno >= logging.WARNING] + + +@pytest.mark.parametrize("block", GOOD_BLOCKS) +def test_task_json_with_a_good_block_has_no_problem(block): + assert _provenance_problems({"provenance": block, "version": "1"}) == [] + + +def test_task_json_without_the_keys_has_no_problem(): + assert _provenance_problems({"id": "t"}) == [] + + +@pytest.mark.parametrize("record", [{"version": None}, {"provenance": None}]) +def test_task_json_with_a_null_key_is_refused(record): + assert _provenance_problems(record) + + +def test_task_json_with_a_bad_status_names_the_key(): + (problem,) = _provenance_problems({"provenance": {"status": "done"}}) + assert "provenance.status" in problem + + +def test_a_bad_task_toml_metadata_warns_and_still_loads(tmp_path): + task = tmp_path / "t1" + task.mkdir() + (task / "task.toml").write_text('version = "1.0"\n\n[metadata]\nprovenance = { status = "done" }\n') + (task / "instruction.md").write_text("Do it.\n") + with pytest.warns(UserWarning, match="metadata provenance.status must be one of"): + spec = load_task(task) + assert spec.metadata["provenance"] == {"status": "done"} + + +def _cell(tmp_path, name: str, files: dict[str, object]) -> None: + for rel, content in files.items(): + path = tmp_path / "b1" / name / rel + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(content), encoding="utf-8") + + +_RESULT = {"grade": {"status": "ok", "score_only": False, "success": True}, "arm": "a", "model": "m", + "agent": {"timed_out": False, "is_error": False, "over_budget": False, "cost_usd": 0.1, "num_turns": 2}} + + +def test_repo_report_shows_each_tasks_status_and_unknown_without_one(tmp_path, capsys): + _cell(tmp_path, "t1__a__m__r1", {"result.json": {**_RESULT, "task": "t1", "task_status": "validated"}}) + _cell(tmp_path, "t2__a__m__r1", {"result.json": {**_RESULT, "task": "t2"}}) + parser = build_parser() + args = parser.parse_args(["repo", "report", str(tmp_path / "b1"), "--json"]) + args.func(args) + rows = {r["task"]: r["status"] for r in json.loads(capsys.readouterr().out)["data"]["rows"]} + assert rows == {"t1": "validated", "t2": provenance.UNKNOWN} + + +def test_a_task_copy_the_grader_rewrote_does_not_change_the_status(tmp_path, capsys): + # the run dir is read-write for the grader, so its task.json copy is not evidence of the status + _cell(tmp_path, "t4__a__m__r1", {"result.json": {**_RESULT, "task": "t4", "task_status": "candidate"}, + "task.json": {"id": "t4", "provenance": {"status": "frozen"}}}) + parser = build_parser() + args = parser.parse_args(["repo", "report", str(tmp_path / "b1"), "--json"]) + args.func(args) + rows = {r["task"]: r["status"] for r in json.loads(capsys.readouterr().out)["data"]["rows"]} + assert rows == {"t4": "candidate"} + + +def test_a_row_whose_cells_recorded_different_statuses_says_mixed(tmp_path, capsys): + # a batch resumed after the task went from candidate to validated: its cells disagree, the row must say so + _cell(tmp_path, "t5__a__m__r1", {"result.json": {**_RESULT, "task": "t5", "task_status": "candidate"}}) + _cell(tmp_path, "t5__a__m__r2", {"result.json": {**_RESULT, "task": "t5", "task_status": "validated"}}) + parser = build_parser() + args = parser.parse_args(["repo", "report", str(tmp_path / "b1"), "--json"]) + args.func(args) + rows = {r["task"]: r["status"] for r in json.loads(capsys.readouterr().out)["data"]["rows"]} + assert rows == {"t5": "mixed"} diff --git a/dokimi/tests/test_questions_llm_gen.py b/dokimi/tests/test_questions_llm_gen.py index 8e9ecd77e..76f9078ed 100644 --- a/dokimi/tests/test_questions_llm_gen.py +++ b/dokimi/tests/test_questions_llm_gen.py @@ -17,6 +17,7 @@ mock_mod.ChatAnthropic = Mock sys.modules[mod_name] = mock_mod +from dokimi._version import __version__ from dokimi.questions.llm_gen import QuestionGenerator, QUESTION_GENERATION_PROMPT @@ -94,6 +95,15 @@ def test_returns_questions_for_every_persona_and_topic(self, sample_personas, sa for p in sample_personas } assert topics_by_persona == {p["id"]: sorted(sample_topics * 2) for p in sample_personas} + + def test_every_generated_question_carries_the_generator_provenance(self, sample_personas, sample_topics, + mock_llm): + mock_cls = Mock(return_value=mock_llm) + with patch('dokimi.questions.llm_gen.Invoker', mock_cls), \ + patch('dokimi.questions.llm_gen.LANGCHAIN_AVAILABLE', True): + gen = QuestionGenerator() + questions = gen.generate_questions("oneTBB", sample_personas, sample_topics, questions_per_topic=2) + assert {q["provenance"]["source"] for q in questions} == {f"generated:dokimi-questions-persona@{__version__}"} def test_assigns_unique_ids(self, sample_personas, sample_topics, mock_llm): mock_cls = Mock(return_value=mock_llm) diff --git a/dokimi/tests/test_repo_run_task_status.py b/dokimi/tests/test_repo_run_task_status.py new file mode 100644 index 000000000..6d84aedf7 --- /dev/null +++ b/dokimi/tests/test_repo_run_task_status.py @@ -0,0 +1,22 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""The host writes a task's status into result.json, the key the report reads.""" + +from __future__ import annotations + +import json + +from tests.test_repo_run import _run, setup # noqa: F401 # the fixture and the runner of test_repo_run + + +def test_the_host_writes_the_tasks_status_into_result_json(setup): # noqa: F811 + # the report reads the status from this key only: a run that drops the write shows every task as unknown + task_json = setup["root"] / ".agent-eval/tasks/fix_add/task.json" + task = json.loads(task_json.read_text()) + task["provenance"] = {"source": "hand-written", "status": "validated"} + task_json.write_text(json.dumps(task)) + _run(setup, "--arm", "guidance:raw", source=str(setup["root"])) + rec = json.loads((setup["out"] / "b1/fix_add__guidance-raw__m-1__r1/result.json").read_text()) + assert rec["task_status"] == "validated" + diff --git a/dokimi/tests/test_task_runs_report.py b/dokimi/tests/test_task_runs_report.py index 6accbc922..f86fe6749 100644 --- a/dokimi/tests/test_task_runs_report.py +++ b/dokimi/tests/test_task_runs_report.py @@ -113,7 +113,7 @@ def test_rollups_keep_distinct_harnesses(): # both harness ids appear as distinct columns; per-task row shows 0/1 and 1/1 assert "`claude-code` pass" in md assert "`codex` pass" in md - assert "| t1 | easy | 0/1 |" in md # claude-code column = 0/1, not merged with codex + assert "| t1 | easy | unknown | 0/1 |" in md # claude-code column = 0/1, not merged with codex def test_per_task_cost_unknown_renders_dash(): @@ -135,7 +135,7 @@ def test_per_task_cost_unknown_renders_dash(): }, "comparisons": {}}, } md = render_task_runs_report(data) - assert "| t1 | easy | 1/1 | — |" in md + assert "| t1 | easy | unknown | 1/1 | — |" in md assert "$0.0000" not in md @@ -165,7 +165,7 @@ def row(task, harness, passed, error=None): md = render_task_runs_report(data) assert "| `skill` | 100.00% (1/1) | 1 |" in md # headline: scored cells, errored column assert "missing data, not a 0" in md - assert "| t2 | easy | 1/1 | — | — (+1 err) | — |" in md + assert "| t2 | easy | unknown | 1/1 | — | — (+1 err) | — |" in md assert "— (arms lost different cells)" in md # unpaired deltas withheld assert "| `skill` | 1 | null | ERR |" in md assert "(harness error)" in md @@ -198,7 +198,7 @@ def row(task, harness, **extra): } md = render_task_runs_report(data) assert "| `base` | 100.00% (1/1) | 1 |" in md # headline agrees: 1 errored - assert "| t1 | easy | — (+1 err) | — | 1/1 | — |" in md + assert "| t1 | easy | unknown | — (+1 err) | — | 1/1 | — |" in md assert "| `base` | 1 | 1 | ERR |" in md