Skip to content

feat(provenance): optional version and provenance on tasks and question-set items - #800

Open
napetrov wants to merge 23 commits into
intel:mainfrom
napetrov:feat/task-question-provenance
Open

napetrov wants to merge 23 commits into
intel:mainfrom
napetrov:feat/task-question-provenance

Conversation

@napetrov

@napetrov napetrov commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

task.json / questions.v2 item
  version                       # optional, absent = unknown
  provenance                    # optional {source, reviewed_by, status}
    source     hand-written | generated:dokimi-<tool>@<ver> | imported:<format>
    status     candidate | validated | frozen

dokimi/provenance.py            # parse + validate; absent is never an error
  eval tasks generate           # stamps generated:dokimi-tasks-generate@<version>
  eval questions generate       # stamps generated:dokimi-questions-persona@<version>
  pipeline chunk questions      # stamps generated:dokimi-questions-chunk@<version>
  repo report, task-runs report # show a status per task

Evidence

  • No version / provenance on tasks or question-set items before this PR; the reports had no status column.
  • tests/test_repo_run_task_status.py covers the host writing task_status into result.json; tests/test_provenance.py covers the parser and the validation; the task-runs golden (tests/fixtures/report_fmt.json) carries the new Status column.

Merge Danger

Door: two-way. The new keys are optional, and the schema change only adds two properties. Reverting drops the fields without touching existing records.

Blast Radius: two kinds of change.

  • Reports: the Status column is always added to the task-runs report and to repo report text, and repo report JSON rows always carry a status key; a task with no provenance shows unknown. Anything that reads those tables by column position shifts by one. Golden fixtures are regenerated for this.
  • Validation tightening on task.json: repo validate now refuses a malformed version (a float such as 2.0, a null, or a non-string, non-integer value) and a malformed provenance block. Before this PR task.json accepted any value for these keys. A content repo that already sets either key with a malformed value fails repo validate after upgrading until it fixes the value; task.toml only warns in that case. Question-set items only warn, never refuse.

Open questions:

  • questions.v2 version has no type in the schema, so any value passes; the parser warns on a float. Kept as warn-only.
  • The trigger-set round trip drops the new keys. Outside this diff; not fixed here.

🤖 Generated with Claude Code

…on-set items

A repo task.json and a questions.v2 item may carry `version` and
`provenance` ({source, reviewed_by, status}). Absent is unknown, never an
error. `eval tasks generate` and the question generator stamp
`generated:dokimi-<tool>@<version>`. The repo report and the task-runs
report show each task's status, so results can be split by it.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov requested a review from a team as a code owner November 16, 2026 05:27
@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.79%. Comparing base (b62613a) to head (dc2c209).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff            @@
##             main     #800    +/-   ##
========================================
  Coverage   95.79%   95.79%            
========================================
  Files         432      433     +1     
  Lines       57204    57309   +105     
  Branches     9858     9876    +18     
========================================
+ Hits        54796    54901   +105     
  Misses       1497     1497            
  Partials      911      911            
Files with missing lines Coverage Δ
dokimi/dokimi/harnesses/tasks.py 97.38% <100.00%> (+0.05%) ⬆️
dokimi/dokimi/provenance.py 100.00% <100.00%> (ø)
dokimi/dokimi/questions/artifact.py 97.79% <100.00%> (+0.21%) ⬆️
dokimi/dokimi/questions/llm_gen.py 94.62% <100.00%> (+0.30%) ⬆️
dokimi/dokimi/report/task_runs_report.py 93.50% <100.00%> (+0.06%) ⬆️
dokimi/dokimi/repos/graph_tasks.py 95.61% <100.00%> (+0.01%) ⬆️
dokimi/dokimi/repos/manifest.py 94.93% <100.00%> (+0.14%) ⬆️
dokimi/dokimi/repos/replay/authoring.py 98.46% <100.00%> (+0.01%) ⬆️
dokimi/dokimi/repos/replay/build.py 99.36% <100.00%> (+<0.01%) ⬆️
dokimi/dokimi/repos/report.py 98.07% <100.00%> (+0.04%) ⬆️
... and 1 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@exai-bot

exai-bot commented Nov 16, 2026 •

Copy link
Copy Markdown

OpenCodeReview: Commit dc2c209 was not reviewed. This PR has reached its review limit (6 reviews).

CI at `dc2c209`: 12 still running, 4 passing, 1 skipped
Check Status
Build docs (strict) still running
CLI interface diff still running
CodeQL (actions) still running
CodeQL (python) still running
External catalog smoke test (uxlfoundation/skills) still running
Platforms and Python versions still running
Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com... still running
benchmark still running
mypy still running
self / repo scan still running

2 more not listed.

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (a682ece): 1 medium</summary>

Open threads from the bot: 0

</details>

Comment thread dokimi/docs/repo-evaluation-contract.md Resolved
Comment thread dokimi/dokimi/provenance.py Resolved
Comment thread dokimi/dokimi/provenance.py Resolved

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Changes needed: 1 must-fix · reviewed 7f37022

Next: @napetrov: fix the 1 must-fix finding and answer 4 open threads from exai-bot; then push for a re-review.

Blocking

  • Still holds on this head: thread by exai-bot – [metadata] provenance = {...} on one line is rejected by tomllib, so the example in the doc does not load.
  • Still holds on this head: thread by exai-bot – repo check is not a registered subcommand (repo validate is); python cli.py repo check x exits 2 in the contract row and the changelog fragment.
  • Still holds on this head: thread by exai-bot – parse_provenance({1: 'legacy', 'extra': 'value'}) raises TypeError from sorted, which _check_provenance does not catch, instead of ProvenanceError.
  • Still holds on this head: thread by exai-bot – status_of({'provenance': {'source': 'made-by-hand', 'status': 'validated'}}) returns validated though parse_provenance refuses the block; expected unknown.

Verified on 7f37022: ✅ changed tests: 226 passed, fail on main · ✅ full suite with main merged in: 21827 passed, its only failures are main's too · ✅ merges onto main · 1 mutation tried · 3 edge inputs tried

3 checks behind it, each with its evidence
Area What holds How
New tests The changed tests, including tests/test_provenance.py with its good and bad block cases, pass on the head and the new module fails to import on main tests on the head, edge input (structured)
Merge The head merges onto main with no conflict merge onto main
Version check Changing the bool exclusion in parse_version so True is accepted is caught by tests/test_provenance.py broke the code, tests caught it, edge input (text)

Comment thread dokimi/dokimi/schemas/questions.v2.json Resolved
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863y56l
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863y56q
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… as unknown

Thread: PRRT_kwDOVBg7g863y56x

Thread: PRRT_kwDOVBg7g863y567
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

Changes needed: 1 must-fix
Waiting on a decision before I change this: the one must-fix is the questions.v2 item in its thread, and I need your call there. The four exai-bot points are fixed in this round's commits.

Comment thread dokimi/dokimi/repos/report.py Resolved
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

# Conflicts:
#	dokimi/dokimi/repos/report.py
A question written before the keys existed loaded fine, and the schema
refused it once it carried version 1.5 or a string provenance. The schema
now accepts any value; the parser logs a WARNING for a malformed one and
keeps it as written, so one value no longer refuses the whole set.

Thread: PRRT_kwDOVBg7g8630Azk
Thread: PRR_kwDOVBg7g88AAAABYKCnEQ
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The status came from the cell's task.json copy in the run dir, which the
grader had read-write and could rewrite to any status. The runner now
records the task's provenance status in result.json, which the host
writes, and `repo report` reads it there.

Thread: PRRT_kwDOVBg7g864E-ZT
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The question's provenance is not read as a status anywhere, so the schema
description no longer says it reads as unknown.

Thread: PRRT_kwDOVBg7g8630Azk
Thread: PRR_kwDOVBg7g88AAAABYKCnEQ
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…upt one

The report no longer opens the run dir's task.json, so the test of a corrupt
copy did not test anything. The no-warning test now asserts no WARNING record.

Thread: PRRT_kwDOVBg7g864E-ZT
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

1 must-fix: the questions.v2 item
Fixed in 9f3a95c: old files load.

The batch golden pins every key of result.json. task_status is written only
when the task carries a status, so a task without provenance writes the same
record as before; the report reads an absent key as unknown.

Thread: PRRT_kwDOVBg7g864E-ZT
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Comment thread dokimi/dokimi/repos/runner.py Resolved
Comment thread dokimi/dokimi/repos/runner.py Resolved
Comment thread dokimi/dokimi/repos/runner.py Resolved
Comment thread dokimi/dokimi/repos/runner.py Resolved
Comment thread dokimi/dokimi/repos/runner.py Resolved
Comment thread dokimi/dokimi/repos/runner.py Resolved
…-question-provenance

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864IAxy: already fixed in afcde91
Thread: PRRT_kwDOVBg7g864IAyA: already fixed in afcde91
Thread: PRRT_kwDOVBg7g864IAyI: already fixed in afcde91
Thread: PRRT_kwDOVBg7g864IAyP: already fixed in afcde91
Thread: PRRT_kwDOVBg7g864IAyW: already fixed in afcde91
Thread: PRRT_kwDOVBg7g864IAyb: already fixed in afcde91
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at a682ece. The feature works as described, and the points from the earlier rounds are fixed. What is left: the host-side task_status write has no test, and two labels are misleading. Details are in the line comments.

What I ran, at a682ece:

  • make ci-local --offline: 0 failed, 23 passed, 20 skipped (network-only jobs); the test suite had 22,332 passed and 72 skipped.
  • A mutation run: I removed the task_status write in run_cell and the provenance stamps in replay/build.py and replay/authoring.py. The full suite still passed.

Checked, no issues

  • Earlier threads: the TOML example now has [metadata] on its own line. The docs say repo validate. Non-string keys raise ProvenanceError, not TypeError. status_of returns unknown for a malformed block. questions.v2 accepts any value and only warns. repo report reads the status from the host-written result.json, not from the run dir's task.json copy. result.json has no new key for a task without a status, so the golden key list does not change.
  • Compatibility: no score, default or prompt changes. The questions.v2 change only widens the schema. typecheck's string version still passes parse_version.
  • Docs and changelog: the fragment is under Added, has the license header, and names the commands and the warn/fail split correctly.

Comment on lines +311 to +312
if status_of(task.raw) != UNKNOWN: # absent means unknown to the report: a task without one writes no key
record["task_status"] = status_of(task.raw)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No test covers the host writing task_status into result.json, and nothing else in the status path is tested end to end.

The repo-report tests build result.json by hand. With these two lines removed, and the provenance stamps removed from replay/build.py:221 and replay/authoring.py:141, the full suite still passes (22,332 passed), while repo report would show unknown for every task. The task-runs report tests also only ever render unknown.

Suggestion: a run_cell test with a fake harness on a task with provenance.status, checking result.json; a stamp check for the two replay generators; one task-runs report row with a known status.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a: Test added: result.json carries task_status. b: Replay stamp tests are outside this PR, in #896.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a: Test added: result.json carries task_status. b: Accepted as a follow-up in #896; this thread stays open for the reviewer to resolve.

Comment thread dokimi/dokimi/questions/llm_gen.py Resolved
Comment thread dokimi/dokimi/repos/report.py Resolved
…venance

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

# Conflicts:
#	dokimi/dokimi/report/task_runs_report.py
…enance

Thread: PRRT_kwDOVBg7g864ZXoB
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864ZXoM
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864ZXoN
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Verified against a mutant that drops the write: the test fails.

Thread: PRRT_kwDOVBg7g864ZXn1
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… merge

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
test_repo_run.py is at the 1000-line cap (scripts/check_growth.py); the test moves to test_repo_run_task_status.py.

Thread: PRRT_kwDOVBg7g864ZXn1
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

Item: PRRT_kwDOVBg7g864ZXn1.a
Review: PRR_kwDOVBg7g88AAAABYePnSw
The test is the same as before; the fixture import is marked for F811.

Thread: PRRT_kwDOVBg7g864ZXn1
Review: PRR_kwDOVBg7g88AAAABYePnSw
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vlad-perevezentsev

What is left: the task_status test gap and two labels
Test added and both labels fixed, see the threads.

The methodology page and the changelog entry named one source stamp
for every generated question. The code writes generated:dokimi-questions-persona@<version> for persona questions and generated:dokimi-questions-chunk@<version> for chunk-grounded ones.

Thread: PRRT_kwDOVBg7g864ZXoB
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov force-pushed the feat/task-question-provenance branch from 269353c to dc2c209 Compare November 18, 2026 00:02
@napetrov

Copy link
Copy Markdown
Contributor Author

What is left: the task_status test gap and two labels
Test added and both labels fixed, see the threads.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants