Repository navigation
Conversation
…on-set items
A repo task.json and a questions.v2 item may carry `version` and
`provenance` ({source, reviewed_by, status}). Absent is unknown, never an
error. `eval tasks generate` and the question generator stamp
`generated:dokimi-<tool>@<version>`. The repo report and the task-runs
report show each task's status, so results can be split by it.
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #800 +/- ##
========================================
Coverage 95.79% 95.79%
========================================
Files 432 433 +1
Lines 57204 57309 +105
Branches 9858 9876 +18
========================================
+ Hits 54796 54901 +105
Misses 1497 1497
Partials 911 911
🚀 New features to boost your workflow:
|
|
OpenCodeReview: Commit dc2c209 was not reviewed. This PR has reached its review limit (6 reviews). CI at `dc2c209`: 12 still running, 4 passing, 1 skipped
2 more not listed. To review this commit: re-request review from exai-bot in the Reviewers list. <details><summary>Last review (a682ece): 1 medium</summary> Open threads from the bot: 0 </details> |
vchamarthi
left a comment
There was a problem hiding this comment.
🔴 Changes needed: 1 must-fix · reviewed 7f37022
Next: @napetrov: fix the 1 must-fix finding and answer 4 open threads from exai-bot; then push for a re-review.
Blocking
- Still holds on this head: thread by exai-bot –
[metadata] provenance = {...}on one line is rejected bytomllib, so the example in the doc does not load. - Still holds on this head: thread by exai-bot –
repo checkis not a registered subcommand (repo validateis);python cli.py repo check xexits 2 in the contract row and the changelog fragment. - Still holds on this head: thread by exai-bot –
parse_provenance({1: 'legacy', 'extra': 'value'})raisesTypeErrorfromsorted, which_check_provenancedoes not catch, instead ofProvenanceError. - Still holds on this head: thread by exai-bot –
status_of({'provenance': {'source': 'made-by-hand', 'status': 'validated'}})returnsvalidatedthoughparse_provenancerefuses the block; expectedunknown.
Verified on 7f37022: ✅ changed tests: 226 passed, fail on main · ✅ full suite with main merged in: 21827 passed, its only failures are main's too · ✅ merges onto main · 1 mutation tried · 3 edge inputs tried
3 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| New tests | The changed tests, including tests/test_provenance.py with its good and bad block cases, pass on the head and the new module fails to import on main |
tests on the head, edge input (structured) |
| Merge | The head merges onto main with no conflict | merge onto main |
| Version check | Changing the bool exclusion in parse_version so True is accepted is caught by tests/test_provenance.py |
broke the code, tests caught it, edge input (text) |
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863y56l Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863y56q Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… as unknown Thread: PRRT_kwDOVBg7g863y56x Thread: PRRT_kwDOVBg7g863y567 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/dokimi/repos/report.py
A question written before the keys existed loaded fine, and the schema refused it once it carried version 1.5 or a string provenance. The schema now accepts any value; the parser logs a WARNING for a malformed one and keeps it as written, so one value no longer refuses the whole set. Thread: PRRT_kwDOVBg7g8630Azk Thread: PRR_kwDOVBg7g88AAAABYKCnEQ Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The status came from the cell's task.json copy in the run dir, which the grader had read-write and could rewrite to any status. The runner now records the task's provenance status in result.json, which the host writes, and `repo report` reads it there. Thread: PRRT_kwDOVBg7g864E-ZT Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The question's provenance is not read as a status anywhere, so the schema description no longer says it reads as unknown. Thread: PRRT_kwDOVBg7g8630Azk Thread: PRR_kwDOVBg7g88AAAABYKCnEQ Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…upt one The report no longer opens the run dir's task.json, so the test of a corrupt copy did not test anything. The no-warning test now asserts no WARNING record. Thread: PRRT_kwDOVBg7g864E-ZT Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
The batch golden pins every key of result.json. task_status is written only when the task carries a status, so a task without provenance writes the same record as before; the report reads an absent key as unknown. Thread: PRRT_kwDOVBg7g864E-ZT Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…-question-provenance Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864IAxy: already fixed in afcde91 Thread: PRRT_kwDOVBg7g864IAyA: already fixed in afcde91 Thread: PRRT_kwDOVBg7g864IAyI: already fixed in afcde91 Thread: PRRT_kwDOVBg7g864IAyP: already fixed in afcde91 Thread: PRRT_kwDOVBg7g864IAyW: already fixed in afcde91 Thread: PRRT_kwDOVBg7g864IAyb: already fixed in afcde91 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at a682ece. The feature works as described, and the points from the earlier rounds are fixed. What is left: the host-side task_status write has no test, and two labels are misleading. Details are in the line comments.
What I ran, at a682ece:
make ci-local --offline: 0 failed, 23 passed, 20 skipped (network-only jobs); the test suite had 22,332 passed and 72 skipped.- A mutation run: I removed the
task_statuswrite inrun_celland theprovenancestamps inreplay/build.pyandreplay/authoring.py. The full suite still passed.
Checked, no issues
- Earlier threads: the TOML example now has
[metadata]on its own line. The docs sayrepo validate. Non-string keys raiseProvenanceError, notTypeError.status_ofreturnsunknownfor a malformed block.questions.v2accepts any value and only warns.repo reportreads the status from the host-written result.json, not from the run dir's task.json copy. result.json has no new key for a task without a status, so the golden key list does not change. - Compatibility: no score, default or prompt changes. The
questions.v2change only widens the schema.typecheck's stringversionstill passesparse_version. - Docs and changelog: the fragment is under
Added, has the license header, and names the commands and the warn/fail split correctly.
| if status_of(task.raw) != UNKNOWN: # absent means unknown to the report: a task without one writes no key | ||
| record["task_status"] = status_of(task.raw) |
There was a problem hiding this comment.
No test covers the host writing task_status into result.json, and nothing else in the status path is tested end to end.
The repo-report tests build result.json by hand. With these two lines removed, and the provenance stamps removed from replay/build.py:221 and replay/authoring.py:141, the full suite still passes (22,332 passed), while repo report would show unknown for every task. The task-runs report tests also only ever render unknown.
Suggestion: a run_cell test with a fake harness on a task with provenance.status, checking result.json; a stamp check for the two replay generators; one task-runs report row with a known status.
There was a problem hiding this comment.
a: Test added: result.json carries task_status. b: Replay stamp tests are outside this PR, in #896.
There was a problem hiding this comment.
a: Test added: result.json carries task_status. b: Accepted as a follow-up in #896; this thread stays open for the reviewer to resolve.
…venance Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/dokimi/report/task_runs_report.py
…enance Thread: PRRT_kwDOVBg7g864ZXoB Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864ZXoM Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864ZXoN Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Verified against a mutant that drops the write: the test fails. Thread: PRRT_kwDOVBg7g864ZXn1 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… merge Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
test_repo_run.py is at the 1000-line cap (scripts/check_growth.py); the test moves to test_repo_run_task_status.py. Thread: PRRT_kwDOVBg7g864ZXn1 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Item: PRRT_kwDOVBg7g864ZXn1.a Review: PRR_kwDOVBg7g88AAAABYePnSw
The test is the same as before; the fixture import is marked for F811. Thread: PRRT_kwDOVBg7g864ZXn1 Review: PRR_kwDOVBg7g88AAAABYePnSw Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Re: review by vlad-perevezentsev
|
The methodology page and the changelog entry named one source stamp for every generated question. The code writes generated:dokimi-questions-persona@<version> for persona questions and generated:dokimi-questions-chunk@<version> for chunk-grounded ones. Thread: PRRT_kwDOVBg7g864ZXoB Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
269353c to
dc2c209
Compare
|
Summary
Evidence
version/provenanceon tasks or question-set items before this PR; the reports had no status column.tests/test_repo_run_task_status.pycovers the host writingtask_statusinto result.json;tests/test_provenance.pycovers the parser and the validation; the task-runs golden (tests/fixtures/report_fmt.json) carries the new Status column.Merge Danger
Door: two-way. The new keys are optional, and the schema change only adds two properties. Reverting drops the fields without touching existing records.
Blast Radius: two kinds of change.
repo reporttext, andrepo reportJSON rows always carry astatuskey; a task with no provenance showsunknown. Anything that reads those tables by column position shifts by one. Golden fixtures are regenerated for this.task.json:repo validatenow refuses a malformedversion(a float such as2.0, a null, or a non-string, non-integer value) and a malformedprovenanceblock. Before this PRtask.jsonaccepted any value for these keys. A content repo that already sets either key with a malformed value failsrepo validateafter upgrading until it fixes the value;task.tomlonly warns in that case. Question-set items only warn, never refuse.Open questions:
questions.v2versionhas no type in the schema, so any value passes; the parser warns on a float. Kept as warn-only.🤖 Generated with Claude Code