Repository navigation
fix(novelty): routing partial-file guard reads status "partial" alone - #764
Conversation
…intel#700) `_is_partial_file` in `novelty_routing_report.py` tested only the `partial` key, so a routing file holding `{"status": "partial"}` was taken for a finished result and a capped run wrote its partial result to a sidecar instead of replacing it. It now tests `"partial" in data or data.get("status") == "partial"`, the marker the atoms reader and `artifacts.py` use. `_refuse_partial` in `novelty/items.py` already refuses a probe file marked by status alone (landed with intel#633, pinned by `test_novelty_items_refuses_a_probe_file_marked_partial_by_status_alone`), so only the routing guard changes here. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 3 selected item(s). CI at Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #764 +/- ##
=======================================
Coverage 95.72% 95.72%
=======================================
Files 415 415
Lines 55363 55363
Branches 9516 9516
=======================================
Hits 52994 52994
Misses 1466 1466
Partials 903 903
🚀 New features to boost your workflow:
|
vchamarthi
left a comment
There was a problem hiding this comment.
Ready to merge at 740119e: _is_partial_file now also treats {"status": "partial"} as a partial file, matching the atoms reader and load_artifact; it does so, and nothing to fix was found.
Verified
- Status-only marker:
{"status": "partial"}is a partial file on the head and not on main; the new test passes on the head and fails on main. - Changed tests: All 51 tests in
tests/test_novelty_routing_spend_cap_partial.pypass on the head. - Test pins the clause: Dropping the
statusclause from_is_partial_filefails the new test. - Finished and odd files:
{"status": "ok"},[],nulland{}are not partial files on the head. - Merge with main: Merges onto main with no conflict and the whole local run passes.
napetrov
left a comment
There was a problem hiding this comment.
Change class: not computed (no receipt profile for this repo, or receipt.py failed)
Approve with nits at 740119e: no blocking finding; CI green on head (0 failing jobs). 1 nit inline.
Pre-existing, follow-up: _is_partial_file catches OSError/ValueError but not RecursionError: a ~20000-deep [ file makes save_result raise after a paid capped run (same on base); other readers (read_json_input, eval/cells.py) catch it.
Coverage: plain + edge pass on head and merge base, probes run; full suite and py3.14 leg not run locally (CI covers them).
| @@ -531,7 +531,7 @@ def _is_partial_file(path: Path) -> bool: | |||
| earlier = json.loads(path.read_text(encoding="utf-8")) | |||
There was a problem hiding this comment.
Anchored at line 531; the finding is at line 528, outside the diff.
nit: the change also reaches the atoms, probe and analyze partial sidecars, not just novelty routing
dokimi/dokimi/commands/novelty_routing_report.py:528: _is_partial_file is shared through _partial_destinations/_takes_partial, used by novelty_budget.partial_destination/partial_write. So a status-only <stem>.partial.json is now overwritten in place by capped atoms/probe/analyze runs; base wrote <stem>.partial2.json.
How to check: atoms.partial.json = {"status":"partial"}, capped atoms run → head writes atoms.partial.json (base: atoms.partial2.json, changed).
Expected: Probably intended consistency; the PR body and changelog fragment name routing only.
Fix: Mention the sidecar paths in the changelog fragment, and add one test on partial_destination with a status-only file.
…793) Settles the review point of #764 that `_is_partial_file` caught OSError and ValueError but not RecursionError: a routing file holding about 20000 `[` made `save_result` raise after a paid, capped run. `_is_partial_file` now also catches RecursionError, so such a file is treated like one this cannot read: not a partial result, and the capped run writes its partial result to the sidecar beside it. A test pins both the guard and `save_result` over a 20000-deep file. Changelog fragment (Fixed). Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Summary
Two result-file guards read a file marked only by
"status": "partial"as finished, where the same file with apartialblock is treated as partial (#700, the same cause #642/#702 fixed in the atoms reader).mainnovelty/items.py_refuse_partial{"status": "partial"}ProbeFileErrorsince #633 (items.py:271), pinned bytest_novelty_items_refuses_a_probe_file_marked_partial_by_status_alonecommands/novelty_routing_report.py_is_partial_file{"status": "partial"}False: a capped run saw a finished result and wrote its partial result torouting.partial.jsonTrue: the capped run replaces the fileload_artifactinartifacts.py:341already tests"partial" in data or data.get("status") == "partial", as does the atoms reader (atoms.py:786). The remaining reads ofpartialinnovelty_routing_report.py(:392,:477,:647) are on in-memory payloads the run built, not on files.Closes #700
Changes
_is_partial_fileindokimi/commands/novelty_routing_report.pytests"partial" in earlier or earlier.get("status") == "partial", the marker the atoms reader uses. Files with apartialblock, with neither marker, and non-dict JSON are handled as before.tests/test_novelty_routing_spend_cap_partial.py:_is_partial_fileis true for{"status": "partial"}, andsave_resultwrites a partial payload over it.Risk
A routing file holding only
{"status": "partial"}is now overwritten by a capped run instead of kept with the partial result going to a sidecar. Files written by dokimi carry both markers, so only a hand-edited or foreign file is affected.Testing
On Linux, at
740119e(onmain7137604):make ci-local: 33 passed, 0 failed, 14 skipped (steps needing micromamba, semgrep, other Python versions or a PR context). Not run: the matrix legs on other Pythons.main's code atassert _is_partial_file(marked)and passes here. The guard tests (test_docs_code_refs,test_growth_ratchet,test_audit_test_quality,test_import_architecture) pass.assert _is_partial_file(marked); droppingnot _is_partial_file(out)fromsave_resultfailsassert save_result(...) == marked.fix/cache-dir-unusableonmain7137604:pytest -n auto20449 passed, 19 skipped,mypyclean,ruff checkclean.Revert
A plain revert restores the old guard. Nothing is stored.
Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.md.github/.licenserc.yaml)