Skip to content

fix: settle two leftovers of the reviews of #628 and #828 - #856

Merged
antonwolfy merged 4 commits into
intel:mainfrom
antonwolfy:test/formal-and-report-leftovers
Nov 17, 2026
Merged

antonwolfy merged 4 commits into
intel:mainfrom
antonwolfy:test/formal-and-report-leftovers

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Follow-up to the reviews of #628 and #828, which left two small points open on main.

Changes

  • _SOURCE_EXIT (the description of repo run and eval check), the tasks generate help, four paragraphs of docs/repo-evaluation-contract.md (the repo run/eval check one and the --from replay, --from graph and --from prs sections) and the exit-code paragraph of docs/quickstart.md name "a --cache-dir that is not a usable directory" among the exit-2 cases.
  • tests/test_cli_help.py: the --help of repo run and eval check, and the eval tasks --help listing that holds the tasks generate description, name the --cache-dir case.
  • tests/test_repo_report_unreadable_cell.py: the mixed-execution refusal starts with the batch name the cells record, and with where when they record none.
  • Changelog fragment (Fixed), one bullet for the help text and docs.

Risk

None: help and doc text and two tests. The behaviour of every command is unchanged.

Testing

On a Linux review host, at 66536e1 (on main 08f393c):

  • make ci-local: 34 passed, 0 failed, 14 skipped (matrix legs on other Pythons, semgrep, conda-forge, fuzz, the CLI-surface package step and the jobs that do not run on a PR). 22392 tests passed, 19 skipped; changed-line coverage 100% (no measurable changed line).
  • The help test and the doc, growth, test-quality and import-architecture guards: 1467 passed.
  • Mutants: {batch or where} to {where} and to {batch} each fail test_the_mixed_execution_refusal_starts_with_the_batch_name_or_else_where (one of its two cases); removing the --cache-dir phrase from _SOURCE_EXIT fails both cases of test_the_exit_2_help_of_a_source_command_names_the_cache_dir_case; removing it from the tasks generate help fails its case.
  • A --cache-dir naming a regular file exits 2 with "--cache-dir ... is not a directory" for repo run, eval check, tasks generate --from replay and --from prs; --from graph exits 2 on a bad index first, and reads the cache directory after it.
  • Not run: the other Python versions (3.11, 3.12, 3.14), semgrep.

Open questions

None.

Revert

A plain revert restores the old help and doc text and drops the two tests. Nothing is stored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

Settles review point B1.1 of intel#628: the formal-model README and the RepoBatch.tla comment said the batch result is written with mkstemp + os.replace, but the code writes a `<name>.<pid>.<thread>.tmp` sibling of the target and then calls os.replace (artifacts.py). Only the README row and the TLA+ comment change; no state or action of the model is touched.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…nd eval check

Settles the review of intel#828 that left the exit-2 text stale: `_SOURCE_EXIT` (the help of `repo run` and `eval check`), the `tasks generate` help and the repository-evaluation contract now name a `--cache-dir` that is not a usable directory, which these commands refuse with exit 2. A test pins the phrase in the help of both commands.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Settles review point B2.1 of intel#628: nothing pinned the `{batch or where}` label that starts the refusal in refuse_mixed_executions, so replacing it with `{where}` kept the suite green. The new test checks that the message starts with the batch name the cells record, and with `where` when they record none. Adds the changelog fragment for this and the help text.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy
antonwolfy requested a review from a team as a code owner November 17, 2026 10:28
@antonwolfy antonwolfy self-assigned this Nov 17, 2026
@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 7 selected item(s).

CI at 66536e1: 28 passing, 3 skipped.

Review details
  • Commit reviewed: 66536e1 against merge base 08f393c
  • Coverage: 7 files in 2 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 187k tokens (137k from cache), 19s
  • Tool calls: 21 (13 file_read, 6 code_search, 1 ci_status, 1 local_check_results)
  • Cache-directory CLI help cleanup: 5 files
  • Repository evaluation error contract and unreadable-cell coverage: 2 files

@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.76%. Comparing base (08f393c) to head (66536e1).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #856   +/-   ##
=======================================
  Coverage   95.76%   95.76%           
=======================================
  Files         424      425    +1     
  Lines       56511    56472   -39     
  Branches     9730     9718   -12     
=======================================
- Hits        54115    54079   -36     
+ Misses       1487     1486    -1     
+ Partials      909      907    -2     
Files with missing lines Coverage Δ
dokimi/dokimi/commands/repo_run.py 95.80% <ø> (ø)
dokimi/dokimi/commands/tasks_generate.py 93.63% <ø> (ø)

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 9e5c257. The --cache-dir exit-2 case is right in the help and in §4.6/§4.8 of the contract, and the new tests pin it. But the formal-model change describes a temp file the runner does not use, and two docs still miss the case. Details are in the line comments.

What I ran, at 9e5c257:

  • make ci-local --offline: 0 failed, 23 passed, 20 skipped; tests 22,338 passed, 72 skipped.
  • eval tasks generate --from graph with a bad --cache-dir: the index is read first (exit 2 on a bad index); after it, _source_tree checks the cache dir, so the graph path has the exit-2 case too.

Checked, no issues

  • Help texts: _SOURCE_EXIT feeds both CHECK_EXIT and RUN_EXIT, so eval check and repo run both get the clause; the new help test fails on the base, which lacks it.
  • Mixed-execution test: refuse_mixed_executions starts with the recorded batch, else the label it is given; the test checks both.

Comment thread dokimi/formal/repo_batch/README.md Resolved
Comment thread dokimi/formal/repo_batch/RepoBatch.tla Resolved
Comment thread dokimi/docs/repo-evaluation-contract.md Resolved
Comment thread dokimi/tests/test_cli_help.py Resolved
…o more docs

Settles the review threads on intel#856: the formal-model README row and the RepoBatch.tla comment describe the write of result.json as mkstemp + os.replace again, since the runner writes through batch_ledger.write_json, which uses mkstemp.

The --from graph section of the repository-evaluation contract (§4.7) and the exit-code paragraph of the quickstart now list a --cache-dir that is not a usable directory among the exit-2 cases.

The help test also covers the eval tasks listing that holds the generate description, and the changelog fragment drops the bullet about the pinned mixed-execution label, which changes nothing a user sees.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy antonwolfy changed the title fix: settle three leftovers of the reviews of #628 and #828 fix: settle two leftovers of the reviews of #628 and #828 Nov 17, 2026
@antonwolfy

Copy link
Copy Markdown
Contributor Author

(commented) Review at 9e5c257. The --cache-dir exit-2 case is right in the help and in §4.6/§4.8 of the c

@vlad-perevezentsev (#856 (review)): The line comments are answered in their threads (66536e1): the mkstemp wording is restored in the README and RepoBatch.tla, and the --from graph section of the contract and the quickstart name the --cache-dir case.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I checked 66536e1: every point of my first round is fixed. The formal model reads "mkstemp + os.replace" again, as the code does, and the --cache-dir exit-2 case is now in the --from graph section and in the quickstart too.

Merge: safe now: it merges cleanly into current main, 5 commits behind, and its tests pass on the merge (test_cli_help.py, test_repo_report_unreadable_cell.py, test_docs_code_refs.py: 357 passed). No open PR conflicts with it once it is in.

What I ran, at 66536e1:

  • make ci-local --offline: 1 failed, 22 passed, 20 skipped. The failure is schema_compat: it compares against a newer main that has repo_event.v1.json, which this branch predates. This PR changes no schema.
  • eval tasks --help shows the new tasks generate text, which the new test case checks.

@antonwolfy
antonwolfy merged commit 2bc3c97 into intel:main Nov 17, 2026
31 checks passed
@antonwolfy
antonwolfy deleted the test/formal-and-report-leftovers branch November 17, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants