Repository navigation
Conversation
A suite_manifest.v1 file in the eval tree lists tasks whose base is a commit of another repository (a shared scaffold), each repository referenced as <url>@<sha>. `repo run --suite` resolves each listed task's base in the repository it names. A repository that cannot be opened, or a base that is not one of its commits, leaves those cells unsupported (unsupported.json), never failed or missing. Revisions must be full SHAs; a branch, tag or ref needs --allow-floating-suite-revs and is pinned in the batch (key `suite`). Tests use local git repositories only. Contract section 2.1. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): ~ `dokimi eval check` parameters changed:
+ added option `--allow-floating-suite-revs`
+ added option `--suite`
~ `dokimi repo run` parameters changed:
+ added option `--allow-floating-suite-revs`
+ added option `--suite`Updated on every push; the |
|
OpenCodeReview: Commit d1229e1 was not reviewed. This PR has reached its review limit (6 reviews). CI at `d1229e1`: 15 still running, 5 passing, 1 skipped
5 more not listed. To review this commit: re-request review from exai-bot in the Reviewers list. <details><summary>Last review (4c791c9): 2 medium</summary> Open threads from the bot: 0 </details> |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #783 +/- ##
==========================================
+ Coverage 95.79% 95.80% +0.01%
==========================================
Files 432 433 +1
Lines 57204 57395 +191
Branches 9858 9887 +29
==========================================
+ Hits 54796 54988 +192
+ Misses 1497 1496 -1
Partials 911 911
🚀 New features to boost your workflow:
|
antonwolfy
left a comment
There was a problem hiding this comment.
The base resolution itself works: a suite task takes its base from the named repository, a missing repository leaves only that task unsupported, and meta.json records base_repo. The problems are in the batch pin, the self-check exit code and the SHA check, and each of them contradicts §2.1 as written. Inline comments cover the ones I would fix before merge.
What I ran, on a Linux review host at eb89306:
make ci-local: 30 passed, 3 failed, 14 skipped. The 3 failures are not the PR's: the gitleaks download returned HTTP 504 and the commit scan that depends on it did not run, and the tests step had 20533 passed with 2 pytest-timeout errors intest_audit_test_quality.py, a file this PR does not touch, which passes alone (172 passed)- Reproductions for the findings below with local git repositories and a fake runner, no network.
- A mutation run of
tests/test_repo_suite.pyagainst scratch copies of the new code: 4 of 10 mutants of thecheckand_preparepaths pass the suite (no test callseval check --suite), and 4 of 4 changes to_SHA_REpass it.
Open bot points I confirm:
- exai-bot is right that a resumed floating batch does not reuse the pinned commits, contrary to §2.1 (
repo-evaluation-contract.md:342). New evidence: withsc@main, the first run pinnedsuite: {"sc": "bbb43d6d…"}; after one new commit onmain, the same command and batch exits 1 withbatch b1 was run from suite {'sc': 'bbb43d6d…'}, not suite {'sc': 'c302794b…'}; use another --batch. - exai-bot is right that
repo tasks checkexits 1 when every selected task is a suite gap, against §2.1 (repo-evaluation-contract.md:347). New evidence: one scaffold unreachable and--tasknaming only that task gives0/1 ok, then✗ incomplete: no task passed the self-check (0 ok, 0 unsupported)and exit 1; adding a passing task exits 0. - exai-bot is right that the pin must carry the task-to-repository mapping. New evidence: two suite files in one tree with the same
reposand differenttasksare both accepted when the same batch is resumed.
Checked, no issues
- Schema registration:
suite_manifest.v1is an input schema likerepo_manifest.v1, so it is correctly absent fromSCHEMA_VERSIONS. The fixture exists andtest_schema_compat.pypasses. - Example and docs:
data/suites/example-suite.yamlparses withparse_suite; the contract page is already inmkdocs.yml;--suiteand--allow-floating-suite-revsare registered incli_flags.pyand the CLI reference is regenerated. - Sign-off: the single commit eb89306 carries a
Signed-off-by. - Mutants killed: dropping the duplicate-task, undeclared-repo, unpinned or floating checks, the
_refuse_escapecall, the batch-pin stamp, thebase_repometa, therun_cellgit_dir_for, or therun_batchgap verdict fails the suite.
Bot points I checked that no longer apply at this head (1)
- exai-bot's point on the egress precedence in
selfcheck.pydoes not apply: a task that declares egress undernetwork: nonefailsrepo tasks checkas egress-unsupported with or without this PR, so the exit code is unchanged and only the reason text leaves out the missing repository.
Fixes on the suite manifest: - a pin must be a full 40 or 64 hex commit; a short SHA or a hex-named branch is no longer taken as one - a pinned local path that does not exist leaves its tasks unsupported instead of refusing the whole suite - a base missing from a warm remote cache is fetched once before it is recorded as a gap - the batch pin records the suite file's path, the task-to-repository mapping and the repository locations; resuming with another suite file, or without --suite, is refused - only the suite's tasks are loaded, so an unrelated malformed task.json does not stop the run - the guidance revision of an arm is read from the eval tree while the base is checked out from the scaffold - check_arms and the .mcp.json check skip suite tasks, whose base is not in the eval tree, so a gap no longer aborts the batch - repo tasks check exits 0 when every selected task is a suite gap, and the gap is judged before the egress requirement - a suite file that is not UTF-8 is a SuiteError Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
|
Pushed 1 commit to Changes
Needs @napetrov
Threads and comments
Not this PR's
Verified (12)
|
- check_arms and check_mcp_configs read each available suite task's base in its scaffold repository; only suite gaps are skipped - a pin that resolves to another commit (a branch named like a hash) is refused, except under --allow-floating-suite-revs, which resolves it - a repos key that is not a string is a SuiteError, not a TypeError - the non-string name check is a helper, so parse_suite stays under the complexity cap - the scaffold MCP test runs through repo run, so it fails when the suite base dirs are not passed to check_mcp_configs Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
|
🟡 Waiting on @napetrov · 1 commit on Next: @napetrov: answer the 1 question below. @napetrov: correct the PR description (below). Changes
Needs @napetrov
Threads and comments
Verified on the new head: ✅ tests of the fix: 5 passed · ✅ full suite with main merged in: 21795 passed, 3 failures that main has too · ✅ second reader: 2 raised, all fixed What each test shows
|
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The review threads on this PR are answered by commits already on the branch (c873266 and 4b80490) and by a merge of intel/main. This commit names them. Thread: PRRT_kwDOVBg7g863m7sG: already fixed in c873266 Thread: PRRT_kwDOVBg7g863m7sI: already fixed in c873266 Thread: PRRT_kwDOVBg7g863m7sJ: already fixed in c873266 Thread: PRRT_kwDOVBg7g863m7sN: already fixed in c873266 Thread: PRRT_kwDOVBg7g863m7sQ: already fixed in c873266 Thread: PRRT_kwDOVBg7g863m7sS: already fixed in c873266 Thread: PRRT_kwDOVBg7g863m7sT: already fixed in c873266 Thread: PRRT_kwDOVBg7g863m7sV: already fixed in c873266 Thread: PRRT_kwDOVBg7g863m7sX: already fixed in c873266 Thread: PRRT_kwDOVBg7g863m7sa: already fixed in c873266 Thread: PRRT_kwDOVBg7g863m7sd: already fixed in c873266 Thread: PRRT_kwDOVBg7g863paUe: already fixed in c873266 Thread: PRRT_kwDOVBg7g863paUg: already fixed in c873266 Thread: PRRT_kwDOVBg7g863paUk: already fixed in c873266 Thread: PRRT_kwDOVBg7g863paUn: already fixed in c873266 Thread: PRRT_kwDOVBg7g863paUq: already fixed in c873266 Thread: PRRT_kwDOVBg7g863paUr: already fixed in c873266 Thread: PRRT_kwDOVBg7g863paUt: already fixed in c873266 Thread: PRRT_kwDOVBg7g863paUw: already fixed in c873266 Thread: PRRT_kwDOVBg7g863sVHe: already fixed in 4b80490 Thread: PRRT_kwDOVBg7g863sVHh: already fixed in 4b80490 Thread: PRRT_kwDOVBg7g863sVHk: already fixed in 4b80490 Thread: PRR_kwDOVBg7g88AAAABYE3Cvg: already fixed in c873266 Thread: IC_kwDOVBg7g88AAAABm3m9qg: accept, PR description corrected to the resume behaviour Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A floating suite revision is resolved when the run starts and pinned in the batch. A resume after the branch moved is refused, so the contract now says so instead of promising reuse. The PR description says the same. Thread: PRRT_kwDOVBg7g863m7sD: accept, contract changed to refuse the resume Thread: IC_kwDOVBg7g88AAAABmzV5cw: accept, same decision Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
|
|
|
Not in this PR, left as follow-ups (no issues opened):
|
vchamarthi
left a comment
There was a problem hiding this comment.
🔴 Changes needed: 2 must-fixes · reviewed 10dcfaf
Next: @napetrov: fix the 2 must-fix findings; then push for a re-review.
Must-fix
-
dokimi/dokimi/commands/repo_run.py:391– A suite task whose repository is missing and which has egress aborts the whole run underimage.network: noneinstead of gettingunsupported.json.Input:
repo run --suitewithfix_add(egressexample.com, repository missing) andreview_add,image.network: none. On the head the preflight loop callsdocker_network(..., task.egress)for every task, raisesEgressNotImplementedand the run exits 1 beforerun_batch;review_adddoes not run. Expected:fix_addgetsunsupported.jsonandreview_addruns, ascheck_taskalready gives a suite gap precedence. Skip tasks inctx.base_gaps()in this loop.Also at:
dokimi/dokimi/repos/runner.py:383.Suggestion: Skip tasks in
ctx.base_gaps()in the preflight loop.
Verified on 10dcfaf: ✅ changed tests: 240 passed, fail on main · ✅ full suite with main merged in: 21810 passed, its only failures are main's too · ✅ merges onto main · 2 mutations tried · 4 edge inputs tried
9 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Suite tests | The changed tests (test_repo_host.py, test_repo_suite.py, test_repo_typecheck_grader.py) pass on the head, 240 cases, and the new suite module fails to import on main |
tests on the head |
| Merge | The PR merges onto current main with no conflict | merge onto main |
| Full suite | The whole local check passes with main merged in | full suite, main merged in |
| Base checkout | Pointing prepare_workspace at git_dir instead of base_git_dir fails 4 cases in tests/test_repo_suite.py, so the per-task base repository is pinned |
broke the code, tests caught it |
| Check exit code | Dropping + len(unavailable) from the exit test in cmd_repo_tasks_check fails 2 cases, so exit 0 for suite gaps is pinned |
broke the code, tests caught it |
| Suite shapes | None, a list, a string, an int and {} as the suite file each raise SuiteError, not a traceback |
ran it, edge input (structured) |
| Crlf suite file | A suite file with CRLF line endings loads and gives its tasks | ran it, edge input (text) |
| Suite path | A directory or a missing file given as the suite path raises SuiteError |
ran it, edge input (paths) |
| Base ids | An empty base, -x and a task absent from bases become gaps; HEAD resolves, in resolve_suite |
ran it, edge input (collections) |
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…uite An explicit empty path disabled suite mode, so a new run used eval-tree bases. Thread: PRRT_kwDOVBg7g863zTxO Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A task whose suite base is missing stopped the whole batch under no network. Thread: PRRT_kwDOVBg7g863zTxX Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…e it resolves A branch named like a commit kept its old pin in the warm cache after it moved. Thread: PRRT_kwDOVBg7g863zTxY Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A base that is the parent of a pinned commit was recorded as unsupported when the remote cache held only the pin. Thread: PRRT_kwDOVBg7g863zTxd Thread: PRRT_kwDOVBg7g863z-U9 Review: PRR_kwDOVBg7g88AAAABYKBVag Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A short SHA base missing from a warm remote cache is fetched by refreshing the branches and tags first, since git cannot fetch an abbreviated name. A floating pin is refreshed whatever its spelling, so a branch named like a full hash that moved is resolved again; a commit on no branch is still fetched by its name. Thread: PRRT_kwDOVBg7g864E4pz Thread: PRRT_kwDOVBg7g864E4qA Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at 4c791c9. The earlier threads are fixed on this head: the base comes from the scaffold, a gap is unsupported in run and check, the pin carries the suite plan. Two problems are left in the suite cache and the batch pin, in the line comments.
What I ran, at 4c791c9:
make ci-local --offline: 0 failed, 23 passed, 20 skipped (network-only steps).- A repro with a
file://scaffold that allows filters: after one short-SHA base,.full.gitgetsremote.origin.promisorand a later base has missing blobs. - A repro of a resume after a suite repository came back: refused with
use another --batch. - The catalog smoke (
docker-oracle,docker-probe): 1/1 pass each.
Checked, no issues
- Base vs. guidance repository:
prepare_workspacechecks the base out ofbase_git_dirand reads guidance revisions from the eval tree;check_armsandcheck_mcp_configsread each suite base in its own repository and skip only gaps. - Gap handling:
run_batchandcheck_taskjudge a suite gap before host and egress; the preflight loop skips gaps. - Schema and docs:
suite_manifest.v1is an input schema with a fixture, the example parses, the flags are incli_flags.pyand the CLI reference; new files carry the SPDX header; the changelog fragment is under Added.
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/dokimi/repos/source.py
… every repository A short-SHA base is fetched into the suite's full clone without a blob filter, since the filter would make every later fetch of that clone partial. A full-SHA pin is recorded for its repository whether or not the repository opened, so a repository that is down does not change the batch pin of a resume. Thread: PRRT_kwDOVBg7g864ZRyU Thread: PRRT_kwDOVBg7g864ZRyZ Review: PRR_kwDOVBg7g88AAAABYeMkog Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The pin lists every declared repository since 39554f3; this test still expected only the opened one. Found by the local ci-local run. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A declared repository always has a rev (validation refuses none), so the pin is the rev wherever it is set. The mypy job of ci-local failed on the old comprehension. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Re: review by vlad-perevezentsev
|
A self-check that checked nothing is incomplete, as repo run reads it, not a pass. Thread: PRRT_kwDOVBg7g864ZRyi Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Summary
A task of an eval tree can now take its
basecheckout from a scaffold repository that is not the eval tree.A suite manifest (
suite_manifest.v1) lists those tasks and names, for each, the repository<url>@<sha>itcomes from.
repo run --suite <path>opens each repository once, resolves each listed task's base there, and runsthe cells as before.
unsupported (
unsupported.json), not failed and not missing.repo reportcounts them neither way.--allow-floating-suite-revsis given;then it is resolved once, when the run starts, and the batch pins the commit (
_eval/batch.json, keysuite).repo tasks checkreports the suite-unsupported tasks as "not checked" and does not fail on them.data/suites/example-suite.yaml.Not in this PR:
repo validatedoes not check suite bases;{source_dir}stays the eval-tree source for suitetasks;
guidance:rev:<sha>arms are pinned against the eval-tree source. All three are in §2.1.Evidence
tests/test_repo_suite.py, 20 tests: pin refusal (branch, tag, ref,missing rev), the floating-rev flag, schema refusals, per-task base from the named repo, gap recording
(absent base, unreachable repo), and end-to-end
repo run(meta.jsonbase_repo,_eval/batch.jsonpin,unsupported.jsonfor a missing repo while the other task still runs, path escape refused).make ci-localon eb89306: 0 failed, 33 passed, 14 skipped (exit 0).and the escape refusal (see the review file in the scratch directory).
Merge Danger
suitepin. Resuming itonce the repository is present refuses with the "was run from ... use another --batch" message. Loud, but it
surprises an operator who expects a resume to continue. Not changed in this PR.
--allow-floating-suite-revs, a branch that moved between runs refuses aresume the same way (
use another --batch). The batch does not re-resolve and does not reuse a later commit.--suiteand--allow-floating-suite-revsare indokimi/cli_flags.py(conceptsuite) and the CLI snapshot is regenerated.suite_manifest.v1is a persisted format. The fixture is intests/fixtures/artifacts/.🤖 Generated with Claude Code