Skip to content

feat(repo): suite manifest for tasks whose base is in another repository - #783

Open
napetrov wants to merge 18 commits into
intel:mainfrom
napetrov:feat/suite-manifest
Open

napetrov wants to merge 18 commits into
intel:mainfrom
napetrov:feat/suite-manifest

Conversation

@napetrov

@napetrov napetrov commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A task of an eval tree can now take its base checkout from a scaffold repository that is not the eval tree.
A suite manifest (suite_manifest.v1) lists those tasks and names, for each, the repository <url>@<sha> it
comes from. repo run --suite <path> opens each repository once, resolves each listed task's base there, and runs
the cells as before.

  • A repository that cannot be opened, or a base that is not a commit of it, leaves that task's cells
    unsupported (unsupported.json), not failed and not missing. repo report counts them neither way.
  • Revisions must be pinned SHAs. A branch, tag or ref is refused unless --allow-floating-suite-revs is given;
    then it is resolved once, when the run starts, and the batch pins the commit (_eval/batch.json, key suite).
  • repo tasks check reports the suite-unsupported tasks as "not checked" and does not fail on them.
  • Contract §2.1 documents the schema, the pins and the limitations. Example: data/suites/example-suite.yaml.

Not in this PR: repo validate does not check suite bases; {source_dir} stays the eval-tree source for suite
tasks; guidance:rev:<sha> arms are pinned against the eval-tree source. All three are in §2.1.

Evidence

  • Offline tests with local git repos only: tests/test_repo_suite.py, 20 tests: pin refusal (branch, tag, ref,
    missing rev), the floating-rev flag, schema refusals, per-task base from the named repo, gap recording
    (absent base, unreachable repo), and end-to-end repo run (meta.json base_repo, _eval/batch.json pin,
    unsupported.json for a missing repo while the other task still runs, path escape refused).
  • make ci-local on eb89306: 0 failed, 33 passed, 14 skipped (exit 0).
  • Self-review (local mode, lane L3): one pass, no blocking finding; probes for the empty base, symbolic bases
    and the escape refusal (see the review file in the scratch directory).

Merge Danger

  • Batch pin on resume. A batch first run while a suite repository is missing has no suite pin. Resuming it
    once the repository is present refuses with the "was run from ... use another --batch" message. Loud, but it
    surprises an operator who expects a resume to continue. Not changed in this PR.
  • Floating revs move the pin. With --allow-floating-suite-revs, a branch that moved between runs refuses a
    resume the same way (use another --batch). The batch does not re-resolve and does not reuse a later commit.
  • No new CLI spelling without registry. --suite and --allow-floating-suite-revs are in
    dokimi/cli_flags.py (concept suite) and the CLI snapshot is regenerated.
  • Schema is new. suite_manifest.v1 is a persisted format. The fixture is in tests/fixtures/artifacts/.

🤖 Generated with Claude Code

A suite_manifest.v1 file in the eval tree lists tasks whose base is a
commit of another repository (a shared scaffold), each repository
referenced as <url>@<sha>. `repo run --suite` resolves each listed
task's base in the repository it names. A repository that cannot be
opened, or a base that is not one of its commits, leaves those cells
unsupported (unsupported.json), never failed or missing.

Revisions must be full SHAs; a branch, tag or ref needs
--allow-floating-suite-revs and is pinned in the batch (key `suite`).
Tests use local git repositories only. Contract section 2.1.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov requested a review from a team as a code owner November 15, 2026 06:24
@github-actions github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Nov 15, 2026
@github-actions

Copy link
Copy Markdown

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

~ `dokimi eval check` parameters changed:
    + added option `--allow-floating-suite-revs`
    + added option `--suite`
~ `dokimi repo run` parameters changed:
    + added option `--allow-floating-suite-revs`
    + added option `--suite`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

@exai-bot

exai-bot commented Nov 15, 2026 •

Copy link
Copy Markdown

OpenCodeReview: Commit d1229e1 was not reviewed. This PR has reached its review limit (6 reviews).

CI at `d1229e1`: 15 still running, 5 passing, 1 skipped
Check Status
Build docs (strict) still running
CLI interface diff still running
CodeQL (actions) still running
CodeQL (python) still running
External catalog smoke test (uxlfoundation/skills) still running
Linux Python 3.11 still running
Linux Python 3.14 still running
Linux Wheels Python 3.11 still running
Linux Wheels Python 3.14 still running
Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com... still running

5 more not listed.

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (4c791c9): 2 medium</summary>

Open threads from the bot: 0

</details>

Comment thread dokimi/docs/repo-evaluation-contract.md Resolved
Comment thread dokimi/docs/repo-evaluation-contract.md Resolved
Comment thread dokimi/dokimi/commands/repo_run.py Resolved
Comment thread dokimi/dokimi/commands/repo_run.py Resolved
Comment thread dokimi/dokimi/commands/repo_run.py Resolved
Comment thread dokimi/dokimi/repos/selfcheck.py Resolved
Comment thread dokimi/dokimi/repos/suite.py Resolved
Comment thread dokimi/dokimi/repos/suite.py Resolved
Comment thread dokimi/dokimi/repos/suite.py Resolved
Comment thread dokimi/dokimi/repos/suite.py Resolved
@codecov-commenter

codecov-commenter commented Nov 15, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 95.29915% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.80%. Comparing base (b62613a) to head (d1229e1).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
dokimi/dokimi/repos/source.py 40.00% 5 Missing and 1 partial ⚠️
dokimi/dokimi/repos/arms.py 88.88% 1 Missing and 1 partial ⚠️
dokimi/dokimi/repos/bases.py 66.66% 1 Missing and 1 partial ⚠️
dokimi/dokimi/repos/workspace.py 80.00% 1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #783      +/-   ##
==========================================
+ Coverage   95.79%   95.80%   +0.01%     
==========================================
  Files         432      433       +1     
  Lines       57204    57395     +191     
  Branches     9858     9887      +29     
==========================================
+ Hits        54796    54988     +192     
+ Misses       1497     1496       -1     
  Partials      911      911              
Files with missing lines Coverage Δ
dokimi/dokimi/cli_flags.py 100.00% <ø> (ø)
dokimi/dokimi/commands/repo_run.py 96.10% <100.00%> (+0.30%) ⬆️
dokimi/dokimi/repos/grading.py 96.10% <100.00%> (+0.10%) ⬆️
dokimi/dokimi/repos/runner.py 97.15% <100.00%> (+0.02%) ⬆️
dokimi/dokimi/repos/selfcheck.py 97.14% <100.00%> (+0.08%) ⬆️
dokimi/dokimi/repos/suite.py 100.00% <100.00%> (ø)
dokimi/dokimi/repos/workspace.py 93.60% <80.00%> (+0.78%) ⬆️
dokimi/dokimi/repos/arms.py 96.15% <88.88%> (-0.88%) ⬇️
dokimi/dokimi/repos/bases.py 95.50% <66.66%> (-2.17%) ⬇️
dokimi/dokimi/repos/source.py 95.74% <40.00%> (-0.63%) ⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The base resolution itself works: a suite task takes its base from the named repository, a missing repository leaves only that task unsupported, and meta.json records base_repo. The problems are in the batch pin, the self-check exit code and the SHA check, and each of them contradicts §2.1 as written. Inline comments cover the ones I would fix before merge.

What I ran, on a Linux review host at eb89306:

  • make ci-local: 30 passed, 3 failed, 14 skipped. The 3 failures are not the PR's: the gitleaks download returned HTTP 504 and the commit scan that depends on it did not run, and the tests step had 20533 passed with 2 pytest-timeout errors in test_audit_test_quality.py, a file this PR does not touch, which passes alone (172 passed)
  • Reproductions for the findings below with local git repositories and a fake runner, no network.
  • A mutation run of tests/test_repo_suite.py against scratch copies of the new code: 4 of 10 mutants of the check and _prepare paths pass the suite (no test calls eval check --suite), and 4 of 4 changes to _SHA_RE pass it.

Open bot points I confirm:

  • exai-bot is right that a resumed floating batch does not reuse the pinned commits, contrary to §2.1 (repo-evaluation-contract.md:342). New evidence: with sc@main, the first run pinned suite: {"sc": "bbb43d6d…"}; after one new commit on main, the same command and batch exits 1 with batch b1 was run from suite {'sc': 'bbb43d6d…'}, not suite {'sc': 'c302794b…'}; use another --batch.
  • exai-bot is right that repo tasks check exits 1 when every selected task is a suite gap, against §2.1 (repo-evaluation-contract.md:347). New evidence: one scaffold unreachable and --task naming only that task gives 0/1 ok, then ✗ incomplete: no task passed the self-check (0 ok, 0 unsupported) and exit 1; adding a passing task exits 0.
  • exai-bot is right that the pin must carry the task-to-repository mapping. New evidence: two suite files in one tree with the same repos and different tasks are both accepted when the same batch is resumed.

Checked, no issues

  • Schema registration: suite_manifest.v1 is an input schema like repo_manifest.v1, so it is correctly absent from SCHEMA_VERSIONS. The fixture exists and test_schema_compat.py passes.
  • Example and docs: data/suites/example-suite.yaml parses with parse_suite; the contract page is already in mkdocs.yml; --suite and --allow-floating-suite-revs are registered in cli_flags.py and the CLI reference is regenerated.
  • Sign-off: the single commit eb89306 carries a Signed-off-by.
  • Mutants killed: dropping the duplicate-task, undeclared-repo, unpinned or floating checks, the _refuse_escape call, the batch-pin stamp, the base_repo meta, the run_cell git_dir_for, or the run_batch gap verdict fails the suite.
Bot points I checked that no longer apply at this head (1)
  • exai-bot's point on the egress precedence in selfcheck.py does not apply: a task that declares egress under network: none fails repo tasks check as egress-unsupported with or without this PR, so the exit code is unchanged and only the reason text leaves out the missing repository.

Comment thread dokimi/dokimi/commands/repo_run.py Resolved
Comment thread dokimi/dokimi/commands/repo_run.py Resolved
Comment thread dokimi/dokimi/commands/repo_run.py Resolved
Comment thread dokimi/dokimi/commands/repo_run.py Resolved
Comment thread dokimi/dokimi/repos/runner.py Resolved
Comment thread dokimi/dokimi/repos/suite.py Resolved
Comment thread dokimi/dokimi/repos/suite.py Resolved
Comment thread dokimi/dokimi/repos/suite.py Resolved
Fixes on the suite manifest:
- a pin must be a full 40 or 64 hex commit; a short SHA or a hex-named
  branch is no longer taken as one
- a pinned local path that does not exist leaves its tasks unsupported
  instead of refusing the whole suite
- a base missing from a warm remote cache is fetched once before it is
  recorded as a gap
- the batch pin records the suite file's path, the task-to-repository
  mapping and the repository locations; resuming with another suite
  file, or without --suite, is refused
- only the suite's tasks are loaded, so an unrelated malformed task.json
  does not stop the run
- the guidance revision of an arm is read from the eval tree while the
  base is checked out from the scaffold
- check_arms and the .mcp.json check skip suite tasks, whose base is not
  in the eval tree, so a gap no longer aborts the batch
- repo tasks check exits 0 when every selected task is a suite gap, and
  the gap is judged before the egress requirement
- a suite file that is not UTF-8 is a SuiteError

Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
@vchamarthi

Copy link
Copy Markdown
Contributor

Pushed 1 commit to feat/suite-manifest (eb89306..c873266): Waiting on @napetrov: 1 question below.

Changes

  • A suite file that is not UTF-8 is a SuiteError (fix already on f92ead5), now with a test.
  • Batch pin carries the suite mapping and refuses a resume without --suite.
  • A suite gap task no longer has its eval-tree base read for a skill collision.
  • A suite file that is not UTF-8 is a SuiteError, with a test.
  • _SHA_RE requires a full commit.
  • Batch pin carries the suite file's path and mapping, and refuses a resume without --suite.
  • The all-gap self-check exits 0 and a gap is judged before egress.
  • The revision-arm test asserts the guidance is installed from the eval tree.

Needs @napetrov

  1. Should a resumed floating batch reuse the commits stored in _eval/batch.json, or keep refusing with use another --batch as it does now? (dokimi/docs/repo-evaluation-contract.md:342)
    Contract §2.1 says a resumed batch reuses the pinned commits. With sc@main, after one new commit on main, the same command and batch exits 1 with batch b1 was run from suite {'sc': ...}, not suite {...}. Reusing needs _suite_bases to read the stored commits before it resolves; refusing needs the sentence changed.
    • A: Reuse the stored commits on resume: the batch keeps its first commits, and a repository that was missing and is now available is also pinned.
    • B: Keep the refusal and change §2.1 to say a moved floating revision refuses the resume.

Threads and comments

  • Fixed: 19, each answered in its thread.
  • Fixed (comment by @antonwolfy): No test calls eval check --suite; mutants of the check and _prepare paths and of _SHA_RE survive – Tests added for the all-gap exit, the gap before egress, the guidance revision arm and the SHA length; each fails when its fix is reverted.

Not this PR's

  • Fails on main too: 1 test in tests/test_repo_compare_interval.py – float sum 2.0000000000000004e+16 where the test expects 2e+16, on main as well.
  • Fails on main too: 1 test in tests/test_repo_compare_interval.py – -0.0001 where the test expects 0.0, on main as well.
Verified (12)
  • all-gap exit: passes.
  • gap before egress: passes.
  • resume refused: passes.
  • malformed unrelated task: passes.
  • guidance revision arm: passes.
  • sha pin: passes.
  • full sha is a pin: passes.
  • missing local repo: passes.
  • warm cache base: passes.
  • arm preflight skips gaps: passes.
  • utf-8 suite file: passes.
  • Full suite with main merged in: 21789 passed.

Comment thread dokimi/dokimi/commands/repo_run.py Resolved
Comment thread dokimi/dokimi/commands/repo_run.py Resolved
Comment thread dokimi/dokimi/repos/suite.py Resolved
- check_arms and check_mcp_configs read each available suite task's base in
  its scaffold repository; only suite gaps are skipped
- a pin that resolves to another commit (a branch named like a hash) is
  refused, except under --allow-floating-suite-revs, which resolves it
- a repos key that is not a string is a SuiteError, not a TypeError
- the non-string name check is a helper, so parse_suite stays under the
  complexity cap
- the scaffold MCP test runs through repo run, so it fails when the suite
  base dirs are not passed to check_mcp_configs

Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
@vchamarthi

Copy link
Copy Markdown
Contributor

🟡 Waiting on @napetrov · 1 commit on feat/suite-manifest (c873266..4b80490)

Next: @napetrov: answer the 1 question below. @napetrov: correct the PR description (below).

Changes

  • A repos key that is not a string raises SuiteError naming it.
  • A branch named like a full SHA resolves under --allow-floating-suite-revs instead of being refused.

Needs @napetrov

  1. Should a resumed floating batch reuse the commits stored in _eval/batch.json, or should the contract say it refuses? (dokimi/docs/repo-evaluation-contract.md:342)
    --allow-floating-suite-revs with sc@main, then a new commit on main, then the same command and batch: _pin_batch exits 1 with use another --batch. §2.1 says a resumed batch reuses the pinned commits.
    • A: Reuse: load the stored suite pins before resolve_suite; about 30 lines with a resume test
    • B: Refuse: change §2.1 and the Merge Danger text to say a moved branch refuses the resume
  • PR description says "The batch keeps the commit it first resolved to; it does not re-resolve."; it should read: With a moved floating ref the resume is refused with use another --batch.

Threads and comments

  • 3 fixed; each answered in its thread.

Verified on the new head: ✅ tests of the fix: 5 passed · ✅ full suite with main merged in: 21795 passed, 3 failures that main has too · ✅ second reader: 2 raised, all fixed

What each test shows
  • integer repos key: passes.
  • arm check reads the scaffold base: passes.
  • mcp check reads the scaffold base: passes.
  • hash-named branch is not a pin: passes.
  • hash-named branch resolves when floating revs are allowed: passes.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The review threads on this PR are answered by commits already on the branch
(c873266 and 4b80490) and by a merge of intel/main. This commit names them.

Thread: PRRT_kwDOVBg7g863m7sG: already fixed in c873266
Thread: PRRT_kwDOVBg7g863m7sI: already fixed in c873266
Thread: PRRT_kwDOVBg7g863m7sJ: already fixed in c873266
Thread: PRRT_kwDOVBg7g863m7sN: already fixed in c873266
Thread: PRRT_kwDOVBg7g863m7sQ: already fixed in c873266
Thread: PRRT_kwDOVBg7g863m7sS: already fixed in c873266
Thread: PRRT_kwDOVBg7g863m7sT: already fixed in c873266
Thread: PRRT_kwDOVBg7g863m7sV: already fixed in c873266
Thread: PRRT_kwDOVBg7g863m7sX: already fixed in c873266
Thread: PRRT_kwDOVBg7g863m7sa: already fixed in c873266
Thread: PRRT_kwDOVBg7g863m7sd: already fixed in c873266
Thread: PRRT_kwDOVBg7g863paUe: already fixed in c873266
Thread: PRRT_kwDOVBg7g863paUg: already fixed in c873266
Thread: PRRT_kwDOVBg7g863paUk: already fixed in c873266
Thread: PRRT_kwDOVBg7g863paUn: already fixed in c873266
Thread: PRRT_kwDOVBg7g863paUq: already fixed in c873266
Thread: PRRT_kwDOVBg7g863paUr: already fixed in c873266
Thread: PRRT_kwDOVBg7g863paUt: already fixed in c873266
Thread: PRRT_kwDOVBg7g863paUw: already fixed in c873266
Thread: PRRT_kwDOVBg7g863sVHe: already fixed in 4b80490
Thread: PRRT_kwDOVBg7g863sVHh: already fixed in 4b80490
Thread: PRRT_kwDOVBg7g863sVHk: already fixed in 4b80490
Thread: PRR_kwDOVBg7g88AAAABYE3Cvg: already fixed in c873266
Thread: IC_kwDOVBg7g88AAAABm3m9qg: accept, PR description corrected to the resume behaviour
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A floating suite revision is resolved when the run starts and pinned in the batch.
A resume after the branch moved is refused, so the contract now says so instead of
promising reuse. The PR description says the same.

Thread: PRRT_kwDOVBg7g863m7sD: accept, contract changed to refuse the resume
Thread: IC_kwDOVBg7g88AAAABmzV5cw: accept, same decision
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by antonwolfy

The problems are in the batch pin, the self-check exit code and the SHA check
Those three are the threads above, all fixed in c873266 and checked on the merged head. The mutants in your list are covered by the tests named there.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: comment by vchamarthi

Correct the PR description (the floating-revs paragraph)
Done: it now says a resume after the branch moved is refused with use another --batch, and the batch does not re-resolve. The resume question is decided in its own reply.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: comment by vchamarthi

Should a resumed floating batch reuse the commits stored in _eval/batch.json, or refuse?
Decided: refuse, as the code does. §2.1 and the PR description now say so (10dcfaf).

@napetrov

Copy link
Copy Markdown
Contributor Author

Not in this PR, left as follow-ups (no issues opened):

  • repo tasks check and repo run differ when every selected task is a suite gap: check exits 0, run exits 1 with "no cell has a result". §2.1 states the check side only.
  • _suite_bases loads every task the suite lists, so a malformed listed task that this run does not select still refuses the run. The code comment overstates how narrow the load is.
  • A symbolic task base (e.g. main) in a local scaffold follows that repo's refs, not the pinned @sha.
  • _SHA_RE accepts lowercase hex only, so an uppercase full SHA is refused as a ref.
  • suite_plan.path stores the --suite path as typed, so ./s.yaml and s.yaml count as different files on resume.

Comment thread dokimi/dokimi/commands/repo_run.py Resolved
Comment thread dokimi/dokimi/commands/repo_run.py Resolved
Comment thread dokimi/dokimi/repos/suite.py Resolved
Comment thread dokimi/dokimi/repos/suite.py Resolved

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Changes needed: 2 must-fixes · reviewed 10dcfaf

Next: @napetrov: fix the 2 must-fix findings; then push for a re-review.

Must-fix

  • dokimi/dokimi/commands/repo_run.py:391 – A suite task whose repository is missing and which has egress aborts the whole run under image.network: none instead of getting unsupported.json.

    Input: repo run --suite with fix_add (egress example.com, repository missing) and review_add, image.network: none. On the head the preflight loop calls docker_network(..., task.egress) for every task, raises EgressNotImplemented and the run exits 1 before run_batch; review_add does not run. Expected: fix_add gets unsupported.json and review_add runs, as check_task already gives a suite gap precedence. Skip tasks in ctx.base_gaps() in this loop.

    Also at: dokimi/dokimi/repos/runner.py:383.

    Suggestion: Skip tasks in ctx.base_gaps() in the preflight loop.

Verified on 10dcfaf: ✅ changed tests: 240 passed, fail on main · ✅ full suite with main merged in: 21810 passed, its only failures are main's too · ✅ merges onto main · 2 mutations tried · 4 edge inputs tried

9 checks behind it, each with its evidence
Area What holds How
Suite tests The changed tests (test_repo_host.py, test_repo_suite.py, test_repo_typecheck_grader.py) pass on the head, 240 cases, and the new suite module fails to import on main tests on the head
Merge The PR merges onto current main with no conflict merge onto main
Full suite The whole local check passes with main merged in full suite, main merged in
Base checkout Pointing prepare_workspace at git_dir instead of base_git_dir fails 4 cases in tests/test_repo_suite.py, so the per-task base repository is pinned broke the code, tests caught it
Check exit code Dropping + len(unavailable) from the exit test in cmd_repo_tasks_check fails 2 cases, so exit 0 for suite gaps is pinned broke the code, tests caught it
Suite shapes None, a list, a string, an int and {} as the suite file each raise SuiteError, not a traceback ran it, edge input (structured)
Crlf suite file A suite file with CRLF line endings loads and gives its tasks ran it, edge input (text)
Suite path A directory or a missing file given as the suite path raises SuiteError ran it, edge input (paths)
Base ids An empty base, -x and a task absent from bases become gaps; HEAD resolves, in resolve_suite ran it, edge input (collections)

Comment thread dokimi/dokimi/repos/suite.py Resolved
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…uite

An explicit empty path disabled suite mode, so a new run used eval-tree bases.

Thread: PRRT_kwDOVBg7g863zTxO
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A task whose suite base is missing stopped the whole batch under no network.

Thread: PRRT_kwDOVBg7g863zTxX
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…e it resolves

A branch named like a commit kept its old pin in the warm cache after it moved.

Thread: PRRT_kwDOVBg7g863zTxY
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A base that is the parent of a pinned commit was recorded as unsupported when the
remote cache held only the pin.

Thread: PRRT_kwDOVBg7g863zTxd
Thread: PRRT_kwDOVBg7g863z-U9
Review: PRR_kwDOVBg7g88AAAABYKBVag
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

Changes needed: 2 must-fixes
Both fixed in db5e8df: batch gap abort, parent base.

Comment thread dokimi/dokimi/repos/suite.py Resolved
Comment thread dokimi/dokimi/repos/suite.py Resolved
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A short SHA base missing from a warm remote cache is fetched by refreshing
the branches and tags first, since git cannot fetch an abbreviated name.
A floating pin is refreshed whatever its spelling, so a branch named like a
full hash that moved is resolved again; a commit on no branch is still
fetched by its name.

Thread: PRRT_kwDOVBg7g864E4pz
Thread: PRRT_kwDOVBg7g864E4qA
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 4c791c9. The earlier threads are fixed on this head: the base comes from the scaffold, a gap is unsupported in run and check, the pin carries the suite plan. Two problems are left in the suite cache and the batch pin, in the line comments.

What I ran, at 4c791c9:

  • make ci-local --offline: 0 failed, 23 passed, 20 skipped (network-only steps).
  • A repro with a file:// scaffold that allows filters: after one short-SHA base, .full.git gets remote.origin.promisor and a later base has missing blobs.
  • A repro of a resume after a suite repository came back: refused with use another --batch.
  • The catalog smoke (docker-oracle, docker-probe): 1/1 pass each.

Checked, no issues

  • Base vs. guidance repository: prepare_workspace checks the base out of base_git_dir and reads guidance revisions from the eval tree; check_arms and check_mcp_configs read each suite base in its own repository and skip only gaps.
  • Gap handling: run_batch and check_task judge a suite gap before host and egress; the preflight loop skips gaps.
  • Schema and docs: suite_manifest.v1 is an input schema with a fixture, the example parses, the flags are in cli_flags.py and the CLI reference; new files carry the SPDX header; the changelog fragment is under Added.

Comment thread dokimi/dokimi/repos/suite.py Resolved
Comment thread dokimi/dokimi/repos/suite.py Resolved
Comment thread dokimi/dokimi/commands/repo_run.py Resolved
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

# Conflicts:
#	dokimi/dokimi/repos/source.py
… every repository

A short-SHA base is fetched into the suite's full clone without a blob
filter, since the filter would make every later fetch of that clone
partial. A full-SHA pin is recorded for its repository whether or not
the repository opened, so a repository that is down does not change the
batch pin of a resume.

Thread: PRRT_kwDOVBg7g864ZRyU
Thread: PRRT_kwDOVBg7g864ZRyZ
Review: PRR_kwDOVBg7g88AAAABYeMkog
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The pin lists every declared repository since 39554f3; this test still
expected only the opened one. Found by the local ci-local run.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A declared repository always has a rev (validation refuses none), so the pin is
the rev wherever it is set. The mypy job of ci-local failed on the old
comprehension.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vlad-perevezentsev

Two problems are left in the suite cache and the batch pin
Both are fixed in 39554f3, each with a test that fails on the old code.

A self-check that checked nothing is incomplete, as repo run reads it, not a pass.

Thread: PRRT_kwDOVBg7g864ZRyi
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli-interface-change Changes dokimi's user-facing CLI surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants