Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
eb89306
feat(repo): suite manifest for tasks whose base is in another repository
napetrov Nov 15, 2026
c873266
fix(repo): pin and resolve suite bases strictly
vchamarthi Nov 15, 2026
4b80490
fix(repo): check suite bases in their repositories before a run
vchamarthi Nov 15, 2026
936b445
Merge branch 'main' of https://github.com/intel/dokimi into pr783-local
napetrov Nov 16, 2026
f7110eb
Address review: record the suite fixes already on this branch
napetrov Nov 16, 2026
10dcfaf
docs(repo): a resumed suite batch refuses a moved floating revision
napetrov Nov 16, 2026
cd4d746
Merge branch 'main' of https://github.com/intel/dokimi into pr783-local
napetrov Nov 16, 2026
a4131e0
fix(repo): refuse an empty --suite path instead of reading it as no s…
napetrov Nov 16, 2026
d72786d
fix(repo): a suite gap is unsupported in the batch, not a run abort
napetrov Nov 16, 2026
43a1fe9
fix(repo): a floating suite pin re-fetches a hash-shaped branch befor…
napetrov Nov 16, 2026
db5e8df
fix(repo): a suite base in a warm cache is fetched before it is a gap
napetrov Nov 16, 2026
17f1089
Merge commit '9aa10c7101429cbabfa61ae77eaf3fa5f124a16f' into pr783-local
napetrov Nov 16, 2026
4c791c9
Address review: fetch an abbreviated suite base; refresh floating pins
napetrov Nov 16, 2026
69fdedd
Merge branch 'main' of https://github.com/intel/dokimi into pr783-r10
napetrov Nov 17, 2026
39554f3
fix(repo): a short base leaves the full clone full, and the pin lists…
napetrov Nov 17, 2026
a793554
test(repo): expect the pin of a down repository in the suite test
napetrov Nov 17, 2026
40e596f
fix(repo): narrow the suite pin type so mypy passes
napetrov Nov 17, 2026
d1229e1
fix(repo): tasks check exits 1 when every selected task is a suite gap
napetrov Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Added

- **Suite manifests for tasks whose base is in another repository** — `suite_manifest.v1` lists eval-tree tasks whose base is a commit of a scaffold repository, referenced as `<url>@<sha>`. `repo run --suite <path>` resolves each base there; a repository that cannot be opened leaves its cells unsupported, not failed. Floating refs need `--allow-floating-suite-revs`. `repo tasks check` exits 1 (incomplete) when every selected task is such a gap, since nothing was checked. See `docs/repo-evaluation-contract.md` §2.1.
12 changes: 12 additions & 0 deletions dokimi/data/suites/example-suite.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Copyright (C) 2026 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
#
# Example suite_manifest.v1 (contract §2.1): `repos` pins each scaffold, `tasks` names the tasks whose base is in one.
schema: suite_manifest.v1
id: linreg-cpu-solver
repos:
sklearn-scaffold: https://github.com/example-org/sklearn-scaffold@9f8e7d6c5b4a39281706f5e4d3c2b1a098765432
numpy-scaffold: https://github.com/example-org/numpy-scaffold@0a1b2c3d4e5f60718293a4b5c6d7e8f901234567
tasks:
- {id: solver-pick-cholesky, repo: sklearn-scaffold}
- {id: solver-pick-lstsq, repo: numpy-scaffold}
4 changes: 4 additions & 0 deletions dokimi/docs/reference/cli-interface.json
Original file line number Diff line number Diff line change
Expand Up @@ -565,6 +565,7 @@
"allow_abbrev": true,
"params": [
{"key": "source", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "argument", "nargs": null, "position": 0, "required": true, "type": null},
{"key": "--allow-floating-suite-revs", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--allow-unsupported", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--arm", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--build-network", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
Expand All @@ -582,6 +583,7 @@
{"key": "--out-dir", "action": "_StoreAction", "choices": null, "const": null, "default": "repo-check", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--rebuild-image", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--source-dir", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--suite", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--task", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--warm", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null}
],
Expand Down Expand Up @@ -1372,6 +1374,7 @@
"params": [
{"key": "source", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "argument", "nargs": null, "position": 0, "required": true, "type": null},
{"key": "--agent-network", "action": "_StoreAction", "choices": ["'bridge'", "'host'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--allow-floating-suite-revs", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--arm", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": true, "type": null},
{"key": "--batch", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": true, "type": null},
{"key": "--bin", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
Expand All @@ -1398,6 +1401,7 @@
{"key": "--rebuild-image", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--reps", "action": "_StoreAction", "choices": null, "const": null, "default": 1, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--source-dir", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--suite", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--task", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--timeout", "action": "_StoreAction", "choices": null, "const": null, "default": 2700, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--tools", "action": "_StoreAction", "choices": null, "const": null, "default": "all", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null}
Expand Down
52 changes: 50 additions & 2 deletions dokimi/docs/repo-evaluation-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ SPDX-License-Identifier: Apache-2.0
> Implemented so far (slice S1): the `repo_manifest.v1` schema, validation of
> the manifest and of every `task.json` against one repository snapshot
> (with git history, every task base must resolve and the guidance globs are
> evaluated at each base), the `repo_static.v1` schema (§4.4), and the `repo`
> subject kind:
> evaluated at each base), the `repo_static.v1` schema (§4.4), the suite
> manifest for bases in other repositories (§2.1), and the `repo` subject kind:
>
> ```bash
> python cli.py repo validate <dir | dir@rev | url@rev> --manifest <path> [--json] [--out FILE] [--fail-on-warning]
Expand Down Expand Up @@ -313,6 +313,54 @@ Rules:
harness short-lived credentials scoped to the model endpoint. Every
connection through the proxy is logged with the run.

### 2.1 Suite manifest (`suite_manifest.v1`)

A task's `base` is normally a commit of the eval tree's source (§1.1). A task
whose base is in another repository, such as a scaffold that a benchmark shares
or a library at a fixed revision, is listed in a suite file kept in the eval
tree:

```yaml
schema: suite_manifest.v1
id: linreg-cpu-solver
repos:
sklearn-scaffold: https://github.com/example/scikit-learn-scaffold@<40-hex sha>
numpy-scaffold: /path/to/numpy-scaffold@<40-hex sha>
tasks:
- id: solver-pick-cholesky
repo: sklearn-scaffold
```

- `repo run --suite <path>` reads the file from the eval tree; the path is
relative to its root and may not leave it. Each listed task takes its base
checkout from the repository it names; its `task.json` keeps the rest of the
task. A task the suite does not list keeps its eval-tree base. Every listed
task must be a task of `tasks` (§2), or the run stops before any cell.
- Each repository is `<url>@<rev>` or a local `<path>@<rev>`, as in §1.1. The
revision must be a full commit SHA. A branch, tag or other ref is refused
unless the run passes `--allow-floating-suite-revs`; it is resolved again
when each run starts, and the batch pin (`_eval/batch.json`, key `suite`)
records the commits. A resume is refused with `use another --batch` when the
revision now resolves to another commit: the batch does not follow a
moved branch or tag, and it does not reuse a later commit. The pin also
records the suite file's path, the task-to-repository mapping and the
repository locations (key `suite_plan`): a resume with another suite file, or
without `--suite`, is refused.
- A repository that cannot be opened, or a base that is not a commit of it,
leaves that task's cells unsupported (§9): `unsupported.json` gives the reason,
and `repo report` counts the cells neither as failed nor as missing. A cell
that runs records `base_repo` (`{"repo", "source"}`) in `meta.json`. A base that
the cached clone of a remote lacks is fetched once before it counts as missing.
- `repo tasks check` reports these tasks as not checked when their repository is
missing and fails only on the other bad tasks; when every selected task is
not checked it exits 1 (incomplete), as `repo run` does, since nothing was checked.
- Not covered yet: `repo validate` checks the bases of the eval tree only; a suite
task's `{source_dir}` is still the eval-tree source; and a `guidance:rev:<sha>`
arm is pinned against the eval-tree source.

The schema is `dokimi/schemas/suite_manifest.v1.json`; an example is
`data/suites/example-suite.yaml`.

## 3. `task.json`

The framework interprets the **core** and **workspace** keys. Every other key
Expand Down
7 changes: 7 additions & 0 deletions dokimi/dokimi/cli_flags.py
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,13 @@ class Concept:
canonical=(Flag("--network", "[PHASE=]MODE[:hosts]", "text"),),
banned=("--agent-network", "--build-network"),
),
Concept(
name="suite",
help="a suite manifest: tasks whose base is in another repository (contract §2.1)",
canonical=(Flag("--suite", "PATH", "text", meaning="the suite file, in the eval tree"),
Flag("--allow-floating-suite-revs", kind="flag",
meaning="resolve a branch or tag of a suite repository when the run starts")),
),
Concept(
name="online",
help="network checks of a catalog's skills, opt in",
Expand Down
Loading
Loading