Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Added

- **`catalog provenance` matches each skill copy to the upstream version it came from** — offline, from the `--upstream` git checkouts or `--upstream-manifest` files you name (dokimi ships no upstream list). A copy is matched by its git blob sha against the upstream history, then by frontmatter name and word Jaccard similarity ≥ 0.8. The report gives the origin skill and version, how many commits and months behind upstream head, and whether the copy was edited.
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Added

- **`catalog check --upstream` and `--upstream-manifest` add provenance findings** — each skill that matches an upstream version gets `upstream-stale` (note: the copy is not at the upstream head), `license-dropped-from-upstream` (warn: an edited copy drops the upstream's `license`) and `risk-inherited-from-upstream` (warn: the copy keeps unrestricted Bash, `pipe-to-shell` or `permission-bypass-flag` that its upstream version also has). Offline, from the upstreams you name; not combined with `--per-skill` or `--kind tasks`.
58 changes: 58 additions & 0 deletions dokimi/docs/external-content.md
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,64 @@ combine with `--format json`; any other misuse exits 2. From Python,
`dokimi.external.review_skill_dir(path)` returns the `SkillReview` of the one
skill in `path`.

### Provenance: which upstream a copy came from (`catalog provenance`)

A skill copied from another repository drifts from it: the upstream is updated and the
copy is not, or the copy is edited and drops the licence. `catalog provenance` matches each
loaded skill under `--root` to a version of an upstream you name, offline. Dokimi ships no
upstream list: give it the upstreams yourself.

- `--upstream CHECKOUT` (repeatable) is the top level of a local git checkout; a folder inside another repository is refused. Clone it yourself; a full clone
gives the exact count of commits behind, a shallow one says it sees only its own history.
- `--upstream-manifest FILE` (repeatable) is a JSON file of one upstream, for an upstream that
is not a git checkout. Its `schema` field is the version-1 upstream schema string shown here:

```json
{"schema": "dokimi.upstream.v1", "name": "acme-skills",
"head": {"commit": "2c5958e", "date": "2026-04-01T00:00:00+00:00"},
"skills": [{"path": "skills/pdf/SKILL.md", "content": "---\nname: pdf\n...", "head": true,
"commit": "9f1c2ab", "date": "2026-03-01T00:00:00+00:00"}]}
```

`content` is the bytes of the file; `head`, `commit` and `date` are optional. The git blob
sha is computed from `content`, so a manifest and a checkout agree.

```text
$ dokimi catalog provenance --root ./my-plugin --upstream ~/src/acme-skills
# upstream acme-skills: head 446e0e560cff 2026-04-01, history full
# skills (2 under ./my-plugin: 1 exact, 1 similar, 0 unmatched; 1 edited; 0 at head; median 3.0 months behind)
skill match upstream version behind edited
skills/pdf-copy exact acme-skills:pdf/SKILL.md 88fa89b3fcc7 2026-01-01 1 commits, 3.0 mo no
skills/pdf-edit similar acme-skills:pdf/SKILL.md 88fa89b3fcc7 2026-01-01 1 commits, 3.0 mo yes
```

A copy is matched by its bytes first: an identical `SKILL.md` at the upstream head or in its
history is `exact` (not edited), and the newest such version is the one reported. Otherwise
an upstream version with the same frontmatter `name` and a word Jaccard similarity of at least
0.8 is `similar` (edited). Anything else is `unmatched`. `behind` counts the commits that
touched the file after the matched version, and the months run from that version's commit date
to the upstream head. `--format json` adds the upstreams, one row per skill and a `summary`.

### Upstream findings in `catalog check` (`--upstream`, `--upstream-manifest`)

Give `catalog check` the same upstreams as `catalog provenance` and each loaded skill that matches one gets
up to three more findings. Nothing is fetched, and none of them is an error:

- `upstream-stale` (note): the copy matches an upstream version that is not the upstream head. A note does not

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

documentation · medium
Must-fix — Impact: users can interpret the absence of upstream-stale as evidence that an old copy is current, despite supplying a manifest that cannot establish this. Trigger: a manifest has a top-level head but no entry with "head": true for the matched path (including when another path is flagged). Evidence: Index.head_known() requires a flagged entry for that same path, and upstream_findings() suppresses the note otherwise; test_a_manifest_with_no_head_flag_says_nothing_is_stale and test_a_flagged_head_on_one_path_does_not_make_another_path_stale explicitly assert this behavior. Document that prerequisite and distinguish the top-level repository head from each skill path's head. Workaround: flag the current entry of each relevant path with "head": true, or use a checkout. Severity: medium because this is a bounded mismatch in the diagnostic contract, with no security or data-loss impact.

Check run: inconclusive, the result is not shown as a pass.

Suggestion:

Suggested change
- `upstream-stale` (note): the copy matches an upstream version that is not the upstream head. A note does not
- `upstream-stale` (note): the copy matches a non-head version of a path whose head is known. For manifests,
that requires an entry of the same path with `"head": true`; the top-level `head` block alone is insufficient. A note does not

change the state or the exit code. The message says how many commits and months behind the head it is, where the
history gives them.
- `license-dropped-from-upstream` (warn): the copy is edited, its upstream version declares a `license`, and the copy
declares none. An edited copy that changes the license to another value is not reported. An unedited copy has the
upstream bytes, so it cannot drop one.
- `risk-inherited-from-upstream` (warn): the copy keeps a risk its upstream version has too. The risks are the ones the
scan already reports: unrestricted Bash (`allowed-tools-unrestricted-bash`), `pipe-to-shell` and
`permission-bypass-flag`. The copy's side is its own findings, bundled files included. The upstream side is that
version's `SKILL.md` alone, since an upstream version is one file, not a folder.

The warnings follow the usual rules: they do not fail `--strict`, and `--ignore` takes each code. The flags do not
combine with `--per-skill` (the upstream index is built once, in one process) or with `--kind tasks`.


## 2. The awareness half: question sets

A question set is `<questions-root>/<product_key>_golden.json`, in the format
Expand Down
17 changes: 16 additions & 1 deletion dokimi/docs/reference/cli-interface.json
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,9 @@
{"key": "--root", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": true, "type": null},
{"key": "--skip-nested", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--strict", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--tasks-subdir", "action": "_StoreAction", "choices": null, "const": null, "default": "evaluation/harbor/tasks", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null}
{"key": "--tasks-subdir", "action": "_StoreAction", "choices": null, "const": null, "default": "evaluation/harbor/tasks", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--upstream", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--upstream-manifest", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null}
],
"path": "dokimi catalog check",
"subcommand_required": null,
Expand Down Expand Up @@ -352,6 +354,19 @@
"subcommand_required": null,
"subcommands": {}
},
"provenance": {
"allow_abbrev": true,
"params": [
{"key": "--format", "action": "_StoreAction", "choices": ["'json'", "'text'"], "const": null, "default": "text", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--help/-h", "action": "_HelpAction", "choices": null, "const": null, "default": "==SUPPRESS==", "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--root", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": true, "type": null},
{"key": "--upstream", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--upstream-manifest", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null}
],
"path": "dokimi catalog provenance",
"subcommand_required": null,
"subcommands": {}
},
"routing": {
"allow_abbrev": true,
"params": [
Expand Down
10 changes: 10 additions & 0 deletions dokimi/dokimi/cli_flags.py
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,16 @@ class Concept:
note="The subject is a positional. --product, --library and --target name it, so new "
"commands do not use them.",
),
Concept(
name="upstream",
help="the upstream copies a skill is matched against (catalog provenance)",
canonical=(
Flag("--upstream", "CHECKOUT", "path", repeatable=True,
meaning="a local git checkout of an upstream; its history gives the versions"),
Flag("--upstream-manifest", "FILE", "path", repeatable=True,
meaning="a dokimi.upstream.v1 JSON manifest of one upstream"),
),
),
Concept(
name="catalog_impact",
help="the local base revision for catalog change selection",
Expand Down
1 change: 1 addition & 0 deletions dokimi/dokimi/cli_routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ class Tombstone:
Leaf(("catalog", "show"), FREE),
Leaf(("catalog", "intents"), FREE),
Leaf(("catalog", "check"), FREE),
Leaf(("catalog", "provenance"), FREE),
Leaf(("catalog", "eval"), FREE),
Leaf(("catalog", "formats"), FREE),
Leaf(("catalog", "trigger-probe"), PAID),
Expand Down
39 changes: 38 additions & 1 deletion dokimi/dokimi/commands/catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ def _scan_skills(args: argparse.Namespace, root: Path, report: dict, as_json: bo
return catalog_scan.scan_per_skill(args, root, report, as_json, pairs)
skills = [entry for entry, _ in pairs]
reviews = review_skills(skills) # every skill is reviewed: the cross-catalog checks need them all
_upstream_findings(args, skills, reviews)
online = _online_checks(args, skills, reviews, report)
only = getattr(args, "only", None) # `skill check SKILL` reports this one skill (#94 M3)
if only is not None and only not in {entry.name for entry in skills}:
Expand Down Expand Up @@ -86,6 +87,17 @@ def _scan_skills(args: argparse.Namespace, root: Path, report: dict, as_json: bo
return failed, len(skills)


def _upstream_findings(args: argparse.Namespace, skills, reviews) -> None:
"""Add the ``--upstream`` / ``--upstream-manifest`` findings to the reviews; nothing is added without those flags."""
if not (getattr(args, "upstream", None) or getattr(args, "upstream_manifest", None)):
return
from dokimi.external import upstream
from dokimi.external.upstream_findings import add_upstream_findings

index = upstream.build(getattr(args, "upstream", None) or (), getattr(args, "upstream_manifest", None) or ())
add_upstream_findings(index, skills, reviews, ignore=frozenset(getattr(args, "ignore", None) or ()))


def _online_checks(args: argparse.Namespace, skills, reviews, report: dict) -> dict | None:
"""The ``--online`` network checks of the loaded skills, their summary into ``report``; None without the flag."""
if not args.online:
Expand Down Expand Up @@ -197,6 +209,18 @@ def _check_scan_flags(args: argparse.Namespace) -> None:
or args.jobs is not None or args.online or args.ignore):
raise InputError("--per-skill, --skip-nested, --ignore, --jsonl, --jobs, --resume and --online apply to "
"skills; drop them or use --kind skills|all")
_refuse_upstream_with(args)


def _refuse_upstream_with(args: argparse.Namespace) -> None:
"""``--upstream`` and ``--upstream-manifest`` check the skills of the whole catalog, in this process: not per-skill, not tasks."""
if not (getattr(args, "upstream", None) or getattr(args, "upstream_manifest", None)):
return
if args.per_skill:
# The upstream index is built once here; --jobs workers would each need their own copy of it.
raise InputError("--upstream and --upstream-manifest check the whole catalog at once and do not combine with --per-skill")
if args.kind == "tasks":
raise InputError("--upstream and --upstream-manifest apply to skills; drop them or use --kind skills|all")


def cmd_catalog_scan(args: argparse.Namespace) -> None:
Expand Down Expand Up @@ -473,6 +497,8 @@ def register(sub, positive_int, eval_sub) -> None:
library.add_list_show_product(list_p, show_p)
subjects.add_list_show_subject(list_p)

from dokimi import cli_flags

scan_p = catalog_sub.add_parser(
"check", help="Report which skills/tasks under --root load cleanly"
)
Expand Down Expand Up @@ -515,6 +541,16 @@ def register(sub, positive_int, eval_sub) -> None:
help="Leave the findings with this code out of the scan: they are not printed, and they "
"do not set the state or --strict's exit. Repeatable; an unknown code is a usage error",
)
cli_flags.add(
scan_p, "upstream", "--upstream", action="append", default=None, metavar="CHECKOUT",
help="Match each skill to a local git checkout of an upstream (see `catalog provenance`) and add the "
"upstream-stale, license-dropped-from-upstream and risk-inherited-from-upstream findings. Repeatable; "
"nothing is fetched. Not with --per-skill",
)
cli_flags.add(
scan_p, "upstream", "--upstream-manifest", action="append", default=None, metavar="FILE",
help="The same as --upstream, for a dokimi.upstream.v1 JSON manifest of one upstream. Repeatable",
)
scan_p.add_argument(
"--jsonl", metavar="PATH",
help="--per-skill: write one JSON row per skill to PATH as each is reviewed, and print "
Expand Down Expand Up @@ -577,6 +613,7 @@ def register(sub, positive_int, eval_sub) -> None:
ts_p.add_argument("--out", required=True, help="Output JSON path")
ts_p.set_defaults(func=cmd_catalog_trigger_set)

from dokimi.commands import catalog_probe
from dokimi.commands import catalog_probe, catalog_provenance

catalog_probe.register(catalog_sub, positive_int)
catalog_provenance.register(catalog_sub)
119 changes: 119 additions & 0 deletions dokimi/dokimi/commands/catalog_provenance.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# Copyright (C) 2026 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

"""``catalog provenance``: which upstream version each skill copy under ``--root`` came from, and how far behind it is.

Read-only and offline: the upstreams are the caller's (``--upstream`` checkouts, ``--upstream-manifest`` files). The
matching is in :mod:`dokimi.external.upstream`; this module reads the catalog and prints the rows.
"""

from __future__ import annotations

import argparse
import json
from pathlib import Path

from dokimi import cli_flags
from dokimi.utils import InputError


def _rel(skill_dir: Path, root: Path) -> str:
try:
return skill_dir.relative_to(root).as_posix()
except ValueError:
return skill_dir.as_posix()


def _behind_text(row: dict) -> str:
if row["match"] is None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: Negating the match is None check in _behind_text (line 28) still passes the suite: an unmatched copy prints unknown where - is expected.

Input: one unmatched skill and one matched skill, text output. With the condition negated the unmatched row's behind column is unknown and the matched row's is -, and the changed tests all stay green. Expected: - for the unmatched skill, pinned by a text-output case.

Suggestion: Add a text-output case with one unmatched skill and assert its behind column is -.

Patch, run on ce9a58d: the tests now catch the broken variant; tests and lint are no worse.

Patch
--- a/dokimi/tests/test_catalog_provenance.py
+++ b/dokimi/tests/test_catalog_provenance.py
@@ -157,6 +157,14 @@
     assert lines[1].startswith("# skills (1 under ")
     assert lines[2] == "skill\tmatch\tupstream\tversion\tbehind\tedited"
     assert lines[3].split("\t")[:2] == ["skills/pdf", "exact"]
+
+
+def test_text_output_shows_a_dash_for_an_unmatched_skill(upstream_repo, tmp_path) -> None:
+    rewritten = V2.split("# PDF")[0] + "# Sailing\nharbour sails tides anchors keel rudder mast hull wind\n"
+    root = _copy_root(tmp_path, {"skills/pdf/SKILL.md": V1, "skills/sail/SKILL.md": rewritten})
+    text = _run(root, "--upstream", str(upstream_repo), fmt="text")["text"]
+    rows = {line.split("\t")[0]: line.split("\t") for line in text.splitlines()[3:]}
+    assert rows["skills/sail"][1:] == ["unmatched", "-", "-", "-", "-"]
 
 
 def test_a_shallow_clone_is_said_to_see_only_its_own_history(upstream_repo, tmp_path) -> None:

return "-"
if row["at_head"]:
return "at head"
commits = "" if row["behind_commits"] is None else f"{row['behind_commits']} commits"
months = "" if row["behind_months"] is None else f"{row['behind_months']} mo"
return ", ".join(part for part in (commits, months) if part) or "unknown"


def _text(report: dict) -> str:
summary = report["summary"]
lines = []
for up in report["upstreams"]:
head = f"head {up['head_commit']} {up['head_date'] or ''}".strip() if up["head_commit"] else "no head"
note = " (shallow clone: older versions are not seen)" if up["history"] == "shallow" else ""
lines.append(f"# upstream {up['name']}: {head}, history {up['history']}{note}")
median = summary["median_months_behind"]
lines.append(
f"# skills ({summary['skills']} under {report['root']}: {summary['exact']} exact, "
f"{summary['similar']} similar, {summary['unmatched']} unmatched; {summary['edited']} edited; "
f"{summary['at_head']} at head; median {'-' if median is None else median} months behind)"
)
lines.append("skill\tmatch\tupstream\tversion\tbehind\tedited")
for row in report["skills"]:
upstream_text = "-" if row["upstream"] is None else f"{row['upstream']}:{row['path']}"
version = "-" if row["commit"] is None else f"{row['commit']}{' ' + row['date'] if row['date'] else ''}"
if row["version"]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: Negating if row["version"]: in _text (line 54) still passes the suite: a version loses its v prefix, and a skill with no version prints vNone.

Input: one skill with version: 1.0 and one with no version, text output. With the condition negated the first line reads 1.0 <commit> and the second reads vNone <commit>, and the changed tests all stay green. Expected: v1.0 <commit>, and no prefix for the unversioned skill, pinned by a text-output case.

Suggestion: Add a text-output case with a versioned and an unversioned skill and assert the version column.

Patch, run on ce9a58d: the tests now catch the broken variant; tests and lint are no worse.

Patch
--- a/dokimi/tests/test_catalog_provenance.py
+++ b/dokimi/tests/test_catalog_provenance.py
@@ -158,6 +158,14 @@
     assert lines[2] == "skill\tmatch\tupstream\tversion\tbehind\tedited"
     assert lines[3].split("\t")[:2] == ["skills/pdf", "exact"]
 
+
+
+def test_text_output_prefixes_only_a_skill_with_a_version(upstream_repo, tmp_path) -> None:
+    root = _copy_root(tmp_path, {"skills/pdf/SKILL.md": V2, "skills/old/SKILL.md": V1})
+    text = _run(root, "--upstream", str(upstream_repo), fmt="text")["text"]
+    rows = {line.split("\t")[0]: line.split("\t") for line in text.splitlines()[3:]}
+    assert rows["skills/pdf"][3].startswith("v2 ")
+    assert not rows["skills/old"][3].startswith("v")
 
 def test_a_shallow_clone_is_said_to_see_only_its_own_history(upstream_repo, tmp_path) -> None:
     shallow = tmp_path / "shallow"

version = f"v{row['version']} {version}"
edited = "-" if row["edited"] is None else "yes" if row["edited"] else "no"
lines.append(
f"{row['skill']}\t{row['match'] or 'unmatched'}\t{upstream_text}\t{version}\t"
f"{_behind_text(row)}\t{edited}"
)
return "\n".join(lines)


def cmd_catalog_provenance(args: argparse.Namespace) -> None:
from dokimi.external import read_skill_manifest
from dokimi.external import upstream

root = Path(args.root)
if not root.is_dir():
raise InputError(f"--root is not a directory: {root}")
if not args.upstream and not args.upstream_manifest:
raise InputError("catalog provenance needs --upstream CHECKOUT (a git checkout) or "
"--upstream-manifest FILE: dokimi ships no upstream list")
index = upstream.build(args.upstream or (), args.upstream_manifest or ())
try:
entries = read_skill_manifest(root)
except ValueError as exc:
raise InputError(f"skills.yaml: {exc}") from exc
rows = []
for entry in entries:
if not entry.loaded or entry.skill is None:
continue # a skill that does not load has no bytes to match: `catalog check` reports it
skill = entry.skill
match = index.match(skill.path.read_bytes(), skill.name)
rows.append({"skill": _rel(skill.path.parent, root), "name": skill.name, **upstream.to_row(match)})
report = {
"root": str(root),
"upstreams": upstream.upstream_rows(index),
"skills": rows,
"summary": upstream.summary(rows),
}
if args.format == "json":
print(json.dumps(report, indent=2))
else:
print(_text(report))


def register(catalog_sub) -> None:
"""Add ``catalog provenance`` to the ``catalog`` group."""
prov_p = catalog_sub.add_parser(
"provenance",
help="For each skill under --root: the upstream version it copies, how far behind head, and whether it was edited",
)
prov_p.add_argument("--root", required=True, help="Local checkout of the skills to match")
cli_flags.add(
prov_p, "upstream", "--upstream", action="append", default=None, metavar="CHECKOUT",
help="A local git checkout of an upstream (its history gives the versions). Repeatable; the first one "
"wins a tie. Nothing is fetched: clone it yourself, full history for an exact count",
)
cli_flags.add(
prov_p, "upstream", "--upstream-manifest", action="append", default=None, metavar="FILE",
help="A JSON manifest of one upstream: {\"schema\": \"dokimi.upstream.v1\", \"name\", \"head\": {commit, date}, "
"\"skills\": [{path, content, head?, commit?, date?}]}. Repeatable",
)
prov_p.add_argument(
"--format", choices=["text", "json"], default="text",
help="text: one line per skill (default); json: the upstreams, one row per skill and the summary",
)
prov_p.set_defaults(func=cmd_catalog_provenance)
Loading
Loading