Repository navigation
feat(catalog): catalog check --upstream adds provenance findings #849
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
cdf421f
b434f31
185fee6
788459e
666a437
ce9a58d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| <!-- | ||
| Copyright (C) 2026 Intel Corporation | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| --> | ||
|
|
||
| ### Added | ||
|
|
||
| - **`catalog provenance` matches each skill copy to the upstream version it came from** — offline, from the `--upstream` git checkouts or `--upstream-manifest` files you name (dokimi ships no upstream list). A copy is matched by its git blob sha against the upstream history, then by frontmatter name and word Jaccard similarity ≥ 0.8. The report gives the origin skill and version, how many commits and months behind upstream head, and whether the copy was edited. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| <!-- | ||
| Copyright (C) 2026 Intel Corporation | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| --> | ||
|
|
||
| ### Added | ||
|
|
||
| - **`catalog check --upstream` and `--upstream-manifest` add provenance findings** — each skill that matches an upstream version gets `upstream-stale` (note: the copy is not at the upstream head), `license-dropped-from-upstream` (warn: an edited copy drops the upstream's `license`) and `risk-inherited-from-upstream` (warn: the copy keeps unrestricted Bash, `pipe-to-shell` or `permission-bypass-flag` that its upstream version also has). Offline, from the upstreams you name; not combined with `--per-skill` or `--kind tasks`. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,119 @@ | ||
| # Copyright (C) 2026 Intel Corporation | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """``catalog provenance``: which upstream version each skill copy under ``--root`` came from, and how far behind it is. | ||
|
|
||
| Read-only and offline: the upstreams are the caller's (``--upstream`` checkouts, ``--upstream-manifest`` files). The | ||
| matching is in :mod:`dokimi.external.upstream`; this module reads the catalog and prints the rows. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import argparse | ||
| import json | ||
| from pathlib import Path | ||
|
|
||
| from dokimi import cli_flags | ||
| from dokimi.utils import InputError | ||
|
|
||
|
|
||
| def _rel(skill_dir: Path, root: Path) -> str: | ||
| try: | ||
| return skill_dir.relative_to(root).as_posix() | ||
| except ValueError: | ||
| return skill_dir.as_posix() | ||
|
|
||
|
|
||
| def _behind_text(row: dict) -> str: | ||
| if row["match"] is None: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Nit: Negating the Input: one unmatched skill and one matched skill, text output. With the condition negated the unmatched row's behind column is Suggestion: Add a text-output case with one unmatched skill and assert its behind column is Patch, run on Patch--- a/dokimi/tests/test_catalog_provenance.py
+++ b/dokimi/tests/test_catalog_provenance.py
@@ -157,6 +157,14 @@
assert lines[1].startswith("# skills (1 under ")
assert lines[2] == "skill\tmatch\tupstream\tversion\tbehind\tedited"
assert lines[3].split("\t")[:2] == ["skills/pdf", "exact"]
+
+
+def test_text_output_shows_a_dash_for_an_unmatched_skill(upstream_repo, tmp_path) -> None:
+ rewritten = V2.split("# PDF")[0] + "# Sailing\nharbour sails tides anchors keel rudder mast hull wind\n"
+ root = _copy_root(tmp_path, {"skills/pdf/SKILL.md": V1, "skills/sail/SKILL.md": rewritten})
+ text = _run(root, "--upstream", str(upstream_repo), fmt="text")["text"]
+ rows = {line.split("\t")[0]: line.split("\t") for line in text.splitlines()[3:]}
+ assert rows["skills/sail"][1:] == ["unmatched", "-", "-", "-", "-"]
def test_a_shallow_clone_is_said_to_see_only_its_own_history(upstream_repo, tmp_path) -> None: |
||
| return "-" | ||
| if row["at_head"]: | ||
| return "at head" | ||
| commits = "" if row["behind_commits"] is None else f"{row['behind_commits']} commits" | ||
| months = "" if row["behind_months"] is None else f"{row['behind_months']} mo" | ||
| return ", ".join(part for part in (commits, months) if part) or "unknown" | ||
|
|
||
|
|
||
| def _text(report: dict) -> str: | ||
| summary = report["summary"] | ||
| lines = [] | ||
| for up in report["upstreams"]: | ||
| head = f"head {up['head_commit']} {up['head_date'] or ''}".strip() if up["head_commit"] else "no head" | ||
| note = " (shallow clone: older versions are not seen)" if up["history"] == "shallow" else "" | ||
| lines.append(f"# upstream {up['name']}: {head}, history {up['history']}{note}") | ||
| median = summary["median_months_behind"] | ||
| lines.append( | ||
| f"# skills ({summary['skills']} under {report['root']}: {summary['exact']} exact, " | ||
| f"{summary['similar']} similar, {summary['unmatched']} unmatched; {summary['edited']} edited; " | ||
| f"{summary['at_head']} at head; median {'-' if median is None else median} months behind)" | ||
| ) | ||
| lines.append("skill\tmatch\tupstream\tversion\tbehind\tedited") | ||
| for row in report["skills"]: | ||
| upstream_text = "-" if row["upstream"] is None else f"{row['upstream']}:{row['path']}" | ||
| version = "-" if row["commit"] is None else f"{row['commit']}{' ' + row['date'] if row['date'] else ''}" | ||
| if row["version"]: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Nit: Negating Input: one skill with Suggestion: Add a text-output case with a versioned and an unversioned skill and assert the version column. Patch, run on Patch--- a/dokimi/tests/test_catalog_provenance.py
+++ b/dokimi/tests/test_catalog_provenance.py
@@ -158,6 +158,14 @@
assert lines[2] == "skill\tmatch\tupstream\tversion\tbehind\tedited"
assert lines[3].split("\t")[:2] == ["skills/pdf", "exact"]
+
+
+def test_text_output_prefixes_only_a_skill_with_a_version(upstream_repo, tmp_path) -> None:
+ root = _copy_root(tmp_path, {"skills/pdf/SKILL.md": V2, "skills/old/SKILL.md": V1})
+ text = _run(root, "--upstream", str(upstream_repo), fmt="text")["text"]
+ rows = {line.split("\t")[0]: line.split("\t") for line in text.splitlines()[3:]}
+ assert rows["skills/pdf"][3].startswith("v2 ")
+ assert not rows["skills/old"][3].startswith("v")
def test_a_shallow_clone_is_said_to_see_only_its_own_history(upstream_repo, tmp_path) -> None:
shallow = tmp_path / "shallow" |
||
| version = f"v{row['version']} {version}" | ||
| edited = "-" if row["edited"] is None else "yes" if row["edited"] else "no" | ||
| lines.append( | ||
| f"{row['skill']}\t{row['match'] or 'unmatched'}\t{upstream_text}\t{version}\t" | ||
| f"{_behind_text(row)}\t{edited}" | ||
| ) | ||
| return "\n".join(lines) | ||
|
|
||
|
|
||
| def cmd_catalog_provenance(args: argparse.Namespace) -> None: | ||
| from dokimi.external import read_skill_manifest | ||
| from dokimi.external import upstream | ||
|
|
||
| root = Path(args.root) | ||
| if not root.is_dir(): | ||
| raise InputError(f"--root is not a directory: {root}") | ||
| if not args.upstream and not args.upstream_manifest: | ||
| raise InputError("catalog provenance needs --upstream CHECKOUT (a git checkout) or " | ||
| "--upstream-manifest FILE: dokimi ships no upstream list") | ||
| index = upstream.build(args.upstream or (), args.upstream_manifest or ()) | ||
| try: | ||
| entries = read_skill_manifest(root) | ||
| except ValueError as exc: | ||
| raise InputError(f"skills.yaml: {exc}") from exc | ||
| rows = [] | ||
| for entry in entries: | ||
| if not entry.loaded or entry.skill is None: | ||
| continue # a skill that does not load has no bytes to match: `catalog check` reports it | ||
| skill = entry.skill | ||
| match = index.match(skill.path.read_bytes(), skill.name) | ||
| rows.append({"skill": _rel(skill.path.parent, root), "name": skill.name, **upstream.to_row(match)}) | ||
| report = { | ||
| "root": str(root), | ||
| "upstreams": upstream.upstream_rows(index), | ||
| "skills": rows, | ||
| "summary": upstream.summary(rows), | ||
| } | ||
| if args.format == "json": | ||
| print(json.dumps(report, indent=2)) | ||
| else: | ||
| print(_text(report)) | ||
|
|
||
|
|
||
| def register(catalog_sub) -> None: | ||
| """Add ``catalog provenance`` to the ``catalog`` group.""" | ||
| prov_p = catalog_sub.add_parser( | ||
| "provenance", | ||
| help="For each skill under --root: the upstream version it copies, how far behind head, and whether it was edited", | ||
| ) | ||
| prov_p.add_argument("--root", required=True, help="Local checkout of the skills to match") | ||
| cli_flags.add( | ||
| prov_p, "upstream", "--upstream", action="append", default=None, metavar="CHECKOUT", | ||
| help="A local git checkout of an upstream (its history gives the versions). Repeatable; the first one " | ||
| "wins a tie. Nothing is fetched: clone it yourself, full history for an exact count", | ||
| ) | ||
| cli_flags.add( | ||
| prov_p, "upstream", "--upstream-manifest", action="append", default=None, metavar="FILE", | ||
| help="A JSON manifest of one upstream: {\"schema\": \"dokimi.upstream.v1\", \"name\", \"head\": {commit, date}, " | ||
| "\"skills\": [{path, content, head?, commit?, date?}]}. Repeatable", | ||
| ) | ||
| prov_p.add_argument( | ||
| "--format", choices=["text", "json"], default="text", | ||
| help="text: one line per skill (default); json: the upstreams, one row per skill and the summary", | ||
| ) | ||
| prov_p.set_defaults(func=cmd_catalog_provenance) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Must-fix — Impact: users can interpret the absence of
upstream-staleas evidence that an old copy is current, despite supplying a manifest that cannot establish this. Trigger: a manifest has a top-levelheadbut no entry with"head": truefor the matched path (including when another path is flagged). Evidence:Index.head_known()requires a flagged entry for that same path, andupstream_findings()suppresses the note otherwise;test_a_manifest_with_no_head_flag_says_nothing_is_staleandtest_a_flagged_head_on_one_path_does_not_make_another_path_staleexplicitly assert this behavior. Document that prerequisite and distinguish the top-level repository head from each skill path's head. Workaround: flag the current entry of each relevant path with"head": true, or use a checkout. Severity: medium because this is a bounded mismatch in the diagnostic contract, with no security or data-loss impact.Check run: inconclusive, the result is not shown as a pass.
Suggestion: