Skip to content

feat(catalog): catalog check --upstream adds provenance findings - #849

Open
napetrov wants to merge 6 commits into
intel:mainfrom
napetrov:feat/gs-provenance-findings
Open

napetrov wants to merge 6 commits into
intel:mainfrom
napetrov:feat/gs-provenance-findings

Conversation

@napetrov

Copy link
Copy Markdown
Contributor

Summary

Stacked on #838. catalog check gets --upstream CHECKOUT and --upstream-manifest FILE (the same inputs as catalog provenance). Each loaded skill that matches an upstream version gets three findings:

catalog check --root DIR --upstream CHECKOUT
  for each loaded skill:
    match = index.match(SKILL.md bytes, name)        # PR #838
    upstream-stale                 note  match is not the head of its path (the head is known)
    license-dropped-from-upstream  warn  edited copy, upstream declares license, copy does not
    risk-inherited-from-upstream   warn  copy's own risk codes ∩ upstream SKILL.md risk codes

Risk codes are allowed-tools-unrestricted-bash, pipe-to-shell, permission-bypass-flag. The copy side is its own review findings; the upstream side is its SKILL.md alone.

Evidence

  • make ci-local at ce9a58d: 0 failed, 34 passed, 14 skipped.
  • tests/test_catalog_upstream_findings.py (18 functions, 19 cases): stale note, head not stale, unmatched gets nothing, license dropped (edited only), risk inherited (exact and edited), a copy's own risk not inherited, --ignore (also for inherited risks), --strict unchanged, refusal with --per-skill and --kind tasks, bad manifest, no head flag, a head flag on one path not staling another.
  • Red-then-green: the --ignore, no-head-flag and per-path tests fail on the previous commits and pass here.

Decisions

  • Refused with --per-skill: the upstream index is built once in one process; --jobs workers would each need a copy.
  • license-dropped-from-upstream only for dropped licenses: a changed value is not reported.
  • upstream-stale needs a known head for the matched path: always for a checkout; for a manifest only when an entry of that same path is flagged "head": true. A manifest's top-level head block does not say which entry is head, so no note is given. A skill listed under an old path with no flag gets no note either. Conservative: a note is never a guess. It is a note, so it changes neither the state nor the exit.
  • --ignore also removes the ignored risk codes from the inherited check: an ignored code is neither inherited nor named.
  • Upstream risks read from SKILL.md only: an upstream version is one text (blob or manifest entry), not a folder.
  • New findings do not fail --strict: D5 keeps only unrestricted Bash with a risky command as the strict failure.

Merge Danger

Door: two-way. New codes are additive; --upstream is opt-in.

Blast Radius: catalog check output for callers who pass the new flags only. FINDING_CODES grows by three (--ignore accepts them). The CLI snapshot gains two flags. Base moved while this was in review (b434f31a0, 185fee6cd, both on #838); this branch is rebased onto them.

🤖 Generated with Claude Code

… version

`dokimi catalog provenance --root DIR --upstream CHECKOUT|--upstream-manifest FILE`
reports, for each skill under DIR, the upstream skill and version it copies, how
many commits and months behind upstream head it is, and whether it was edited.

Offline: upstreams are named by the caller (local git checkouts, or a
dokimi.upstream.v1 JSON manifest). dokimi ships no upstream list. A copy matches
by git blob sha against the upstream history first, then by frontmatter name and
word Jaccard similarity >= 0.8 (the threshold of the corpus study). The git reads
use the same hardened wrapper as `catalog check`.

Findings (upstream-stale, license-dropped, risk-inherited) come in a follow-up.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ns and duplicate bytes

- A sha-256 checkout prints 64-hex blob ids, so no byte-identical copy matched exactly.
  A copy's bytes are now looked up under both the sha-1 and the sha-256 blob id.
- `log --raw` prints no diff for a merge commit, so a SKILL.md version a merge
  wrote was never indexed and commits-behind was short. Use --diff-merges=first-parent.
- Several versions with identical bytes: the newest commit date wins the exact
  match (it was the first listed), as the module docstring says.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…, as documented

The newest-date key ranked above the upstream order, so a later-dated upstream beat
an earlier one on the command line. Keep the date as the tie-break within one upstream.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
`catalog check --upstream CHECKOUT` and `--upstream-manifest FILE` match each
loaded skill to an upstream version (the catalog provenance matcher) and add:

- upstream-stale (note): the copy matches an upstream version that is not the head.
- license-dropped-from-upstream (warn): an edited copy drops the upstream license.
- risk-inherited-from-upstream (warn): the copy keeps unrestricted Bash,
  pipe-to-shell or permission-bypass-flag that its upstream version also has.

Offline; nothing is fetched. Refused with --per-skill and --kind tasks. The
findings are warnings or notes, so --strict and the exit code are unchanged.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A manifest with no entry flagged as head made every match upstream-stale, and the
note then said the copy was behind the head. Stale is now reported only when the
upstream's head is known: a checkout always, a manifest when an entry is flagged.

--ignore now also drops the ignored risk codes before risk-inherited-from-upstream
is built, so an ignored code is neither inherited nor named in a message.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ream

Index.head_known checks the (upstream, path) of the matched version, so one
flagged entry no longer makes the unflagged paths of that manifest stale-able.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov requested a review from a team as a code owner November 16, 2026 23:30
@github-actions github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Nov 16, 2026
@github-actions

Copy link
Copy Markdown

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

+ added command `dokimi catalog provenance`
    + option `--format`
    + option `--help/-h`
    + option `--root`
    + option `--upstream`
    + option `--upstream-manifest`
~ `dokimi catalog check` parameters changed:
    + added option `--upstream`
    + added option `--upstream-manifest`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

@exai-bot

exai-bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🔍 OpenCodeReview found 6 issue(s) in this PR.

CI at ce9a58d: 25 passing, 3 skipped.

By severity: 1 high, 5 medium

Severity Category Location Finding
high security upstream.py:287 Must-fix — disable signature verification on both new git log calls.
medium documentation external-content.md:484 Must-fix — Impact: users can interpret the absence of upstream-stale as evidence that an old copy is current, despite supplying a manif...
medium bug upstream.py:335 Must-fix — validate commit fields before storing them.
medium bug upstream.py:194 Must-fix — decode all Git C-style path escapes, or use NUL-delimited history output.
medium bug upstream.py:372-373 Must-fix — handle checkout startup errors as InputError.
medium bug upstream.py:258 Must-fix — explicitly request root-commit diffs with --root.
Review details
  • Commit reviewed: ce9a58d against merge base ba74e4a
  • Coverage: 16 files in 2 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 1.38M tokens (1.01M from cache), 2m45s
  • Tool calls: 80 (50 file_read, 23 code_search, 4 code_comment, 3 file_read_diff)
  • Catalog provenance command, CLI integration, and documentation: 7 files
  • Catalog upstream findings and validation: 9 files

Give `catalog check` the same upstreams as `catalog provenance` and each loaded skill that matches one gets
up to three more findings. Nothing is fetched, and none of them is an error:

- `upstream-stale` (note): the copy matches an upstream version that is not the upstream head. A note does not

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

documentation · medium
Must-fix — Impact: users can interpret the absence of upstream-stale as evidence that an old copy is current, despite supplying a manifest that cannot establish this. Trigger: a manifest has a top-level head but no entry with "head": true for the matched path (including when another path is flagged). Evidence: Index.head_known() requires a flagged entry for that same path, and upstream_findings() suppresses the note otherwise; test_a_manifest_with_no_head_flag_says_nothing_is_stale and test_a_flagged_head_on_one_path_does_not_make_another_path_stale explicitly assert this behavior. Document that prerequisite and distinguish the top-level repository head from each skill path's head. Workaround: flag the current entry of each relevant path with "head": true, or use a checkout. Severity: medium because this is a bounded mismatch in the diagnostic contract, with no security or data-loss impact.

Check run: inconclusive, the result is not shown as a pass.

Suggestion:

Suggested change
- `upstream-stale` (note): the copy matches an upstream version that is not the upstream head. A note does not
- `upstream-stale` (note): the copy matches a non-head version of a path whose head is known. For manifests,
that requires an entry of the same path with `"head": true`; the top-level `head` block alone is insufficient. A note does not

body = path[1:-1].encode("utf-8", errors="surrogateescape")
out = bytearray()
i = 0
escapes = {ord("t"): 9, ord("n"): 10, ord('"'): 34, ord("\\"): 92}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — decode all Git C-style path escapes, or use NUL-delimited history output. Trigger: commit two versions of skills/a\r/SKILL.md, where \r is an actual carriage return in the directory name. Evidence: Git quotes that path using \\r, but this mapping leaves it as literal backslash-plus-r; _head_blobs instead reads the real carriage return from its NUL-delimited listing. Consequently first_seen cannot join the HEAD path to its history: the head version loses its commit/date, and older versions are reported with the wrong path. Impact: incorrect provenance attribution and finding messages for valid Git paths. Workaround: rename directories to exclude these control characters. Severity: medium because provenance results are wrong for a bounded input class. The same problem affects Git's \\a, \\b, \\v and \\f escapes; cover history/HEAD agreement in a regression test.

commits that touched each path (newest first). A deletion is a touch with no blob."""
# --diff-merges=first-parent: a merge commit prints no diff by default, so a SKILL.md version that a merge
# resolved (or changed) would never be seen and the commits it counts as behind would be short.
log = _git(root, "-c", "core.quotepath=off", "log", "--raw", "--no-abbrev", "--no-renames", "--full-history",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — explicitly request root-commit diffs with --root. Impact: the index can silently omit the initial version of a skill, losing exact matches and their stale findings. Trigger: an upstream has log.showRoot=false, adds SKILL.md in its root commit, and edits it in a later commit; the catalog contains the initial bytes. Evidence: this log command leaves root-diff behavior to repository configuration, and first_seen only receives blobs from raw diff records. _head_blobs restores only the later HEAD bytes, so the initial blob is absent; if similarity to HEAD is below 0.8, the copy becomes unmatched. Workaround: set log.showRoot=true in the checkout or supply the initial version via a manifest. Severity justification: medium because supported checkout input produces a bounded but materially incorrect provenance verdict. Add --root and a regression test with log.showRoot=false.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, and the result is worse than unmatched. With log.showRoot=false the root version is never indexed, so an unedited copy of it matches the head version as similar: catalog provenance prints at head and edited: yes for a copy that is one change behind. Passing --root to the log in _history fixes it.

if head.returncode:
raise ValueError(f"{root} is not a git checkout with a commit: --upstream takes a git checkout")
head_sha = head.stdout.strip()
head_date_text = _git(root, "log", "-1", "--format=%cI", head_sha).stdout.strip()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security · high
Must-fix — disable signature verification on both new git log calls. Impact: inspecting an upstream can execute a repository-configured verification program with the caller's privileges, violating the data-only inspection contract. Trigger: a checkout containing a signed commit with local log.showSignature=true and gpg.program (or the signature-format-specific program) pointing to an executable. Evidence: neither this call nor _history passes --no-show-signature; the _git wrapper disables hooks/fsmonitor/transports but not signature verification. The existing guidance-history reader explicitly disables this option. Workaround: use a manifest or remove these checkout settings. Severity justification: high because an opt-in inspection of a supplied checkout can execute arbitrary local code. Add --no-show-signature to both calls and test that a configured verifier is never invoked.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: Confirmed: a checkout with log.showSignature=true, gpg.program pointing to a script, and one commit with a gpgsig header runs that script twice during catalog provenance (once per log call). Adding -c log.showSignature=false to catalog_git._git, next to the core.fsmonitor and core.hooksPath overrides, covers both calls and any later log there.

head_raw = data.get("head")
head: dict[str, Any] = head_raw if isinstance(head_raw, dict) else {}
upstream = Upstream(
name=name.strip(), order=order, head_commit=head.get("commit"),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — validate commit fields before storing them. Trigger: a manifest with "head": {"commit": 123} or a matching skill entry with "commit": 123. Evidence: these values pass through load_manifest unchanged, but upstream_rows and to_row later apply [:12]; catalog provenance calls both, producing TypeError and the CLI's infrastructure exit 5 rather than InputError/exit 2. Impact: malformed manifest input crashes report generation instead of receiving a controlled validation error. Workaround: use string or null commit values. Severity: medium because this is a bounded failure of the manifest workflow. Validate both top-level and entry commit values as string-or-null and add regression cases.

Comment on lines +372 to +373
except ValueError as exc:
raise InputError(str(exc)) from exc

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — handle checkout startup errors as InputError. Trigger: catalog check --root VALID_CATALOG --upstream /nonexistent (also catalog provenance), or an upstream path naming a regular file. Evidence: load_checkout immediately invokes _git with that path as cwd; process.git propagates the resulting FileNotFoundError/NotADirectoryError, while this handler catches only ValueError and the CLI handles InputError, not OSError. Impact: bad user input produces a traceback and exit 1 instead of the actionable usage error/exit 2 used for invalid upstreams. Workaround: verify that the checkout directory exists before invoking the command. Severity justification: medium, a bounded failure of the supported CLI input-error contract. Catch expected OSError failures around checkout loading and translate them to InputError with the source path; add missing-path and regular-file cases.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 81.62291% with 77 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.64%. Comparing base (ba74e4a) to head (ce9a58d).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
dokimi/dokimi/external/upstream.py 78.27% 39 Missing and 19 partials ⚠️
dokimi/dokimi/commands/catalog_provenance.py 82.35% 8 Missing and 4 partials ⚠️
dokimi/dokimi/external/upstream_findings.py 87.50% 3 Missing and 4 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #849      +/-   ##
==========================================
- Coverage   95.75%   95.64%   -0.11%     
==========================================
  Files         423      427       +4     
  Lines       56416    56833     +417     
  Branches     9714     9782      +68     
==========================================
+ Hits        54019    54357     +338     
- Misses       1489     1541      +52     
- Partials      908      935      +27     
Files with missing lines Coverage Δ
dokimi/dokimi/cli_flags.py 100.00% <ø> (ø)
dokimi/dokimi/cli_routes.py 97.47% <ø> (ø)
dokimi/dokimi/commands/catalog.py 99.16% <100.00%> (+0.04%) ⬆️
dokimi/dokimi/external/catalog_git.py 89.77% <100.00%> (ø)
dokimi/dokimi/external/command_checks.py 96.79% <100.00%> (+0.07%) ⬆️
dokimi/dokimi/external/upstream_codes.py 100.00% <100.00%> (ø)
dokimi/dokimi/external/validate.py 97.41% <100.00%> (+<0.01%) ⬆️
dokimi/dokimi/external/upstream_findings.py 87.50% <87.50%> (ø)
dokimi/dokimi/commands/catalog_provenance.py 82.35% <82.35%> (ø)
dokimi/dokimi/external/upstream.py 78.27% <78.27%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at ce9a58d. The matching by blob sha and the three findings work on a plain linear history, but the history reader still trusts the checkout's own git config and miscounts merges. The details are in the line comments. This branch also carries all of #838.

Agreed with @exai-bot: gpg.program from the checkout runs via log.showSignature; the root commit is missed with log.showRoot=false; a bad --upstream path gives a traceback; non-string commit values crash; _unquote misses \r and the other escapes; the docs do not say upstream-stale needs a flagged head entry.

What I ran, at ce9a58d:

  • make ci-local --offline: 22,234 passed, 0 failed, 72 skipped.
  • Scratch upstream repos run with catalog provenance and catalog check: a --no-ff merge (2 commits behind for one change), log.showRoot=false (a stale copy reads at head), a fake signed commit with gpg.program set (the script ran twice), a --filter=blob:none clone, a <CR> directory name, a manifest with no head flag, and a missing or file --upstream path.

Checked, no issues

  • Matching: exact matches over sha-1 and sha-256 ids, the newest-version tie-break within one upstream, and the Jaccard fallback by frontmatter name behave as the docs say.
  • catalog check wiring: the findings respect --ignore, skip unreadable skills, and are refused with --per-skill and --kind tasks; the new codes are in FINDING_CODES.
  • Docs and changelog: both fragments are under Added and describe the user-facing change; the CLI snapshot is current; new files carry the SPDX header.

Comment on lines +258 to +259
log = _git(root, "-c", "core.quotepath=off", "log", "--raw", "--no-abbrev", "--no-renames", "--full-history",
"--diff-merges=first-parent", "--format=C %H %cI", "HEAD", "--", _SKILL_PATHSPEC)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

behind counts a change twice when it came in through a merge commit.

--full-history lists the side-branch commit, and --diff-merges=first-parent lists the merge with the same change. Repro: v1 on main, v2 committed on a branch, merge --no-ff; a copy of v1 is reported 2 commits behind, though the file changed once. Commits on a parallel branch are also counted by date order, not by whether they come after the version.

Suggestion: keep the full history to find the versions, but count behind on the first-parent chain only (for a version from a side branch, from the merge that brought it in).

Comment on lines +394 to +396
"at_head": v.head,
"behind_commits": 0 if v.head else v.behind,
"behind_months": v.months_behind,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

catalog provenance reports months behind when the manifest does not say which entry is the head.

For a manifest path with no entry flagged "head": true, to_row gives at_head: false and behind_months from the top-level head date, and the summary median counts it. catalog check with the same manifest treats the head as unknown (Index.head_known) and reports no upstream-stale. Repro: head date 2026-04-01, one unflagged entry dated 2026-01-01: provenance prints 3.0 mo behind, check prints nothing.

Suggestion: pass head_known to to_row and give at_head, behind_commits and behind_months as null (text: unknown) when it is False.

Comment on lines +237 to +239
done = _git(root, "cat-file", "--batch", input=("\n".join(unique) + "\n").encode())
if done.returncode:
raise ValueError(f"{root}: cannot read blobs: {_git_error(done)}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A blobless partial clone fails the whole upstream instead of leaving out the missing blobs.

The docstring says a partial clone may lack some blobs and they are skipped, but cat-file --batch first tries to lazy-fetch a missing blob; protocol.allow=never blocks that and git exits 128. After git clone --filter=blob:none, catalog provenance --upstream <clone> gives cannot read blobs: fatal: could not fetch <sha> from promisor remote.

Suggestion: refuse a partial clone with a clear message and fix the docstring. Skipping the blobs instead would need the history marked partial, as for a shallow clone, since a copy of a missing version then reads as similar and edited.

"""The index over the given checkouts, then manifests (the order is the tie-break). Names must be distinct."""
upstreams: list[Upstream] = []
versions: list[Version] = []
sources = [(load_checkout, Path(p)) for p in checkouts] + [(load_manifest, Path(p)) for p in manifests]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With checkouts and manifests mixed, a tie does not go to the first upstream listed.

build puts every --upstream before every --upstream-manifest, and _exact_rank compares behind before order, where a manifest version's None becomes 1 << 30. So with --upstream-manifest a.json --upstream b and the same non-head bytes in both, b wins, while the --upstream help and the changelog say the first one listed wins.

Suggestion: say in the help and the docs that checkouts come before manifests.

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not approved yet · reviewed ce9a58d

Next: @napetrov: answer 10 open threads from exai-bot, vlad-perevezentsev and settle 1 open question (below). 2 nits optional.

Blocking

  • Still holds on this head: thread by exai-bot – upstream-stale is documented as a note for any copy that matches a non-head version.
  • Still holds on this head: thread by exai-bot – _unquote leaves \r (and \a, \b, \v, \f) as literal backslash pairs, so the history path of a directory with a carriage return does not join its HEAD path: an exact copy of its head version reads commit null.
  • Still holds on this head: thread by exai-bot – With log.showRoot=false the root version of a skill is not indexed: an unedited copy of it matches the head as similar and reads at head.
  • Still holds on this head: thread by exai-bot – A checkout with log.showSignature=true and a gpg.program script: build runs that script, and the marker file the script writes is created.
  • Still holds on this head: thread by exai-bot – A manifest with "commit": 123 on its head or an entry passes it on, and upstream_rows raises TypeError where a controlled InputError is expected.
  • Still holds on this head: thread by exai-bot – A missing --upstream path reaches build as FileNotFoundError (a regular file as NotADirectoryError), and catalog provenance exits 5, not 2.
  • Still holds on this head: thread by @vlad-perevezentsev – A copy of a version that reached main through a --no-ff merge of a branch reads behind 2 for one change, because the side-branch commit and the merge both count.
  • Still holds on this head: thread by @vlad-perevezentsev – A manifest path with no flagged head reads 3.0 months behind (head 2026-04-01, entry 2026-01-01), while catalog check treats its head as unknown.
  • Still holds on this head: thread by @vlad-perevezentsev – build on a --filter=blob:none clone raises InputError (could not fetch ... from promisor remote), where the docstring says missing blobs are skipped.
  • Still holds on this head: thread by @vlad-perevezentsev – A checkout wins an exact tie against a manifest with the same non-head bytes whichever flag comes first, because build lists checkouts before manifests.
  • Open question for @napetrov: When a checkout and a manifest hold the same non-head bytes, the checkout wins even when the manifest flag comes first on the command line, because build lists checkouts before manifests. State that order in the --upstream and --upstream-manifest help and in docs/external-content.md, or give a tie to the flag listed first.

Verified on ce9a58d: ✅ changed tests: 42 passed, fail on main · ✅ full suite with main merged in: 22874 passed · ✅ merges onto main · ⚠️ 12 broken variants of the changed code: 9 caught by the tests, 3 not · 7 mutations tried · 7 edge inputs tried

20 checks behind it, each with its evidence
Area What holds How
Changed tests the 42 tests in tests/test_catalog_provenance.py and tests/test_catalog_upstream_findings.py pass on the head tests on the head
Changed tests on main the same two test files fail to import on main, because dokimi.external.upstream does not exist there, so they pin only the new code same tests on main
Repo checks make ci-local --offline on the head with main merged in has no failing step full suite, main merged in
Merge the branch merges onto main with no conflict merge onto main
Strict exit a copy with an inherited risk gives the same catalog check --strict exit code with and without --upstream-manifest, and prints risk-inherited-from-upstream ran it
Ignored risk code --ignore allowed-tools-unrestricted-bash removes that code from the inherited finding and keeps pipe-to-shell ran it
Per-skill refusal catalog check --per-skill --upstream-manifest exits 2 with the refusal message ran it
License drop an edited copy that changes the license value is not reported, and one that drops the license is ran it
Upstream gate negating the --upstream flag gate in _upstream_findings fails the changed tests broke the code, tests caught it
Jaccard guard negating the empty-set guard in _jaccard fails the changed tests broke the code, tests caught it
Months behind negating the head check in Version.months_behind fails the changed tests broke the code, tests caught it
Provenance root check negating the --root directory check in cmd_catalog_provenance fails the changed tests broke the code, tests caught it
Unreachable raise the raise added after return skill_dir.as_posix() in _rel (catalog_provenance.py:24) is unreachable, so that broken variant behaves the same broke the code, tests caught it
Text _norm_name strips quotes and a carriage return and gives None for a whitespace-only name, and _tokens splits a CRLF text into its words ran it, edge input (text)
Malformed manifest truncated JSON, skills that is null or an object, a non-string content, a null name and a top-level list each raise InputError from load_manifest ran it, edge input (structured)
Manifest paths a missing manifest, a directory and an empty manifest file each raise InputError from load_manifest ran it, edge input (paths)
Empty summary summary of no rows gives zero skills and a null median ran it, edge input (numbers)
Duplicate upstream name the same manifest given twice is refused with InputError ran it, edge input (collections)
No upstream flag catalog provenance with no upstream flag exits 2 and says needs --upstream ran it, edge input (cli)
No name or match a None name gives no match, and to_row(None) gives match: null with the other fields null ran it, edge input (none)



def _behind_text(row: dict) -> str:
if row["match"] is None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: Negating the match is None check in _behind_text (line 28) still passes the suite: an unmatched copy prints unknown where - is expected.

Input: one unmatched skill and one matched skill, text output. With the condition negated the unmatched row's behind column is unknown and the matched row's is -, and the changed tests all stay green. Expected: - for the unmatched skill, pinned by a text-output case.

Suggestion: Add a text-output case with one unmatched skill and assert its behind column is -.

Patch, run on ce9a58d: the tests now catch the broken variant; tests and lint are no worse.

Patch
--- a/dokimi/tests/test_catalog_provenance.py
+++ b/dokimi/tests/test_catalog_provenance.py
@@ -157,6 +157,14 @@
     assert lines[1].startswith("# skills (1 under ")
     assert lines[2] == "skill\tmatch\tupstream\tversion\tbehind\tedited"
     assert lines[3].split("\t")[:2] == ["skills/pdf", "exact"]
+
+
+def test_text_output_shows_a_dash_for_an_unmatched_skill(upstream_repo, tmp_path) -> None:
+    rewritten = V2.split("# PDF")[0] + "# Sailing\nharbour sails tides anchors keel rudder mast hull wind\n"
+    root = _copy_root(tmp_path, {"skills/pdf/SKILL.md": V1, "skills/sail/SKILL.md": rewritten})
+    text = _run(root, "--upstream", str(upstream_repo), fmt="text")["text"]
+    rows = {line.split("\t")[0]: line.split("\t") for line in text.splitlines()[3:]}
+    assert rows["skills/sail"][1:] == ["unmatched", "-", "-", "-", "-"]
 
 
 def test_a_shallow_clone_is_said_to_see_only_its_own_history(upstream_repo, tmp_path) -> None:

for row in report["skills"]:
upstream_text = "-" if row["upstream"] is None else f"{row['upstream']}:{row['path']}"
version = "-" if row["commit"] is None else f"{row['commit']}{' ' + row['date'] if row['date'] else ''}"
if row["version"]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: Negating if row["version"]: in _text (line 54) still passes the suite: a version loses its v prefix, and a skill with no version prints vNone.

Input: one skill with version: 1.0 and one with no version, text output. With the condition negated the first line reads 1.0 <commit> and the second reads vNone <commit>, and the changed tests all stay green. Expected: v1.0 <commit>, and no prefix for the unversioned skill, pinned by a text-output case.

Suggestion: Add a text-output case with a versioned and an unversioned skill and assert the version column.

Patch, run on ce9a58d: the tests now catch the broken variant; tests and lint are no worse.

Patch
--- a/dokimi/tests/test_catalog_provenance.py
+++ b/dokimi/tests/test_catalog_provenance.py
@@ -158,6 +158,14 @@
     assert lines[2] == "skill\tmatch\tupstream\tversion\tbehind\tedited"
     assert lines[3].split("\t")[:2] == ["skills/pdf", "exact"]
 
+
+
+def test_text_output_prefixes_only_a_skill_with_a_version(upstream_repo, tmp_path) -> None:
+    root = _copy_root(tmp_path, {"skills/pdf/SKILL.md": V2, "skills/old/SKILL.md": V1})
+    text = _run(root, "--upstream", str(upstream_repo), fmt="text")["text"]
+    rows = {line.split("\t")[0]: line.split("\t") for line in text.splitlines()[3:]}
+    assert rows["skills/pdf"][3].startswith("v2 ")
+    assert not rows["skills/old"][3].startswith("v")
 
 def test_a_shallow_clone_is_said_to_see_only_its_own_history(upstream_repo, tmp_path) -> None:
     shallow = tmp_path / "shallow"

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli-interface-change Changes dokimi's user-facing CLI surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants