Skip to content

fix(cli): skip runtime by-products when fingerprinting a skill bundle - #5654

Merged
WaterrrForever merged 1 commit into
mainfrom
miao/skills-manifest-tracked-only
Oct 15, 2026
Merged

WaterrrForever merged 1 commit into
mainfrom
miao/skills-manifest-tracked-only

Conversation

@WaterrrForever

Copy link
Copy Markdown
Collaborator

What

hashSkillBundle (packages/cli/src/utils/skillsManifest.ts) now skips files a skill directory picks up from being used rather than authored: __pycache__/, .pytest_cache/, .mypy_cache/, .ruff_cache/, node_modules/ directories, and .DS_Store, Thumbs.db, *.pyc, *.pyo, *.swp, *.swo, *~ files. Previously only .DS_Store was skipped.

Why

The same hash builds the published skills-manifest.json from skills/ and compares an installed skill against it. Every file in the directory went into it, so:

  • Running a skill's Python (e.g. media-use/audio/scripts/lib/gemini-auth_test.py, which imports gemini-auth.py and leaves __pycache__/gemini-auth.cpython-*.pyc) changed the local hash. gen:skills-manifest then wrote a hash CI's clean checkout could not reproduce, and --check failed with "skills-manifest.json is out of date". This happened twice on feat(skills)!: publish seven knowledge skills aligned with Studio and retire the workflow skills #5651.
  • On a user's machine, an installed skill that ever ran a Python script read as outdated permanently, since no update removes the bytecode.

Reproduction on main before this change:

mkdir -p skills/media-use/audio/scripts/lib/__pycache__
printf '\x00junk' > skills/media-use/audio/scripts/lib/__pycache__/gemini-auth.cpython-312.pyc
bun packages/cli/scripts/gen-skills-manifest.ts --check
✗ skills-manifest.json is out of date — a skill changed without regenerating it.
  ~ media-use (c4c3c723aa04bb7d → b0581facb38068ae)

With this change the same command prints ✓ skills-manifest.json is in sync (21 skills).

Related work

Refs #5651. That branch's commit 3cd764422 ("regenerate the skills manifest from tracked files only") changes only skills-manifest.json data (a regeneration after the stray bytecode was removed); it does not change the generator or the hash. This PR does not touch skills-manifest.json, so the two do not conflict, and once this lands a local regeneration on #5651 can no longer pick up bytecode.

How

  • The skip lives in listFilesSorted, which both the source path (buildManifest → gen-skills-manifest) and the installed path (hashInstalled/checkSkills) go through, so the two sides stay symmetric.
  • No tracked file under skills/ matches the skip set (checked with git ls-files skills), so every published hash and file count is unchanged and skills-manifest.json needs no regeneration. Installed skills that are already current stay current.
  • A name-based skip, not a git ls-files walk: the installed side has no git, and the hash must be computed identically on both sides.

Test plan

  • Unit tests added: two cases in packages/cli/src/utils/skillsManifest.test.ts — __pycache__/*.pyc and a stray .pyc leave the hash and file count unchanged while a real .py edit changes it; node_modules/, .DS_Store, .swp, ~ backups leave it unchanged. Both fail without the fix (2 failed / 59 passed) and pass with it.
  • cd packages/cli && bun x vitest run src/utils/skillsManifest.test.ts src/utils/skillsMirror.test.ts src/utils/skillsUpdateCheck.test.ts src/commands/skills.test.ts → 4 files, 133 tests passed
  • bun packages/cli/scripts/gen-skills-manifest.ts --check → ✓ skills-manifest.json is in sync (21 skills), both on a clean tree and with a __pycache__/*.pyc planted in media-use (the reproduction above)
  • bun run lint → 0 warnings, 0 errors
  • bun x oxfmt --check on both changed files → correct format
  • bun x tsc --noEmit -p packages/cli (after bun run --cwd packages/core build) → exit 0, no errors
  • npx -y fallow audit --base origin/main --threads 4 --fail-on-issues → no issues in 2 changed files
  • COMMENT_CHECK_BASE=origin/main node scripts/check-comment-citations.mjs → OK; node scripts/comment-ratchet.mjs → at or under base share
  • Comments follow CONTRIBUTING.md "Comments"

No visible change

CLI internals only: the skill fingerprint ignores runtime by-products. Published hashes are unchanged.

🤖 Generated with Claude Code

The skill hash walked every file in a skill directory, so Python
__pycache__/*.pyc left by running a skill script (or node_modules, editor
swap files) changed it. Locally gen-skills-manifest then wrote a hash CI's
clean checkout could not reproduce, and an installed skill that had run a
script read as outdated forever. Skip those by-products on both the source
and the installed side so the hash stays a function of authored content.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 154 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (1)

  • seqplainkeys-none-pct-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at 2aa4fdf92623ca2a38b9d56e4ce256cbf80af011.

  • One walker for both sides. listFilesSorted is the only file walk behind hashSkillBundle, so the published manifest (buildManifest) and the installed-skill check (hashInstalled) skip the same names. The bundle hash has no other implementation in packages/ or scripts/ (publishProject.ts has its own ignore set, but for a different purpose).
  • Published hashes don't move. Nothing tracked under skills/ matches the skip set: I checked git ls-files skills against every dir name and suffix. So skills-manifest.json needs no regeneration, and installed skills that are current stay current.
  • Tests. The .pyc case also edits the .py and asserts the hash changes, so the skip can't hide a real edit. The second case covers node_modules, .DS_Store, .swp and ~.
  • Scope. Directories are skipped by exact name, files by exact name or suffix. The ~ suffix is the broadest rule, but no skill file ends in ~.

Required CI is green.

— Rames

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 2aa4fdf92623ca2a38b9d56e4ce256cbf80af011. No blocker found.

packages/cli/src/utils/skillsManifest.ts:179 applies the same traversal to source manifest generation and installed-bundle checks, so running a script no longer makes an unchanged install look stale. Authored files still affect the fingerprint.

Independent verification: 61 manifest tests pass; restoring the base implementation fails exactly the two added tests. A separate filesystem probe through real buildManifest → checkSkills kept an install current after adding every excluded cache directory/file class, then marked a real .py edit outdated. No tracked skill path matches the new skip set; the committed 21-skill manifest passes regeneration check without changes. Actual hosted runtime logs execute all 61 manifest tests; current-head checks have no failures or pending jobs.

Limits: synthetic installed directories, not a real Claude/Cursor/Codex profile installation. No install, paid call, merge or release performed.

— Magi (desktop-review)

Verdict: APPROVE
Reasoning: The shared hashing path removes runtime noise symmetrically while retaining authored-change detection, supported by direct integration and negative-control checks.

@WaterrrForever
WaterrrForever added this pull request to the merge queue Oct 15, 2026
Merged via the queue into main with commit 08ee1bd Oct 15, 2026
65 checks passed
@WaterrrForever
WaterrrForever deleted the miao/skills-manifest-tracked-only branch October 15, 2026 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants