Skip to content

fix(tracing): only tag skills a command actually reads or runs - #128

Open
Kiko0301 wants to merge 1 commit into
langfuse:mainfrom
Kiko0301:fix/skill-tag-command-attribution
Open

Kiko0301 wants to merge 1 commit into
langfuse:mainfrom
Kiko0301:fix/skill-tag-command-attribution

Conversation

@Kiko0301

@Kiko0301 Kiko0301 commented Nov 2, 2026

Copy link
Copy Markdown

Summary

skillsForToolCall() scans the raw exec_command text for …/SKILL.md and skills/…/scripts/ paths. It therefore treated a path that a command merely stages or prints as a skill invocation: git add -- .agents/skills/not-loaded/SKILL.md produced a skill:not-loaded trace tag and renamed the tool observation (langfuse/langfuse#19545).

Changes

  • Classify the shell segment that contains the skill path instead of the whole tool text. A path is attributed only when the command reads or executes it:
    • a reader/interpreter (cat, sed, grep, node, python, bash, …);
    • a read-only git subcommand (show, diff, grep, log, blame, cat-file);
    • a bare script path, which Codex records without an interpreter;
    • a nested exec_command({cmd: …}) in code-mode.
  • Unwrap sudo, env, timeout, command, nice, nohup, xargs and shell assignments, and follow bash -c "…", so legitimate reads through wrappers keep working.
  • Paths passed to staging/mutating/printing commands (git add, git status, cp, echo, …) no longer attribute the turn to that skill.
  • Added regression tests for the reported git add cases, wrappers, bash -c, git show, and nested code-mode commands.

Tests

  • node_modules/.bin/vitest run plugins/tracing/test/skills.test.ts: 8 passed (the existing five cases plus the new command-attribution cases).
  • npx prettier@3.9.8 --config .prettierrc.json --check plugins/tracing/src/skills.ts plugins/tracing/test/skills.test.ts: clean.
  • Full workspace typecheck/test was not run locally because the workspace dependencies are not installed in this checkout; the changed file is covered by the Vitest suite above.

This is a command-classification heuristic, not a blanket command allowlist: wrappers, -c shells, bare script paths, and code-mode exec_command calls are handled. It deliberately does not try to interpret arbitrary programs (for example a Python snippet that opens a skill file through a non-command API).

Fixes langfuse/langfuse#19545.

skillsForToolCall() scanned every exec_command text for a skills/ path,
so `git add -- .agents/skills/not-loaded/SKILL.md` produced a
skill:not-loaded tag and renamed the tool observation even though the
command only staged the file.

Classify the shell segment that contains the path: tag it only when the
command reads or executes the path (a reader/interpreter, a read git
subcommand, a bare script path, or a nested exec_command in code-mode).
Unwrap sudo, env, timeout and friends, and follow `bash -c` so
legitimate reads keep working. Commands that only stage, copy, remove,
or print a path no longer attribute the turn to that skill.

Fixes langfuse/langfuse#19545.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Codex skill tags match paths in commands that do not invoke a skill

1 participant