Repository navigation
fix(tp4): anchor findings to the assessed file, align severity, and drop absence-only mismatches - #1069
fix(tp4): anchor findings to the assessed file, align severity, and drop absence-only mismatches#1069Varshith-Kali wants to merge 2 commits into
Conversation
…rop absence-only mismatches TP4 reported every finding at SKILL.md:1 even when the assessed fence lived in another file, escalated severity to HIGH at confidence 0.7 while the README pattern table lists TP4 as MEDIUM, and flagged partial workflow snippets for omitting the rest of the declared workflow. Anchor each TP4 finding to the batch file and lines, which the evidence already carried. TP4 severity is now always MEDIUM, matching the README. Add a partial-snippet criterion to the TP4 prompt, and suppress an assessment deterministically when every mismatched_capabilities entry is phrased as an absence, since that carries no affirmative evidence of a mismatch. Mixed and affirmative assessments still emit. Fixes NVIDIA#1066 Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Decision: Changes Requested (reviewed head e739a9a59f8f022f5a3782b85f6c818b0073d5be).
Value and readiness: Anchoring TP4 to the assessed file and lines fixes actionable evidence, and MEDIUM matches the documented severity. The partial-workflow prompt change is sensible. The deterministic absence guard, however, also discards affirmative evidence of undeclared behavior.
Material finding — blocker, P2: In src/skillspector/nodes/analyzers/mcp_tool_poisoning.py:1172-1185, words such as without, missing, and lacks anywhere in a capability make the entire item count as an absence. _tp4_finding then returns None at lines 1201-1207 if all items satisfy that test. For example, an assessment with is_mismatch=True, confidence 0.9, and one item Uploads source files to an undeclared endpoint without user consent is suppressed because it contains without. That item describes an affirmative undeclared capability; neither an affirmative actual-behavior summary nor explanation can rescue it. The old code emitted the finding.
Please make suppression conservative: only discard an assessment when every item actually states absence of a declared workflow capability, retaining any item that also describes affirmative behavior. Add regressions for an affirmative capability containing without and for an item containing both an affirmative action and a negated clause. The current mixed test puts the affirmative evidence in a separate list item, so it does not cover this failure.
PIC tradeoffs: None requiring a separate decision for this fix.
Verification and gaps: Reviewed the complete three-file diff and pinned source, tracing the new guard through finding construction. The regression assertions cover anchoring, severity, pure absence, separate mixed entries, affirmative entries, and HTTP graph anchoring; they omit affirmative and absence wording within the same entry. All six hosted CI checks passed for this exact head (run). No contributor code, tests, imports, builds, setup, hooks, regex probes, or live model were executed. The finding above follows directly from the literal without matcher and unconditional suppression branch; it is not a claimed runtime reproduction.
…DIA#1069) An absence marker anywhere in a mismatched capability (without, missing, lacks) suppressed the whole assessment, even when the item described an affirmative undeclared capability such as uploading source files to an undeclared endpoint without user consent. Make suppression conservative: an item counts as an absence only when no clause of it describes affirmative behavior, with capability verbs under a marker treated as governed by it. Add regressions for an affirmative capability containing without and for an item holding both an affirmative action and a negated clause. Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
|
Thanks for the catch. Fixed: Added regressions: Evidence: tests/test_mcp_tool_poisoning.py 180 passed (6 deselected); analyzer suite 12,133 passed, 4 skipped, 4 xfailed. New head dd3b48a. |
Fixes #1066.
Three parts, all in the TP4 path:
Anchoring.
_tp4_findinghardcodedfile="SKILL.md"with no line, so a finding onreferences/briefing-authoring.md:27-28reported atSKILL.md:1. Findings now carry the assessed batch file and lines, which the evidence already had. The meta-analyzer formatter needed no change, it already rendersfileandstart_line.Severity. TP4 escalated to HIGH at confidence 0.7, while the README pattern table lists TP4 as MEDIUM. TP4 is now always MEDIUM, matching the documented pattern.
Partial-snippet criterion. A fence implementing one declared workflow step was flagged for omitting the rest of the workflow, with
mismatched_capabilitiesphrased as absences ("No evidence of querying videos"). The TP4 prompt now states a chunk may implement only one declared workflow step and each entry must name what the code does. Deterministically, an assessment is suppressed when everymismatched_capabilitiesentry is phrased as an absence, since that carries no affirmative evidence of a mismatch. Mixed or affirmative assessments still emit.Tests: 13 new regression tests in
tests/test_mcp_tool_poisoning.py(12 failed pre-fix, 13 pass post-fix), including the issue verbatim anchoring case and the video-intel-search absence items.tests/test_mcp_tool_poisoning.py174 passed,tests/unit/test_tp4_openai_http.py51 passed, full suite 20964 passed with 8 failures reproduced on clean main (pre-existing Anthropic provider issues, unrelated). Ruff check and format clean. The prompt addition was kept compact (206 to 238 tokens) so the batch budget tests still pass.Limits, stated plainly: the deterministic guard covers only the pure-absence class. The issue EPUB and PDF findings named affirmative behaviors, and the real pytest finding was mixed, so those still emit and rely on the prompt criterion plus meta-review, which are nondeterministic. No live-LLM verification was run; the LLM-judged part was verified with stubbed inputs only.