Skip to content

fix(tp4): anchor findings to the assessed file, align severity, and drop absence-only mismatches - #1069

Open
Varshith-Kali wants to merge 2 commits into
NVIDIA:mainfrom
Varshith-Kali:fix/tp4-workflow-fences-1066
Open

Varshith-Kali wants to merge 2 commits into
NVIDIA:mainfrom
Varshith-Kali:fix/tp4-workflow-fences-1066

Conversation

@Varshith-Kali

Copy link
Copy Markdown
Contributor

Fixes #1066.

Three parts, all in the TP4 path:

  1. Anchoring. _tp4_finding hardcoded file="SKILL.md" with no line, so a finding on references/briefing-authoring.md:27-28 reported at SKILL.md:1. Findings now carry the assessed batch file and lines, which the evidence already had. The meta-analyzer formatter needed no change, it already renders file and start_line.

  2. Severity. TP4 escalated to HIGH at confidence 0.7, while the README pattern table lists TP4 as MEDIUM. TP4 is now always MEDIUM, matching the documented pattern.

  3. Partial-snippet criterion. A fence implementing one declared workflow step was flagged for omitting the rest of the workflow, with mismatched_capabilities phrased as absences ("No evidence of querying videos"). The TP4 prompt now states a chunk may implement only one declared workflow step and each entry must name what the code does. Deterministically, an assessment is suppressed when every mismatched_capabilities entry is phrased as an absence, since that carries no affirmative evidence of a mismatch. Mixed or affirmative assessments still emit.

Tests: 13 new regression tests in tests/test_mcp_tool_poisoning.py (12 failed pre-fix, 13 pass post-fix), including the issue verbatim anchoring case and the video-intel-search absence items. tests/test_mcp_tool_poisoning.py 174 passed, tests/unit/test_tp4_openai_http.py 51 passed, full suite 20964 passed with 8 failures reproduced on clean main (pre-existing Anthropic provider issues, unrelated). Ruff check and format clean. The prompt addition was kept compact (206 to 238 tokens) so the batch budget tests still pass.

Limits, stated plainly: the deterministic guard covers only the pure-absence class. The issue EPUB and PDF findings named affirmative behaviors, and the real pytest finding was mixed, so those still emit and rely on the prompt criterion plus meta-review, which are nondeterministic. No live-LLM verification was run; the LLM-judged part was verified with stubbed inputs only.

…rop absence-only mismatches

TP4 reported every finding at SKILL.md:1 even when the assessed fence
lived in another file, escalated severity to HIGH at confidence 0.7 while
the README pattern table lists TP4 as MEDIUM, and flagged partial
workflow snippets for omitting the rest of the declared workflow.

Anchor each TP4 finding to the batch file and lines, which the evidence
already carried. TP4 severity is now always MEDIUM, matching the README.
Add a partial-snippet criterion to the TP4 prompt, and suppress an
assessment deterministically when every mismatched_capabilities entry is
phrased as an absence, since that carries no affirmative evidence of a
mismatch. Mixed and affirmative assessments still emit.

Fixes NVIDIA#1066

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Decision: Changes Requested (reviewed head e739a9a59f8f022f5a3782b85f6c818b0073d5be).

Value and readiness: Anchoring TP4 to the assessed file and lines fixes actionable evidence, and MEDIUM matches the documented severity. The partial-workflow prompt change is sensible. The deterministic absence guard, however, also discards affirmative evidence of undeclared behavior.

Material finding — blocker, P2: In src/skillspector/nodes/analyzers/mcp_tool_poisoning.py:1172-1185, words such as without, missing, and lacks anywhere in a capability make the entire item count as an absence. _tp4_finding then returns None at lines 1201-1207 if all items satisfy that test. For example, an assessment with is_mismatch=True, confidence 0.9, and one item Uploads source files to an undeclared endpoint without user consent is suppressed because it contains without. That item describes an affirmative undeclared capability; neither an affirmative actual-behavior summary nor explanation can rescue it. The old code emitted the finding.

Please make suppression conservative: only discard an assessment when every item actually states absence of a declared workflow capability, retaining any item that also describes affirmative behavior. Add regressions for an affirmative capability containing without and for an item containing both an affirmative action and a negated clause. The current mixed test puts the affirmative evidence in a separate list item, so it does not cover this failure.

PIC tradeoffs: None requiring a separate decision for this fix.

Verification and gaps: Reviewed the complete three-file diff and pinned source, tracing the new guard through finding construction. The regression assertions cover anchoring, severity, pure absence, separate mixed entries, affirmative entries, and HTTP graph anchoring; they omit affirmative and absence wording within the same entry. All six hosted CI checks passed for this exact head (run). No contributor code, tests, imports, builds, setup, hooks, regex probes, or live model were executed. The finding above follows directly from the literal without matcher and unconditional suppression branch; it is not a claimed runtime reproduction.

…DIA#1069)

An absence marker anywhere in a mismatched capability (without, missing,
lacks) suppressed the whole assessment, even when the item described an
affirmative undeclared capability such as uploading source files to an
undeclared endpoint without user consent. Make suppression conservative:
an item counts as an absence only when no clause of it describes
affirmative behavior, with capability verbs under a marker treated as
governed by it.

Add regressions for an affirmative capability containing without and for
an item holding both an affirmative action and a negated clause.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
@Varshith-Kali

Copy link
Copy Markdown
Contributor Author

Thanks for the catch. Fixed: _tp4_mismatched_item_is_absence is now conservative. An item counts as an absence only when no clause of it describes affirmative behavior. A capability verb that precedes the first absence marker in its clause, or sits in a clause with no marker, is affirmative evidence, so Uploads source files to an undeclared endpoint without user consent is now emitted, as are items holding both an affirmative action and a negated clause (Uploads files but does not encrypt them, No support for TLS, but sends credentials in plaintext). Pure absence phrasings (No evidence of querying videos, The code does not implement search, Lacks any corpus query, Missing authentication) are still suppressed. Clauses split only on contrast boundaries, so enumerations joined by "and" or commas stay under one leading marker.

Added regressions: _tp4_finding level tests for the exact without example (is_mismatch, confidence 0.9) and for a mixed affirmative plus negated clause item, plus four new cases in the absence-phrasing detection table.

Evidence: tests/test_mcp_tool_poisoning.py 180 passed (6 deselected); analyzer suite 12,133 passed, 4 skipped, 4 xfailed. New head dd3b48a.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TP4 reports a declared workflow's supporting code examples as description-behavior mismatches, at HIGH, and anchors every finding at SKILL.md:1

2 participants