Skip to content

FEAT: add KnownBadSignatureScorer for EICAR / GTUBE / GTphish - #3009

Merged
Roman Lutz (romanlutz) merged 8 commits into
microsoft:mainfrom
abdelhadi703:feat/known-bad-signature-scorer
Oct 10, 2026
Merged

Roman Lutz (romanlutz) merged 8 commits into
microsoft:mainfrom
abdelhadi703:feat/known-bad-signature-scorer

Conversation

@abdelhadi703

Copy link
Copy Markdown
Contributor

Add a known-bad signature scorer (EICAR / GTUBE / GTphish)

PyRIT had no scorer for the standard scanner test signatures, so there was no way to
check whether a target returns a string an antivirus (EICAR), anti-spam (GTUBE) or
anti-phishing (GTphish) scanner should have stopped. Garak covers this with
av_spam_scanning; this adds the PyRIT-side detection.

What I changed:

  • added KnownBadSignatureScorer in pyrit/score/true_false/regex/, built on the
    existing RegexScorer with the three signatures as default patterns
  • exported it from pyrit.score and pyrit.score.true_false.regex
  • added tests/unit/score/regex/test_known_bad_signature_scorer.py

One detail worth flagging: the signatures are the real EICAR/GTUBE/GTphish strings, and
a contiguous literal in a source file trips real-time AV on developer machines and CI
checkouts (the file gets quarantined before it's read). I assemble them from fragments
so the on-disk file stays inert while the runtime value is byte-for-byte the real
signature.

Testing: pytest tests/unit/score/regex/test_known_bad_signature_scorer.py — 10 passed.

Refs #511

@romanlutz Roman Lutz (romanlutz) self-assigned this Oct 9, 2026
Roman Lutz (romanlutz) and others added 2 commits October 9, 2026 02:41
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The scorer was shipped without a mention in doc/code/scoring; tests/unit/docs
requires every registered scorer to appear there. Also applied ruff formatting
to the new module.

Signed-off-by: abdelhadi703 <abdelhadisalmaoui0909@outlook.fr>
Comment thread pyrit/score/true_false/regex/known_bad_signature_scorer.py Outdated
Comment thread pyrit/score/true_false/regex/known_bad_signature_scorer.py Outdated
Review feedback: a hit only proves the marker string is present. A scanner can
detect a marker and still deliver the message, and a target may have no scanner
in front of it at all, so the docstring and the notebook no longer claim the
scorer establishes a missing or misconfigured scanner.

Signed-off-by: abdelhadi703 <abdelhadisalmaoui0909@outlook.fr>
@abdelhadi703

Copy link
Copy Markdown
Contributor Author

Both points addressed in 3b9656e.

The docs entry is in doc/code/scoring/1_true_false_scorers.py with its .ipynb kept in sync — that commit landed just before your review, so the CI failure it describes should already be resolved on the current head.

On the True semantics: agreed, and the wording is fixed. The docstring and the notebook entry now say the scorer reports literal marker presence, and explicitly note that a scanner can detect a marker and still deliver the message (your Cisco GTphish example) and that a target may have no scanner in front of it at all. The guidance is to read a hit against the policy the target is expected to enforce. Only comments and docs changed — matching behaviour is untouched.

Roman Lutz (romanlutz) and others added 2 commits October 9, 2026 07:06
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Restore existing scorer documentation and match the paired Python notebook while preserving recorded outputs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@romanlutz
Roman Lutz (romanlutz) added this pull request to the merge queue Oct 9, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 9, 2026
Wait for actual SDK dispatch before applying the test watchdog. Cover slow startup explicitly and preserve timeout wrapping, no-replay and cleanup assertions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@romanlutz
Roman Lutz (romanlutz) added this pull request to the merge queue Oct 9, 2026
Merged via the queue into microsoft:main with commit 575f74c Oct 10, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants