Skip to content

FIX Stop silently ignoring technique_converters - #3021

Open
Utkarsh Bahuguna (u7k4rs6) wants to merge 2 commits into
microsoft:mainfrom
u7k4rs6:fix/technique-converters-ignored
Open

Utkarsh Bahuguna (u7k4rs6) wants to merge 2 commits into
microsoft:mainfrom
u7k4rs6:fix/technique-converters-ignored

Conversation

@u7k4rs6

Copy link
Copy Markdown
Contributor

Fixes #3020.

  • Doctor and AdversarialBenchmark now pass technique_converters to the matrix builder, so they apply them.
  • Added Scenario.SUPPORTS_TECHNIQUE_CONVERTERS (default True). The scenarios that never applied them set it to False, so they don't declare the parameter and passing it raises the usual unknown-parameter error instead of being dropped. That's AdaptiveScenario (and TextAdaptive), AudioAchillesHeel, Encoding, PackageHallucination, Psychosocial, RedTeamAgent, Scam, SystemPromptExtraction and WebInjection.

Wiring the converters into those nine for real is doable but each builds its attacks differently, so I kept this to making the failure loud. Happy to do that as a follow-up for any of them.

Tests: one checking which scenarios declare the parameter, one that Encoding rejects it, one that Doctor appends the converter to the right technique, and one that the benchmark forwards it to the builder. 13 of them fail on main. tests/unit/scenario and tests/unit/backend: 3459 passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

technique_converters is silently ignored by most scenarios

1 participant