Repository navigation
FIX Stop silently ignoring technique_converters - #3021
Open
Utkarsh Bahuguna (u7k4rs6) wants to merge 2 commits into
Open
Utkarsh Bahuguna (u7k4rs6) wants to merge 2 commits into
Utkarsh Bahuguna (u7k4rs6) wants to merge 2 commits into
Conversation
…g them where they're ignored
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3020.
technique_convertersto the matrix builder, so they apply them.Scenario.SUPPORTS_TECHNIQUE_CONVERTERS(default True). The scenarios that never applied them set it to False, so they don't declare the parameter and passing it raises the usual unknown-parameter error instead of being dropped. That's AdaptiveScenario (and TextAdaptive), AudioAchillesHeel, Encoding, PackageHallucination, Psychosocial, RedTeamAgent, Scam, SystemPromptExtraction and WebInjection.Wiring the converters into those nine for real is doable but each builds its attacks differently, so I kept this to making the failure loud. Happy to do that as a follow-up for any of them.
Tests: one checking which scenarios declare the parameter, one that Encoding rejects it, one that Doctor appends the converter to the right technique, and one that the benchmark forwards it to the builder. 13 of them fail on main. tests/unit/scenario and tests/unit/backend: 3459 passed.