Repository navigation
MAINT: Adding existing scorers to new paradigm (phase 8) - #2899
Merged
Richard Lundeen (richlundeen) merged 13 commits intoOct 8, 2026
Merged
Richard Lundeen (richlundeen) merged 13 commits into
Richard Lundeen (richlundeen) merged 13 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve the upstream empty-target matching fix in the shared matcher. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep deprecated target validation through the generic and message-family bases without storing targets or creating judges. Cover hidden legacy hook rejection across migrated scorers and document required hook migration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
changed the title
Preserve numeric response validation and separate conversation wrapper results from retained intermediate scores. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use get_chat_target() in shared scoring batches, image batches, and scorer evaluation. Cover delegated targets, direct targets, and target-free scorers with configured and unconfigured RPM limits. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
7 tasks done
Resolve scorer, model export, and text matching conflicts. Preserve main's non-editable judge retry support and the scenario completion-order assertion fix; adapt regression tests to explicit judgment requests and async memory. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
hannahwestra25
approved these changes
Oct 7, 2026
hannahwestra25
left a comment
hannahwestra25
left a comment
Contributor
There was a problem hiding this comment.
looks good! some small nits + ghcp finding
Justin Song (jsong468)
approved these changes
Oct 7, 2026
Justin Song (jsong468)
left a comment
Justin Song (jsong468)
left a comment
Contributor
There was a problem hiding this comment.
some additional documentation nits but lgtm!
Preserve shared cancellation cleanup and validate n-gram sizes in the shared text matcher. Reject blank Regex criteria, document per-piece output matching, clarify conversation evidence handling, and retain main's notebook migration-output cleanup. Remove unrelated float-scale notebook output churn. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep analytics compatibility exports and move upstream matcher identifier parameters to pyrit.common.text_matching so substring scorers retain distinct behavioral identities. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
enabled auto-merge
October 8, 2026 04:58
Merge origin/main at c8aa5bd without conflicts, preserving scorer contracts, task cancellation cleanup, and matcher identity. Reuse only the backend test helpers and failing conversation-editor test repair from richlundeen's microsoft#3015 (789802b). Wait for a terminal send state and drain tasks before mock and database teardown; retain all existing assertions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ebae5d0f-7b81-4426-afaf-4e884f12cca3
Merge origin/main at b2e9c78 without conflicts. Preserve scorer evidence persistence and lazy model exports alongside the new scenario result roles. Restore the locked Python and frontend environments and validate the merged state. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ebae5d0f-7b81-4426-afaf-4e884f12cca3
Keep main's AsyncGenerator annotation for the shared lifecycle helper now that the conversation-send reliability fix has landed. Preserve the same terminal-state wait and task cleanup behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ebae5d0f-7b81-4426-afaf-4e884f12cca3
Richard Lundeen (richlundeen)
deleted the
richlundeen-scorer-contract-proposal
branch
October 8, 2026 21:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Existing scorers need to use the shared evidence and expectation contract without keeping target-specific behavior in their base classes. This makes text, conversation, and target-backed scoring consistent while preserving concrete scorer constructors.
Implements Phase 8 of the scorer contract proposal.
Adds typed output matching and whole-conversation snapshots, and migrates existing target-backed scorers to explicit
JudgmentRequestandTargetJudgecomposition. Unifies blocked-judge handling and preserves evidence links and score anchors across conversation entry paths. Updates regression coverage and scoring documentation, with all six scoring notebooks executed and their outputs retained.