FEAT: Add a general scorer registry API for dataset-driven scenarios

[romanlutz]

Is your feature request related to a problem? Please describe.

The dataset explorer in #2744 needs to discover and select configured scorer instances for its custom-scenario flow, rather than hardcoding scorer classes or accepting serialized Python components from the browser. ScorerRegistry already discovers types, exposes constructor metadata, resolves component references, constructs instances, and holds named configured instances. This issue captured the missing general HTTP contract over that registry.

#2834 attempted a specialized objective-presets API for #2755 and closed without merging after Rich's review requested a general registry API first. The subsequent proposal here helped define the backend boundary.

Completed by #2990, merged on October 5, 2026. Rich's final direction on #2755 settled the contract, and #2755 was revised to cover the same general backend scope. This related issue does not represent a second implementation or an additional child in the 17-item dataset roadmap. The earlier contract hold and flat-summary proposal below are superseded by the implemented decision recorded here.

Describe the solution you'd like

Implemented general API

#2990 adds thin typed routes and a service over the existing ScorerRegistry:

Route	Purpose
GET /api/scorers/types	List scorer types with shared constructor-parameter metadata.
GET /api/scorers	List named, configured instances with bounded pagination.
GET /api/scorers/{scorer_registry_name}	Retrieve one configured instance by registry name.
POST /api/scorers	Construct and register a named instance through the shared registry/resolver, returning the same instance model.

The final typed contracts are:

• ScorerTypeEntry: scorer_type, shared parameters: list[Parameter], is_llm_based, and description; type discovery uses an items envelope.
• ScorerInstance: scorer_registry_name, full typed identifier: ScorerIdentifier, and description; list responses use items and the existing pagination model.
• CreateScorerRequest: required name and type, plus params defaulting to an empty object when omitted, using shared registry construction and named component references.

Contract decision: return the full ScorerIdentifier, including child scorers and target identifiers. This deliberately replaces the earlier proposed flat allowlist and blanket prohibition on nested identifiers. Credential values must still be excluded. Identifiers can contain non-secret configuration; they are not a substitute for registry tags, compatible-default selection, or objective-suitability policy.

Keep buildable types distinct from configured instances. A type appearing in the catalog does not prove that its dependencies are initialized or that it is a suitable objective scorer.

Behavior and boundaries

• The core registry and shared resolver own constructor metadata, reference resolution, scorer-specific parameter checks, and construction. The backend handles HTTP validation and error translation, not a second registry or a parallel introspection/compatibility implementation.
• List/detail/type reads do not construct scorers, score content, call targets, fetch datasets, or generate conversations. Creation configures an instance; it is not a scoring operation.
• Failed creation must not register a partial instance. Duplicate-name creation must not silently replace an existing one, including concurrent requests.
• The implementation paginates configured instances in sorted registry-name order. An unknown cursor currently restarts at the first page, following the existing target-service convention; the earlier proposal to return 400 for this case was not the behavior shipped in FEAT: expose ScorerRegistry through the backend API #2990.
• Registry names identify configured instances in the current runtime. They are not persistent saved configurations and may disappear after registry reset/reinitialization or restart. The service cache is reset when runtime registries are replaced.
• Reuse backend-configured dependencies and existing authentication conventions. This work adds no browser credential-entry flow, credential store, memory schema migration, persistent scorer store, or execution engine.

Acceptance criteria completed in #2990:

• Typed request/response contracts distinguish type discovery from configured-instance discovery.
• List, detail, and create use one shared instance model with full nested identifiers.
• Named creation uses the shared resolver and the created instance is retrievable by registry name.
• Pagination, invalid construction input/references, duplicate names, failed construction, and registry lifecycle changes have focused route/service coverage.
• Read endpoints do not construct or invoke scorers, targets, or dataset providers.
• Local fake-registry tests cover nested scorer/target identifiers, credential exclusion, and concurrent duplicate creation without paid calls.
• Existing /scores operations remain separate; no dataset-specific objective-selection or scenario-default policy is added by these routes.

Remaining dataset-explorer consumers

Basic cards, browsing, media previews, and single-example chat imports do not depend on this API. The dependent flow is selected examples -> Configure scenario -> preview/estimate -> confirmed launch.

The requirements not implemented by #2990 are now explicit in the existing roadmap:

• FEAT GUI: Accept exact seed selections in scenario requests #2757: carry scorer_registry_name alongside exact seed selections without serializing components; preserve the effective configuration across preview, estimate, and launch; reject missing/stale/incompatible requested references without silent substitution.
• FEAT: Support custom scenarios over selected dataset examples #2760: reuse compatible configured objective defaults, default_objective_scorer, existing composites/inverters, prerequisite validation, and correct objective-result meaning. Raw refusal=true is not attack success. raise_if_scorer_blocks is a blocked-judging policy, not refusal-result inversion.
• FEAT GUI: Configure a scenario from selected dataset examples #2759: provide the configured-instance picker, tags/default/compatibility metadata through a shared typed core-backed projection, and setup guidance. The general API's full identifier alone does not supply that metadata or policy.

These are retained follow-ups, not acceptance criteria falsely marked complete here. #2757 still waits for #2756 and maintainer promotion; completing #2755/#2990 does not bypass the main explorer/provider-loading sequence.

Describe alternatives you've considered, if relevant

• A specialized /objective-presets endpoint first would establish the separate contract that prompted [FEAT] Expose configured objective-scorer presets #2834's closure.
• A flat instance summary was proposed here, but the final maintainer decision chose full typed identifiers, matching the shared catalog approach. The original comment thread remains as decision history.
• A hardcoded GUI list would miss initializer-provided instances and custom/composite scorers.
• A new general scorer-builder GUI, scorer execution endpoint, arbitrary Python/import support, or update/delete API is outside this completed backend slice.

Additional context

Implementation: #2990, merge commit 5f619c0b87e342ba15b5dd0af29df4858ecce5e9. Related completed roadmap milestone: #2755. The contributor confirmed the general-backend scope and left objective presets and the GUI as follow-ups in this comment.

Rich's broader scorer contract proposal and phase-8 #2899 concern Python evidence/expectation/scorer contracts, not this HTTP endpoint specification. They are related SDK work, not a separate unimplemented copy of the API now shipped in #2990.

Starting points: pyrit\registry\components\scorer_registry.py, pyrit\backend\models\scorers.py, pyrit\backend\services\scorer_service.py, pyrit\models\catalog\scorer.py, ScorerInitializer, ScenarioConfigurationResolver, and Scenario._get_default_objective_scorer().

Follow doc\code\framework.md: registries build and hold components, the backend exposes HTTP contracts, scenarios package selected components, scorers evaluate evidence, and attacks act on scores.

[romanlutz]

Chen Yufeiyang (@feiiiiii5) I'll assign this to you since you expressed interest. [Update: can't assign because you're not on this thread yet 😆] Do you want to propose something here or via PR?

Also, totally fine if you aren't interested in this anymore. I'll keep it assigned to you until 10/8 for now and free it up at that point if there's no response. You can ask for more time, of course. I've labeled it as "not ready yet" only so that nobody else jumps in.

[feiiiiii5]

Yes — I'd rather propose here than open a PR, for the reason your issue already gives: keep it not ready yet until the contract is agreed, and the #2755 hold stands. Coding against an unagreed shape is what got #2834 closed. Below is a proposal against current main; evidence in the collapsed section.

Schemas. Mirror targets, with one deliberate divergence. /types → ScorerTypeEntry{scorer_type, parameters, is_llm_based, description}; GET / → ScorerListResponse{items, pagination}; POST / → CreateScorerRequest{name, type, params} with name required (no legacy UI to accommodate); GET /{scorer_registry_name} returns the same summary POST does.

The divergence: TargetInstance nests a full typed identifier; for scorers I would not. ScorerIdentifier promotes prompt_target and sub_scorers, so serializing it walks into nested target identifiers — exactly what "do not expose raw ScorerIdentifier objects" rules out. Flat allowlist, not an identifier plus a filtered copy.

Duplicate/reserved names and atomicity — already implemented, no new work. create_named_instance validates the name, then constructs, then registers. So a duplicate or reserved name raises before anything is constructed, a failed build never reaches register, and that path never passes replace=True. The route only maps ValueError → 400, like create_target. Note ScorerRegistry passes no reserved_instance_names, unlike TargetRegistry ({"types"}) and ConverterRegistry ({"preview", "types"}) — if scorers get any, that should be deliberate rather than copied.

Pagination and cursor errors. Mirror targets — cursor is the registry name, items in registry order. But the existing _paginate silently falls back to page 1 on an unknown cursor (evidence below). I'd return 400 for the scorer routes and leave targets as a separate call.

Name lifetime. reset_registry_singleton() exists and instances live in an in-memory dict, so names are process-scoped and survive neither a reset nor a restart. The schema should say so plainly.

Which parameters in the first slice. Don't enumerate. Parameter.reference already carries component_type, so a client can distinguish a TARGET reference from a scorer reference. Accept scalars plus resolvable references, reject opaque, let resolution failures surface as 400 — rather than listing names that drift as scorers are added.

Parameter defaults — one concrete problem, evidence below.

Allowlist, not the existing denylist. filter_sensitive_fields matches key names (api_key, token, secret, password, credential, auth, key). Scorer payloads carry their risk under fixed keys — default, identifier, registry_metadata — so it is a no-op on them.

Prerequisite vs #2755. Make this a prerequisite rather than folding it in, and keep it to the four routes with no objective-suitability field. Catalog membership must not imply compatibility.

Evidence, run against main at c6abbc8

1. /types mirroring the target convention would emit live function reprs.

TargetRegistry : 5 types,  0 parameters serialize a live function repr
ScorerRegistry : 49 types, 31 parameters serialize a live function repr

AgentThreatRulesScorer.score_aggregator -> '<function _create_aggregator.<locals>.aggregator at 0x1214b5800>'
DecodingScorer.aggregator               -> '<function _create_aggregator.<locals>.aggregator at 0x1214b5800>'

Parameter.default documents itself as "serialized as a display string for a scalar or a list of display strings". A closure is neither, and the derived parameter contract does not guard against it — _create_aggregator returns a plain nested function. The value is not stable across processes, so it cannot serve as a documented schema example and will churn any client diffing the response, and it exposes an internal closure name plus a heap address. Scorer-specific: targets do not hit this.

The stable value already exists — ScorerIdentifier.score_aggregator holds the aggregator variant name as a string. I would emit that (or the closure's __name__) instead of repr(), and more generally not emit Parameter.default when it is not a scalar or list of scalars.

2. filter_sensitive_fields is a no-op on these payloads.

payload unchanged by the filter: True

Byte-identical before and after, so it cannot be the mechanism behind "return allowlisted summaries only". That needs a typed allowlist model, which is also the second reason I proposed a flat summary rather than reusing TargetInstance.

3. Unknown cursors silently return page 1.

target_service._paginate scans for the cursor and only reassigns start_idx on a match; with no match it stays 0, and nothing is raised. Combined with process-scoped names, a cursor from a previous run lands the client back on page 1 with no sign anything was wrong — for "create a scorer, then page the list" that is a silently wrong result rather than a visible failure.

Correction. I flagged prompt_format_string's default as prompt text leaking through the types endpoint. That was wrong: the signature default is None, and the format string appears only in the parameter's description. The score_aggregator finding is the real one.

Not verified. I did not exercise create_named_instance end to end, so the atomicity claim is from reading the method. I also have not read #2755's current shape, so the prerequisite point is my read of the sequencing argument in richlundeen's #2834 review rather than anything confirmed against that branch.

[romanlutz]

Closing as completed by #2990, which implements the general scorer registry API also tracked in #2755. I updated this body to the final contract: full typed scorer identifiers, including child scorers and target identifiers, rather than the earlier flat-summary proposal. Credentials still must not be exposed.

The dataset-specific work is retained in #2757 (selection/scorer references), #2760 (objective compatibility and defaults), and #2759 (picker metadata and setup guidance). This closes the overlapping backend work, not those follow-ups, and does not change the #2756 gate on #2757.