Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions doc/code/datasets/0_dataset.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,3 +93,44 @@ an explicit origin other than `local`. Remote dataset providers assign
use `GENERATED`. Origin does not describe upstream authorship. Use `origin=SeedOrigin.USER`
for explicit user entries. Unspecified and legacy origins remain `UNKNOWN`, while edits
preserve the recorded origin.

## Browse stored seeds

The seed browser reads stored seeds from memory only. It does not load providers, open
media or template files, render templates, or generate conversations. Members are
`SeedRecord` projections, not reconstructed execution-ready seeds. Simulated-conversation
configurations remain unchanged in `value`, including legacy file references and
configurations that cannot be executed. Missing files do not remove members or examples.
Stored IDs, hashes, nullable roles and sequences, parameters, objective conditions, and
provenance are retained. Existing `get_seeds_async()` reconstruction is unchanged.

- `GET /api/datasets/seeds?selection_key=<key>` lists one page of logical examples.
- `GET /api/datasets/seeds/{example_id}?selection_key=<key>` returns all members of one example
as stored records, identified by `seed_type`. Configuration fields such as `num_turns`
remain in the stored JSON `value`; browsing does not resolve them into live seed objects.

Get the `selection_key` from `GET /api/datasets`. The unnamed key `dataset:unnamed` includes
NULL and empty dataset names. The example ID is the `prompt_group_id`, or the seed ID when
the seed has no group. Only members in the selected dataset are returned.

The list accepts `limit` (1 to 100), `cursor`, `search`, and repeated `modality`,
`seed_type`, and `harm_category` parameters. Values of one parameter use OR. Different
parameters use AND, and different members of an example can match different parameters.
Harm categories match complete values without case sensitivity. `search` finds literal
text in the values of text prompts and objectives; `%`, `_`, and `[` are not patterns.
SQLite ignores case for ASCII characters only. `search` does not look in
simulated-conversation configurations, because their stored value is JSON. Use
`seed_type=simulated_conversation` to find them.

Examples sort by the earliest member `date_added`, newest first, then by canonical textual
example ID, descending. SQLite and Azure SQL use the same UUID order. A
cursor is valid only for the same `selection_key` and filters. Other cursors return 400.

Each list item has a preview of the first text member: at most 100 characters, with `...`
and `preview_truncated` when it is shortened. Media members show only the file name.
HTTP(S) media URLs are recognized regardless of scheme case or leading whitespace;
their authority, query, and fragment are excluded from the preview.
Standalone absolute paths and URLs stored as text show `[Text reference]` rather than
paths or credentials; detail retains the full stored value. Other types show a type label.
The browser does not render templates or run
simulated conversations.
13 changes: 8 additions & 5 deletions pyrit/backend/mappers/_preview.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,12 @@ def _derive_basename(value: str) -> str | None:
The basename (filename portion) of *value*, or ``None`` if one can't
be derived (e.g. data URI, empty value).
"""
if not value or value.startswith("data:"):
url_value = value.lstrip()
if not url_value or url_value.lower().startswith("data:"):
return None
if value.startswith(("http://", "https://")):
# Strip query string (e.g. SAS tokens) before taking the basename.
parsed = urlparse(value)
if url_value.lower().startswith(("http://", "https://")):
# Use only the URL path, excluding authority, query, and fragment credentials.
parsed = urlparse(url_value)
name = PureWindowsPath(parsed.path).name
return name or None
# Local path — PureWindowsPath treats both ``/`` and ``\`` as separators,
Expand All @@ -66,7 +67,9 @@ def format_last_message_preview(

Media-path data types are rendered as ``[Image: <basename>]`` (and
variants) so the absolute filesystem path of memory artifacts is never
exposed through API responses or UI previews. Error values are replaced
exposed through API responses or UI previews. HTTP(S) URL recognition
ignores scheme case and leading whitespace, and only the path contributes
to the filename. Error values are replaced
with a generic status so persisted exception tracebacks are not exposed.
Text-like data types pass through with truncation and an ellipsis suffix
when they exceed *max_len*.
Expand Down
38 changes: 37 additions & 1 deletion pyrit/backend/models/datasets.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,16 @@

Datasets are seed prompt/objective collections provided by
``SeedDatasetProvider`` subclasses. These models describe the wire format for
listing available datasets.
listing available datasets and browsing their stored seed examples.
"""

from uuid import UUID

from pydantic import BaseModel, Field

from pyrit.backend.models.common import PaginationInfo
from pyrit.models import PromptDataType, SeedRecord, SeedType


class DatasetInfo(BaseModel):
"""Metadata about a single available dataset."""
Expand Down Expand Up @@ -41,3 +46,34 @@ class DatasetListResponse(BaseModel):
"""Response for listing available datasets."""

items: list[DatasetInfo] = Field(..., description="List of available datasets")


class SeedExampleSummary(BaseModel):
"""One logical seed example: the seeds that share a group ID, or one seed without a group."""

example_id: UUID = Field(..., description="The prompt_group_id, or the seed ID of a seed without a group")
name: str | None = Field(None, description="The first member name, if any")
preview: str = Field(..., description="Text preview of at most 100 characters, or a type label")
preview_truncated: bool = Field(..., description="Whether the preview text was shortened")
modalities: list[PromptDataType]
seed_types: list[SeedType]
piece_count: int
objective_count: int
harm_categories: list[str]
has_unlabeled_harm: bool = Field(..., description="Whether any member has no harm category")


class SeedExampleListResponse(BaseModel):
"""One page of logical seed examples."""

items: list[SeedExampleSummary]
pagination: PaginationInfo
total: int = Field(..., description="Number of logical examples that match the filters")


class SeedExampleDetailResponse(SeedExampleSummary):
"""One logical seed example with all of its stored seeds."""

members: list[SeedRecord] = Field(
..., description="Stored seed records without reconstruction, objectives first, then by sequence"
)
88 changes: 84 additions & 4 deletions pyrit/backend/routes/datasets.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,27 @@
"""
Dataset API routes.

Provides an endpoint for listing available seed datasets. Datasets are
discovered from registered ``SeedDatasetProvider`` subclasses.
Lists available seed datasets and browses the stored seed examples of one dataset. Datasets are
discovered from registered ``SeedDatasetProvider`` subclasses and from memory.
"""

from fastapi import APIRouter
from uuid import UUID

from pyrit.backend.models.common import ProblemDetail
from fastapi import APIRouter, HTTPException, Query, status

from pyrit.backend.models.common import MAX_ITEMS, CursorStr, ProblemDetail
from pyrit.backend.models.datasets import (
DatasetListResponse,
SeedExampleDetailResponse,
SeedExampleListResponse,
)
from pyrit.backend.services.dataset_service import get_dataset_service
from pyrit.models import PromptDataType, SeedType

router = APIRouter(prefix="/datasets", tags=["datasets"])

_SELECTION_KEY_DESCRIPTION = "The selection_key of a dataset from GET /datasets"


@router.get(
"",
Expand All @@ -38,3 +45,76 @@ async def list_datasets(loaded_only: bool = False) -> DatasetListResponse: # py
"""
service = get_dataset_service()
return await service.list_datasets_async(loaded_only=loaded_only)


@router.get(
"/seeds",
response_model=SeedExampleListResponse,
responses={
400: {"model": ProblemDetail, "description": "Invalid selection key or cursor"},
},
)
async def list_seed_examples( # pyrit-async-suffix-exempt
selection_key: str = Query(..., description=_SELECTION_KEY_DESCRIPTION),
limit: int = Query(20, ge=1, le=100, description="Maximum examples per page"),
cursor: CursorStr | None = Query(
None,
description="The next_cursor of the previous page. A cursor is valid only with the same "
"selection_key and filters.",
),
search: str | None = Query(
None,
max_length=1000,
description="Case-insensitive literal text to find in text prompt and objective values",
),
modality: list[PromptDataType] | None = Query(None, max_length=MAX_ITEMS, description="Data types, OR-matched"),
harm_category: list[str] | None = Query(
None, max_length=MAX_ITEMS, description="Whole harm categories, case-insensitive, OR-matched"
),
seed_type: list[SeedType] | None = Query(None, max_length=MAX_ITEMS, description="Seed types, OR-matched"),
) -> SeedExampleListResponse:
"""
List one page of the stored logical seed examples of a dataset.

Examples are ordered newest first. Different filters are AND-matched, and different members
of an example can match different filters.

Returns:
SeedExampleListResponse: The page, its pagination data, and the number of matching examples.
"""
return await get_dataset_service().list_seed_examples_async(
selection_key=selection_key,
limit=limit,
cursor=cursor,
search=search,
data_types=modality,
harm_categories=harm_category,
seed_types=seed_type,
)


@router.get(
"/seeds/{example_id}",
response_model=SeedExampleDetailResponse,
responses={
400: {"model": ProblemDetail, "description": "Invalid selection key"},
404: {"model": ProblemDetail, "description": "Seed example not found in the dataset"},
},
)
async def get_seed_example( # pyrit-async-suffix-exempt
example_id: UUID,
selection_key: str = Query(..., description=_SELECTION_KEY_DESCRIPTION),
) -> SeedExampleDetailResponse:
"""
Get one stored logical seed example with all of its members.

Returns:
SeedExampleDetailResponse: The example and its members.

Raises:
HTTPException: If the dataset does not contain the example.
"""
example = await get_dataset_service().get_seed_example_async(selection_key=selection_key, example_id=example_id)
if example is None:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=f"Seed example '{example_id}' not found")
return example
Loading
Loading