Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions .github/workflows/benchmark-rdf.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
name: Benchmark RDF

# Triggers like the main workflow. The tests in tests/perf already run a 1k smoke there; this
# job runs the sizes up to 10k. 100k is left out of CI (run it locally with make benchmark-rdf, or
# pass it in the sizes input). Each run is capped at 120 s, and a size predicted to exceed that
# is skipped, so slow scenarios stay cheap.
on:
push:
branches:
- main
pull_request:
types: [opened, synchronize, reopened, ready_for_review]
workflow_dispatch:
inputs:
sizes:
description: "Triple counts, space separated. Empty uses 1000 2000 4000 10000"
default: ""

jobs:
benchmark-rdf:
if: github.event.pull_request.draft == false
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- name: Check out
uses: actions/checkout@v4

- name: Set up the environment
uses: ./.github/actions/setup-python-env
with:
python-version: "3.12"

- name: Run the RDF benchmark
env:
SIZES: ${{ inputs.sizes || '1000 2000 4000 10000' }}
run: |
# Without pipefail a failing benchmark would be hidden behind tee.
set -o pipefail
# shellcheck disable=SC2086 - SIZES is a space separated list on purpose
PYTHONPATH=tests uv run python -m perf --sizes $SIZES --out benchmark_rdf_results.json \
| tee benchmark_rdf_report.md

- name: Publish the report in the job summary
if: always()
run: |
if [ -f benchmark_rdf_report.md ]; then
cat benchmark_rdf_report.md >> "$GITHUB_STEP_SUMMARY"
fi

- name: Upload the results
if: always()
uses: actions/upload-artifact@v4
with:
name: benchmark-rdf-results
path: |
benchmark_rdf_results.json
benchmark_rdf_report.md
if-no-files-found: warn
retention-days: 90
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,7 @@ __marimo__/
# Benchmarks (pytest-benchmark)
.benchmarks/
benchmark_results.json
benchmark_rdf_results.json
baseline_benchmark.json
current_benchmark.json
benchmark_comparison.txt
Expand Down
5 changes: 5 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,11 @@ benchmark: ## Run performance benchmarks with pytest-benchmark
--benchmark-autosave \
--benchmark-json=benchmark_results.json

.PHONY: benchmark-rdf
benchmark-rdf: ## Benchmark RDF export and import at 1k, 10k and 100k triples (own harness, not pytest-benchmark)
@echo "🚀 Benchmarking RDF export and import"
@PYTHONPATH=tests uv run python -m perf --out benchmark_rdf_results.json

.PHONY: build
build: clean-build ## Build wheel file
@echo "🚀 Creating wheel file"
Expand Down
13 changes: 13 additions & 0 deletions docs/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,19 @@ The tests marked `postgres` need a live server and skip unless `OOLD_TEST_POSTGR
OOLD_TEST_POSTGRES_URL=postgresql://user:password@localhost:5432/postgres make test
```

### Benchmarking RDF performance

A separate harness measures RDF export, store and import at 1k, 10k and 100k triples. It runs each scenario in a fresh subprocess and records time, CPU, peak memory, GC pauses and the scaling exponent. It is not `pytest-benchmark`, which `make benchmark` uses.

```bash
make benchmark-rdf # 1k to 100k, each run capped at 120 s, writes benchmark_rdf_results.json
PYTHONPATH=tests uv run python -m perf --sizes 1000 2000 --scenarios export store
PYTHONPATH=tests uv run python -m perf.worker import 1000 --profile 30 # cProfile, top 30 by cumulative time
uv run pytest -m scaling --no-cov # scaling guards, skipped by default
```

The code lives in `tests/perf/`. CI runs the 1k sizes with the normal tests. The `Benchmark RDF` workflow runs sizes up to 10k, triggers like the main workflow and can also be started from the Actions tab (100k is left out of CI, use `make benchmark-rdf` locally), which publishes the report in the job summary and uploads `benchmark_rdf_results.json` as an artifact.

### Docs

Doc sources live in `docs/`. The site is configured in `zensical.toml`.
Expand Down
3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,7 @@ dev = [
"pytest-cov>=7.0.0",
"pytest-benchmark",
"pytest-playwright",
"psutil>=5.9",
"jsondiff",
"panel",
"pre-commit>=4.5.1",
Expand Down Expand Up @@ -227,6 +228,8 @@ markers = [
"parity: compares this port against the reference harness",
# Needs a live PostgreSQL; skips unless OOLD_TEST_POSTGRES_URL is set.
"postgres: runs against a live PostgreSQL server",
# Slow scaling checks over the RDF scenarios; they skip unless selected with `-m scaling`.
"scaling: slow checks of how the RDF scenarios scale",
]

[tool.deptry]
Expand Down
65 changes: 35 additions & 30 deletions src/oold/backend/sparql.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
from typing import Any

from pydantic import ConfigDict
from rdflib import Graph
from rdflib import Graph, URIRef
from SPARQLWrapper import JSON, JSONLD, SPARQLWrapper

from oold.backend.auth import UserPwdCredential, get_credential
Expand Down Expand Up @@ -30,6 +30,19 @@
"""


_EX_NAMESPACE = "https://example.com/"


def _subject(iri: str) -> URIRef:
"""Full IRIs pass through, ``ex:`` expands to the one namespace this module assumes."""
if iri.startswith("http"):
return URIRef(iri)
prefix, _, local = iri.partition(":")
if prefix != "ex":
raise ValueError(f"Unknown namespace prefix : {prefix}")
return URIRef(_EX_NAMESPACE + local)


class LocalSparqlResolver(Resolver):
model_config = ConfigDict(arbitrary_types_allowed=True)

Expand All @@ -55,41 +68,33 @@ def query(self, param: QueryParam) -> ResolveResult:
return self.resolve(ResolveParam(iris=iris, model_cls=model_cls))

def resolve_iris(self, iris: list[str]) -> dict[str, dict]:
# sparql query to get a node by IRI with all its properties
# using CONSTRUCT to get the full node
# format the result as json-ld
"""Every triple of each subject as JSON-LD, or None for a subject with no triples.

A direct index lookup rather than a CONSTRUCT with a FILTER: the FILTER
is evaluated against every triple of the graph, once per IRI, which made
resolving quadratic in the size of the graph.
"""
jsonld_dicts = {}
for iri in iris:
iri_filter = f"FILTER (?s = {iri})"
# check if the iri is a full IRI or a prefix
if iri.startswith("http"):
iri_filter = f"FILTER (?s = <{iri}>)"
# todo: build full iri / prefix mapping from model context
qres = self.graph.query(_construct_node(iri_filter, "PREFIX ex: <https://example.com/>"))
jsonld_dict = json.loads(qres.serialize(format="json-ld"))[0]
jsonld_dicts[iri] = jsonld_dict
triples = list(self.graph.triples((_subject(iri), None, None)))
if not triples:
jsonld_dicts[iri] = None
continue
node = Graph(bind_namespaces="none")
for triple in triples:
node.add(triple)
jsonld_dicts[iri] = json.loads(node.serialize(format="json-ld"))[0]
return jsonld_dicts


class LocalSparqlBackend(LocalSparqlResolver, Backend):
def store_jsonld_dicts(self, jsonld_dicts: dict[str, dict]) -> StoreResult:
# delete all triples with the given iris as subject
for iri in jsonld_dicts:
iri_filter = f"{iri}"
# check if the iri is a full IRI or a prefix
if iri.startswith("http"):
iri_filter = f"<{iri}>"
query = """
PREFIX ex: <https://example.com/>
DELETE WHERE {
{{{iri_filter}}} ?p ?o .
}
""".replace("{{{iri_filter}}}", iri_filter)
self.graph.update(query)
# convert jsonld_dict to rdflib triples and add to graph
g = Graph()
g.parse(data=json.dumps(jsonld_dicts[iri]), format="json-ld")
self.graph += g
# replace: drop the subject's triples, then parse the new document into the graph itself.
# None means delete, like the other stores.
for iri, jsonld_dict in jsonld_dicts.items():
self.graph.remove((_subject(iri), None, None))
if jsonld_dict is not None:
self.graph.parse(data=json.dumps(jsonld_dict), format="json-ld")
return StoreResult(success=True)


Expand Down Expand Up @@ -212,7 +217,7 @@ def _post_process(self, jsonld_dict: dict) -> dict:


def _construct_node(iri_filter: str, prefixes: str = "") -> str:
"""CONSTRUCT every triple of one subject. Shared by all three resolvers."""
"""CONSTRUCT every triple of one subject. Used by the remote resolvers."""
return f"""
{prefixes}
CONSTRUCT {{ ?s ?p ?o . }}
Expand Down
Empty file added tests/perf/__init__.py
Empty file.
39 changes: 39 additions & 0 deletions tests/perf/__main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
"""``python -m perf``: run the scenarios over several sizes, print a report, write JSON."""

import argparse
import sys

from .dataset import BENCHMARK_SIZES
from .harness import dump
from .report import format_report
from .runner import DEFAULT_TIMEOUT_S, SLOW_RUN_S, collect
from .scenarios import SCENARIOS


def main(argv=None) -> None:
parser = argparse.ArgumentParser(prog="python -m perf")
parser.add_argument("--sizes", type=int, nargs="+", default=list(BENCHMARK_SIZES))
parser.add_argument("--scenarios", nargs="+", choices=sorted(SCENARIOS), default=list(SCENARIOS))
parser.add_argument("--repeats", type=int, default=3)
parser.add_argument("--timeout", type=float, default=DEFAULT_TIMEOUT_S, help="seconds per run")
parser.add_argument(
"--slow", type=float, default=SLOW_RUN_S, help="runs slower than this are not repeated or traced"
)
parser.add_argument("--out", default="benchmark_rdf_results.json")
args = parser.parse_args(argv)

collected = collect(
args.sizes,
args.scenarios,
repeats=args.repeats,
timeout=args.timeout,
slow_s=args.slow,
progress=lambda message: print(message, file=sys.stderr, flush=True),
)
dump(collected.results, args.out, [s.as_tuple() for s in collected.skipped])
print(format_report(collected))
print(f"\nwrote {args.out}", file=sys.stderr)


if __name__ == "__main__":
main()
18 changes: 18 additions & 0 deletions tests/perf/conftest.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
from pathlib import Path

import pytest

HERE = Path(__file__).parent
RDFLIB_NOISE = "ignore:ConjunctiveGraph is deprecated:DeprecationWarning"


def pytest_collection_modifyitems(config, items):
"""Silence rdflib's deprecation noise here, and skip ``scaling`` tests unless selected with ``-m scaling``."""
selected = "scaling" in (config.getoption("-m") or "")
skip = pytest.mark.skip(reason="slow, run with `pytest -m scaling`")
for item in items:
if not item.path.is_relative_to(HERE):
continue
item.add_marker(pytest.mark.filterwarnings(RDFLIB_NOISE))
if item.get_closest_marker("scaling") and not selected:
item.add_marker(skip)
85 changes: 85 additions & 0 deletions tests/perf/dataset.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
"""Deterministic datasets for the RDF performance scenarios.

The scale unit is the triple, not the entity: each person contributes one type
triple, one name triple and one triple per ``knows`` edge.
"""

from pydantic import ConfigDict, Field

TRIPLE_SIZES = (1_000, 10_000, 100_000)
# The intermediate sizes give a scaling exponent even for a scenario too slow to reach 10k.
BENCHMARK_SIZES = (1_000, 2_000, 4_000, 10_000, 100_000)
PREFIX = "perf"
NAMESPACE = "https://perf.example.org/"
BASE_TRIPLES = 2


def build_person_model():
"""Define the Person model. Call it inside the test or run that needs it.

Defining it registers the class by type IRI, so the conftest fixture that
restores the registries must be active.
"""
from oold.model import LinkedBaseModel

class Entity(LinkedBaseModel):
model_config = ConfigDict(
json_schema_extra={
"@context": {
"id": "@id",
"type": "@type",
"schema": "https://schema.org/",
PREFIX: NAMESPACE,
"name": "schema:name",
},
"$id": f"{NAMESPACE}Entity",
}
)
type: str | None = f"{PREFIX}:Entity"
name: str

def get_iri(self):
return f"{PREFIX}:{self.name}"

class Person(Entity):
model_config = ConfigDict(
json_schema_extra={
"@context": [
f"{NAMESPACE}Entity",
{"knows": {"@id": "schema:knows", "@type": "@id", "@container": "@set"}},
],
"$id": f"{NAMESPACE}Person",
}
)
type: str | None = f"{PREFIX}:Person"
knows: list["Person"] | None = Field(None, json_schema_extra={"range": f"{PREFIX}:Person"})

return Person


def edges_per_person(triples: int, knows: int = 2) -> list[int]:
"""Edge count per person so the dataset has exactly ``triples`` triples."""
per_person = BASE_TRIPLES + knows
people = triples // per_person
if people <= knows + 1:
raise ValueError(f"{triples} triples is too few for {knows} edges per person")
counts = [knows] * people
for i in range(triples - people * per_person):
counts[i] += 1
return counts


def make_people(person_cls, triples: int, knows: int = 2) -> list:
"""Persons ``p0..pN`` where person i knows the next persons, wrapping around.

Edges are IRI strings, so the people can be built in any order.
"""
counts = edges_per_person(triples, knows)
total = len(counts)
return [
person_cls(
name=f"p{i}",
knows=[f"{PREFIX}:p{(i + step) % total}" for step in range(1, edges + 1)],
)
for i, edges in enumerate(counts)
]
Loading
Loading