Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,58 @@ jobs:
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

tests-postgres:
# Real PostgreSQL via a service container, which only Linux runners support. One Python
# version is enough: the matrix above already covers the interpreters.
name: tests-postgres (PostgreSQL ${{ matrix.postgres }})
runs-on: ubuntu-latest
strategy:
matrix:
postgres: ["16", "17", "18"]
fail-fast: false
services:
postgres:
image: postgres:${{ matrix.postgres }}
env:
POSTGRES_PASSWORD: postgres
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U postgres"
--health-interval 5s
--health-timeout 5s
--health-retries 10
env:
OOLD_TEST_POSTGRES_URL: postgresql://postgres:postgres@localhost:5432/postgres
steps:
- name: Check out
uses: actions/checkout@v4

- name: Set up the environment
uses: ./.github/actions/setup-python-env
with:
python-version: "3.12"

- name: Run the postgres tests
run: >-
uv run python -m pytest tests -m postgres --cov --cov-config=pyproject.toml
--cov-report=xml --benchmark-disable --junitxml="$RUNNER_TEMP/postgres-junit.xml"

- name: Upload coverage reports to Codecov on PostgreSQL 17
# Codecov merges this with the main upload, where these tests skip.
uses: codecov/codecov-action@v4
if: ${{ matrix.postgres == '17' }}
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

- name: Fail if any postgres test was skipped
# A wrong URL or a missing driver skips these tests, and a skip counts as a pass.
run: |
if grep -q "<skipped" "$RUNNER_TEMP/postgres-junit.xml"; then
echo "postgres tests were skipped, so nothing ran against PostgreSQL ${{ matrix.postgres }}" >&2
exit 1
fi

validate-schemas:
# Runs the OO-LD validator over the committed fixtures, offline, so a regression in the
# validation pipeline fails CI even when the unit tests still pass.
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,7 @@ Built-in backends for entity persistence and resolution:
| --------------------------- | ------------------------------------------------ | --------------- |
| **SimpleDictDocumentStore** | In-memory dict, optional JSON file (`file_path`) | Filter by field |
| **SqliteDocumentStore** | SQLite database (default `format=JSON`) | - |
| **PostgresDocumentStore** | PostgreSQL JSONB (`oold[postgres]`) | - |
| **LocalSparqlBackend** | In-memory RDF graph (rdflib) | SPARQL |

```python
Expand Down
1 change: 1 addition & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ All backends implement the `Backend` interface from `backend/interface.py`:
|---|---|---|
| `SimpleDictDocumentStore` | In-memory dict, optional JSON file | No |
| `SqliteDocumentStore` | SQLite database | No |
| `PostgresDocumentStore` | PostgreSQL JSONB | No |
| `LocalSparqlBackend` | In-memory RDFLib graph | Yes |

Backends are registered per IRI prefix via `set_resolver` / `set_backend`, so multiple backends can coexist in one application.
Expand Down
8 changes: 8 additions & 0 deletions docs/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,14 @@ Requires `uv` and `git`.
uv run python -m pytest --cov --cov-config=pyproject.toml --cov-report=xml
```

### PostgreSQL tests

The tests marked `postgres` need a live server and skip unless `OOLD_TEST_POSTGRES_URL` is set. CI runs them against PostgreSQL 16, 17 and 18. Each test uses its own table and drops it afterwards, so any database you can write to will do:

```bash
OOLD_TEST_POSTGRES_URL=postgresql://user:password@localhost:5432/postgres make test
```

### Docs

Doc sources live in `docs/`. The site is configured in `zensical.toml`.
Expand Down
40 changes: 38 additions & 2 deletions docs/how-to/backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ Persists entities in a SQLite database. Drop-in replacement for `SimpleDictDocum
```python
from oold.backend.document_store import SqliteDocumentStore

store = SqliteDocumentStore(file_path="./data/entities.db")
store = SqliteDocumentStore(db_path="./data/entities.db") # or ":memory:"

set_resolver(SetResolverParam(iri="ex", resolver=store))
set_backend(SetBackendParam(iri="ex", backend=store))
Expand All @@ -99,6 +99,42 @@ alice = Person["ex:alice"]

---

## PostgresDocumentStore

Persists entities as JSONB rows in PostgreSQL, one row per IRI. It needs the `postgres` extra:

=== "uv (recommended)"

```bash
uv add "oold[postgres]"
```

=== "pip"

```bash
pip install "oold[postgres]"
```

```python
from oold.backend.postgres import PostgresDocumentStore

store = PostgresDocumentStore(dsn="postgresql://user:password@localhost:5432/mydb")

set_resolver(SetResolverParam(iri="ex", resolver=store))
set_backend(SetBackendParam(iri="ex", backend=store))

store.store(StoreParam(nodes={"ex:alice": alice}))
alice = Person["ex:alice"]

store.close() # the connection stays open until you close it
```

- Entities live in table `entities` (`id TEXT PRIMARY KEY, data JSONB`), created on first use. Pass `table="..."` to use another name (letters, digits and underscores, up to 63 characters).
- Storing an IRI again overwrites its row, and a batch is written in one transaction.
- Querying with `Model[Model.field == value]` is not supported yet.

---

## LocalSparqlBackend

An in-memory RDF graph (via [rdflib](https://rdflib.readthedocs.io/)) that supports SPARQL queries.
Expand Down Expand Up @@ -153,7 +189,7 @@ Register different backends for different IRI prefixes:

```python
local_store = SimpleDictDocumentStore()
remote_store = SqliteDocumentStore(file_path="remote.db")
remote_store = SqliteDocumentStore(db_path="remote.db")

set_resolver(SetResolverParam(iri="local", resolver=local_store))
set_resolver(SetResolverParam(iri="remote", resolver=remote_store))
Expand Down
7 changes: 7 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,9 @@ mcp = [
"referencing>=0.30",
"click>=8.1",
]
postgres = [
"psycopg[binary]>=3.2",
]
UI-panel = [
"panel",
"jupyter_bokeh",
Expand Down Expand Up @@ -104,6 +107,7 @@ all = [
"referencing>=0.30",
"click>=8.1",
"mcp>=1.2",
"psycopg[binary]>=3.2",
]

[dependency-groups]
Expand All @@ -114,6 +118,7 @@ dev = [
"referencing>=0.30",
"click>=8.1",
"mcp>=1.2",
"psycopg[binary]>=3.2",
"pytest",
"pytest-cov>=7.0.0",
"pytest-benchmark",
Expand Down Expand Up @@ -220,6 +225,8 @@ markers = [
# Opt-in parity tests against a local oold-schema checkout; see
# tests/test_validation/test_parity_live.py. They skip unless OOLD_SCHEMA_DIR is set.
"parity: compares this port against the reference harness",
# Needs a live PostgreSQL; skips unless OOLD_TEST_POSTGRES_URL is set.
"postgres: runs against a live PostgreSQL server",
]

[tool.deptry]
Expand Down
54 changes: 54 additions & 0 deletions src/oold/backend/postgres.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
try:
import psycopg
from psycopg import sql
from psycopg.types.json import Jsonb
except ImportError as exc:
raise ImportError("PostgresDocumentStore needs psycopg, install it with: pip install 'oold[postgres]'") from exc

from pydantic import Field

from oold.backend.interface import Backend, LinkedDataFormat, StoreResult


class PostgresDocumentStore(Backend):
"""Persist entities as JSONB rows in PostgreSQL, one row per IRI."""

dsn: str
table: str = Field(default="entities", pattern=r"^[A-Za-z_][A-Za-z0-9_]{0,62}$")
format: LinkedDataFormat = LinkedDataFormat.JSON
_conn: psycopg.Connection | None = None

def __init__(self, **kwargs):
super().__init__(**kwargs)
with self._connection().cursor() as cur:
cur.execute(
sql.SQL("CREATE TABLE IF NOT EXISTS {} (id TEXT PRIMARY KEY, data JSONB)").format(
sql.Identifier(self.table)
)
)

def _connection(self) -> psycopg.Connection:
if self._conn is None or self._conn.closed:
self._conn = psycopg.connect(self.dsn, autocommit=True)
return self._conn

def close(self):
"""Close the connection if it is open."""
if self._conn is not None:
self._conn.close()
self._conn = None

def resolve_iris(self, iris: list[str]) -> dict[str, dict]:
query = sql.SQL("SELECT id, data FROM {} WHERE id = ANY(%s::text[])").format(sql.Identifier(self.table))
with self._connection().cursor() as cur:
cur.execute(query, (iris,))
return dict(cur.fetchall())

def store_json_dicts(self, json_dicts: dict[str, dict]) -> StoreResult:
query = sql.SQL(
"INSERT INTO {} (id, data) VALUES (%s, %s) ON CONFLICT (id) DO UPDATE SET data = EXCLUDED.data"
).format(sql.Identifier(self.table))
conn = self._connection()
with conn.transaction(), conn.cursor() as cur:
cur.executemany(query, [(iri, Jsonb(d)) for iri, d in json_dicts.items()])
return StoreResult(success=True)
26 changes: 26 additions & 0 deletions tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
"""

import importlib.util
import os
import uuid

import pytest

Expand Down Expand Up @@ -62,6 +64,7 @@ def _restore_global_registries():

registries = [
interface._resolvers,
interface._backends,
_types,
_types_v1,
_controller_types,
Expand All @@ -77,6 +80,29 @@ def _restore_global_registries():
registry.update(snapshot)


@pytest.fixture
def postgres_store():
"""A PostgresDocumentStore on its own table, dropped afterwards.

Skips unless ``OOLD_TEST_POSTGRES_URL`` points at a live server, so the
default suite and the OS/Python matrix never need one.
"""
pytest.importorskip("psycopg")
dsn = os.environ.get("OOLD_TEST_POSTGRES_URL")
if not dsn:
pytest.skip("OOLD_TEST_POSTGRES_URL is not set")

from psycopg import sql

from oold.backend.postgres import PostgresDocumentStore

table = f"entities_{uuid.uuid4().hex}"
store = PostgresDocumentStore(dsn=dsn, table=table)
yield store
store._connection().execute(sql.SQL("DROP TABLE IF EXISTS {}").format(sql.Identifier(table)))
store.close()


def pytest_configure(config):
"""Register the ``benchmark`` mark so it is not an unknown-mark warning."""
config.addinivalue_line("markers", "benchmark(**kwargs): pytest-benchmark group settings")
Expand Down
33 changes: 12 additions & 21 deletions tests/test_backends.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,38 +86,35 @@ def _run(store: Backend):
_store_procedure(store, pydantic_version="v2")


# Test names are the benchmark baseline keys (scripts/compare_benchmarks.py matches on them),
# so keep them stable instead of parametrizing.
@pytest.mark.benchmark(group="backend")
def test_simple_dict_document_store(benchmark):
store = SimpleDictDocumentStore()

if benchmark is not None:
benchmark(_run, store)
else:
_run(store)
benchmark(_run, SimpleDictDocumentStore())


@pytest.mark.benchmark(group="backend")
def test_sqlite_document_store(benchmark):
from oold.backend.document_store import SqliteDocumentStore

store = SqliteDocumentStore(db_path=":memory:")

if benchmark is not None:
try:
benchmark(_run, store)
else:
_run(store)
finally:
store.close()


@pytest.mark.benchmark(group="backend")
def test_local_sparql_store(benchmark):
from oold.backend.sparql import LocalSparqlBackend

store = LocalSparqlBackend()
benchmark(_run, LocalSparqlBackend())

if benchmark is not None:
benchmark(_run, store)
else:
_run(store)

@pytest.mark.benchmark(group="backend")
@pytest.mark.postgres
def test_postgres_document_store(postgres_store, benchmark):
benchmark(_run, postgres_store)


def test_simple_dict_file_persistence(tmp_path):
Expand Down Expand Up @@ -158,9 +155,3 @@ def test_simple_dict_file_no_file():
store = SimpleDictDocumentStore()
assert store.file_path is None
assert store._store == {}


if __name__ == "__main__":
test_simple_dict_document_store(None)
test_sqlite_document_store(None)
test_local_sparql_store(None)
Loading
Loading