Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,12 +235,12 @@ Parameters:

Built-in backends for entity persistence and resolution:

| Backend | Storage | Query support |
| --------------------------- | ------------------------------------------------ | --------------- |
| **SimpleDictDocumentStore** | In-memory dict, optional JSON file (`file_path`) | Filter by field |
| **SqliteDocumentStore** | SQLite database (default `format=JSON`) | - |
| **PostgresDocumentStore** | PostgreSQL JSONB (`oold[postgres]`) | - |
| **LocalSparqlBackend** | In-memory RDF graph (rdflib) | SPARQL |
| Backend | Storage | Query support |
| --------------------------- | ------------------------------------------------ | ----------------------------- |
| **SimpleDictDocumentStore** | In-memory dict, optional JSON file (`file_path`) | Filter by field, nested paths |
| **SqliteDocumentStore** | SQLite database (default `format=JSON`) | - |
| **PostgresDocumentStore** | PostgreSQL JSONB (`oold[postgres]`) | Filter by field, nested paths |
| **LocalSparqlBackend** | In-memory RDF graph (rdflib) | SPARQL |

```python
from oold.backend.document_store import SimpleDictDocumentStore
Expand Down
59 changes: 56 additions & 3 deletions docs/how-to/backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,9 +129,11 @@ alice = Person["ex:alice"]
store.close() # the connection stays open until you close it
```

- Entities live in table `entities` (`id TEXT PRIMARY KEY, data JSONB`), created on first use. Pass `table="..."` to use another name (letters, digits and underscores, up to 63 characters).
- Entities live in table `entities` (`id TEXT PRIMARY KEY, data JSONB`), created on first use. Pass `table="..."` to use another name (letters, digits and underscores, up to 58 characters).
- Storing an IRI again overwrites its row, and a batch is written in one transaction.
- Querying with `Model[Model.field == value]` is not supported yet.
- `index=True` (the default) adds a GIN index on `data` that serves equality conditions. Pass `index=False` to skip it.
- `query_limit=100` caps the rows a query returns. The default is no limit.
- Querying works as described in [Querying document stores](#querying-document-stores).

---

Expand All @@ -156,6 +158,56 @@ The backend serializes each entity to JSON-LD before inserting it into the RDF g

---

## Querying document stores

`SimpleDictDocumentStore` and `PostgresDocumentStore` answer the query DSL, so a condition on a model class subscripts straight into the store:

```python
Person[Person.age > 30]
Person[(Person.name == "Ada") & (Person.age >= 18)]
Person[(Person.name == "Ada") | (Person.name == "Bob")]
```

`==`, `!=`, `<`, `<=`, `>`, `>=`, `&` and `|` are supported. Put parentheses around each side of `&` and `|`.

### Nested fields

A field that embeds a model chains, so you can filter on what is inside it:

```python
from pydantic import BaseModel

class Address(BaseModel):
city: str | None = None

class Person(LinkedBaseModel):
id: str
address: Address | None = None

Person[Person.address.city == "Berlin"]
```

The condition carries the dotted path `address.city`. You can also write it by hand with `Condition(field="address.city", ...)`. A document where the path is missing never matches, not even for `!=`.

Paths reach embedded objects only:

- A link is an IRI reference, not an embedded object. Compare it with an IRI (`Person.knows == "ex:bob"`). `Person.knows.name` raises, because following a link means looking up another document.
- A list of models cannot be indexed, so `Person.previous.city` raises.
- Chaining needs the default descriptor binding. With `OOLD_DESCRIPTOR_BINDING=0` it raises `NotImplementedError`.

### Differences between the stores

| | SimpleDictDocumentStore | PostgresDocumentStore |
| --- | --- | --- |
| A key that contains a dot | matches as a literal key first | always read as a path |
| A range on mixed types (`age < 36` on a string) | raises `TypeError` | does not match |
| Strings in a range | Python ordering | database collation |
| Result order | not defined | by IRI |

On PostgreSQL, equality is jsonb containment and uses the GIN index. Range conditions scan the table. A range only compares values of the same JSON type, so a JSON `null` or a string never matches `age < 36`.

---

## Querying a SPARQL backend

`SparqlResolver`, `LocalSparqlResolver` and `WikiDataSparqlResolver` translate
Expand All @@ -170,7 +222,8 @@ Person[Person.name == "Tim Berners-Lee"]
```

`eq`, `ne`, `lt`, `le`, `gt`, `ge` and `&` are supported. Anything else raises
rather than returning the wrong rows. The predicate and the literal are taken
rather than returning the wrong rows. That includes `|` and nested field paths
such as `Person.address.city`. The predicate and the literal are taken
from the model's own JSON-LD context, so a term scoped to a language produces a
language-tagged literal and one with an `@type` produces a typed one.

Expand Down
24 changes: 23 additions & 1 deletion src/oold/backend/document_store.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,29 @@
ResolveResult,
StoreResult,
apply_operator,
field_path,
)

_MISSING = object()


def _lookup(document: Any, field: str) -> Any:
"""The value a condition field points at, or ``_MISSING``.

A literal key wins, so a document whose key contains a dot keeps matching
as before. Otherwise the field is a dotted path into embedded objects.
"""
if not isinstance(document, dict):
return _MISSING
if field in document:
return document[field]
value: Any = document
for segment in field_path(field):
if not isinstance(value, dict) or segment not in value:
return _MISSING
value = value[segment]
return value


class SimpleDictDocumentStore(Backend):
"""In-memory document store backed by a Python dict.
Expand Down Expand Up @@ -71,7 +92,8 @@ def _filter(
# key = context[key]
matched_entities = set()
for iri, jsonld_dict in data.items():
if key in jsonld_dict and apply_operator(operator, jsonld_dict[key], value):
found = _lookup(jsonld_dict, key)
if found is not _MISSING and apply_operator(operator, found, value):
matched_entities.add(iri)
return matched_entities

Expand Down
31 changes: 24 additions & 7 deletions src/oold/backend/interface.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,14 @@ class ResolveResult(BaseModel):
nodes: dict[str, None | GenericLinkedBaseModel]


def field_path(field: str) -> list[str]:
"""Split a condition field into path segments: ``"address.city"`` -> ``["address", "city"]``."""
segments = field.split(".")
if not all(segments):
raise ValueError(f"Invalid field path {field!r}: empty segment")
return segments


class Query(BaseModel):
op1: Union["Query", "Condition"]
operator: str
Expand All @@ -65,6 +73,10 @@ class Query(BaseModel):
def __and__(self, other):
return Query(op1=self, operator="and", op2=other)

# override the | operator
def __or__(self, other):
return Query(op1=self, operator="or", op2=other)


class Condition(BaseModel):
field: str
Expand All @@ -81,6 +93,10 @@ def __eq__(self, other):
def __and__(self, other):
return Query(op1=self, operator="and", op2=other)

# override the | operator
def __or__(self, other):
return Query(op1=self, operator="or", op2=other)


class QueryParam(BaseModel):
query: Query | Condition
Expand All @@ -100,16 +116,17 @@ class Resolver(BaseModel):
def resolve_iris(self, iris: list[str]) -> dict[str, dict]:
pass

def resolve(self, request: ResolveParam):
# print("RESOLVE", request)

model_cls = request.model_cls
if model_cls is None:
model_cls = self.model_cls
def _model_cls_for(self, model_cls: type[GenericLinkedBaseModel] | None) -> type[GenericLinkedBaseModel]:
model_cls = model_cls or self.model_cls
if model_cls is None:
raise ValueError("No model_cls provided in request or resolver")
return model_cls

def resolve(self, request: ResolveParam):
model_cls = self._model_cls_for(request.model_cls)
return self._to_nodes(self.resolve_iris(request.iris), model_cls)

jsonld_dicts = self.resolve_iris(request.iris)
def _to_nodes(self, jsonld_dicts: dict[str, dict], model_cls: type[GenericLinkedBaseModel]) -> "ResolveResult":
nodes = {}
for iri, jsonld_dict in jsonld_dicts.items():
if jsonld_dict is None:
Expand Down
94 changes: 87 additions & 7 deletions src/oold/backend/postgres.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,27 +5,91 @@
except ImportError as exc:
raise ImportError("PostgresDocumentStore needs psycopg, install it with: pip install 'oold[postgres]'") from exc

from typing import Any

from pydantic import Field

from oold.backend.interface import Backend, LinkedDataFormat, StoreResult
from oold.backend.interface import (
Backend,
ComparisonOperator,
Condition,
LinkedDataFormat,
Query,
QueryParam,
ResolveResult,
StoreResult,
field_path,
)

# Fragments only: field names and values are always bound parameters.
_COMPARISONS = {
ComparisonOperator.LT: "<",
ComparisonOperator.LE: "<=",
ComparisonOperator.GT: ">",
ComparisonOperator.GE: ">=",
}


def _json_type(value: Any) -> str:
return "string" if isinstance(value, str) else "null" if value is None else "number"


def _where(node: Query | Condition) -> tuple[str, list[Any]]:
"""Render a query tree as a WHERE clause and its parameters.

Equality is jsonb containment, which a GIN index serves. Ranges compare the
value at the path and only against values of the same JSON type: jsonb
orders across types (null sorts below every number), Python raises.
"""
if isinstance(node, Query):
if node.operator not in ("and", "or"):
raise NotImplementedError(f"Operator {node.operator} not implemented")
left, left_params = _where(node.op1)
right, right_params = _where(node.op2)
return f"({left}) {node.operator.upper()} ({right})", left_params + right_params
if not isinstance(node, Condition):
raise TypeError("Invalid query type")

path = field_path(node.field)
operator = node.operator or ComparisonOperator.EQ
if operator == ComparisonOperator.EQ:
contained: Any = node.value
for segment in reversed(path):
contained = {segment: contained}
return "data @> %s", [Jsonb(contained)]
if operator == ComparisonOperator.NE:
return "(data #> %s::text[]) <> %s", [path, Jsonb(node.value)]
return (
f"(jsonb_typeof(data #> %s::text[]) = %s AND (data #> %s::text[]) {_COMPARISONS[operator]} %s)",
[path, _json_type(node.value), path, Jsonb(node.value)],
)


class PostgresDocumentStore(Backend):
"""Persist entities as JSONB rows in PostgreSQL, one row per IRI."""
"""Persist entities as JSONB rows in PostgreSQL, one row per IRI.

``index`` adds a GIN index (``jsonb_path_ops``), which serves equality
conditions on any path; range conditions scan the table.
"""

dsn: str
table: str = Field(default="entities", pattern=r"^[A-Za-z_][A-Za-z0-9_]{0,62}$")
table: str = Field(default="entities", pattern=r"^[A-Za-z_][A-Za-z0-9_]{0,57}$")
format: LinkedDataFormat = LinkedDataFormat.JSON
index: bool = True
query_limit: int | None = Field(default=None, gt=0)
_conn: psycopg.Connection | None = None

def __init__(self, **kwargs):
super().__init__(**kwargs)
table = sql.Identifier(self.table)
with self._connection().cursor() as cur:
cur.execute(
sql.SQL("CREATE TABLE IF NOT EXISTS {} (id TEXT PRIMARY KEY, data JSONB)").format(
sql.Identifier(self.table)
cur.execute(sql.SQL("CREATE TABLE IF NOT EXISTS {} (id TEXT PRIMARY KEY, data JSONB)").format(table))
if self.index:
cur.execute(
sql.SQL("CREATE INDEX IF NOT EXISTS {} ON {} USING GIN (data jsonb_path_ops)").format(
sql.Identifier(f"{self.table}_gin"), table
)
)
)

def _connection(self) -> psycopg.Connection:
if self._conn is None or self._conn.closed:
Expand All @@ -52,3 +116,19 @@ def store_json_dicts(self, json_dicts: dict[str, dict]) -> StoreResult:
with conn.transaction(), conn.cursor() as cur:
cur.executemany(query, [(iri, Jsonb(d)) for iri, d in json_dicts.items()])
return StoreResult(success=True)

def _select(self, query: Query | Condition) -> tuple[sql.Composable, list[Any]]:
where, params = _where(query)
statement = sql.SQL("SELECT id, data FROM {} WHERE ").format(sql.Identifier(self.table)) + sql.SQL(where)
statement += sql.SQL(" ORDER BY id")
if self.query_limit is not None:
statement += sql.SQL(" LIMIT %s")
params.append(self.query_limit)
return statement, params

def query(self, param: QueryParam) -> ResolveResult:
model_cls = self._model_cls_for(param.model_cls)
statement, params = self._select(param.query)
with self._connection().cursor() as cur:
cur.execute(statement, params)
return self._to_nodes(dict(cur.fetchall()), model_cls)
18 changes: 18 additions & 0 deletions src/oold/backend/sparql.py
Original file line number Diff line number Diff line change
Expand Up @@ -265,12 +265,30 @@ def _expand_term(model_cls, field: str, value) -> tuple[str, str]:
return predicate, literal


def _reject_nested_path(model_cls, field: str) -> None:
"""Refuse ``address.city`` when ``address`` embeds a model.

Left alone, a context with ``@vocab`` expands the dotted name into a made-up
predicate and the query quietly matches nothing. A dotted term the context
defines itself is no nested path and still translates.
"""
from oold.model._paths import embedded_model, model_fields

head, _, rest = field.partition(".")
if rest and embedded_model((model_fields(model_cls) or {}).get(head)) is not None:
raise NotImplementedError(
f"Nested field path {field!r} is not supported by the SPARQL resolvers yet. "
"Query a top-level field, or use a document store."
)


def _translate(node, model_cls, counter: list[int]) -> str:
"""Render a Condition or Query as SPARQL graph patterns."""
if isinstance(node, Query):
if node.operator != "and":
raise NotImplementedError(f"Unsupported query operator: {node.operator}")
return _translate(node.op1, model_cls, counter) + "\n" + _translate(node.op2, model_cls, counter)
_reject_nested_path(model_cls, node.field)
predicate, literal = _expand_term(model_cls, node.field, node.value)
operator = node.operator or ComparisonOperator.EQ
if operator == ComparisonOperator.EQ:
Expand Down
2 changes: 2 additions & 0 deletions src/oold/model/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
get_backend,
get_resolver,
)
from oold.model._paths import reject_nested_path
from oold.static import (
GenericLinkedBaseModel,
export_jsonld,
Expand Down Expand Up @@ -97,6 +98,7 @@ def __bool__(self):
return bool(d)

def __getattr__(self, name):
reject_nested_path(object.__getattribute__(self, "parent"), object.__getattribute__(self, "name"), name)
d = object.__getattribute__(self, "_default")
if d is not None and d is not PydanticUndefined:
return getattr(d, name)
Expand Down
Loading
Loading