Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,20 @@ The rendered figures are exposed via `view.figures` so a host app can add its
own annotations; `view.export_series()` returns the plotted series as tidy
records.

Export covers every checked channel, numeric and text-log alike, while the plot
HTML is numeric-only. Column names must stay unique: composite sub-fields share
their parent channel label, and a duplicate name makes `df[label]` return a
DataFrame instead of a Series, which breaks the per-column `dequantify()`.
Composite series are therefore labelled `<channel>/<field>`, with a dedupe
safety net in `_series_to_dataframe`.

HTML export builds a fresh set of figures rather than reusing the live ones:
`bokeh.embed.file_html` raises `RuntimeError: Models must be owned by only a
single document` for a figure already rendered in a `pn.pane.Bokeh`, and
`copy.deepcopy` on a figure fails too. Both views therefore split figure
construction into a pure `_make_figures()`, used for the live panes and reused
by `_export_figures()`.

### Configuration and URL sync

The config is the complete, JSON-serializable state of a view, composed to
Expand All @@ -203,6 +217,55 @@ aggregate model.
*Two views composed under one aggregate config, URL-synced in PLAIN_KEYS mode
(readable `?app.reactor...` params in the address bar).*

### Implementation notes

Behaviour of the Panel/Bokeh stack that the view code depends on. Read this
before touching figure building, zoom/reset or layout CSS in
`src/opensemantic/base/view/`.

**Linked x-axes need one shared range at creation.** Create a single
`DataRange1d` and pass it to every figure while building it:

```python
shared_x = DataRange1d()
fig = bk_figure(x_range=shared_x, ...) # every figure
```

Reassigning `f.x_range` afterwards races the render and leaves some plots
unlinked, so panning moves only part of them. The y-ranges stay per figure, one
scale per characteristic.

**Reset events do not reach the server.** `figure.on_event(Reset, cb)` never
fires under `pn.pane.Bokeh`: Bokeh *events* are not forwarded, while model
*property* changes sync reliably. Bridge it through a property:

```python
self._reset_bridge = ColumnDataSource(data={"n": [0]})
self._reset_bridge.on_change("data", self._on_reset_bridge)
fig.js_on_event(Reset, CustomJS(args={"bridge": self._reset_bridge},
code="bridge.data = {n: [(bridge.data.n[0] || 0) + 1]};"))
```

`m.reset.emit()` does fire `js_on_event`, but client-side only, so it is useful
in tests and not a substitute for the bridge.

**Shadow-DOM CSS.** Page-level `pn.config.raw_css` does not reliably reach
elements inside a component's shadow root. For a Panel `AnyWidgetComponent` such
as Wunderbaum, pass `stylesheets=[css]`, which injects into the component's own
shadow root. That is how the tree grid's hardcoded inline `width: 800px` is
overridden.

**Content scrolling is owned by panelini.** `.content` lacked the `overflow-y:
auto` that `.sidebar` has, so tall stacked plots spilled past the wallpaper;
fixed upstream in
[panelini#37](https://github.com/OpenSemanticWorld/panelini/pull/37). With that
in place, leave `_plot_col` uncapped, otherwise a `max_height` plus `scroll=True`
adds a second, nested scrollbar.

**Stale Bokeh models.** After a pane rebuild, old figures and ranges linger in
`doc._all_models`. When inspecting through JS, filter to the live ones or you
will read a phantom range.

## Server-side downsampling

Large time series are downsampled on the server so the plot only transports the
Expand Down Expand Up @@ -413,3 +476,21 @@ PostgREST integration tests require a running [pgstack](https://github.com/opens
1. Start pgstack: `docker compose -f docker-compose.yml -f docker-compose.example-tsdb.override.yml up -d`
2. Copy `tests/.env.example` to `tests/.env` and fill in `TEST_PGRST_URL` and `TEST_PGRST_JWT_SECRET`
3. Run tests - PostgREST tests are skipped unless both env vars are set

### Browser tests

`tests/test_export_browser.py` drives `panel serve` with Playwright. Four things
it has to get right:

- `examples/datatool_dashboard.py` pins its SQLite database next to the script,
so the test serves a **copy in a temp directory** rather than contending with
a running demo on the shared file.
- Bokeh 3 renders widgets in a shadow root, so find buttons with
`get_by_role("button", name=...)`, which pierces it, not a light-DOM
`querySelectorAll`.
- Readiness is a raw TCP connect, not an HTTP GET: a corporate proxy may divert
localhost GETs.
- Port cleanup parses `netstat` by column, not by the state word, which is
localized (German Windows prints `ABHOEREN`, not `LISTENING`).
- `FileDownload` generates its payload lazily on click, so wrap the click in
`expect_download`.
97 changes: 97 additions & 0 deletions examples/dataset_from_remote_csv.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
"""Wrap a CSV that lives on the web in a Dataset.

A ``Dataset`` carries its rows either inline in ``data`` or by reference,
through a ``Distribution`` that names the file holding them. This builds the
second form over a CSV nobody here controls, then reads the rows back into the
typed model with one call.

``AddressRow`` gives the columns their types. That is what makes the dataset
more than a file: ``materialize_data()`` knows what each column means, so the
rows come back as models rather than strings in a frame.

Run from the package root::

python examples/dataset_from_remote_csv.py
"""

from pathlib import Path
from typing import List, Optional

from pydantic import BaseModel

from opensemantic.base import Dataset, Distribution
from opensemantic.core import Label, LocalFile, RemoteFile

CSV_URL = (
"https://raw.githubusercontent.com/w3c/csvw/refs/heads/gh-pages/"
"examples/dm-example-1/addresses.csv"
)


class AddressRow(BaseModel):
"""One line of the CSV. Field names match the column headers."""

ID: Optional[int] = None
city: Optional[str] = None
state: Optional[str] = None


class AddressDataset(Dataset):
"""A dataset whose rows are addresses.

``Dataset`` is the controller that implements the model, so this subclass
carries the conversions - ``to_df``, ``materialize_data`` - without naming
them.
"""

data: Optional[List[AddressRow]] = None


def main() -> None:
# 1. The dataset refers to the file instead of holding its content. The
# file entity is a RemoteFile, so reading it needs no local copy.
dataset = AddressDataset(
label=[Label(text="W3C CSVW example addresses")],
distributions=[
Distribution(
file=RemoteFile(
label=[Label(text="addresses.csv")], download_url=CSV_URL
),
media_type="text/csv",
)
],
)
print(f"{len(dataset.distributions)} distribution, data: {dataset.data}")

# 2. Read the file and build the typed rows from it.
rows = dataset.materialize_data()
for row in rows:
print(
f" {type(row).__name__}(ID={row.ID!r}, city={row.city!r}, "
f"state={row.state!r})"
)

# 3. The rows are now inline, and the frame form carries the same columns.
print(f"\n{dataset.to_df()}")

# 4. Back to CSV. The second row names each column's unit and is empty
# where a column has none, so a table of plain values writes and reads
# like any other CSV.
print(f"\n{dataset.serialize_data('csv').decode('utf-8')}")

# 5. The whole loop: write the rows to a local file, let a Distribution
# describe it, drop the inline copy, and read them back.
target = Path(__file__).parent / "addresses.roundtrip.csv"
dataset.externalize_data(
LocalFile(label=[Label(text=target.name)], local_path=str(target)),
fmt="csv",
)
print(f"data after externalize: {dataset.data}")

restored = dataset.materialize_data(dataset.distributions[-1])
print(f"read back from {target.name}: {restored}")
target.unlink()


if __name__ == "__main__":
main()
5 changes: 4 additions & 1 deletion setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,10 @@ python_requires = >=3.10
# For more information, check out https://semver.org/.
install_requires =
opensemantic>=0.2.4
opensemantic.core>=0.53.2
# TEMPORARY: the file controllers the dataset controller needs are merged
# but not released. Replace with opensemantic.core>=0.59.0 once a version
# tag is cut.
opensemantic.core @ git+https://github.com/OpenSemanticWorld-Packages/opensemantic.core-python@main
pydantic[email]

[options.packages.find]
Expand Down
22 changes: 21 additions & 1 deletion src/opensemantic/base/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,26 @@
from opensemantic.base._model import * # noqa

try:
from opensemantic.base import _controller # noqa: F401
from opensemantic.base._controller import * # noqa
except ImportError:
except ImportError as _error: # pragma: no cover - an optional dependency is missing
# Without this the controllers are simply absent and every lookup returns
# the pure model, which reads as "no controller registered" rather than as
# a broken install.
import logging as _logging

# info, not warning: the controllers are an optional extra, so a
# model-only install is a supported configuration - but a missing
# dependency must still be nameable, instead of the controllers simply
# not being there
_logging.getLogger(__name__).info(
"opensemantic.base controllers are unavailable: %s", _error
)

# Assignments, not imports: isort sorts an import above the star import, which
# would put the pure models back. These names resolve to the controllers that
# implement them.
try:
Dataset = _controller.DatasetController
except NameError: # pragma: no cover - controllers unavailable, see above
pass
20 changes: 20 additions & 0 deletions src/opensemantic/base/_controller.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,11 @@
DownsampleParams,
TSDCMixin,
)
from opensemantic.base._dataset_controller_mixin import ( # noqa: F401 (re-export)
DatasetControllerMixin,
)
from opensemantic.base._model import Database as _Database
from opensemantic.base._model import Dataset as _Dataset
from opensemantic.base._model import DataTool as _DataTool


Expand Down Expand Up @@ -175,3 +179,19 @@ async def start_offline_sync(self):
DeleteToolParams = TSDCMixin.DeleteToolParams
WriteToolChannelRawParams = TSDCMixin.WriteToolChannelRawParams
ReadToolChannelRawParams = TSDCMixin.ReadToolChannelRawParams


class DatasetController(DatasetControllerMixin, _Dataset):
"""Dataset controller (v2)."""


# A controller stands in for the model it implements, so a Dataset a caller
# builds can convert and materialize itself; it needs nothing beyond what the
# model requires and its __init__ does nothing but construct.
#
# DataToolController and the database controllers keep their own names. The
# first indexes channels, warns about unloaded characteristics and may open an
# archive database on construction, which is not what building a data model
# should do. Of the latter there are three for one Database, and the local one
# needs a db_path, so which applies is the caller's choice, not a default.
Dataset = DatasetController
10 changes: 7 additions & 3 deletions src/opensemantic/base/_controller_mixin.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,9 +80,13 @@ def __init__(self, *args, **data):
# Auto-init archive database from storage_locations
_archive_db = getattr(self, "archive_database", None)
_needs_init = _archive_db is None or isinstance(_archive_db, dict)
_storage = getattr(self, "storage_locations", None)
if self.auto_archive and _needs_init and _storage:
self.archive_database = self._init_archive_database(_storage[0])
if self.auto_archive and _needs_init:
# read last: storage_locations is a link field, so reading it
# resolves its IRIs and needs a registered resolver. Constructing
# a tool that does not archive must not require one.
_storage = getattr(self, "storage_locations", None)
if _storage:
self.archive_database = self._init_archive_database(_storage[0])

# __setattr__ for private attrs inherited from BaseController
# get_osw_id() and get_iri() are inherited from OswBaseModel
Expand Down
Loading
Loading