Skip to content

adapter.json: Speed up JSON (de-)serialization - #648

Open
zrgt wants to merge 6 commits into
eclipse-basyx:developfrom
rwth-iat:perf/json-adapter
Open

zrgt wants to merge 6 commits into
eclipse-basyx:developfrom
rwth-iat:perf/json-adapter

Conversation

@zrgt

@zrgt zrgt commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Reading and writing large AAS JSON files was dominated by avoidable
work found by profiling (see #647): the serializer/parser mappings
were rebuilt for every object, json.dump() always uses the
pure-Python encoder, the BCP 47 regex was compiled on every language
tag check, and error strings were built even when no error occurred.

The public API and the produced JSON stay unchanged.

  • write_aas_json_file(): use json.dumps() (C-accelerated). The
    whole JSON string is now held in memory before it is written.
  • AASToJsonEncoder/AASFromJsonDecoder: build the class mapping
    once per class. The cache is looked up in cls.__dict__, so
    subclasses overriding the mapping get their own.
  • LangStringSet: compile the BCP 47 regex once at import.
  • _expect_type(): take the context as an object and only call
    str() if the check fails.
  • AASFromJsonDecoder.object_hook(): in failsafe mode, only
    pprint.pformat() the object if ERROR logging is enabled.

Measured on 70 real-world AAS JSON files (54 MB), best of 7:

develop this PR speedup
read_aas_json_file() 744 ms 606 ms 1.23x
write_aas_json_file() 964 ms 314 ms 3.07x

For 2000 sampled real-world files, output and error messages are
identical to develop (lists sorted, as set-based attribute order
varies). Tests added for subclass mapping overrides.

Part of #647

Claude Opus 5.4

Profiling `read_aas_json_file()` and `write_aas_json_file()` on real-world
AAS JSON files revealed some avoidable overhead:

- `write_aas_json_file()` used `json.dump()`, which always falls back to
  the pure-Python encoder. Use `json.dumps()` instead, which uses the
  C-accelerated encoder. The downside is that the whole JSON string is
  held in memory before writing.
- `AASToJsonEncoder.default()` and `AASFromJsonDecoder.object_hook()`
  rebuilt their class mapping for every single object. Cache it once
  per encoder/decoder class, so subclasses that override
  `_get_aas_class_serializers()` / `_get_aas_class_parsers()` still work.
- `LangStringSet._check_language_tag_constraints()` rebuilt the BCP 47
  regex from its parts for every language tag. Compile it once at
  import time.
- `_expect_type()` received `str(ret)` as context, which computes the
  id_short path of the parent for every child element, although it is
  only needed if the check fails. Pass the object and let the error
  message stringify it lazily.
- In failsafe mode, `object_hook()` formatted the defective JSON object
  with `pprint.pformat()` even if the log message was discarded. Only
  format it, if the logger is enabled for ERROR.

Measured on 70 real-world JSON files (54 MB), single process:
load 744 ms -> 606 ms (1.23x), save 964 ms -> 314 ms (3.07x).
The (order-independent) output and error messages are unchanged for
2000 sampled files.
Calling a `staticmethod` object in the class body only works since
Python 3.10, so we call its underlying function via `__func__` instead.
Also, quote the `re.Pattern[str]` return annotation, which can't be
evaluated at runtime before Python 3.9.
@zrgt
zrgt marked this pull request as draft September 25, 2026 07:46
The package requires Python 3.11, so the `__func__` workaround for
Python < 3.10 in the class body is obsolete.
@zrgt
zrgt marked this pull request as ready for review November 17, 2026 09:52
@zrgt
zrgt requested a review from s-heppner November 17, 2026 09:53

@s-heppner s-heppner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd merge #660 first and rebase this PR on it. #660 adds about 10 call sites that rebuild the
mappings for every object (_get_aas_class_no_children_serializers(), _remove_children(),
the _*_to_json_no_children() methods). After the rebase, please cache all three mappings
behind a small helper again, and run the benchmark again on top of #660's sorting.

Comment on lines 847 to 858
def write_aas_json_file(
file: _generic.PathOrIO,
data: model.AbstractObjectStore,
stripped: bool = False,
encoder: Optional[Type[AASToJsonEncoder]] = None,
**kwargs,
) -> None:
"""
Write a set of AAS objects to an Asset Administration Shell JSON file according to 'Details of the Asset
Administration Shell', chapter 5.5

:param file: A filename or file-like object to write the JSON-serialized data to

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole JSON string is now held in memory before it is written. Only a code comment says so. Callers who write very large files will read the docstring, not the comment, so one sentence here would help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants