Skip to content

sdk/server: Implement missing serialization modifier - #660

Open
hpoeche wants to merge 49 commits into
eclipse-basyx:developfrom
rwth-iat:feat/serialization-formats
Open

hpoeche wants to merge 49 commits into
eclipse-basyx:developfrom
rwth-iat:feat/serialization-formats

Conversation

@hpoeche

@hpoeche hpoeche commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Previously, our SDK missed the JSON serialization formats "Metadata", "ValueOnly" and "Path",
that are defined in the IDTA-01001 specification. The StrippedAASToJsonEncoder implemented
a different serialization format, but followed none of the described ones (see #651).
Therefore, our server implementation responded with 501 Not Implemented to GET requests
on /$value and /$path routes, ignored ?extent and wrongly interpreted the /$metadata and
?level=core modifiers (see #650, #651).

Additionally, the order of objects in JSON arrays, which are not required to be ordered by the specification,
was not explicitly defined during serialization, which lead to different JSON strings for the same objects
(see #573).

Finally, during implementation we discovered a bug in the current serialization of SubmodelElementList
and their children. Because internally each child gets assigned a generated id_short this value was never
serialized, even when manually set.

Changes

We need to support all serialization formats, defined in IDTA-01001, as well as all serialization modifiers
from IDTA-01002 which are defined on top of these formats. We therefore chose to keep all
serialization logic in the SDK and also implement level and extent as parts of the encoder classes.

SDK changes - basyx.aas.adapter.json

With this change, the SDK offers a JSON encoder for every serialization format of IDTA-01001,
each one a subclass of AASToJsonEncoder in the new subpackage adapter.json.formats:
MetadataJsonEncoder, ValueOnlyJsonEncoder, ReferenceJsonEncoder and PathJsonEncoder.
To support the level and extent modifier introduced in IDTA-01002, AASToJsonEncoder itself
gets the init argument deep for level=core and the class attribute with_blob_value for
extent=WithoutBLOBValue.
All formats and modifier are tested in sdk/test/adapter/json/formats using unittest.

Collections that are unordered according to the specification are now serialized in a
canonical order, defined in adapter._ordering. For Referableobjects,modelType`
is now always the first key in JSON serialization.

Elements of SubmodelElementList now output their idShort, if it is a non generated one. This
holds for JSON and XML serialization and uses a central serializable_id_short() function that hides
generated idShort values of elements in SML as None and lets all other values pass.

Server changes

The server parses the modifiers centrally in app.adapter.modifiers and selects the matching
JSON encoder, so all content modifiers as well as level and extent are supported on the
Submodel Repository GET routes. Combinations that IDTA-01002 does not allow are rejected with
400 Bad Request.
Requests that expect an XML response fail fast with 406 Not Acceptable before object fetching, when
a serialization different from format Normal with level=deep and extent=WithBlobValue is present.
This changes existing behavior, which however was never defined in the specification.

All endpoints are tested for correct behavior under all modifier values.

Decisions we made

Where the specification is ambiguous or contradicts itself, the following decisions were made:

  • Operations and Capabilities have neither a ValueOnly nor a Metadata representation. They
    are omitted from listings, and requesting them directly results in 400 Bad Request.
  • In ValueOnly, elements without a value are omitted, except within a SubmodelElementList,
    where they are kept as null to preserve the indices. Further assumptions are documented
    in the docstring of ValueOnlyJsonEncoder.
  • The idShortPaths of $path start with the idShort of the requested element. Therefore, an
    element of a SubmodelElementList without an idShort has no Path representation
    (400 Bad Request).
  • GET .../submodel-elements/$value returns a single object with the idShorts as names, so that
    each value stays associated with its element.
  • The API spec states the response to /submodels/$value as one SubmodelValue object, which
    can only hold values for one submodel, because the idShorts can duplicate across submodels.
    Because this is a known error of the spec submodels-value-issue, that is corrected in a slightly
    later version, we chose to implement this correct version directly.

Changed behavior

The implementation changes result in changes in the existing behavior as follows:

  • Normal JSON responses of the Submodel Repository omit Blob values unless
    ?extent=WithBLOBValue is given, as the specification defines WithoutBLOBValue as the default.
  • XML only supports $reference and the Normal serialization with level=deep and
    extent=WithBLOBValue. Any other combination, including a request without ?extent=WithBLOBValue,
    is rejected with 406 Not Acceptable. This breaks existing clients.
  • ?limit=0 is rejected with 400 Bad Request.

AI Usage

This contribution was written with the help of Claude Opus 5.5.

Fixes #446
Fixes #573
Fixes #643
Fixes #644
From #650 and #651, GET requests are covered while PATCH requests are still wrong

Closes #649
Closes #623
Closes #621

thammel and others added 5 commits August 31, 2026 17:58
All `$value` routes previously responded with `501 Not Implemented`.

The specification leaves some points open, which are resolved as
follows:

- Property values are mapped to the JSON type of their `valueType`, so
  booleans and numbers are not quoted. Values without a JSON literal
  are returned as their XSD representation.
  `xs:integer` is unbounded in both Python and JSON and is
  therefore passed on as it is.
- Elements without a value are omitted from Submodels, collections and
  Entity statements, but kept as `null` within a SubmodelElementList, as
  omitting them would shift the indices of the remaining elements.
- Operations and Capabilities have no ValueOnly representation. They are
  skipped in containers and result in `400 Bad Request` when requested
  directly, which is how the `$metadata` route handles them.
- `level=core` returns the direct children of the requested resource and
  their nested containers as empty, matching the examples of the
  specification.
- The entries of `GET /submodels/{id}/submodel-elements/$value` are
  mapped by idShort, as a bare `SubmodelElementValue` would lose the
  association between a value and its element. Submodels are returned
  as-is, since their entries are already named.
- The languages of a MultiLanguageProperty are sorted, to keep the order
  of the response deterministic.

The ValueOnly serialization is only defined for JSON, so the routes
respond with `406 Not Acceptable` if XML is requested. `extent` remains
unimplemented, as on all other routes. `PATCH …/$value`, the `$path`
routes and the operation invocation routes are not part of this change.

Fixes eclipse-basyx#446
Previously, the ValueOnly serialization was method based in the
server package. These changes shift the code from serve package to
the sdk, as ValueOnly serialiaztion is defined in Part 1 of the spec.
Additionally, I restructured the code to live inside a `JSONEncoder`
subclass, so it can be used the same way as `AASToJsonEncoder`.

Still TODO:
- Implement `level=core` serialization
- Integrate this serialiazer into the server
Showcases collapse of rules and handler methods using converter and modifiers

@hpoeche hpoeche left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the idea of the ContentModifierConverter for the endpoint definition! I found some small things we should discuss, so I can mirror them for the ValueOnly encoder.

Comment thread server/app/adapter/modifiers.py Resolved
Comment thread server/app/adapter/modifiers.py Resolved
JAB1305 and others added 13 commits October 19, 2026 20:36
Previously, `PathJsonEncoder` carried `deep` as a class variable
  and the `?level=core` variant was expressed by a dedicated
  `ShallowPathJsonEncoder` subclass that flipped it to `False`.
  Composing this axis with other content modes required another
  `Shallow*` sibling per encoder, so the class count scaled with
  the Cartesian product of axes.

  With this change, `deep` is passed to `PathJsonEncoder.__init__`
  as a keyword argument and stored on the instance. The encoder
  uses `default()` directly instead of extending the classmethod-
  based `_get_aas_class_serializers` dispatch, which avoids
  `cls`/`self` mismatches along the MRO when carrying per-request
  state. On the server side, `pick_encoder` now returns
  `(cls, kwargs)` and `JsonResponse.serialize` forwards the kwargs
  through `json.dumps` into the encoder constructor;
  `ResultShallowPathJsonEncoder` is removed since the single
  `ResultPathJsonEncoder` now covers both levels via the kwarg.
These changes implement a core serialization for the `AASToJsonEncoder`
that is backward compatible to custom subclasses of this class.
For all five types, that are affected by core serialization, addtional
methods were introduced that call the standard serialization method
and afterwards serialize all children, omitting the grandchildren.

This solution allows us to pass the `deep` parameter via the `__init__()`
method, which helps to avoid addtional subclass constructs.

The `ValueOnlyJsonEncoder` overwrites these methods to also implement
core serialization. Test are written for both classes in core serialization
mode.
Fixes __init__ ordering by using local imports
Previously, the order of elements in JSON arrays was
non-deterministic for the same AAS. This overcomplicated
easy change detection of serialized objects.

With these changes, this behavior is changed. All JSON
arrays are now ordered, either because the metamodel requires
them to be sorted (e.g. SubmodelElementList) or sorting is
done during serialization (e.g. elements of
SubmodelElementCollections are now ordered by their `idShort`).

The order of keys in the JSON objects is already defined by
their insertion order in the Python `dict`. From now on,
objects with equal content will result in identical JSON
representations.

Fixes eclipse-basyx#573
JAB1305 and others added 19 commits November 5, 2026 22:16
Address review comments on the ported test_metadata.py:
- Drop XML support, expect 406 for XML requests
- Check responses are actually serialized as Metadata
- Use FormatClient for all requests
- Derive element counts from the example submodel
- Add extent and 404 tests, check all level values
Address review comments on the ported test_value.py
General overhaul following metadata testing and encoder redesign

- Drop tests and helpers duplicating SDK Encoder tests
- Replace 501 extent test expecting functional blob toggle
- Add a shared value endpoint test base to enforce proper modifier combination handling
- Flag list endpoint shape with TODO for now
Replace the separate handlers for $metadata, $value, $reference and
$path with single handlers driven by the parsed serialization
modifiers, which now select the response encoder.

Extract the result serialization into a mixin, so each format can
combine it directly with its SDK encoder. Listings of
submodel-elements are wrapped so that each format controls the shape
of its result.

Filtering and pagination of $path listings is still to be resolved.
Previously, in path serialization the pagination was still applied
to the SubmodelElements. Because each SubmodelElement is mapped to
multiple idShortPath strings, the paginated results exceeded the
limit.

Now the pagination is done on the already encoded idShortPaths.
Therefore the `ResultPathJsonEncoder` is extended with a `iter_paths()`
method, that returns an iterator for the paths, that is then paginated
using the existing `_get_slice()`.
Elements added to a SubmodelElementList without an idShort get an
internal `generated_submodel_list_hack_...` idShort. Previously,
this could not leak because elements in SubmodelElementList were not
allowed to have idShorts. When this contstrain (AASd-120) was dropped,
this lead to the following problems:

- Normal + Metadata Serialization (JSON + XML):
  Previously the idShort of an element inside a SML was never serialized.
  This wrongly hid existing idShorts. To fix this, we introduce a central
  `serializable_id_short()` function that only hides generated idShorts
  of elements in SML as `None` and lets all other values pass.

- Path Serialization (JSON):
  When elements of a SML are serialized directly, their idShort leads
  the idShortPaths that are generated. At this point the generated
  isShorts could leak, so we use the `serializable_id_short()` helper
  and raise a `ValueError` on generated or missing idShorts.

- Server `POST .../submodel-elements/<idShortPath>`:
  When a new SubmodelElement is created the response header `Location`
  contians the address that should return this element. This was manually
  contructed from the new idShort, leaking generated values, which lead
  to a wrong address. We now use the `Referable.get_id_short_path_as_list()`
  method, which handles children of SML correctly.

Test cases are added, that ensure no generated idShorts are leaking
and all manually set idShorts are serialized.
- `limit=0` returned an empty page whose cursor points to the same start.
  `_get_slice` now rejects `limit < 1` with 400 for all routes.
- By-path `$path` on an element without idShort (child of a
  SubmodelElementList) raised a ValueError (500). It now responds with 400.
  List children with an idShort are still supported.
- Use a format client in all `$path` tests instead of the bare client.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hpoeche
hpoeche marked this pull request as ready for review November 12, 2026 22:11
@s-heppner

Copy link
Copy Markdown
Member

Regarding #621, which implemented the fixed JSON order as opt-in and here, which implemented it as always-on; I'd like to document here which way we go and why.

Always-on gives everyone reproducible output, but every serialization now becomes more expensive due to the sorting, and every existing user gets a changed (although equivalent) JSON output. Do we accept that, or do we want an opt-out?

This change contains some minor fixes from the review:

- The `adapter.json.formats` subpackage was missing its
  `__init__.py`, unlike every other package in `basyx`.
- The `406 Not Acceptable` message of `XmlResponse` printed
  `/$Content.METADATA` instead of `/$metadata`, since an f-string of
  a `(str, Enum)` member uses `str()` from Python 3.12 on.
- The embedded data specifications were sorted with a `lambda` that
  only wrapped the sort key function.
- A line in `ValueOnlyJsonEncoder._file_to_value_only()` was
  indented too far.
if isinstance(value, (float, model.datatypes.Decimal)):
if not math.isfinite(float(value)):
return model.datatypes.xsd_repr(value)
return float(value)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

xs:decimal loses precision here, at serialization time. The stdlib json cannot serialize Decimal, so it is converted to float first, which rounds it to the nearest IEEE double:

  • Decimal("0.10000000000000000000001") → 0.1
  • Decimal(2**53 + 1) → 9007199254740992.0 (also off by one, and gets a .0)

The spec allows this in principle: IDTA-01001 maps xs:decimal to a JSON number and says that values that cannot be represented in JSON "cannot be used" (IDTA-01001 v3.1.2, Note 1). So this is not a spec violation, but the integer types are handled differently: an int is passed through unchanged, and json writes it exactly at any size (2**53 + 1 → 9007199254740993). For integers, precision is only lost if the client parses JSON numbers as doubles. For decimals, we lose it ourselves before the client sees anything.

The spec covers both approaches, so I don't mind which one we choose, as long as all numeric types are treated the same way:

  • Keep the exact value for all types: write the Decimal digits as a JSON number instead of going through float, and leave any rounding to the client, as we already do for integers.
  • Round all types: also convert the unbounded integer types to float, so that every number in our output can be represented as an IEEE double.

Whichever we choose, let's document it in the class docstring next to the NaN/INF note.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right here, we should handle Decimal different from float and serialize it with full precision, like this is done already in the normal encoder. I will adapt the code accordingly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After taking a closer look at this, this is not as easy as it seems. The base json.JSONEncoder only supports str, int, float, bool, None, listand dict. As far as I (and Claude) can see, we have the following options:

  1. Map a decimal.Decimal to a float and accept precision loss.
  2. Overwrite the internal method iterencode() of the json.JSONEncoder, thereby relying on undocumented behavior (I dislike this very much).
  3. Switch to the simplejson implementation, which supports decimal.Decimal out of the box. This would require some rework on the other encoders and is a breaking change to downstream subclasses any users implemented.

We should discuss this in a meeting.

@hpoeche

hpoeche commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

Regarding #621, which implemented the fixed JSON order as opt-in and here, which implemented it as always-on; I'd like to document here which way we go and why.

Always-on gives everyone reproducible output, but every serialization now becomes more expensive due to the sorting, and every existing user gets a changed (although equivalent) JSON output. Do we accept that, or do we want an opt-out?

We discussed this in the meeting and chose to always sort by default, because ordered instead of no order at all is not a real change. The computational overhead due to the sorting should be small, because sorting is optimized in C. Or do we need a performance analysis here?

@s-heppner

Copy link
Copy Markdown
Member

We discussed this in the meeting and chose to always sort by default, because ordered instead of no order at all is not a real change. The computational overhead due to the sorting should be small, because sorting is optimized in C. Or do we need a performance analysis here?

No need for performance analysis needed. #648 should more than make up for any performance losses we incur here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants