Repository navigation
Conversation
All `$value` routes previously responded with `501 Not Implemented`.
The specification leaves some points open, which are resolved as
follows:
- Property values are mapped to the JSON type of their `valueType`, so
booleans and numbers are not quoted. Values without a JSON literal
are returned as their XSD representation.
`xs:integer` is unbounded in both Python and JSON and is
therefore passed on as it is.
- Elements without a value are omitted from Submodels, collections and
Entity statements, but kept as `null` within a SubmodelElementList, as
omitting them would shift the indices of the remaining elements.
- Operations and Capabilities have no ValueOnly representation. They are
skipped in containers and result in `400 Bad Request` when requested
directly, which is how the `$metadata` route handles them.
- `level=core` returns the direct children of the requested resource and
their nested containers as empty, matching the examples of the
specification.
- The entries of `GET /submodels/{id}/submodel-elements/$value` are
mapped by idShort, as a bare `SubmodelElementValue` would lose the
association between a value and its element. Submodels are returned
as-is, since their entries are already named.
- The languages of a MultiLanguageProperty are sorted, to keep the order
of the response deterministic.
The ValueOnly serialization is only defined for JSON, so the routes
respond with `406 Not Acceptable` if XML is requested. `extent` remains
unimplemented, as on all other routes. `PATCH …/$value`, the `$path`
routes and the operation invocation routes are not part of this change.
Fixes eclipse-basyx#446
Previously, the ValueOnly serialization was method based in the server package. These changes shift the code from serve package to the sdk, as ValueOnly serialiaztion is defined in Part 1 of the spec. Additionally, I restructured the code to live inside a `JSONEncoder` subclass, so it can be used the same way as `AASToJsonEncoder`. Still TODO: - Implement `level=core` serialization - Integrate this serialiazer into the server
Functionality follows eclipse-basyx#649
Showcases collapse of rules and handler methods using converter and modifiers
hpoeche
left a comment
There was a problem hiding this comment.
I really like the idea of the ContentModifierConverter for the endpoint definition! I found some small things we should discuss, so I can mirror them for the ValueOnly encoder.
Previously, `PathJsonEncoder` carried `deep` as a class variable and the `?level=core` variant was expressed by a dedicated `ShallowPathJsonEncoder` subclass that flipped it to `False`. Composing this axis with other content modes required another `Shallow*` sibling per encoder, so the class count scaled with the Cartesian product of axes. With this change, `deep` is passed to `PathJsonEncoder.__init__` as a keyword argument and stored on the instance. The encoder uses `default()` directly instead of extending the classmethod- based `_get_aas_class_serializers` dispatch, which avoids `cls`/`self` mismatches along the MRO when carrying per-request state. On the server side, `pick_encoder` now returns `(cls, kwargs)` and `JsonResponse.serialize` forwards the kwargs through `json.dumps` into the encoder constructor; `ResultShallowPathJsonEncoder` is removed since the single `ResultPathJsonEncoder` now covers both levels via the kwarg.
These changes implement a core serialization for the `AASToJsonEncoder` that is backward compatible to custom subclasses of this class. For all five types, that are affected by core serialization, addtional methods were introduced that call the standard serialization method and afterwards serialize all children, omitting the grandchildren. This solution allows us to pass the `deep` parameter via the `__init__()` method, which helps to avoid addtional subclass constructs. The `ValueOnlyJsonEncoder` overwrites these methods to also implement core serialization. Test are written for both classes in core serialization mode.
Fixes __init__ ordering by using local imports
Following API Specification Part 2
…iat/basyx-python-sdk into feat/serialization-formats
Previously, the order of elements in JSON arrays was non-deterministic for the same AAS. This overcomplicated easy change detection of serialized objects. With these changes, this behavior is changed. All JSON arrays are now ordered, either because the metamodel requires them to be sorted (e.g. SubmodelElementList) or sorting is done during serialization (e.g. elements of SubmodelElementCollections are now ordered by their `idShort`). The order of keys in the JSON objects is already defined by their insertion order in the Python `dict`. From now on, objects with equal content will result in identical JSON representations. Fixes eclipse-basyx#573
Address review comments on the ported test_metadata.py: - Drop XML support, expect 406 for XML requests - Check responses are actually serialized as Metadata - Use FormatClient for all requests - Derive element counts from the example submodel - Add extent and 404 tests, check all level values
Address review comments on the ported test_value.py General overhaul following metadata testing and encoder redesign - Drop tests and helpers duplicating SDK Encoder tests - Replace 501 extent test expecting functional blob toggle - Add a shared value endpoint test base to enforce proper modifier combination handling - Flag list endpoint shape with TODO for now
Replace the separate handlers for $metadata, $value, $reference and $path with single handlers driven by the parsed serialization modifiers, which now select the response encoder. Extract the result serialization into a mixin, so each format can combine it directly with its SDK encoder. Listings of submodel-elements are wrapped so that each format controls the shape of its result. Filtering and pagination of $path listings is still to be resolved.
Previously, in path serialization the pagination was still applied to the SubmodelElements. Because each SubmodelElement is mapped to multiple idShortPath strings, the paginated results exceeded the limit. Now the pagination is done on the already encoded idShortPaths. Therefore the `ResultPathJsonEncoder` is extended with a `iter_paths()` method, that returns an iterator for the paths, that is then paginated using the existing `_get_slice()`.
Elements added to a SubmodelElementList without an idShort get an internal `generated_submodel_list_hack_...` idShort. Previously, this could not leak because elements in SubmodelElementList were not allowed to have idShorts. When this contstrain (AASd-120) was dropped, this lead to the following problems: - Normal + Metadata Serialization (JSON + XML): Previously the idShort of an element inside a SML was never serialized. This wrongly hid existing idShorts. To fix this, we introduce a central `serializable_id_short()` function that only hides generated idShorts of elements in SML as `None` and lets all other values pass. - Path Serialization (JSON): When elements of a SML are serialized directly, their idShort leads the idShortPaths that are generated. At this point the generated isShorts could leak, so we use the `serializable_id_short()` helper and raise a `ValueError` on generated or missing idShorts. - Server `POST .../submodel-elements/<idShortPath>`: When a new SubmodelElement is created the response header `Location` contians the address that should return this element. This was manually contructed from the new idShort, leaking generated values, which lead to a wrong address. We now use the `Referable.get_id_short_path_as_list()` method, which handles children of SML correctly. Test cases are added, that ensure no generated idShorts are leaking and all manually set idShorts are serialized.
- `limit=0` returned an empty page whose cursor points to the same start. `_get_slice` now rejects `limit < 1` with 400 for all routes. - By-path `$path` on an element without idShort (child of a SubmodelElementList) raised a ValueError (500). It now responds with 400. List children with an idShort are still supported. - Use a format client in all `$path` tests instead of the bare client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Regarding #621, which implemented the fixed JSON order as opt-in and here, which implemented it as always-on; I'd like to document here which way we go and why. Always-on gives everyone reproducible output, but every serialization now becomes more expensive due to the sorting, and every existing user gets a changed (although equivalent) JSON output. Do we accept that, or do we want an opt-out? |
This change contains some minor fixes from the review: - The `adapter.json.formats` subpackage was missing its `__init__.py`, unlike every other package in `basyx`. - The `406 Not Acceptable` message of `XmlResponse` printed `/$Content.METADATA` instead of `/$metadata`, since an f-string of a `(str, Enum)` member uses `str()` from Python 3.12 on. - The embedded data specifications were sorted with a `lambda` that only wrapped the sort key function. - A line in `ValueOnlyJsonEncoder._file_to_value_only()` was indented too far.
| if isinstance(value, (float, model.datatypes.Decimal)): | ||
| if not math.isfinite(float(value)): | ||
| return model.datatypes.xsd_repr(value) | ||
| return float(value) |
There was a problem hiding this comment.
xs:decimal loses precision here, at serialization time. The stdlib json cannot serialize Decimal, so it is converted to float first, which rounds it to the nearest IEEE double:
Decimal("0.10000000000000000000001")→0.1Decimal(2**53 + 1)→9007199254740992.0(also off by one, and gets a.0)
The spec allows this in principle: IDTA-01001 maps xs:decimal to a JSON number and says that values that cannot be represented in JSON "cannot be used" (IDTA-01001 v3.1.2, Note 1). So this is not a spec violation, but the integer types are handled differently: an int is passed through unchanged, and json writes it exactly at any size (2**53 + 1 → 9007199254740993). For integers, precision is only lost if the client parses JSON numbers as doubles. For decimals, we lose it ourselves before the client sees anything.
The spec covers both approaches, so I don't mind which one we choose, as long as all numeric types are treated the same way:
- Keep the exact value for all types: write the
Decimaldigits as a JSON number instead of going throughfloat, and leave any rounding to the client, as we already do for integers. - Round all types: also convert the unbounded integer types to
float, so that every number in our output can be represented as an IEEE double.
Whichever we choose, let's document it in the class docstring next to the NaN/INF note.
There was a problem hiding this comment.
You are right here, we should handle Decimal different from float and serialize it with full precision, like this is done already in the normal encoder. I will adapt the code accordingly.
There was a problem hiding this comment.
After taking a closer look at this, this is not as easy as it seems. The base json.JSONEncoder only supports str, int, float, bool, None, listand dict. As far as I (and Claude) can see, we have the following options:
- Map a
decimal.Decimalto afloatand accept precision loss. - Overwrite the internal method
iterencode()of thejson.JSONEncoder, thereby relying on undocumented behavior (I dislike this very much). - Switch to the
simplejsonimplementation, which supportsdecimal.Decimalout of the box. This would require some rework on the other encoders and is a breaking change to downstream subclasses any users implemented.
We should discuss this in a meeting.
We discussed this in the meeting and chose to always sort by default, because ordered instead of no order at all is not a real change. The computational overhead due to the sorting should be small, because sorting is optimized in C. Or do we need a performance analysis here? |
No need for performance analysis needed. #648 should more than make up for any performance losses we incur here. |
Motivation
Previously, our SDK missed the JSON serialization formats "Metadata", "ValueOnly" and "Path",
that are defined in the IDTA-01001 specification. The
StrippedAASToJsonEncoderimplementeda different serialization format, but followed none of the described ones (see #651).
Therefore, our server implementation responded with
501 Not Implementedto GET requestson
/$valueand/$pathroutes, ignored?extentand wrongly interpreted the/$metadataand?level=coremodifiers (see #650, #651).Additionally, the order of objects in JSON arrays, which are not required to be ordered by the specification,
was not explicitly defined during serialization, which lead to different JSON strings for the same objects
(see #573).
Finally, during implementation we discovered a bug in the current serialization of
SubmodelElementListand their children. Because internally each child gets assigned a generated
id_shortthis value was neverserialized, even when manually set.
Changes
We need to support all serialization formats, defined in IDTA-01001, as well as all serialization modifiers
from IDTA-01002 which are defined on top of these formats. We therefore chose to keep all
serialization logic in the SDK and also implement level and extent as parts of the encoder classes.
SDK changes -
basyx.aas.adapter.jsonWith this change, the SDK offers a JSON encoder for every serialization format of IDTA-01001,
each one a subclass of
AASToJsonEncoderin the new subpackageadapter.json.formats:MetadataJsonEncoder,ValueOnlyJsonEncoder,ReferenceJsonEncoderandPathJsonEncoder.To support the level and extent modifier introduced in IDTA-01002,
AASToJsonEncoderitselfgets the init argument
deepforlevel=coreand the class attributewith_blob_valueforextent=WithoutBLOBValue.All formats and modifier are tested in
sdk/test/adapter/json/formatsusingunittest.Collections that are unordered according to the specification are now serialized in a
canonical order, defined in
adapter._ordering. ForReferableobjects,modelType`is now always the first key in JSON serialization.
Elements of
SubmodelElementListnow output theiridShort, if it is a non generated one. Thisholds for JSON and XML serialization and uses a central
serializable_id_short()function that hidesgenerated
idShortvalues of elements in SML asNoneand lets all other values pass.Server changes
The server parses the modifiers centrally in
app.adapter.modifiersand selects the matchingJSON encoder, so all content modifiers as well as
levelandextentare supported on theSubmodel Repository GET routes. Combinations that IDTA-01002 does not allow are rejected with
400 Bad Request.Requests that expect an XML response fail fast with
406 Not Acceptablebefore object fetching, whena serialization different from format
Normalwithlevel=deepandextent=WithBlobValueis present.This changes existing behavior, which however was never defined in the specification.
All endpoints are tested for correct behavior under all modifier values.
Decisions we made
Where the specification is ambiguous or contradicts itself, the following decisions were made:
are omitted from listings, and requesting them directly results in
400 Bad Request.SubmodelElementList,where they are kept as
nullto preserve the indices. Further assumptions are documentedin the docstring of
ValueOnlyJsonEncoder.$pathstart with the idShort of the requested element. Therefore, anelement of a
SubmodelElementListwithout anidShorthas no Path representation(
400 Bad Request).GET .../submodel-elements/$valuereturns a single object with theidShortsas names, so thateach value stays associated with its element.
/submodels/$valueas oneSubmodelValueobject, whichcan only hold values for one submodel, because the
idShorts can duplicate across submodels.Because this is a known error of the spec submodels-value-issue, that is corrected in a slightly
later version, we chose to implement this correct version directly.
Changed behavior
The implementation changes result in changes in the existing behavior as follows:
Blobvalues unless?extent=WithBLOBValueis given, as the specification definesWithoutBLOBValueas the default.$referenceand the Normal serialization withlevel=deepandextent=WithBLOBValue. Any other combination, including a request without?extent=WithBLOBValue,is rejected with
406 Not Acceptable. This breaks existing clients.?limit=0is rejected with400 Bad Request.AI Usage
This contribution was written with the help of Claude Opus 5.5.
Fixes #446
Fixes #573
Fixes #643
Fixes #644
From #650 and #651, GET requests are covered while PATCH requests are still wrong
Closes #649
Closes #623
Closes #621