feat(codegen): preserve callable contracts

.github/workflows/ci.yml

@@ -101,7 +101,7 @@ jobs:
       - name: Run Python codec and frame tests
         run: |
           python -m pip install pytest
-          python -m pytest test/python/test_bridge_codec.py test/python/test_frame_codec.py
+          python -m pytest test/python/test_bridge_codec.py test/python/test_bridge_core.py test/python/test_frame_codec.py
 
   python-suite-data:
     if: github.event_name != 'schedule'
@@ -223,7 +223,7 @@ jobs:
           PIP_PROGRESS_BAR: 'off'
         run: |
           python -m pip install -e tywrap_ir
-          python -m pip install numpy==2.3.5 pandas==3.0.2 pyarrow==24.0.0 scipy==1.16.3 scikit-learn==1.8.0
+          python -m pip install numpy==2.3.5 pandas==3.0.2 pyarrow==24.0.0 scipy==1.16.3 scikit-learn==1.8.0 pydantic==2.7.4
           python -m pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cpu
       - name: Run pinned scientific menagerie
         env:

docs/guide/runtimes/http.md

@@ -49,6 +49,11 @@ setRuntimeBridge(
 payloads. You must provide a compatible server endpoint. Each call is an
 independent POST request because the protocol is stateless.
 
+Generated async callables have typed results when their return annotations have
+supported value conversions. The in-repo Node and Pyodide bridges await Python
+coroutines. For `HttpBridge`, your server must await coroutines before it sends
+the result. `HttpBridge` cannot verify a remote server's coroutine support.
+
 tywrap does not include an HTTP server. See the [API reference](/reference/api/)
 for the expected request and response format.
 

docs/maintainers/callable-contracts.md

@@ -0,0 +1,114 @@
+# Callable compilation contracts
+
+`tywrap_ir` emits IR 0.4. `validateIrContract()` checks required fields in
+unknown JSON. Each diagnostic has a JSON path. Validation stops after 100,000
+array entries.
+
+`compileContract()` is an internal pure step. It accepts validated IR,
+exported-name selection, value conversions, capabilities, and a generator. It
+maps callable types from the validated IR. The selection input cannot change
+those types. It returns the resolved module, callable records, diagnostics, and
+TypeScript and declaration content.
+
+The compiler does not read files. It does not start Python. It does not read a
+cache. It does not write files. It does not access a runtime registry.
+
+`generate()` owns Python discovery, cache reads, output comparison, and writes.
+It validates IR and maps annotations, then calls `compileContract()` before
+emission. Offline contract files omit extractor metadata for stable bytes. The
+reader restores an empty metadata object.
+
+## Resolved callable data
+
+The compiler records each parameter and result, including overloads. Each record
+has a Python type, direction, source path, and conversion result. One resolved
+module model serves declarations, call wrappers, and return validators.
+
+Each value conversion uses revision 2 of
+[value-contracts.v2.json](value-contracts.v2.json). Do not infer a wire rule
+from the TypeScript type.
+
+| Case                  | Result                                                                                |
+| --------------------- | ------------------------------------------------------------------------------------- |
+| `int`                 | JSON number, JavaScript number, safe-integer constraint                               |
+| `bytes`               | Existing base64 envelope, decoded as `Uint8Array`                                     |
+| nested `dict[str, T]` | JSON record with the child conversion at every value path                             |
+| `str \| int`          | Selected union alternative with one child contract per option                         |
+| `tuple[int, str]`     | Exact JSON array with a contract for each index                                       |
+| float16 ndarray       | Existing Arrow or JSON-fallback envelope, decoded as a numeric scalar or nested array |
+| float16 Torch tensor  | Outer `torch.float16` envelope with a nested `float16` ndarray                        |
+
+The compiler requires an explicit float16 dtype. An untyped ndarray or Torch
+tensor stays unresolved.
+
+For an evaluated PyTorch annotation, use `torch.HalfTensor`. The compiler
+accepts this exact class and leaves other Torch tensor classes unresolved.
+
+NumPy can spell `NDArray[np.float16]` as
+`numpy.ndarray[tuple[Any, ...], numpy.dtype[numpy.float16]]`. The compiler reads
+the dtype argument, not the shape wildcard. Only a supported scientific result
+clears its matching external-annotation IR warning. Other external annotations
+remain warnings.
+
+The codec records scalar ndarray dtype and rank for one call. A generated return
+validator accepts a scalar only with matching proof. Nested proof follows its
+parent and array index or record key. Array index `0` and `"0"` refer to the
+same child. Equal numbers from separate calls do not share proof. Array and
+tensor objects keep their identity-based marker proof.
+
+The compiler keeps Arrow int64 behavior outside this prototype. Existing Arrow
+int64 values can decode as `bigint`.
+
+Selected local non-generic `TypedDict` classes use revision-2 record contracts
+when every field has a supported conversion. Nested records and selected simple
+aliases compose through the same resolver. Foreign and recursive names remain
+unresolved.
+
+## Diagnostics and fallback
+
+Known unsupported cases create error diagnostics. The compiler changes the
+affected generated type to `unknown`. `generate()` returns each diagnostic as a
+warning, so `--fail-on-warn` rejects strict builds.
+
+- The default compiler capability covers coroutine execution in the in-repo Node
+  and Pyodide bridges. A caller can mark this capability unavailable when it
+  uses an older or custom bridge. The compiler then emits `unknown` and an error
+  diagnostic for async results.
+- Dataclasses require an adapter owned by #339. The default capability stays
+  unavailable. An explicit output record conversion can use the adapter
+  capability when enabled.
+- Non-string record keys require an explicit conversion.
+
+Unresolved annotations create warning diagnostics. Their result type is
+`unknown`, including an output that contains `object` or a type variable. Python
+`object` can hold a primitive value. A type variable has no validated conversion
+to the decoded result. Neither annotation justifies a precise TypeScript return.
+
+The compiler keeps an existing best-effort return check when it widens a
+declaration. For example, an unresolved `list[object]` still rejects a non-array
+result, and a `DataFrame` annotation still requires a decoded marker.
+
+`Literal` returns keep their exact-value validator, but stay `unknown` until a
+literal value contract exists. Unresolved literal overloads cannot select a
+precise return at runtime.
+
+IR overload signatures stay separate from the implementation signature. The
+extractor collects type variables from overload annotations. Generated
+declarations retain each input-to-result relation. The hidden TypeScript
+implementation accepts `unknown` values so it can cover every declared overload.
+
+At runtime, the wrapper selects the first supported overload that matches bound
+arguments. This follows TypeScript declaration order. The compiler warns when
+overload inputs may overlap, including an omitted optional argument. Python 3.10
+cannot expose registered overloads through `typing.get_overloads`; it keeps the
+implementation signature and emits an IR warning.
+
+## Deferred work
+
+Revision 2 does not describe references or recursive aliases. The compiler
+reports these cases as unresolved. A later value-contract revision can give them
+a supported conversion.
+
+The client-binding work may consume the resolved callable model later. It must
+remain opt-in. It must not add files to default generation or mutate the global
+runtime registry.

docs/maintainers/value-contract-fixtures.v2.json

@@ -0,0 +1,63 @@
+{
+  "revision": 2,
+  "integerCases": [
+    { "decimal": "-9007199254740991", "outcome": "number" },
+    { "decimal": "9007199254740991", "outcome": "number" },
+    { "decimal": "-9007199254740992", "outcome": "reject" },
+    { "decimal": "9007199254740992", "outcome": "reject" },
+    { "decimal": "9007199254740993", "outcome": "reject" },
+    { "decimal": "-9223372036854775808", "outcome": "reject" },
+    { "decimal": "9223372036854775807", "outcome": "reject" },
+    { "decimal": "265252859812191058636308480000000", "outcome": "reject" }
+  ],
+  "bytesCases": [
+    {
+      "logicalHex": "00ff80",
+      "requestEnvelope": { "__tywrap_bytes__": true, "b64": "AP+A" },
+      "responseEnvelope": { "__type__": "bytes", "encoding": "base64", "data": "AP+A" },
+      "decodedAs": "Uint8Array[0,255,128]"
+    },
+    {
+      "logicalHex": "",
+      "requestEnvelope": { "__tywrap_bytes__": true, "b64": "" },
+      "responseEnvelope": { "__type__": "bytes", "encoding": "base64", "data": "" },
+      "decodedAs": "Uint8Array[]"
+    }
+  ],
+  "binary16Cases": [
+    { "wordHex": "0000", "outcome": "0" },
+    { "wordHex": "8000", "outcome": "-0" },
+    { "wordHex": "0001", "outcome": "5.960464477539063e-8" },
+    { "wordHex": "03ff", "outcome": "0.00006097555160522461" },
+    { "wordHex": "0400", "outcome": "0.00006103515625" },
+    { "wordHex": "3e00", "outcome": "1.5" },
+    { "wordHex": "c080", "outcome": "-2.25" },
+    { "wordHex": "7bff", "outcome": "65504" },
+    { "wordHex": "fbff", "outcome": "-65504" },
+    { "wordHex": "7c00", "outcome": "reject" },
+    { "wordHex": "fc00", "outcome": "reject" },
+    { "wordHex": "7e00", "outcome": "reject" }
+  ],
+  "compositionCases": [
+    {
+      "path": "result.outer[0].value",
+      "logicalValue": "9007199254740993",
+      "outcome": "reject"
+    },
+    {
+      "path": "result.outer[0].tensor.value",
+      "logicalValue": "float16[1.5,-2.25]",
+      "outcome": "number[1.5,-2.25]"
+    }
+  ],
+  "unionCases": [
+    { "contract": "str|int", "value": "label", "outcome": "string" },
+    { "contract": "str|int", "value": 42, "outcome": "safe-integer number" },
+    { "contract": "str|int", "value": 9007199254740992, "outcome": "reject" }
+  ],
+  "tupleCases": [
+    { "contract": "tuple[str,int]", "value": ["label", 42], "outcome": "accept" },
+    { "contract": "tuple[str,int]", "value": ["label"], "outcome": "reject-length" },
+    { "contract": "tuple[str,int]", "value": ["label", 9007199254740992], "outcome": "reject-integer" }
+  ]
+}

docs/maintainers/value-contracts.md

@@ -0,0 +1,113 @@
+# Value contracts
+
+The frozen internal prototype is
+[value-contracts.v2.json](value-contracts.v2.json). That JSON file owns the
+policy. The TypeScript interface mirrors it and has revision 2.
+
+Each contract names a logical value, its wire encoding, its decoded JavaScript
+value, and its constraints. Sequences, fixed tuples, records, and unions compose
+child contracts. A union has two to 32 alternatives. The compiler rejects a
+union if one wire value could decode two ways. Contract depth stops at 64. Named
+records can inline within that limit. Recursive aliases stay unsupported until a
+reviewed reference graph exists. The prototype covers float16 ndarrays and Torch
+tensors. Other Arrow types keep their existing rules. In particular, Arrow int64
+may decode as `bigint`. Unsupported conversions carry a reason and guidance.
+
+The wire stays at `tywrap/1` and scientific `codecVersion: 1`. Existing Arrow
+envelopes, JSON fallback, and transports keep their shape. The producer rejects
+unsafe Python integers before it writes JSON. A generated wrapper that consumes
+this interface must keep `number` for Python `int` and add a safe-integer return
+constraint. Regenerate wrappers after that compiler change. Do not return
+`bigint` under an existing `Promise<number>`.
+
+The bytes leaf keeps the existing base64 envelopes. Requests use
+`__tywrap_bytes__`; responses use `__type__: bytes`. Both decode to
+`Uint8Array`. The Torch float16 contract names `torch.float16` on the outer
+tensor and `float16` on its nested version 1 ndarray. The decoder rejects a
+different nested dtype before it decodes the data.
+
+Migration for users with large Python integers: convert them to strings in
+Python with an explicit return annotation, or keep integer columns in Arrow
+tables. JSON fallback for ndarray and pandas values still rejects unsafe
+integers. This is a behavior change from silent rounding to a call error.
+
+The [fixed proof fixtures](value-contract-fixtures.v2.json) cover integer
+boundaries, binary16 values, unions, fixed tuples, and nested paths. The focused
+tests also compare every finite binary16 storage word with an independent
+float32 bit reference. They test signed zero with `Object.is`.
+
+The Python serializer checks plain integers, NumPy scalar integers, and model
+dumps during its existing value walk. It also checks terminal estimator
+parameters and Series name metadata before JSON output. A JSON ndarray or sparse
+matrix checks integer data before it creates an envelope. Sparse metadata checks
+shapes and indices. The TypeScript decoder converts float16 Arrow storage during
+extraction, before it reshapes the ndarray. Torch uses that same ndarray path.
+These checks do not walk Arrow payloads again for schema validation.
+
+## Exact bigint extension for #337
+
+This extension is a design sketch. The 0.11 policy still rejects unsafe plain
+Python integers.
+
+Choose an explicit callable option, `integerMode: 'bigint'`. When enabled, map
+Python `int` to TypeScript `bigint` for both parameters and returns. The return
+validator must require `typeof value === 'bigint'`, including every record and
+sequence member. Keep `bool` separate. Calls without the option keep `number`
+and the safe range.
+
+Add a version 2 integer value envelope with a canonical signed decimal string. A
+sample value is
+`{"__tywrap__":"integer","codecVersion":2,"encoding":"decimal","value":"18446744073709551617"}`.
+The request encoder and Python request decoder must validate the same grammar
+and digit limit. The response encoder and TypeScript decoder must do the same.
+Use `BigInt` only after validation. Keep Arrow int64 columns on their current
+Arrow path.
+
+The bridge must advertise exact integer support during capability negotiation. A
+generated bigint wrapper must reject an older bridge before its first call. The
+new bridge must emit version 2 integer envelopes only when the call opts in. An
+older client therefore continues to receive the 0.11 safe integer policy.
+Unknown integer envelopes, mixed versions, oversized strings, and malformed
+decimal values must fail with a path. Do not guess a conversion.
+
+Acceptance evidence still needed: a bounded prototype must round trip positive
+and negative values beyond 64 bits through nested records and arrays. It must
+test bigint inputs, malformed tags, payload limits, and both mixed-version
+directions. An independent review must approve the wire grammar, capability
+rule, generated types, and migration cost before #337 can close.
+
+## Dataclass return extension for #339
+
+This extension is a design sketch. Dataclass inputs remain unsupported and need
+a separate input contract. Users must pass a plain record through an explicitly
+typed function if they need that behavior now.
+
+For an annotated dataclass return, compile a structural TypeScript record from
+`dataclasses.fields`. Include each instance field in the output, including
+fields with defaults or `init=False`. A default does not make a returned field
+optional. A field annotated `Optional[T]` permits `null`. Compose nested
+dataclasses, records, and sequences only when every field has a supported value
+contract. Reject ordinary class instances and unsupported fields with their
+result paths.
+
+Use a version 2 dataclass envelope with a declared type identity and a `fields`
+record. For example, `Point(1, 2)` would carry
+`{"__tywrap__":"dataclass","codecVersion":2,"encoding":"fields","type":"fixtures.values_torture.Point","fields":{"x":1,"y":2}}`.
+The TypeScript decoder returns `{x: 1, y: 2}`. The generated validator must
+check the type identity, exact field names, field values, and required fields.
+It must reject extra fields unless the compiled contract allows them. The Python
+encoder must use the current bounded value walk. It must detect cycles and
+enforce depth, node, and payload limits without calling `dataclasses.asdict` as
+a shortcut.
+
+The bridge must advertise dataclass value support. A generated dataclass wrapper
+must reject an older bridge before calling it. A new bridge must send the
+envelope only when the wrapper requests it. This leaves existing value RPC and
+ordinary class behavior unchanged.
+
+Acceptance evidence still needed: a generated `Promise<Point>` wrapper must
+return the exact plain record and reject missing, extra, or wrong typed fields.
+Tests must cover optional and defaulted fields, nested values, cycles,
+unsupported fields, depth, and payload limits. An independent review must
+approve the type identity, version rule, and migration cost before #339 can
+close.