Annotated Parquet reads can corrupt nested Arrow arrays with null children #1143

Priority Level

Medium (has a workaround)

Describe the bug

Loading valid JSON-annotated Parquet artifacts through DD's default Arrow-backed reader can produce malformed nested arrays. An unencoded nested side-effect column containing null children is valid in the original Arrow table, but pandas' ArrowDtype conversion performs an identity cast that can corrupt its child-array lengths with PyArrow 24. Subsequent validation or materialization then fails.

Steps/Code to reproduce bug

Reproduced with DD 0.10.0, pandas 2.3.3 and PyArrow 24.0.0:

from pathlib import Path
from tempfile import TemporaryDirectory

import pandas as pd
import pyarrow.parquet as pq

from data_designer.config.utils.storage_encoding import encode_storage_table, decode_storage_table
from data_designer.config.utils.storage_schema import JsonStoragePlan

values = [
    [{"entries": [{"label": "synthetic", "optional": None} for _ in range(count)]}]
    for count in (1, 3, 2)
]
frame = pd.DataFrame({"side_effect": values, "payload": [{"x": 1}] * 3})
table = encode_storage_table(frame, JsonStoragePlan(columns={"payload": [[]]}))
with TemporaryDirectory() as directory:
    path = Path(directory) / "batch.parquet"
    pq.write_table(table, path)
    original = pq.read_table(path)
    original.validate(full=True)  # passes
    restored = decode_storage_table(original)
    restored["side_effect"].array.__arrow_array__().validate(full=True)  # fails

Representative error:

ArrowInvalid: ... Struct child array #1 has length smaller than expected for struct array (3 < 6)

Expected behavior

Reading a valid artifact preserves nested values and valid Arrow buffers, while decoding only explicitly JSON-encoded paths. Native columns remain Arrow-backed and JSON storage metadata is preserved.

Agent Diagnostic / Prior Investigation

• The original Parquet table validates and materializes successfully. The corruption appears during pandas' ArrowDtype conversion; an identity array.cast(array.type, safe=True) also reproduces it independently of DD.
• Variable-length (1, 3, 2) and uniform-length (3, 3, 3) nested-list fixtures reproduce the invalid child lengths; the all-empty fixture is a control.
• Proposed fix avoids sending nested columns through pandas conversion and wraps their original Arrow arrays directly, without first materializing discarded Python objects.
• The proposed regression exercises the public Parquet reader. Additional tests cover all-nested/empty tables, row indexes, column order, and sliced/chunked arrays.
• Related dtype-backend PR feat: add dtype_backend to result loading #1128 changes backend selection, not this default-path corruption.

Additional context

Fix PR: #1142

Discovered during downstream Anonymizer #305 integration. The fixture above is entirely synthetic and contains no benchmark records or provider data.

Checklist

• I reproduced this issue or provided a minimal example.
• I searched the docs/issues myself, or had my agent do so.
• I included the agent's diagnostics above.