[binaryaaron]

Fixes the multi-batch failure Andre raised on #375 (storage.py:84). With more than one row group, a registered side-effect column (_validated_seed_entities, _validation_candidates, _entities_for_replace) can come back from DataDesigner as object arrays, and the next seed write raises InvalidFileFormatError. On main + #375, Substitute and Rewrite both fail this way with buffer_size=2, two entity-free rows, then two with entities.

What changes

• prepare_json_storage_seed rebuilds every planned column from plain Python values, not only Arrow-backed ones. Numpy arrays and scalars become lists and Python scalars, and a top-level NaN, pd.NA or None becomes None.
• The converter is the existing _to_python from rewrite/parsers.py, moved to engine/schemas/shared.py as to_python so engine.ndd does not import engine.rewrite. normalize_payload behaves the same.
• pyarrow is no longer imported in storage.py.

Checks

• Two new tests in tests/engine/test_ndd_storage.py:

  • a two-row-group handoff of a side-effect column through the real adapter;
  • a NaN and numpy seed written to parquet.

  All three cases fail on caab2130 with InvalidFileFormatError.

• The full suite, ruff and make typecheck pass.

• On main + fix(ndd): preserve nested JSON storage across workflow handoffs #375, the end-to-end Substitute and Rewrite run above passes with this commit and fails without it.

When #375 is rebased onto main, this commit conflicts with #367 in schemas/shared.py. Keep #367's tail and add to_python after it; the result passes the full suite.

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 5/5

[Medium impact] This PR appears safe to merge; no actionable issues were found.

Summary

This PR converts every planned seed column to plain Python values before DataDesigner writes it.

• Workflow seeds turn registered structured columns into plain Python values.

No actionable issues found. Tests were inspected but not run; DataDesigner was unavailable in the review environment.

_{Reviews (1) · Last reviewed commit: "fix(ndd): encode registered seed columns..." · Reviewed by Greptile}