Skip to content

PyDABs: keep empty nested objects, as YAML does - #7790

Open
Daniel114510 wants to merge 2 commits into
mainfrom
fs3100-pydabs-keep-empty-objects
Open

Daniel114510 wants to merge 2 commits into
mainfrom
fs3100-pydabs-keep-empty-objects

Conversation

@Daniel114510

@Daniel114510 Daniel114510 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Changes

Python resources now keep a nested object that is set but empty, such as StreamingMode() or ExternalTrigger(), and emit it as {}, as YAML does. Unset fields (None) and empty lists and maps are still omitted.

Why

The Python serializer drops every empty value, so it can't tell an object that was set but has no fields from one that was never set. YAML keeps it. Where the empty object is itself the choice, a resource defined in Python comes out wrong:

  • A class object specifying no arguments, e.g. streaming_mode=StreamingMode() for a materialized feature member in a materialization set, will drop the streaming_mode in the request sent to the service, which will be rejected. The same declaration in YAML deploys.
  • A class with no fields, such as ExternalTrigger or a data quality monitor's SnapshotConfig, can't be expressed from Python at all.

Only Python code that writes an empty object changes, and it now gets what the same {} does in YAML. For example, continuous=Continuous() now makes a job continuous.

Tests

  • Acceptance tests:
    • acceptance/bundle/python/materialization_sets-support: empty streaming, table and external triggers, from YAML and Python
    • acceptance/bundle/python/data_quality_monitors-support: golden now keeps snapshot: {}
  • Unit tests:
    • test_transform_to_json.py: empty, non-empty and unset nested dataclasses, and a Variable field
    • test_transform.py: empty object dict round trip
  • Real workspace: with this wheel on CLI v1.26.0, bundle plan of a Python-defined set plans its create with streaming_mode: {} and table_trigger: {}, as for the same set in YAML

Python resources now emit a nested object that is set but empty, such as
StreamingMode() or ExternalTrigger(), as {} instead of dropping it like an
unset field. Unset fields and empty lists and maps are still omitted.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added DABs DABs related issues PyDABs labels Nov 19, 2026
@Daniel114510
Daniel114510 marked this pull request as ready for review November 19, 2026 05:57
@Daniel114510
Daniel114510 requested review from a team as code owners November 19, 2026 05:57
@Daniel114510
Daniel114510 requested a review from denik November 19, 2026 05:57
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 575456c24 | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

4 tests: regression=0 unreleased=0 coverage=4 failing=0
test branch main (06e25a1) latest (v1.27.0)
TestAccept/bundle/python/data_quality_monitors-support/DMS=/PYDAB_VERSION=current ✅ ✅ ✅
TestAccept/bundle/python/data_quality_monitors-support/DMS=true/PYDAB_VERSION=current ✅ ✅ ✅
TestAccept/bundle/python/materialization_sets-support/DMS=/PYDAB_VERSION=current ✅ ✅ ✅
TestAccept/bundle/python/materialization_sets-support/DMS=true/PYDAB_VERSION=current ✅ ✅ ✅

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a user-visible bug fix (Python-defined bundles with empty nested objects like StreamingMode() now serialize correctly), but there is no .nextchanges/ fragment in the diff. Can you add one under .nextchanges/bundles/?

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 575456c

Run: 44621910167

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 21 5:27
💚​ aws-windows-latest-4core-16gb 1 274 19 4:52
💚​ azure linux-2core-8gb 1 271 21 4:52
💚​ azure-windows-latest-4core-16gb 1 273 19 4:18
💚​ gcp linux-2core-8gb 1 272 21 6:02
💚​ gcp-windows-latest-4core-16gb 1 274 19 5:29
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:14 aws-windows-latest-4core-16gb TestAccept
4:04 gcp-windows-latest-4core-16gb TestAccept
4:03 azure-windows-latest-4core-16gb TestAccept

A Variable is also a dataclass, but it serializes to a non-empty string, so it
no longer reaches the dataclass check. Also add the changelog fragment.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues PyDABs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants