Skip to content

Apply resource lifecycle metadata in config-remote-sync change detection - #7645

Merged
ilyakuz-db merged 4 commits into
mainfrom
ilyakuz-db/configsync-lifecycle-metadata
Nov 16, 2026
Merged

ilyakuz-db merged 4 commits into
mainfrom
ilyakuz-db/configsync-lifecycle-metadata

Conversation

@ilyakuz-db

Copy link
Copy Markdown
Contributor

Changes

  • config-remote-sync now applies the direct engine's lifecycle metadata to every change, whatever action the plan gave it (bundle/configsync/lifecycle.go). A remote value is never written back when it falls under ignore_remote_changes (output-only, input-only, backend-managed), or when it's a backend default the config doesn't set.
  • When the plan reports a whole object (one side is nil), those fields are left out of it, and the change is skipped if nothing else differs. Inside a remote-only object, only backend defaults pinned to allowed values are dropped. A field like node_type_id in a task added in the UI may be the user's pick, so it's kept.
  • Job run_as keys are still synced, so a principal kind switch keeps working ([DECO-28500] Fix config-remote-sync writing run_as: {} on a principal kind switch #7493).
  • Moved the backend-default matcher from bundle/direct to dresources.ResourceLifecycleConfig.IsBackendDefault so the plan and config sync share it. The plan's behavior doesn't change.

Why

Config sync only dropped changes the plan marked as skipped. Metadata-covered fields still reached the YAML whenever the plan reported an update, or reported the object that contains them:

  • A genie space's etag is output_only, but OverrideChangeDesc promotes etag drift to an update for deploy, so the etag ended up in the YAML.
  • Input-only fields are never returned by GET, so sync removed them when the config had an undeployed edit (a job's trigger.table_update.condition), or when the whole object was missing remotely (a serving endpoint's telemetry_config.table_names).
  • Output-only and backend-managed fields inside an object added in the UI were written as-is (for example aws_attributes under a for-each task's cluster).
  • Backend defaults were added back: a job's format once the config dropped it, or spark.sql.ansi.enabled inside a pipeline configuration map added remotely.

Tests

  • New unit test in diff_test.go covering each case above, plus the run_as switch and a backend default with a non-default value. It fails without the fix.
  • go test ./bundle/..., the config-remote-sync acceptance tests and golangci-lint pass locally.

This pull request and its description were written by Isaac.

ilyakuz-db and others added 4 commits November 15, 2026 23:47
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@ilyakuz-db
ilyakuz-db requested review from a team as code owners November 16, 2026 00:08
@ilyakuz-db ilyakuz-db added DABs DABs related issues software-factory Deprecated. Use factory-generated instead. ai-auto Deprecated: use factory-generated instead. labels Nov 16, 2026
@ilyakuz-db
ilyakuz-db requested a review from denik November 16, 2026 00:08
@ilyakuz-db ilyakuz-db self-assigned this Nov 16, 2026
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 2d69c06

Run: 43778640015

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 15 3:35
💚​ aws-windows-latest-4core-16gb 1 272 13 3:04
💚​ azure linux-2core-8gb 1 269 15 3:25
💚​ azure-windows-latest-4core-16gb 1 271 13 3:04
💚​ gcp linux-2core-8gb 1 270 15 3:47
💚​ gcp-windows-latest-4core-16gb 1 272 13 3:35
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:31 gcp-windows-latest-4core-16gb TestAccept
3:00 aws-windows-latest-4core-16gb TestAccept
3:00 azure-windows-latest-4core-16gb TestAccept

@andrewnester
andrewnester self-requested a review November 16, 2026 08:42

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings from a full pass; I'll take a manual look at the direct changes before approving.

@ilyakuz-db
ilyakuz-db added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 9603bbb Nov 16, 2026
28 checks passed
@ilyakuz-db
ilyakuz-db deleted the ilyakuz-db/configsync-lifecycle-metadata branch November 16, 2026 14:28
petrapostelnicu-db added a commit that referenced this pull request Nov 17, 2026
Resolve conflicts with #7645 (lifecycle metadata) and #7536 (permissions):
- convertPlanChange applies skipJobRunAs before applyLifecycle.
- applyLifecycle no longer drops a job's whole run_as as a backend default;
  skipJobRunAs already limits it to jobs passed in --sync-run-as-ids.

Also validate --sync-run-as-ids inside PostStateFunc so an invalid value
is reported through WriteResult (status: failed with -o json) and telemetry,
like --select-ids.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added factory-generated and removed software-factory Deprecated. Use factory-generated instead. ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants