Skip to content

Add acceptance tests for the engine user-agent tag on a terraform state - #7775

Merged
denik merged 4 commits into
mainfrom
denik/engine-ua-terraform-tests
Nov 19, 2026
Merged

denik merged 4 commits into
mainfrom
denik/engine-ua-terraform-tests

Conversation

@denik

@denik denik commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Acceptance tests under acceptance/bundle/migrate/user-agent/ that record the engine/ user-agent tag of every request when a bundle still has a terraform state:

  • race: the local terraform state disappears between the state pull and the auto-migration, as when two deploys run at once. The deploy tags engine/terraform and then panics with DeploymentState must be opened first.
  • bind-unbind: deployment bind and unbind only pull the state, so they tag engine/terraform before rejecting it.
  • engine-pinned: a terraform engine pinned through DATABRICKS_BUNDLE_ENGINE or bundle.engine.

The shared acceptance/bundle/migrate/script.prepare gains the print_engines helper and keeps LOG.* files out of the bundle upload, for all migrate tests.

Why

Telemetry classifies bundle requests by this tag, and engine/terraform still shows up from CLI >= 1.20. These tests pin down current behavior so changes to the tagging (#7690) and a fix for the panic show up as golden diffs.

Tests

Test-only change; goldens record what main does today.

This pull request and its description were written by Isaac.

@denik
denik requested review from a team as code owners November 18, 2026 14:04
@denik
denik requested review from anton-107 and rugpanov November 18, 2026 14:04
@github-actions github-actions Bot added DABs DABs related issues test-only-fix PRs that update failing of flaky tests without changing CLI code. labels Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 7f4741346 | Workflow run

Regression test report

Selected 3 changed tests (limit=20, 0 not selected)

3 tests: regression=0 unreleased=0 coverage=3 failing=0
test branch main (83ed6fa) latest (v1.27.0)
TestAccept/bundle/migrate/user-agent/bind-unbind/DMS= ✅ ✅ ✅
TestAccept/bundle/migrate/user-agent/engine-pinned/DMS= ✅ ✅ ✅
TestAccept/bundle/migrate/user-agent/race/DMS= ✅ ✅ ✅

@@ -0,0 +1,110 @@
{

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

terraform.tfstate (110 lines) and terraform-requests.json (19 lines) are byte-for-byte identical in all three subdirectories. They could live once in the user-agent/ parent and each script would reference ../terraform-requests.json ../terraform.tfstate. That keeps them in sync and drops ~260 lines; replay_tfstate.py takes plain relative paths so it just works. If a test later needs a different fixture it can fork at that point.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 2f9d1c9: the fixtures now live once in acceptance/bundle/migrate/user-agent/. One correction: tests run in a temp copy of their directory, so a bare ../ doesn't resolve. The scripts read them via $TESTDIR/../, like bundle/validate/empty_resources.

This reply was written by Isaac.

@andrewnester
andrewnester self-requested a review November 18, 2026 14:34
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 7f47413

Run: 44503512551

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 22 4:12
💚​ aws-windows-latest-4core-16gb 1 274 20 3:43
💚​ azure linux-2core-8gb 1 271 22 3:54
💚​ azure-windows-latest-4core-16gb 1 273 20 4:10
💚​ gcp linux-2core-8gb 1 272 22 4:01
💚​ gcp-windows-latest-4core-16gb 1 274 20 3:25
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:05 azure-windows-latest-4core-16gb TestAccept
3:20 gcp-windows-latest-4core-16gb TestAccept
3:19 aws-windows-latest-4core-16gb TestAccept

Record which engine tag each request carries when a bundle still has
terraform state:

- race: the local terraform state disappears between the state pull and
  the auto-migration (as with two concurrent deploys). The deploy is
  tagged engine/terraform and then panics.
- bind-unbind: pull-only commands tag engine/terraform before rejecting
  the terraform state.
- engine-pinned: a terraform engine pinned via DATABRICKS_BUNDLE_ENGINE or
  bundle.engine.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik force-pushed the denik/engine-ua-terraform-tests branch from db32498 to 2495c84 Compare November 18, 2026 17:52
BSD uniq pads the count differently from GNU uniq, so the
migrate/user-agent goldens failed on the macOS CI leg.

Co-authored-by: Isaac <no-reply@databricks.com>

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Follow-up review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

No new issues found.

denik and others added 2 commits November 19, 2026 09:28
The three tests replayed byte-identical terraform-requests.json and
terraform.tfstate copies; keep one in the parent directory and read it via
$TESTDIR/.. like other acceptance tests with shared fixtures.

Co-authored-by: Isaac <no-reply@databricks.com>
- race: require the failure with musterr and assert the panic with
  contains.py on LOG.deploy instead of a bare errcode piped into grep.
- bind-unbind: read the job id with read_id.py.
- engine-pinned: send summary output to LOG.summary, not /dev/null.
- Move print_engines to bundle/migrate/script.prepare so other migrate
  tests can use it, and keep LOG.* files out of the bundle upload.

Co-authored-by: Isaac <no-reply@databricks.com>

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Follow-up review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

No new issues found.

@denik
denik added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit f696aca Nov 19, 2026
27 checks passed
@denik
denik deleted the denik/engine-ua-terraform-tests branch November 19, 2026 10:45
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

❌ Integration test report

Commit: f696aca

Run: 44508658055

Env 💥​PANIC ❌​FAIL 🤯​MISS 💚​RECOVERED ✅​pass 🙈​skip Time
💥​ aws linux-2core-8gb 1 3 441 26 1159 1390 320:00
💥​ aws-windows-latest-4core-16gb 1 4 320 23 1198 1418 320:01
❌​ azure linux-2core-8gb 3 312 20 1275 1398 320:00
❌​ azure-windows-latest-4core-16gb 3 410 20 1093 1426 320:01
💥​ gcp linux-2core-8gb 1 371 21 1224 1396 320:00
💥​ gcp-windows-latest-4core-16gb 1 246 23 1263 1424 320:00
967 interesting tests: 943 MISS, 15 RECOVERED, 5 FAIL, 4 PANIC
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
🤯​ TestAccept 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/apps/job_permissions ✅​p 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/apps/job_permissions/DMS= ✅​p 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/apps/job_permissions/DMS=true ✅​p 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_doesnot_exist ✅​p 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_doesnot_exist/DMS= ✅​p 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_doesnot_exist/DMS=true ✅​p 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_not_deployed ✅​p 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_not_deployed/DMS= ✅​p 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/artifacts/artifact_path_with_volume/volume_not_deployed/DMS=true ✅​p 🤯​M 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cli_defaults ✅​p 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/cli_defaults/DMS= ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cli_defaults/DMS=true ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_cross_target 🤯​M 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_cross_target/DMS= 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_cross_target/DMS=true 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_permissions ✅​p 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_permissions/DMS= ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_permissions/DMS=true ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition ✅​p 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition/DMS= ✅​p 🤯​M 🤯​M
💥​ TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition/DMS=true ✅​p 🤯​M 💥​P
🤯​ TestAccept/bundle/config-remote-sync/config_edits ✅​p 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/config_edits/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/config_edits/DMS=true ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/dashboard_etag ✅​p 🙈​s 🤯​M 🙈​s 🤯​M 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/dashboard_etag/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/dashboard_etag/DMS=true ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/config-remote-sync/external_cluster_policy_cross_target ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/external_cluster_policy_cross_target/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/external_cluster_policy_cross_target/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/flushed_cache ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/flushed_cache/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/flushed_cache/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/formatting_preserved ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/formatting_preserved/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/formatting_preserved/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/job_fields ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/job_fields/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/job_fields/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/job_multiple_tasks ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/job_multiple_tasks/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/job_multiple_tasks/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/job_params_variables ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/job_params_variables/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/job_params_variables/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/job_pipeline_task ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/job_pipeline_task/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/job_pipeline_task/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/json_status_missing_state ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/json_status_missing_state/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/json_status_missing_state/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/json_status_selectors ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/json_status_selectors/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/json_status_selectors/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/multiple_files ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/multiple_files/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/multiple_files/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/multiple_resources ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/multiple_resources/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/output_json ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/output_json/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/output_json/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/output_no_changes ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/output_no_changes/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/output_no_changes/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/pipeline_fields ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/pipeline_fields/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/pipeline_fields/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/resolve_variables ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/resolve_variables/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/resolve_variables/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/select_basic ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/select_basic/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/select_basic/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/select_multiple ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/select_multiple/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/select_multiple/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/skip_deployment ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/skip_deployment/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/skip_deployment/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/skip_permissions ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/skip_permissions/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/skip_permissions/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/target_override ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/target_override/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/target_override/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/task_rename_revert ✅​p 🙈​s 🤯​M 🙈​s ✅​p 🙈​s
🤯​ TestAccept/bundle/config-remote-sync/task_rename_revert/DMS= ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/config-remote-sync/task_rename_revert/DMS=true ✅​p 🤯​M ✅​p
🤯​ TestAccept/bundle/debug/fetch-repository-info ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/debug/fetch-repository-info/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/debug/fetch-repository-info/DMS=true ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/empty-bundle ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=/DMS=true ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=true/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=true/DMS=true ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/files/no-snapshot-sync ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/files/no-snapshot-sync/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/files/no-snapshot-sync/DMS=true ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/basic ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/basic/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/breakglass ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/breakglass/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/can-manage ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/can-manage/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/destroy-redeploy ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/destroy-redeploy/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/dev-mode ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/dev-mode/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/legacy-state ✅​p ✅​p 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/legacy-state/DMS= ✅​p ✅​p 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/missing-snapshot ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/missing-snapshot/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/no-artifacts ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/no-artifacts/DMS=/READPLAN= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/no-artifacts/DMS=/READPLAN=1 ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/permissions-change ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/permissions-change/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/select ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/immutable/select/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/mlops-stacks ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/mlops-stacks/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/mlops-stacks/DMS=true ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/spark-jar-task ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/spark-jar-task/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deploy/spark-jar-task/DMS=true ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deployment/bind/alert 💚​R 🤯​M 🤯​M 🤯​M 💚​R 🤯​M
🤯​ TestAccept/bundle/deployment/bind/alert/DMS= 💚​R 🤯​M 🤯​M 🤯​M 💚​R 🤯​M
🤯​ TestAccept/bundle/deployment/bind/alert/DMS=true 💚​R 🤯​M 🤯​M 🤯​M 💚​R 🤯​M
🤯​ TestAccept/bundle/deployment/bind/catalog ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deployment/bind/catalog/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deployment/bind/catalog/DMS=true ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deployment/bind/cluster ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deployment/bind/cluster/DMS= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deployment/bind/cluster/DMS=true ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deployment/bind/dashboard ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deployment/bind/dashboard/DMS=/READPLAN= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deployment/bind/dashboard/DMS=/READPLAN=1 ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deployment/bind/dashboard/DMS=true/READPLAN= ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
🤯​ TestAccept/bundle/deployment/bind/dashboard/DMS=true/READPLAN=1 ✅​p 🤯​M 🤯​M 🤯​M ✅​p 🤯​M
Top 50 slowest tests (at least 2 minutes):
duration env testname
15:40 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
15:02 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
14:54 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
12:30 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
12:23 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
11:57 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
11:37 azure-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=job_apply_policy_default_values_for_each_task.yml.tmpl/READPLAN=1
11:36 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
11:13 azure linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=model_serving_endpoint.yml.tmpl/READPLAN=
11:10 gcp-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=job_with_depends_on.yml.tmpl/READPLAN=1
10:23 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
10:21 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
10:04 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:47 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:39 azure linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=job_table_update_trigger.yml.tmpl/READPLAN=1
9:27 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
9:26 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=schema_empty_grants.yml.tmpl/READPLAN=
9:25 aws linux-2core-8gb TestAccept/bundle/templates/default-python/combinations/classic/DLT=yes/DMS=/NBOOK=no/PY=yes/READPLAN=
9:24 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=secret.yml.tmpl/READPLAN=
9:00 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=pipeline_config_dots.yml.tmpl/READPLAN=1
8:56 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:54 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=schema_tf_compat.yml.tmpl/READPLAN=
8:50 azure linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=job_table_update_trigger.yml.tmpl/READPLAN=
8:45 gcp linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
8:44 gcp linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
8:41 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
8:40 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
8:37 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:36 azure-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=dashboard.yml.tmpl/READPLAN=1
8:33 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:30 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
8:30 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=schema_empty_grants.yml.tmpl/READPLAN=1
8:23 azure linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=model.yml.tmpl/READPLAN=1
8:18 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
8:17 azure-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
8:11 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
8:08 aws linux-2core-8gb TestAccept/bundle/resources/volumes/rename_out_of_band/DMS=true/READPLAN=1
8:06 azure linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=volume_uppercase_name.yml.tmpl/READPLAN=
8:05 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true
8:04 gcp linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
8:02 azure-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=genie_space.yml.tmpl/READPLAN=1
7:59 azure linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=job_with_task.yml.tmpl/READPLAN=1
7:56 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:55 azure linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=job_with_depends_on.yml.tmpl/READPLAN=1
7:53 azure-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=job_apply_policy_default_values_task_cluster.yml.tmpl/READPLAN=1
7:50 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:45 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:39 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true
7:38 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=vector_search_index.yml.tmpl/READPLAN=
7:36 aws linux-2core-8gb TestAccept/bundle/resources/volumes/move_from_renamed_schema/DMS=true/READPLAN=
(821 table rows omitted to keep the report under 60000 bytes)

idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
…migration (databricks#7778)

## Changes
`statemgmt.Migrate` now returns an error when the local terraform state
is missing, instead of reporting that there is nothing to migrate. Its
`bool` return was only false on that path, so it is dropped. The failure
is recorded as a new deploy telemetry key,
`direct_migrate_state_missing`, separate from `direct_migrate_error` (a
conversion failure).

## Why
Migration only runs when the pulled state is terraform, and the pull
always leaves that state on disk, so a missing file means it changed
after the pull, e.g. a concurrent deploy migrated it. Before, the deploy
then continued on the terraform engine with no direct state open: it
took the deploy lock, uploaded files, sent `engine/terraform` in the
user agent, and panicked in `CalculatePlan` with `DeploymentState must
be opened first`. This is what one production deploy on v1.24.0 looked
like.

## Tests
The `migrate/user-agent/race` golden from
databricks#7775 (this PR is stacked on it)
changes from the panic to the error, and the deploy now stops after the
state pull.

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
…abricks#7690)

## Changes
Commands that read and need the state (deploy, plan, destroy, run,
summary) now tag the user agent with `engine/direct` before the bundle
is loaded, instead of after the state is pulled. Pull-only commands
(bind, unbind, `debug states`, `deployment migrate`) keep tagging the
pulled state's engine.

## Why
Requests made while loading the bundle, initializing, pulling the state
and resolving `lookup` variables (`scim/v2/Me`, `workspace/export`,
`workspace/get-status`, `jobs/list`, `pipelines`, ...) went out without
an engine tag, as did the telemetry upload of runs that failed early. On
CLI >= 1.20 that's 2.3% of `bundle_deploy` requests and 0.78% of deploy
runs over the last 30 days, and dashboards that classify by engine count
them as terraform. These commands always run on the direct engine since
v1.20.0: a terraform state is migrated first, and a failed migration
(including a state removed after the pull,
databricks#7778) is fatal.

## Tests
In `acceptance/bundle/user_agent`, the opening requests flip from MISS
to OK; the only untagged ones left are `/.well-known/databricks-config`,
which is sent before there is a command context, and `bundle validate`,
which doesn't read state. `state_present` and
`migrate/auto/run-no-commit` only change the tag order.
`migrate/auto/plan-failure` asserts that every request of a deploy whose
migration fails is tagged `engine/direct`; its deploy telemetry still
reports `state_engine: terraform`, the state it started from.

Stacked on databricks#7778, which is stacked
on databricks#7775. In databricks#7775's tests, the
`race` deploy is now tagged `engine/direct`, and deploys rejected for a
terraform engine pinned via `DATABRICKS_BUNDLE_ENGINE` or
`bundle.engine` now tag their telemetry upload `engine/direct`.
`bind-unbind` is unchanged: those commands still tag `engine/terraform`.

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants