Skip to content

acc: wait for RUNNING before planning in clusters/resize-terminated-fallback - #7751

Merged
denik merged 2 commits into
mainfrom
denik/acc-resize-wait-running
Nov 18, 2026
Merged

denik merged 2 commits into
mainfrom
denik/acc-resize-wait-running

Conversation

@denik

@denik denik commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Changes

bundle/resources/clusters/resize-terminated-fallback plans a resize and then terminates the cluster to exercise the resize to edit fallback. The planner chooses resize only if the cluster is RUNNING at plan time (OverrideChangeDesc in bundle/direct/dresources/cluster.go). Deploy waits for RUNNING after create, but on cloud the cluster is sometimes not RUNNING yet when the plan runs. The plan then says update, the saved plan skips resize, and the golden doesn't match.

This PR:

  • re-runs bundle plan (up to 60 × 10s, via the retry helper) until it plans resize. The saved plan decides the apply path, so the cluster's state after the plan is saved doesn't matter. Waiting for RUNNING before planning would not be enough: the cluster can still leave RUNNING before the plan reads it;
  • records the planned action (resize) in the output.

The CLI behavior is correct: a non-running cluster should get edit, not resize. Only the test setup was racy.

Why

It flakes on main, on aws and azure. Both variants are affected:

Dashboard Failure
error/2021 -Resized clusters.test_cluster / +Updated clusters.test_cluster

Tests: TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=, TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true

Tests

Local run passes. The commit message has a cloud: line, so this PR's cloud CI runs the test on every cloud.

This pull request and its description were written by Isaac.

…allback

The planner chooses resize only when the cluster is RUNNING at plan time.
On cloud the cluster is sometimes not RUNNING right after deploy, so the
plan says update and the golden (Resized + /clusters/resize) mismatches.
Wait for RUNNING before planning and record the planned action.

cloud: TestAccept/bundle/resources/clusters/resize-terminated-fallback

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 18, 2026
@github-actions github-actions Bot added DABs DABs related issues test-only-fix PRs that update failing of flaky tests without changing CLI code. labels Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 88b192dab | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

2 tests: regression=0 unreleased=0 coverage=2 failing=0
test branch main (30ea3bf) latest (v1.26.0)
TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true ✅ ✅ ✅

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 88b192d

Run: 44283514995

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 275 19 5:37
💚​ aws-windows-latest-4core-16gb 1 277 17 5:27
💚​ azure linux-2core-8gb 1 274 19 4:56
💚​ azure-windows-latest-4core-16gb 1 276 17 5:41
💚​ gcp linux-2core-8gb 1 275 19 5:23
💚​ gcp-windows-latest-4core-16gb 1 277 17 6:29
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 11 slowest tests (at least 2 minutes):
duration env testname
3:30 gcp-windows-latest-4core-16gb TestAccept
3:23 aws-windows-latest-4core-16gb TestAccept
3:22 azure-windows-latest-4core-16gb TestAccept
3:03 aws linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
2:54 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=
2:29 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
2:28 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
2:19 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=
2:12 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
2:02 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=
2:02 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=

@denik
denik marked this pull request as ready for review November 18, 2026 08:38
@denik
denik requested review from a team as code owners November 18, 2026 08:38
@denik
denik requested review from Sankalp-Mittal, andrewnester, anton-107 and rugpanov and removed request for anton-107 and rugpanov November 18, 2026 08:38
@denik
denik enabled auto-merge November 18, 2026 08:38

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me — no issues found.

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

Waiting for the first RUNNING still left a window where the cluster could
leave RUNNING before the plan read it. Retry the plan itself until it
says resize, using the retry helper (MSYS_NO_PATHCONV-safe on Windows).

cloud: TestAccept/bundle/resources/clusters/resize-terminated-fallback

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 75c0a4c Nov 18, 2026
27 checks passed
@denik
denik deleted the denik/acc-resize-wait-running branch November 18, 2026 10:45
@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants