Repository navigation
Wait for app compute to settle before bundle run - #7460
Closed
andrewnester wants to merge 4 commits into
Closed
andrewnester wants to merge 4 commits into
andrewnester wants to merge 4 commits into
Conversation
bundle run read the app compute state once, so it could act on a transient state. An app whose compute was UPDATING failed with "not in RUNNING state", and a compute that went from STARTING back to STOPPED left the run waiting for ACTIVE. Poll until the compute is ACTIVE, STOPPED or ERROR before starting or deploying the app. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: 4bd5d5d
Top 7 slowest tests (at least 2 minutes):
|
andrewnester
requested review from
denik,
janniklasrose and
pietern
and removed request for
rclarey
November 11, 2026 12:49
waitForComputeToSettle polled through every state it did not list, so a DELETING app kept bundle run waiting for the full 20 minute budget before it failed anyway. GET returns the app (not 404) for up to ~20 minutes after DELETE, so there is nothing to wait for; halt with an error pointing at redeploying the bundle, matching how IsGone in the direct engine treats a DELETING app. Reword the doc comment so it states the contract: ACTIVE, STOPPED and ERROR return, DELETING fails, every other state is polled through. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
approved these changes
Nov 12, 2026
andrewnester
enabled auto-merge
November 13, 2026 10:22
pietern
reviewed
Nov 13, 2026
| } | ||
|
|
||
| if app.ComputeStatus == nil { | ||
| return app, nil |
Contributor
There was a problem hiding this comment.
Is this supposed to be a terminal condition, or can it happen as an initial transient condition?
Contributor
Author
There was a problem hiding this comment.
Terminal. The API always returns compute_status (the SDK waiters and dresources dereference it unconditionally, a no_compute app reports STOPPED), so nil is just a guard; added a comment saying why.
Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
Closed as duplicate of #7709 |
auto-merge was automatically disabled
November 17, 2026 10:19
Pull request was closed
idelafuente-oc
pushed a commit
to idelafuente-oc/cli
that referenced
this pull request
Nov 20, 2026
…tabricks#7606) ## Changes In the direct engine, app update (`CreateUpdate`) and delete now retry when the Apps API rejects them with 400 BAD_REQUEST because the compute is UPDATING. The CLI matches the two messages seen in cloud runs: "compute is in UPDATING state" and "not terminal with state UPDATING". Before each retry it logs a warning and polls the app until the compute leaves UPDATING/STARTING/STOPPING, for at most 20 minutes per wait. The attempt limit and pause come from `libs/retry.With`. - DELETING is not retried; `IsGone` and the apply-time `deleteConfirmedGone` already handle it. - If the app disappears while a delete is waiting, the delete succeeds. An update fails with a not-found error. - If the wait itself fails (for example a GET error), the result is still the API 400, so the apply layer's error classification does not change. - After waiting, the update refreshes the compute started state that `lifecycle` uses, because the plan-time value may be stale. - The normal path makes no extra requests, so request recordings don't change. ## Why The Apps API rejects update and delete while the app compute is in a transitional state, and the compute can enter UPDATING on its own a few seconds after a create or update returned. Cloud runs on main hit this in two tests: the second deploy right after the create fails, and so does the destroy that follows. Decision: the retry is triggered by the error message, not only by the compute state read afterwards. Otherwise a compute that settled before the follow-up read would not be retried, and that is exactly when a retry succeeds. Other 400s fail at once, without an extra GET. [databricks#7460](databricks#7460) adds a similar wait for `bundle run`; the two could share a helper once both land. ## Tests - Unit tests in `bundle/direct/dresources/app_test.go`: - delete: retries after UPDATING; retries when the compute settled before the first read; succeeds when the app is gone while waiting; returns the last 400 when retries run out; keeps the 400 when the wait fails; does not retry DELETING or other 400s. - update: retries after UPDATING; after the wait, starts an app that settled to STOPPED, and does not stop or start an app whose state already matches `lifecycle.started`; fails with not-found when the app is gone. - Local acceptance tests `bundle/apps`, `bundle/resources/apps`, `bundle/run/app*`, `bundle/invariant` pass with no output changes. - `testme-aws`: `bundle/apps/job_permissions/DMS=true` and `bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=app.yml.tmpl` pass. The race is rare (2 failures each in 14 days), so one run doesn't reproduce it; the last commit names these tests for the PR's cloud run. ## Dashboard | error | snippet | |---|---| | [error/3291](http://go/cli/dashboard/error/3291?main=1&pr=1&end=2026-11-13T16:00:00Z&last=14d) | `Error: cannot delete resources.apps.[NAME]: deleting id=test[UNIQUE_NAME]: Cannot delete app test[UNIQUE_NAME] as it is not terminal with state UPDATING, and was updated less than 20 minutes ago. Please wait before trying again. (400 BAD_REQUEST)` | | [error/3267](http://go/cli/dashboard/error/3267?main=1&pr=1&end=2026-11-13T16:00:00Z&last=14d) | `Error: cannot update resources.apps.[NAME]: updating id=app-[UNIQUE_NAME]: Cannot update app app-[UNIQUE_NAME] as its compute is in UPDATING state. App compute needs to be ACTIVE or STOPPED to update. (400 BAD_REQUEST)` | Tests: [TestAccept/bundle/apps/job_permissions/DMS=true](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/apps/job_permissions/DMS=true?main=1&pr=1&end=2026-11-13T16:00:00Z&last=14d), [TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=app.yml.tmpl/READPLAN=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=app.yml.tmpl/READPLAN=?main=1&pr=1&end=2026-11-13T16:00:00Z&last=14d) This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
bundle runon an app now polls the app until its compute is ACTIVE, STOPPED or ERROR (or the app has no compute status) before it starts or deploys the app. Transient states (STARTING, UPDATING, STOPPING) are polled through. The separateWaitGetAppActivecall for STARTING is removed, because the poll covers it.Files:
bundle/run/app.go: newwaitForComputeToSettlehelper replaces the singleApps.GetinappRunner.Run.bundle/run/app_test.go: table-driven test covering UPDATING, STARTING and STOPPING transitions..nextchanges/bundles/fix-app-run-compute-state.md: changelog fragment.Why
appRunner.Runchecked the compute state once. If it read UPDATING, it tried to deploy straight away and failed with "not in RUNNING state". If it read STARTING and the compute then went back to STOPPED,WaitGetAppActivefailed instead of starting the app again.Tests
go test ./bundle/run/...passes, including the newTestAppRunWaitsForComputeToSettlecases.go test ./acceptance -run 'TestAccept/bundle/run/app|TestAccept/bundle/resources/apps|TestAccept/bundle/apps/'passes locally.golangci-lintreports no issues on./bundle/run/....The transient-state behaviour is covered by unit tests with mocked transitions. The fake app server in
libs/testserverreturns a fixed compute state, so an acceptance test cannot reproduce the transition without a stateful change to the fake.Fixes #7451
This pull request and its description were written by Isaac.