Skip to content

Wait for app compute to settle before bundle run - #7460

Closed
andrewnester wants to merge 4 commits into
mainfrom
fix/7451
Closed

andrewnester wants to merge 4 commits into
mainfrom
fix/7451

Conversation

@andrewnester

Copy link
Copy Markdown
Contributor

Changes

bundle run on an app now polls the app until its compute is ACTIVE, STOPPED or ERROR (or the app has no compute status) before it starts or deploys the app. Transient states (STARTING, UPDATING, STOPPING) are polled through. The separate WaitGetAppActive call for STARTING is removed, because the poll covers it.

Files:

  • bundle/run/app.go: new waitForComputeToSettle helper replaces the single Apps.Get in appRunner.Run.
  • bundle/run/app_test.go: table-driven test covering UPDATING, STARTING and STOPPING transitions.
  • .nextchanges/bundles/fix-app-run-compute-state.md: changelog fragment.

Why

appRunner.Run checked the compute state once. If it read UPDATING, it tried to deploy straight away and failed with "not in RUNNING state". If it read STARTING and the compute then went back to STOPPED, WaitGetAppActive failed instead of starting the app again.

Tests

  • go test ./bundle/run/... passes, including the new TestAppRunWaitsForComputeToSettle cases.
  • go test ./acceptance -run 'TestAccept/bundle/run/app|TestAccept/bundle/resources/apps|TestAccept/bundle/apps/' passes locally.
  • golangci-lint reports no issues on ./bundle/run/....

The transient-state behaviour is covered by unit tests with mocked transitions. The fake app server in libs/testserver returns a fixed compute state, so an acceptance test cannot reproduce the transition without a stateful change to the fake.

Fixes #7451

This pull request and its description were written by Isaac.

bundle run read the app compute state once, so it could act on a
transient state. An app whose compute was UPDATING failed with
"not in RUNNING state", and a compute that went from STARTING back to
STOPPED left the run waiting for ACTIVE. Poll until the compute is
ACTIVE, STOPPED or ERROR before starting or deploying the app.

Co-authored-by: Isaac <no-reply@databricks.com>
@andrewnester
andrewnester requested review from a team as code owners November 11, 2026 12:02
@andrewnester
andrewnester requested a review from rclarey November 11, 2026 12:02
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the DABs DABs related issues label Nov 11, 2026
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 4bd5d5d

Run: 43413352516

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 15 9:38
💚​ aws-windows-latest-4core-16gb 1 272 13 6:36
💚​ azure linux-2core-8gb 1 269 15 8:13
💚​ azure-windows-latest-4core-16gb 1 271 13 5:38
💚​ gcp linux-2core-8gb 1 270 15 9:00
💚​ gcp-windows-latest-4core-16gb 1 272 13 6:29
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 7 slowest tests (at least 2 minutes):
duration env testname
3:55 gcp linux-2core-8gb TestAccept
3:36 azure-windows-latest-4core-16gb TestAccept
3:36 aws-windows-latest-4core-16gb TestAccept
3:30 gcp-windows-latest-4core-16gb TestAccept
3:20 aws linux-2core-8gb TestAccept
3:16 aws linux-2core-8gb TestTelemetryEndpoint
3:13 azure linux-2core-8gb TestAccept

@andrewnester
andrewnester requested review from denik, janniklasrose and pietern and removed request for rclarey November 11, 2026 12:49
Comment thread bundle/run/app.go Resolved
Comment thread bundle/run/app.go Resolved
waitForComputeToSettle polled through every state it did not list,
so a DELETING app kept bundle run waiting for the full 20 minute
budget before it failed anyway. GET returns the app (not 404) for up
to ~20 minutes after DELETE, so there is nothing to wait for; halt with
an error pointing at redeploying the bundle, matching how IsGone in the
direct engine treats a DELETING app.

Reword the doc comment so it states the contract: ACTIVE, STOPPED and
ERROR return, DELETING fails, every other state is polled through.

Co-authored-by: Isaac <no-reply@databricks.com>
Comment thread bundle/run/app.go
}

if app.ComputeStatus == nil {
return app, nil

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this supposed to be a terminal condition, or can it happen as an initial transient condition?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Terminal. The API always returns compute_status (the SDK waiters and dresources dereference it unconditionally, a no_compute app reports STOPPED), so nil is just a guard; added a comment saying why.

Co-authored-by: Isaac <no-reply@databricks.com>
@andrewnester andrewnester added the ai-auto Deprecated: use factory-generated instead. label Nov 13, 2026
@andrewnester

Copy link
Copy Markdown
Contributor Author

Closed as duplicate of #7709

auto-merge was automatically disabled November 17, 2026 10:19

Pull request was closed

@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
…tabricks#7606)

## Changes

In the direct engine, app update (`CreateUpdate`) and delete now retry
when the Apps API rejects them with 400 BAD_REQUEST because the compute
is UPDATING. The CLI matches the two messages seen in cloud runs:
"compute is in UPDATING state" and "not terminal with state UPDATING".
Before each retry it logs a warning and polls the app until the compute
leaves UPDATING/STARTING/STOPPING, for at most 20 minutes per wait. The
attempt limit and pause come from `libs/retry.With`.

- DELETING is not retried; `IsGone` and the apply-time
`deleteConfirmedGone` already handle it.
- If the app disappears while a delete is waiting, the delete succeeds.
An update fails with a not-found error.
- If the wait itself fails (for example a GET error), the result is
still the API 400, so the apply layer's error classification does not
change.
- After waiting, the update refreshes the compute started state that
`lifecycle` uses, because the plan-time value may be stale.
- The normal path makes no extra requests, so request recordings don't
change.

## Why

The Apps API rejects update and delete while the app compute is in a
transitional state, and the compute can enter UPDATING on its own a few
seconds after a create or update returned. Cloud runs on main hit this
in two tests: the second deploy right after the create fails, and so
does the destroy that follows.

Decision: the retry is triggered by the error message, not only by the
compute state read afterwards. Otherwise a compute that settled before
the follow-up read would not be retried, and that is exactly when a
retry succeeds. Other 400s fail at once, without an extra GET.
[databricks#7460](databricks#7460) adds a similar wait
for `bundle run`; the two could share a helper once both land.

## Tests

- Unit tests in `bundle/direct/dresources/app_test.go`:
- delete: retries after UPDATING; retries when the compute settled
before the first read; succeeds when the app is gone while waiting;
returns the last 400 when retries run out; keeps the 400 when the wait
fails; does not retry DELETING or other 400s.
- update: retries after UPDATING; after the wait, starts an app that
settled to STOPPED, and does not stop or start an app whose state
already matches `lifecycle.started`; fails with not-found when the app
is gone.
- Local acceptance tests `bundle/apps`, `bundle/resources/apps`,
`bundle/run/app*`, `bundle/invariant` pass with no output changes.
- `testme-aws`: `bundle/apps/job_permissions/DMS=true` and
`bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=app.yml.tmpl` pass. The
race is rare (2 failures each in 14 days), so one run doesn't reproduce
it; the last commit names these tests for the PR's cloud run.

## Dashboard

| error | snippet |
|---|---|
|
[error/3291](http://go/cli/dashboard/error/3291?main=1&pr=1&end=2026-11-13T16:00:00Z&last=14d)
| `Error: cannot delete resources.apps.[NAME]: deleting
id=test[UNIQUE_NAME]: Cannot delete app test[UNIQUE_NAME] as it is not
terminal with state UPDATING, and was updated less than 20 minutes ago.
Please wait before trying again. (400 BAD_REQUEST)` |
|
[error/3267](http://go/cli/dashboard/error/3267?main=1&pr=1&end=2026-11-13T16:00:00Z&last=14d)
| `Error: cannot update resources.apps.[NAME]: updating
id=app-[UNIQUE_NAME]: Cannot update app app-[UNIQUE_NAME] as its compute
is in UPDATING state. App compute needs to be ACTIVE or STOPPED to
update. (400 BAD_REQUEST)` |

Tests:
[TestAccept/bundle/apps/job_permissions/DMS=true](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/apps/job_permissions/DMS=true?main=1&pr=1&end=2026-11-13T16:00:00Z&last=14d),
[TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=app.yml.tmpl/READPLAN=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=app.yml.tmpl/READPLAN=?main=1&pr=1&end=2026-11-13T16:00:00Z&last=14d)

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bundle run on an app fails with "not in RUNNING state" when the app's compute is UPDATING

5 participants