Skip to content

direct: treat starting an already-active app as success - #7709

Open
denik wants to merge 9 commits into
mainfrom
denik/app-start-idempotent
Open

denik wants to merge 9 commits into
mainfrom
denik/app-start-idempotent

Conversation

@denik

@denik denik commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Changes

In the direct deployment engine, starting an app whose compute is already ACTIVE now counts as success. On a 400 from POST /api/2.0/apps/{name}/start, the app is read once. If its compute is ACTIVE, the deploy continues to the app deployment. If it is STARTING, the CLI waits for ACTIVE with Apps.WaitGetAppActive, as bundle run does. Otherwise the original error is returned.

libs/testserver now rejects a start of an ACTIVE app with the same 400 the backend returns.

Why

DeploymentUnit.Update in bundle/direct/apply.go retries the whole DoUpdate on transient errors (from #7129). For an app toggled from lifecycle.started: false to true, DoUpdate calls Start and then waits for the compute with GETs. On GCP, one of those GETs returned 504 Gateway Timeout. The retry ran DoUpdate again with the plan-time state (compute stopped) and sent Start a second time. The backend rejected it because the first start had already gone through:

Warn: deploying resources.apps.mykey: retrying after 504 Gateway Timeout from GET /api/2.0/apps/[UNIQUE_NAME]
Error: cannot update resources.apps.mykey: updating id=[UNIQUE_NAME]: Cannot start app [UNIQUE_NAME] as its compute is in ACTIVE state. App compute needs to be STOPPED to start. (400 BAD_REQUEST)

Decisions:

  • Only an ACTIVE compute, or a STARTING one that then becomes ACTIVE, turns the 400 into success. A retry comes 15s after the failure, so the first start may still be in progress. Any other state, such as DELETING or STOPPING, fails at once, as it did before. This PR is independent of direct: retry app update and delete while app compute is UPDATING #7606, which retries update and delete while the compute is UPDATING, STARTING or STOPPING.
  • The status is checked instead of the error message. Matching on the message alone would be more fragile, and the state is what decides whether the rejected start is harmless.
  • The fresh compute state is read only after a rejected start, not before every Start. Reading it before every Start would add a request to every lifecycle deploy and change the recorded requests in the app acceptance tests.
  • Stop has the same retry pattern. It is left unchanged because I could not confirm how the backend responds to a stop of an already-stopped app.

Tests

  • Table-driven unit test TestAppStartApp_RejectedStart: a rejected start succeeds for ACTIVE, waits and succeeds for STARTING, and fails for STOPPED, DELETING and a missing compute status.
  • New local acceptance test bundle/resources/apps/lifecycle-start-retried: fault.py --after-handler answers the first start with a 504 after the handler ran, so the retried start is rejected because the compute is already ACTIVE. The deploy succeeds. Without the fix, the test fails with the error from GCP.
  • All local acceptance tests pass with the testserver change.
  • bundle/resources/apps/lifecycle-started-toggle passes on AWS with testme-aws. The 504 itself cannot be reproduced on demand.

Dashboard:

error key failure lines
error/2561 Error: cannot update resources.apps.mykey: updating id=[UNIQUE_NAME]: Cannot start app [UNIQUE_NAME] as its compute is in ACTIVE state. App compute needs to be STOPPED to start. (400 BAD_REQUEST)

Test: TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS= (2 flaky runs on GCP main, 2026-11-13)

This pull request and its description were written by Isaac.

apply retries DoUpdate on transient errors. When the app start waiter's
GET returned 504, the retry sent POST /apps/{name}/start again, and the
backend rejected it with 400 because the first start had already brought
the compute to ACTIVE.

On a 400 from Start, read the app once and continue if its compute is
ACTIVE. libs/testserver now rejects a start of an ACTIVE app as the
backend does.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 17, 2026
cloud: TestAccept/bundle/resources/apps/lifecycle-started-toggle

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the DABs DABs related issues label Nov 17, 2026
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 99b39f6

Run: 44353579045

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 20 13:39
💚​ aws-windows-latest-4core-16gb 1 274 18 9:39
💚​ azure linux-2core-8gb 1 271 20 9:53
💚​ azure-windows-latest-4core-16gb 1 273 18 7:55
💚​ gcp linux-2core-8gb 1 272 20 10:43
💚​ gcp-windows-latest-4core-16gb 1 274 18 8:45
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 24 slowest tests (at least 2 minutes):
duration env testname
4:05 aws linux-2core-8gb TestFilerRecursiveDelete/workspace_files
3:31 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
3:30 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
3:22 aws linux-2core-8gb TestFilerRecursiveDelete/workspace_files_extensions
3:12 azure-windows-latest-4core-16gb TestAccept
3:11 gcp-windows-latest-4core-16gb TestAccept
3:11 aws-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files
2:58 aws-windows-latest-4core-16gb TestFilerReadWrite/workspace_files
2:46 aws-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
2:38 aws-windows-latest-4core-16gb TestAccept
2:34 aws-windows-latest-4core-16gb TestFilerWorkspaceNotebook/rNb.r
2:33 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
2:21 azure-windows-latest-4core-16gb TestSyncFullFileSync
2:18 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
2:17 azure linux-2core-8gb TestFilerRecursiveDelete/workspace_files
2:16 azure-windows-latest-4core-16gb TestImportDirDoesNotOverwrite
2:16 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:14 aws linux-2core-8gb TestFilerReadWrite/workspace_files
2:09 gcp linux-2core-8gb TestSyncFullFileSync
2:06 aws linux-2core-8gb TestSyncIncrementalFileOverwritesFolder
2:02 aws-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
2:02 aws linux-2core-8gb TestImportDirDoesNotOverwrite
2:02 aws-windows-latest-4core-16gb TestImportDirDoesNotOverwrite
2:01 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete

@denik
denik marked this pull request as ready for review November 17, 2026 08:15
@denik
denik requested review from a team as code owners November 17, 2026 08:15
@denik
denik requested review from Sankalp-Mittal, andrewnester and pietern and removed request for pietern November 17, 2026 08:15

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

Comment thread bundle/direct/dresources/app.go Resolved
Comment thread libs/testserver/apps.go Resolved

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings from a full pass; I'll take a manual look at the direct changes before approving.

A retried start can arrive while the first start is still bringing the
compute up. Wait for it to become ACTIVE, as bundle run does, instead of
failing with the 400.

Add a local acceptance test that loses the first start's response with a
504 after the handler ran, so the retried start reaches the testserver's
ACTIVE rejection.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 99b39f641 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0
test branch main (61f1fc5) latest (v1.26.0)
TestAccept/bundle/resources/apps/lifecycle-start-retried/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/apps/lifecycle-start-retried/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/apps/lifecycle-start-retried/DMS=: branch ✅ | main (61f1fc5) ❌ | latest (v1.26.0) ❌

main (61f1fc5):

acceptance_test.go:1334: Suppressed from output.txt:
        Warn: deploying resources.apps.mykey: retrying after 504 Gateway Timeout from POST /api/2.0/apps/myapp/start
    acceptance_test.go:1388: Diff:
        --- bundle/resources/apps/lifecycle-start-retried/output.txt
        +++ /tmp/TestAcceptbundleresourcesappslifecycle-start-retriedDMS=1239239505/002/output.txt
        @@ -11,36 +11,15 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/lifecycle-start-retried/default/files...
        -✓ Deployment succeeded
        -Updated apps.mykey
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        +Error: cannot update resources.apps.mykey: updating id=myapp: Cannot start app myapp as its compute is in ACTIVE state. App compute needs to be STOPPED to start. (400 BAD_REQUEST)
         
        -=== start was sent twice; the second was rejected because the compute was already ACTIVE
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/apps/myapp/start
        +HTTP Status: 400 Bad Request
        +API error_code: BAD_REQUEST
        +API message: Cannot start app myapp as its compute is in ACTIVE state. App compute needs to be STOPPED to start.
         
        ->>> print_requests.py //apps ^//api/2.0/bundle --del-body url
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/apps/myapp/start",
        -  "body": {}
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/apps/myapp/start",
        -  "body": {}
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/apps/myapp/deployments",
        -  "body": {
        -    "mode": "SNAPSHOT",
        -    "source_code_path": "/Workspace/Users/[USERNAME]/.bundle/lifecycle-start-retried/default/files/app"
        -  }
        -}
        +Files: 1 uploaded, 0 deleted
         
        ->>> [CLI] apps get myapp
        -"ACTIVE"
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.apps.mykey
        @@ -48,3 +24,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/lifecycle-start-retried/default
         
         Destroy: 1 deleted
        +
        +Exit code: 1
        
FAIL

latest (v1.26.0):

acceptance_test.go:1334: Suppressed from output.txt:
        Warn: deploying resources.apps.mykey: retrying after 504 Gateway Timeout from POST /api/2.0/apps/myapp/start
    acceptance_test.go:1388: Diff:
        --- bundle/resources/apps/lifecycle-start-retried/output.txt
        +++ /tmp/TestAcceptbundleresourcesappslifecycle-start-retriedDMS=3420922510/002/output.txt
        @@ -11,36 +11,15 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/lifecycle-start-retried/default/files...
        -✓ Deployment succeeded
        -Updated apps.mykey
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        +Error: cannot update resources.apps.mykey: updating id=myapp: Cannot start app myapp as its compute is in ACTIVE state. App compute needs to be STOPPED to start. (400 BAD_REQUEST)
         
        -=== start was sent twice; the second was rejected because the compute was already ACTIVE
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/apps/myapp/start
        +HTTP Status: 400 Bad Request
        +API error_code: BAD_REQUEST
        +API message: Cannot start app myapp as its compute is in ACTIVE state. App compute needs to be STOPPED to start.
         
        ->>> print_requests.py //apps ^//api/2.0/bundle --del-body url
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/apps/myapp/start",
        -  "body": {}
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/apps/myapp/start",
        -  "body": {}
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/apps/myapp/deployments",
        -  "body": {
        -    "mode": "SNAPSHOT",
        -    "source_code_path": "/Workspace/Users/[USERNAME]/.bundle/lifecycle-start-retried/default/files/app"
        -  }
        -}
        +Files: 1 uploaded, 0 deleted
         
        ->>> [CLI] apps get myapp
        -"ACTIVE"
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.apps.mykey
        @@ -48,3 +24,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/lifecycle-start-retried/default
         
         Destroy: 1 deleted
        +
        +Exit code: 1
TestAccept/bundle/resources/apps/lifecycle-start-retried/DMS=true: branch ✅ | main (61f1fc5) ❌ | latest (v1.26.0) ❌

main (61f1fc5):

acceptance_test.go:1334: Suppressed from output.txt:
        Warn: deploying resources.apps.mykey: retrying after 504 Gateway Timeout from POST /api/2.0/apps/myapp/start
    acceptance_test.go:1388: Diff:
        --- bundle/resources/apps/lifecycle-start-retried/output.txt
        +++ /tmp/TestAcceptbundleresourcesappslifecycle-start-retriedDMS=tru590620917/002/output.txt
        @@ -11,36 +11,15 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/lifecycle-start-retried/default/files...
        -✓ Deployment succeeded
        -Updated apps.mykey
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        +Error: cannot update resources.apps.mykey: updating id=myapp: Cannot start app myapp as its compute is in ACTIVE state. App compute needs to be STOPPED to start. (400 BAD_REQUEST)
         
        -=== start was sent twice; the second was rejected because the compute was already ACTIVE
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/apps/myapp/start
        +HTTP Status: 400 Bad Request
        +API error_code: BAD_REQUEST
        +API message: Cannot start app myapp as its compute is in ACTIVE state. App compute needs to be STOPPED to start.
         
        ->>> print_requests.py //apps ^//api/2.0/bundle --del-body url
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/apps/myapp/start",
        -  "body": {}
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/apps/myapp/start",
        -  "body": {}
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/apps/myapp/deployments",
        -  "body": {
        -    "mode": "SNAPSHOT",
        -    "source_code_path": "/Workspace/Users/[USERNAME]/.bundle/lifecycle-start-retried/default/files/app"
        -  }
        -}
        +Files: 1 uploaded, 0 deleted
         
        ->>> [CLI] apps get myapp
        -"ACTIVE"
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.apps.mykey
        @@ -48,3 +24,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/lifecycle-start-retried/default
         
         Destroy: 1 deleted
        +
        +Exit code: 1

latest (v1.26.0):

acceptance_test.go:1334: Suppressed from output.txt:
        Warn: deploying resources.apps.mykey: retrying after 504 Gateway Timeout from POST /api/2.0/apps/myapp/start
    acceptance_test.go:1388: Diff:
        --- bundle/resources/apps/lifecycle-start-retried/output.txt
        +++ /tmp/TestAcceptbundleresourcesappslifecycle-start-retriedDMS=tru1898827686/002/output.txt
        @@ -11,36 +11,15 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/lifecycle-start-retried/default/files...
        -✓ Deployment succeeded
        -Updated apps.mykey
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        +Error: cannot update resources.apps.mykey: updating id=myapp: Cannot start app myapp as its compute is in ACTIVE state. App compute needs to be STOPPED to start. (400 BAD_REQUEST)
         
        -=== start was sent twice; the second was rejected because the compute was already ACTIVE
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/apps/myapp/start
        +HTTP Status: 400 Bad Request
        +API error_code: BAD_REQUEST
        +API message: Cannot start app myapp as its compute is in ACTIVE state. App compute needs to be STOPPED to start.
         
        ->>> print_requests.py //apps ^//api/2.0/bundle --del-body url
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/apps/myapp/start",
        -  "body": {}
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/apps/myapp/start",
        -  "body": {}
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/apps/myapp/deployments",
        -  "body": {
        -    "mode": "SNAPSHOT",
        -    "source_code_path": "/Workspace/Users/[USERNAME]/.bundle/lifecycle-start-retried/default/files/app"
        -  }
        -}
        +Files: 1 uploaded, 0 deleted
         
        ->>> [CLI] apps get myapp
        -"ACTIVE"
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.apps.mykey
        @@ -48,3 +24,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/lifecycle-start-retried/default
         
         Destroy: 1 deleted
        +
        +Exit code: 1
        
FAIL

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Follow-up review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

Each DMS variant waited the default 15s before the retried DoUpdate.

Co-authored-by: Isaac <no-reply@databricks.com>

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Follow-up review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

No new issues found.

denik and others added 2 commits November 18, 2026 11:40
Resolve the conflict in bundle/direct/dresources/app.go: keep startApp and
take main's updated hasAppChanges comment from #6399.

Co-authored-by: Isaac <no-reply@databricks.com>
Resolve conflicts with #7606 in bundle/direct/dresources: keep both the
libs/retry import and log import in app.go, and keep both sets of tests in
app_test.go. #7606 refreshes the compute state before manageLifecycle;
startApp still handles a rejected start, so the two changes compose.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik

denik commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

AI agent (opus5.5, isaac_claude): merged main twice to clear conflicts (no rebase or force-push). The second merge (d769cae) overlapped #7606 in bundle/direct/dresources/app.go and app_test.go: it kept both the libs/retry and libs/log imports and both sets of tests, with no logic changes. The two changes compose: #7606 refreshes alreadyStarted from the settled compute before manageLifecycle, and manageLifecycle then calls this PR's startApp for a rejected start. All TestApp* unit tests and the local app acceptance tests pass, and CI is green on d769cae.

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.

Comment thread bundle/direct/dresources/app.go Resolved
If the GET after a rejected start failed, startApp returned only the
start's 400, so apply did not retry a transient read failure (such as
a 504). Return the read error, with the 400 as context.

Co-authored-by: Isaac <no-reply@databricks.com>

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings from a full pass; I'll take a manual look at the direct changes before approving.

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Follow-up review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.

No new issues found.

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

startCalls := 0
server.Handle("POST", "/api/2.0/apps/{name}/start", func(req testserver.Request) any {
startCalls++
return testserver.Response{

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] nit: nit: this builds the 400 response and the client inline, while the next test uses appBadRequest and newAppForServer from this file. Can this one use them too?

// is in ACTIVE state"). That happens when apply retries DoUpdate after a transient error in the
// start waiter: the first start already went through, so the compute is ACTIVE, or still
// STARTING if the retry came before it finished.
func (r *ResourceApp) startApp(ctx context.Context, id string) (*apps.App, error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a similar logic in bundle/run/app.go function start, can we consolidate in single helper function maybe?

@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants