Skip to content

direct: use the Lakebase project a timed-out create request created - #7747

Merged
denik merged 2 commits into
mainfrom
pgproj-create-504
Nov 18, 2026
Merged

denik merged 2 commits into
mainfrom
pgproj-create-504

Conversation

@denik

@denik denik commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Changes

When the request to create a postgres project fails with a transient error (5xx, 408, inactivity timeout), the direct engine now looks for the project before repeating the request, as #7602 does for branches:

  • The project exists and our request created it (by create_time, allowing a minute of clock skew): wait for its default branch to leave INIT, as the create operation on the success path would have, then use the project and log that at Info. A wait that times out, or a default branch that can't be read, only warns: the project is ours, so state records it.
  • The project exists but predates our first request: fail with the original error plus project ... exists but was not created by this deployment. It isn't taken over.
  • The project is missing: pause for the standard retry interval (with the usual retrying after ... warning), then repeat the create, up to retry.MaxRetries times. A repeat that fails because the project already exists means the earlier request has landed in the meantime; the next look finds that project and uses it.
  • A read that fails returns the original create error, with the read error as a warning. A cancelled context is returned as is.

Before this PR, the project create error was marked retrySafe (#7557) and DeploymentUnit.Create repeated the request without looking. When the 504'd request had created the project, the repeat failed with "already exists", and the deploy failed.

The look-then-repeat loop from postgres_branch.go moves to postgres_create_recovery.go (recoverPostgresCreate) and both resources use it. Branch behavior is unchanged; its unit tests and create_504 acceptance test pass as before.

The real API answers a duplicate project create with 400 BAD_REQUEST "A project with this resource name already exists in the workspace.", not 409 ALREADY_EXISTS (branches use 409). The loop accepts both (isPostgresAlreadyExists), and libs/testserver now answers a duplicate project with the same 400 as the cloud.

Judgment calls:

  • Projects have no ready state of their own, so the recovered project waits on its default branch, reusing the branch's INIT wait (waitBranchReady).
  • The "already exists" check is shared, so a repeated branch create that gets a 400 mentioning "already exists" now also goes back to the lookup instead of failing at once. The result is the same (a branch is used only if found and created by this deploy), with at most retry.MaxRetries extra requests.
  • Without project_id, a failed create is not repeated any more: there is no name to look up, and the API rejects such a request with 400 anyway (without_project_id/out.deploy.txt).

Why

On Azure, a project create was answered with a 504 after the project had been created. The retry then hit "already exists" and the deploy failed, along with every test that deploys a project first.

Dashboard:

error snippet
error/3516 Warn: deploying resources.postgres_projects.my_project: retrying after 504 Gateway Timeout from POST /api/2.0/postgres/projects then Error: cannot create resources.postgres_projects.my_project: A project with this resource name already exists in the workspace. [TraceId: [TRACE_ID]] (400 BAD_REQUEST)

Tests: postgres_databases/recreate/DMS=true, postgres_projects/add_default_endpoint_settings/DMS=, postgres_projects/purge_on_delete/DMS=, postgres_snapshot_schedules/basic/DMS=true.

Tests

  • New unit tests in bundle/direct/dresources/postgres_project_test.go drive DoCreate against a testserver with scripted responses: the project is found and used; missing then a successful repeat; missing then a repeat that gets the 400 "already exists", then the project is found; missing on every attempt; a project that predates the request; a failed read; a non-transient error (no read); no project_id (no read, no repeat); the default branch in INIT then READY; a default branch that can't be read.
  • New local acceptance test bundle/resources/postgres_projects/create_504: the project was created despite the 504 and is used (one POST, the project GET and its default branch GET); the read fails and the deploy fails with the original 504.
  • postgres_projects/create_transient_500 now records the GET before the repeated POST.
  • The local bundle/resources/postgres_* and bundle/deploy acceptance tests and the bundle/direct/..., libs/retry and libs/testserver unit tests pass.
  • The last commit asks CI to run postgres_projects, postgres_databases/recreate and postgres_snapshot_schedules/basic on Azure.

This pull request and its description were written by Isaac.

@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 18, 2026
A project create that fails with a transient error (504 "The create request
timed out and may still be completing") is no longer repeated blindly: the
direct engine first looks for the project, uses it if our request created it,
and repeats the request only while it is missing, as #7602 does for branches.
The lookup loop moves to a shared helper used by both resources.

The testserver now answers a duplicate project create with 400 BAD_REQUEST,
as the real API does.

azure: TestAccept/bundle/resources/postgres_projects TestAccept/bundle/resources/postgres_databases/recreate TestAccept/bundle/resources/postgres_snapshot_schedules/basic

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the DABs DABs related issues label Nov 18, 2026
@denik
denik force-pushed the pgproj-create-504 branch from 24a1366 to 8c55b77 Compare November 18, 2026 05:25
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Commit: c6294b811 | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

36 tests: regression=4 unreleased=2 coverage=30 failing=0
test branch main (a9494c0) latest (v1.26.0)
TestAccept/bundle/resources/postgres_databases/recreate/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_databases/recreate/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/add_default_endpoint_settings/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/add_default_endpoint_settings/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/basic/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/create_504/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/postgres_projects/create_504/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/postgres_projects/create_transient_500/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/postgres_projects/create_transient_500/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/postgres_projects/delete_transient_500/DMS= ✅ ✅ ❌
TestAccept/bundle/resources/postgres_projects/delete_transient_500/DMS=true ✅ ✅ ❌
TestAccept/bundle/resources/postgres_projects/factcheck/update_mask_spec ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/factcheck/update_mask_star ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/purge_on_delete/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/purge_on_delete/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/purge_on_delete_transitions/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/purge_on_delete_transitions/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/recreate/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/recreate/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/remove_history_retention/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/remove_history_retention/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/update_default_endpoint_autoscaling/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/update_default_endpoint_autoscaling/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/update_default_endpoint_suspend/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/update_default_endpoint_suspend/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/update_display_name/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/update_display_name/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/update_history_retention/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/update_history_retention/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/update_keeps_children/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/update_keeps_children/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/without_project_id/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/without_project_id/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_snapshot_schedules/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_snapshot_schedules/basic/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/create_504/DMS=: branch ✅ | main (a9494c0) ❌ | latest (v1.26.0) ❌

main (a9494c0):

acceptance_test.go:1384: Diff:
        --- bundle/resources/postgres_projects/create_504/output.txt
        +++ /tmp/TestAcceptbundleresourcespostgres_projectscreate_504DMS=2821829469/002/output.txt
        @@ -3,31 +3,14 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created postgres_projects.my_project
        -Files: 0 uploaded, 0 deleted
        -Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        +Warn: deploying resources.postgres_projects.my_project: retrying after 504 Gateway Timeout from POST /api/2.0/postgres/projects
        +Error: cannot create resources.postgres_projects.my_project: A project with this resource name already exists in the workspace. (400 BAD_REQUEST)
         
        ->>> print_requests.py --get //postgres/projects
        -POST /api/2.0/postgres/projects
        -GET /api/2.0/postgres/projects/test-pg-proj
        -GET /api/2.0/postgres/projects/test-pg-proj/branches/production
        -
        -=== The project cannot be read after the create request times out: deploy fails with the create error
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Warn: deploying resources.postgres_projects.my_project: checking whether projects/test-pg-proj was created: Fault injected by test.
        -Error: cannot create resources.postgres_projects.my_project: Fault injected by test. (504 DEADLINE_EXCEEDED)
        -
         Endpoint: POST [DATABRICKS_URL]/api/2.0/postgres/projects?project_id=test-pg-proj
        -HTTP Status: 504 Gateway Timeout
        -API error_code: DEADLINE_EXCEEDED
        -API message: Fault injected by test.
        +HTTP Status: 400 Bad Request
        +API error_code: BAD_REQUEST
        +API message: A project with this resource name already exists in the workspace.
         
         Files: 0 uploaded, 0 deleted
         
         Exit code: 1
        -
        ->>> print_requests.py --get //postgres/projects
        -POST /api/2.0/postgres/projects
        -GET /api/2.0/postgres/projects/test-pg-proj

latest (v1.26.0):

acceptance_test.go:1384: Diff:
        --- bundle/resources/postgres_projects/create_504/output.txt
        +++ /tmp/TestAcceptbundleresourcespostgres_projectscreate_504DMS=2060849047/002/output.txt
        @@ -3,20 +3,6 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created postgres_projects.my_project
        -Files: 0 uploaded, 0 deleted
        -Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py --get //postgres/projects
        -POST /api/2.0/postgres/projects
        -GET /api/2.0/postgres/projects/test-pg-proj
        -GET /api/2.0/postgres/projects/test-pg-proj/branches/production
        -
        -=== The project cannot be read after the create request times out: deploy fails with the create error
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Warn: deploying resources.postgres_projects.my_project: checking whether projects/test-pg-proj was created: Fault injected by test.
         Error: cannot create resources.postgres_projects.my_project: Fault injected by test. (504 DEADLINE_EXCEEDED)
         
         Endpoint: POST [DATABRICKS_URL]/api/2.0/postgres/projects?project_id=test-pg-proj
        @@ -27,7 +13,3 @@
         Files: 0 uploaded, 0 deleted
         
         Exit code: 1
        -
        ->>> print_requests.py --get //postgres/projects
        -POST /api/2.0/postgres/projects
        -GET /api/2.0/postgres/projects/test-pg-proj
TestAccept/bundle/resources/postgres_projects/create_504/DMS=true: branch ✅ | main (a9494c0) ❌ | latest (v1.26.0) ❌

main (a9494c0):

acceptance_test.go:1384: Diff:
        --- bundle/resources/postgres_projects/create_504/output.txt
        +++ /tmp/TestAcceptbundleresourcespostgres_projectscreate_504DMS=tru1757053447/002/output.txt
        @@ -3,31 +3,14 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created postgres_projects.my_project
        -Files: 0 uploaded, 0 deleted
        -Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        +Warn: deploying resources.postgres_projects.my_project: retrying after 504 Gateway Timeout from POST /api/2.0/postgres/projects
        +Error: cannot create resources.postgres_projects.my_project: A project with this resource name already exists in the workspace. (400 BAD_REQUEST)
         
        ->>> print_requests.py --get //postgres/projects
        -POST /api/2.0/postgres/projects
        -GET /api/2.0/postgres/projects/test-pg-proj
        -GET /api/2.0/postgres/projects/test-pg-proj/branches/production
        -
        -=== The project cannot be read after the create request times out: deploy fails with the create error
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Warn: deploying resources.postgres_projects.my_project: checking whether projects/test-pg-proj was created: Fault injected by test.
        -Error: cannot create resources.postgres_projects.my_project: Fault injected by test. (504 DEADLINE_EXCEEDED)
        -
         Endpoint: POST [DATABRICKS_URL]/api/2.0/postgres/projects?project_id=test-pg-proj
        -HTTP Status: 504 Gateway Timeout
        -API error_code: DEADLINE_EXCEEDED
        -API message: Fault injected by test.
        +HTTP Status: 400 Bad Request
        +API error_code: BAD_REQUEST
        +API message: A project with this resource name already exists in the workspace.
         
         Files: 0 uploaded, 0 deleted
         
         Exit code: 1
        -
        ->>> print_requests.py --get //postgres/projects
        -POST /api/2.0/postgres/projects
        -GET /api/2.0/postgres/projects/test-pg-proj
        
FAIL

latest (v1.26.0):

acceptance_test.go:1384: Diff:
        --- bundle/resources/postgres_projects/create_504/output.txt
        +++ /tmp/TestAcceptbundleresourcespostgres_projectscreate_504DMS=tru399800374/002/output.txt
        @@ -3,20 +3,6 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created postgres_projects.my_project
        -Files: 0 uploaded, 0 deleted
        -Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py --get //postgres/projects
        -POST /api/2.0/postgres/projects
        -GET /api/2.0/postgres/projects/test-pg-proj
        -GET /api/2.0/postgres/projects/test-pg-proj/branches/production
        -
        -=== The project cannot be read after the create request times out: deploy fails with the create error
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Warn: deploying resources.postgres_projects.my_project: checking whether projects/test-pg-proj was created: Fault injected by test.
         Error: cannot create resources.postgres_projects.my_project: Fault injected by test. (504 DEADLINE_EXCEEDED)
         
         Endpoint: POST [DATABRICKS_URL]/api/2.0/postgres/projects?project_id=test-pg-proj
        @@ -27,7 +13,3 @@
         Files: 0 uploaded, 0 deleted
         
         Exit code: 1
        -
        ->>> print_requests.py --get //postgres/projects
        -POST /api/2.0/postgres/projects
        -GET /api/2.0/postgres/projects/test-pg-proj
TestAccept/bundle/resources/postgres_projects/create_transient_500/DMS=: branch ✅ | main (a9494c0) ❌ | latest (v1.26.0) ❌

main (a9494c0):

acceptance_test.go:1384: Diff:
        --- bundle/resources/postgres_projects/create_transient_500/output.txt
        +++ /tmp/TestAcceptbundleresourcespostgres_projectscreate_transient_51959950288/002/output.txt
        @@ -9,7 +9,6 @@
         
         >>> print_requests.py --get //postgres
         POST /api/2.0/postgres/projects
        -GET /api/2.0/postgres/projects/test-pg-proj
         POST /api/2.0/postgres/projects
         GET /api/2.0/postgres/projects/test-pg-proj/operations/[UUID]
         GET /api/2.0/postgres/projects/test-pg-proj/operations/[UUID]

latest (v1.26.0):

acceptance_test.go:1384: Diff:
        --- bundle/resources/postgres_projects/create_transient_500/output.txt
        +++ /tmp/TestAcceptbundleresourcespostgres_projectscreate_transient_5573095923/002/output.txt
        @@ -1,27 +1,13 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Warn: deploying resources.postgres_projects.my_project: retrying after 500 Internal Server Error from POST /api/2.0/postgres/projects
        -Warn: deploying resources.postgres_projects.my_project: retrying after 500 Internal Server Error from GET /api/2.0/postgres/projects/test-pg-proj/operations/[UUID]
        -Created postgres_projects.my_project
        -Files: 0 uploaded, 0 deleted
        -Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        +Error: cannot create resources.postgres_projects.my_project: Fault injected by test. (500 INTERNAL_ERROR)
         
        ->>> print_requests.py --get //postgres
        -POST /api/2.0/postgres/projects
        -GET /api/2.0/postgres/projects/test-pg-proj
        -POST /api/2.0/postgres/projects
        -GET /api/2.0/postgres/projects/test-pg-proj/operations/[UUID]
        -GET /api/2.0/postgres/projects/test-pg-proj/operations/[UUID]
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/postgres/projects?project_id=test-pg-proj
        +HTTP Status: 500 Internal Server Error
        +API error_code: INTERNAL_ERROR
        +API message: Fault injected by test.
         
        ->>> [CLI] bundle destroy --auto-approve
        -The following resources will be deleted:
        -  delete resources.postgres_projects.my_project
        +Files: 0 uploaded, 0 deleted
         
        -This action will result in the deletion of the following Lakebase projects along with
        -all their branches, databases, and endpoints. All data stored in them will be permanently lost:
        -  delete resources.postgres_projects.my_project
        -
        -All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
        -
        -Destroy: 1 deleted
        +Exit code: 1
TestAccept/bundle/resources/postgres_projects/create_transient_500/DMS=true: branch ✅ | main (a9494c0) ❌ | latest (v1.26.0) ❌

main (a9494c0):

acceptance_test.go:1384: Diff:
        --- bundle/resources/postgres_projects/create_transient_500/output.txt
        +++ /tmp/TestAcceptbundleresourcespostgres_projectscreate_transient_52881721099/002/output.txt
        @@ -9,7 +9,6 @@
         
         >>> print_requests.py --get //postgres
         POST /api/2.0/postgres/projects
        -GET /api/2.0/postgres/projects/test-pg-proj
         POST /api/2.0/postgres/projects
         GET /api/2.0/postgres/projects/test-pg-proj/operations/[UUID]
         GET /api/2.0/postgres/projects/test-pg-proj/operations/[UUID]

latest (v1.26.0):

acceptance_test.go:1384: Diff:
        --- bundle/resources/postgres_projects/create_transient_500/output.txt
        +++ /tmp/TestAcceptbundleresourcespostgres_projectscreate_transient_53858768811/002/output.txt
        @@ -1,27 +1,13 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Warn: deploying resources.postgres_projects.my_project: retrying after 500 Internal Server Error from POST /api/2.0/postgres/projects
        -Warn: deploying resources.postgres_projects.my_project: retrying after 500 Internal Server Error from GET /api/2.0/postgres/projects/test-pg-proj/operations/[UUID]
        -Created postgres_projects.my_project
        -Files: 0 uploaded, 0 deleted
        -Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        +Error: cannot create resources.postgres_projects.my_project: Fault injected by test. (500 INTERNAL_ERROR)
         
        ->>> print_requests.py --get //postgres
        -POST /api/2.0/postgres/projects
        -GET /api/2.0/postgres/projects/test-pg-proj
        -POST /api/2.0/postgres/projects
        -GET /api/2.0/postgres/projects/test-pg-proj/operations/[UUID]
        -GET /api/2.0/postgres/projects/test-pg-proj/operations/[UUID]
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/postgres/projects?project_id=test-pg-proj
        +HTTP Status: 500 Internal Server Error
        +API error_code: INTERNAL_ERROR
        +API message: Fault injected by test.
         
        ->>> [CLI] bundle destroy --auto-approve
        -The following resources will be deleted:
        -  delete resources.postgres_projects.my_project
        +Files: 0 uploaded, 0 deleted
         
        -This action will result in the deletion of the following Lakebase projects along with
        -all their branches, databases, and endpoints. All data stored in them will be permanently lost:
        -  delete resources.postgres_projects.my_project
        -
        -All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
        -
        -Destroy: 1 deleted
        +Exit code: 1
TestAccept/bundle/resources/postgres_projects/delete_transient_500/DMS=: branch ✅ | main (a9494c0) ✅ | latest (v1.26.0) ❌

latest (v1.26.0):

acceptance_test.go:1384: Diff:
        --- bundle/resources/postgres_projects/delete_transient_500/output.txt
        +++ /tmp/TestAcceptbundleresourcespostgres_projectsdelete_transient_5231079879/002/output.txt
        @@ -15,9 +15,8 @@
         
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
         
        -Warn: destroying resources.postgres_projects.my_project: retrying after 500 Internal Server Error from DELETE /api/2.0/postgres/projects/test-pg-proj
        +Warn: destroying resources.postgres_projects.my_project: Treating resources.postgres_projects.my_project id=projects/test-pg-proj as already deleted despite delete error: Fault injected by test.
         Destroy: 1 deleted
         
         >>> print_requests.py --method DELETE --summary //postgres
         DELETE /api/2.0/postgres/projects/test-pg-proj
        -DELETE /api/2.0/postgres/projects/test-pg-proj
TestAccept/bundle/resources/postgres_projects/delete_transient_500/DMS=true: branch ✅ | main (a9494c0) ✅ | latest (v1.26.0) ❌

latest (v1.26.0):

acceptance_test.go:1384: Diff:
        --- bundle/resources/postgres_projects/delete_transient_500/output.txt
        +++ /tmp/TestAcceptbundleresourcespostgres_projectsdelete_transient_51602717858/002/output.txt
        @@ -15,9 +15,8 @@
         
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
         
        -Warn: destroying resources.postgres_projects.my_project: retrying after 500 Internal Server Error from DELETE /api/2.0/postgres/projects/test-pg-proj
        +Warn: destroying resources.postgres_projects.my_project: Treating resources.postgres_projects.my_project id=projects/test-pg-proj as already deleted despite delete error: Fault injected by test.
         Destroy: 1 deleted
         
         >>> print_requests.py --method DELETE --summary //postgres
         DELETE /api/2.0/postgres/projects/test-pg-proj
        -DELETE /api/2.0/postgres/projects/test-pg-proj

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: c6294b8

Run: 44258164995

Env 🤯​MISS 💚​RECOVERED ✅​pass 🙈​skip Time
🤯​ aws linux-2core-8gb 17 1 272 21 3:59
🤯​ aws-windows-latest-4core-16gb 17 1 274 19 4:26
💚​ azure linux-2core-8gb 1 315 22 5:47
💚​ azure-windows-latest-4core-16gb 1 317 20 8:53
🤯​ gcp linux-2core-8gb 17 1 272 21 4:26
🤯​ gcp-windows-latest-4core-16gb 17 1 274 19 3:57
18 interesting tests: 17 MISS, 1 RECOVERED
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🤯​ TestAccept/bundle/resources/postgres_databases/recreate 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/add_default_endpoint_settings 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/basic 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/delete_transient_500 🤯​M 🤯​M 🙈​s 🙈​s 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/factcheck/update_mask_spec 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/factcheck/update_mask_star 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/purge_on_delete 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/purge_on_delete_transitions 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/recreate 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/remove_history_retention 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/update_default_endpoint_autoscaling 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/update_default_endpoint_suspend 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/update_display_name 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/update_history_retention 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/update_keeps_children 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_projects/without_project_id 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
🤯​ TestAccept/bundle/resources/postgres_snapshot_schedules/basic 🤯​M 🤯​M ✅​p ✅​p 🤯​M 🤯​M
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:20 aws-windows-latest-4core-16gb TestAccept
3:57 azure-windows-latest-4core-16gb TestAccept
3:51 gcp-windows-latest-4core-16gb TestAccept

@denik
denik marked this pull request as ready for review November 18, 2026 06:29
@denik
denik requested review from a team as code owners November 18, 2026 06:29

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me — no issues found.

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.

A project found after a failed create request is now returned once its
default branch leaves INIT, as the create operation on the success path
would have waited for. Address review: the empty project_id guard says the
real reason, and postgresCreateTimeSkew is a const.

azure: TestAccept/bundle/resources/postgres_projects TestAccept/bundle/resources/postgres_databases/recreate TestAccept/bundle/resources/postgres_snapshot_schedules/basic

Co-authored-by: Isaac <no-reply@databricks.com>

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Follow-up review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.

No new issues found.

@denik
denik added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 999f567 Nov 18, 2026
28 checks passed
@denik
denik deleted the pgproj-create-504 branch November 18, 2026 11:24
deco-sdk-tagging Bot added a commit that referenced this pull request Nov 19, 2026
## Release v1.27.0

### Notable Changes

 * `databricks aitools install`, `update` and `uninstall` now exit with code 1 when they do nothing they were asked to. `install` fails when it sets up nothing: no coding agent is detected, every agent is skipped or fails to install, or no skill in the release can be installed (also with `--path`). `update` fails when every update or check it attempts fails, and `uninstall` when every removal it attempts fails. Scripts that ran these commands as a best-effort step and relied on exit code 0 should handle the new exit code. ([#7481](#7481))
 * Add `file:<path>` selectors to `pipelines run` and pipeline `bundle run`. `--refresh` and `--full-refresh` now use space-separated lists, such as `--refresh "orders payments"`, instead of comma-separated lists. Single names containing spaces now require backticks, e.g. ``--refresh 'table:`order details`'``. ([#7184](#7184))
 * The Databricks CLI is now available on PyPI as `databricks`: run it with `uvx databricks`, or install it with `uv tool install databricks`, `pipx install databricks` or `pip install databricks`. The package contains only the prebuilt CLI binary. Uninstall the legacy `databricks-cli` package first, since both install a `databricks` command. ([#7704](#7704), [#7777](#7777))

### CLI

 * Add `databricks sandbox fs` commands (`cp`, `ls`, `cat`, `mkdir`, `rm`) to transfer files between your local filesystem and sandboxes using `<sandbox-id>:/path`. ([#7372](#7372))
 * `databricks fs` commands retry reads (stat, list, open) and deletes on DBFS and Volumes when the server returns a transient error or does not respond within the HTTP timeout. ([#7556](#7556))
 * Retry UC Volumes file operations (`fs cp`, `fs ls`, `fs rm`, bundle uploads to Volumes) when the Files API returns HTTP 408, 429, 502, 503 or 504 (for example "AWS S3 is throttling requests"), or on a transient network error; this regressed in v1.10.0. ([#7559](#7559))
 * Stop warning when the host metadata request (`/.well-known/databricks-config`) times out; the CLI already falls back to your config, and the message is still shown with `--debug`. ([#7752](#7752))
 * Add `databricks aitools auto-install disable` and `enable` to turn off (or back on) the automatic installation of Databricks AI tools on CLI upgrades. A global-scope `databricks aitools uninstall` also turns it off. ([#7701](#7701))
 * Stop printing the Databricks AI tooling install hint on `auth`, `configure`, `current-user me`, `version`, `help`, and shell-completion commands. ([#7452](#7452))
 * When Codex, Cursor, GitHub Copilot CLI, OpenCode, Gemini CLI, Antigravity, Pi, Goose or Kiro run the CLI without Databricks skills installed for that agent, the CLI now prints a one-line hint to install them with `databricks aitools install --agents <agent> --scope global`, previously shown only to Claude Code. ([#7586](#7586))
 * `databricks aitools install` now rejects an unknown `--agents` name before it shows the scope prompt, instead of after. ([#7469](#7469))
 * `databricks aitools install` no longer asks "Proceed?" after the agent picker; the picker selection confirms the plan. The picker title now shows the install scope, its hint says Enter installs, `--skills-only` runs label plugin agents "skills" and pre-check every agent whose config dir exists, and the plan summary after it is reduced to the scope line. ([#7478](#7478))
 * `databricks aitools install --output json` reports more specific `error_category` values: `SKILLS_FETCH_FAILED`, `RELEASE_LOOKUP_FAILED`, `RATE_LIMITED`, `MANIFEST_INVALID`, `STATE_INVALID` and `PLUGIN_CLI_TIMEOUT`. A coding agent CLI that does not answer in time is now reported as `timed out after <duration>`, followed by any stderr it wrote before it was stopped. ([#7475](#7475))
 * `databricks aitools install` no longer prompts when a coding agent runs it, even in a pseudo-terminal; it uses the non-interactive defaults instead and prints a note saying so. ([#7477](#7477))
 * `databricks aitools install` now prints the directory it installed skills into (for example `Installed 1 skill in ~/.databricks/aitools/skills.`), so agents can open the installed `SKILL.md` files without searching for them. ([#7668](#7668))
 * `databricks aitools install` now ends a failed install with a step to recover, in the text output and in a new `recovery` field of `--output json`. ([#7480](#7480))
 * `databricks aitools install` now installs plugins for the other coding agents even when installing skill files fails. ([#7479](#7479))
 * `databricks aitools install` now installs skill files for a detected coding agent whose CLI is not on PATH (for example a Claude Code desktop or IDE install), unless it already has the plugin, says how to switch to the plugin later, and warns when the current project also has the plugin. `databricks aitools update` no longer tries to install the plugin for such an agent, and no longer writes skill files next to a plugin that such an agent already has. ([#7482](#7482))
 * Fix `databricks aitools install` waiting a long time on an unreachable or slow workspace while it resolves authentication for telemetry; that wait is now limited to a few seconds. ([#7468](#7468))
 * `databricks aitools install`, `update` and `list` now refuse a skills manifest that names a skill or file outside the skills directory, before writing any skill files. A skill whose name or file paths are not portable is skipped with a warning, and `update` keeps its installed copy. ([#7722](#7722))
 * `databricks aitools install` and `databricks aitools update` refresh only the marketplace the Databricks plugin is listed in, so another unreachable marketplace no longer blocks them. When that refresh fails, both commands warn, install or update the plugin from the local copy of the marketplace, and record the plugin version as unknown. ([#7473](#7473))
 * `databricks aitools install` and `databricks aitools update` now wait up to 20 seconds instead of 5 for a coding agent's CLI to answer the plugin capability check, so a slow first start is less likely to report the agent as unavailable. ([#7472](#7472))
 * `databricks aitools install` no longer fails a plugin-only install when the lookup of the latest skills release on GitHub fails or is rate-limited. The plugin is installed and its version is recorded as unknown. ([#7474](#7474))
 * `databricks aitools` commands now send usage telemetry, and the CLI records when it recommends installing Databricks AI tools. When the configured workspace is slow or unreachable, `aitools install`, `update` and `uninstall` can take up to about 6 seconds longer, and `list`, `version` and runs that stop early up to about 3 seconds. Set `DATABRICKS_CLI_DISABLE_TELEMETRY=1` to opt out. ([#7467](#7467), [#7469](#7469), [#7470](#7470))
 * `databricks aitools install` and `databricks aitools update` now report a coding agent CLI shim (`.cmd` or `.bat`) on Windows that prints its plugin help and then exits with an error as `shim-exited-with-error`, with the command to run to see the error, instead of reporting the CLI as not on PATH. ([#7476](#7476))
 * `apps init` no longer suggests installing skills when every detected coding agent already has the Databricks plugin. ([#7467](#7467))
 * `databricks auth login` now gives up on the optional workspace lookup after about 5 seconds instead of retrying for about 5 minutes, and explains the fallback to the manual workspace ID prompt. ([#7522](#7522))
 * Throttle Claude Code's native plugin install hint separately from the text hint: it prints at most once per 10 minutes per session, so the install dialog still comes up when a subagent, a hook, or a command outside a Claude Code subprocess runs the session's first `databricks` command. ([#7692](#7692))
 * Print Claude Code's native plugin install hint alongside the text hint, so Claude Code offers a one-click install of the Databricks plugin. Claude Code now counts as having the tooling only when it loads it: skills only in `~/.databricks/aitools/skills` don't count, and a project- or local-scope plugin counts only when the CLI runs inside that project. `auth token`, `auth docker token` and CLI subprocesses no longer print install hints. ([#7585](#7585))
 * Interactive prompts and pagers now work in Git for Windows' bash and other MSYS2 shells when they run under ConPTY (the default since Git for Windows 2.47) or directly in a Windows console. ([#7699](#7699))
 * Add the `databricks feedback` command to send feedback about the CLI to Databricks. ([#7724](#7724))
 * Fix shell completion in `databricks fs` commands for local paths typed with a leading `./` or repeated separators, which previously offered no suggestions. ([#7679](#7679))
 * Fix a panic in `databricks labs` commands when a non-string SDK setting such as `DATABRICKS_DEBUG_HEADERS` or `DATABRICKS_RATE_LIMIT` is set in the environment. ([#7673](#7673))
 * Fix a panic in `databricks labs install` for projects whose `labs.yml` has no `install` section. ([#7673](#7673))
 * `NO_COLOR` no longer disables prompts, spinners or the list pager; they run without color. ([#7700](#7700))
 * Add the Beta `databricks pipelines test` command, which runs the tests of a pipeline, and the `testing` property of pipelines in bundles. ([#7429](#7429))
 * Add `databricks sandbox forward`, which serves an app running in a sandbox at a local URL. ([#7395](#7395))
 * Bound the whole telemetry upload at exit, including config resolution and authentication, by its timeout, so a stalled auth or telemetry endpoint no longer keeps a command from exiting. ([#7467](#7467))
 * `databricks version --check` now suggests `databricks aitools install` on stderr when a coding agent is detected without the Databricks plugin or skills. ([#7471](#7471))

### AI Runtime

 * `air get -o json` now reports `PENDING` while the run waits for compute, matching `air list`, and includes the run's `status_message`. ([#7455](#7455))
 * Wrap long failure reasons and log lines in the `air list` info and logs panes, keep the MLflow link (labeled with its retry) when a run's last retry failed before starting, and point `air logs` at earlier retries when the last one has no logs. ([#7456](#7456))
 * `databricks air logs --node` now reports an out-of-range node index when it can determine the run's node count, instead of looking like a successful run with no logs. ([#7516](#7516))
 * Add experiment, duration, accelerator, and dashboard metadata to `air list --output json`. For retried runs, `started_at` now reports the latest attempt, matching `air get`. ([#7568](#7568))
 * `air list` shortens MLflow run names and unnamed-run IDs by keeping their end behind a leading `…`, and its columns no longer overflow on wide (for example CJK) text; `air get` shows an unnamed run's full MLflow run ID. ([#7582](#7582))
 * Shrink the Experiment and User columns of the interactive `air list` table to fit the terminal width, keeping the end of a shortened experiment name. ([#7583](#7583))
 * Show the experiment, accelerators and configuration in the `air list` info pane for runs where it showed `N/A`. ([#7587](#7587))
 * Strip terminal control characters from the MLflow run, experiment and user names shown by `air list` and `air get`, so a run name cannot clear or retitle the terminal of whoever lists it. ([#7601](#7601))
 * `air logs`, `air get`, and `air run --watch` no longer make an extra credential-check request before fetching the run, and `air get` reuses the configured workspace ID instead of looking it up. ([#7613](#7613))
 * `air list` no longer waits on extra current-user requests before listing runs, which makes the default list about 150ms faster. ([#7615](#7615))
 * `air list --all-status` requests larger index pages, so it makes far fewer round trips once the server supports `page_size`. ([#7616](#7616))
 * Fit the interactive `air list` table to narrow terminals: when the Experiment and User columns are at their floors, shorten Started to `MM-DD HH:MM` and size the MLflow column to its labels, then hide Accelerators, User, MLflow and Progress in that order, naming the hidden columns in the hint line. ([#7629](#7629))
 * `air get -o json` now includes `mlflow_run_id` and `mlflow_experiment_id`, so a run can be handed to MLflow-based steps without parsing `mlflow_url`. They identify the MLflow run `mlflow_url` links to, including one from an earlier retry that `mlflow_retry` names. ([#7636](#7636))
 * `air get` and the `air list` info pane now show the environment of runs on the Databricks AI environment (`databricks_ai_v*`) instead of `N/A`. ([#7637](#7637))
 * `air run` now rejects a `code_source.include_paths` path that selects no files instead of uploading a snapshot without it; glob patterns and pathspec magic are left to git. ([#7643](#7643))
 * `air convert-to-dabs` now suggests the `--output-dir` that converts a working-tree workload whose `code_source.root_path` is outside the bundle directory, instead of suggesting `sync.paths`. ([#7644](#7644))
 * `air run` now lists a `code_source.root_path` outside a git repository relative to `root_path`, so running from a subdirectory of `root_path` no longer uploads an empty or partial snapshot. ([#7646](#7646))
 * `air list --all-status` now fills `--limit` past purged or inaccessible indexed runs. ([#7743](#7743))
 * `air run` now warns about missing PyTorch when a pinned snapshot's `include_paths` selects a Python file. ([#7744](#7744))
 * `air cancel JOB_RUN_ID` now rejects run IDs that are not AI Runtime runs instead of cancelling unrelated Jobs runs. In `-o json` output, the `failed[].error` text for a run that does not exist or that you cannot view now starts with `failed to get run <id>:` instead of the cancel API error. ([#7569](#7569))
 * `air get -o json` now reports `mlflow_retry` when the MLflow link comes from an earlier retry because the run's last retry never reached MLflow. ([#7620](#7620))
 * `databricks air run --help` now shows how to install the databricks-ai-runtime coding agent skill. ([#7652](#7652))
 * `air run --watch -o json` now reports each lifecycle `STATUS` transition once and never reports a non-terminal state after the run has ended, including when logs are served from MLflow artifacts. ([#7621](#7621))
 * `air convert-to-dabs` now explains the `--output-dir` workaround when `code_source.root_path` resolves to the bundle root, and notes when the generated job carries the default `max_retries`. ([#7579](#7579))
 * `air run` now refuses to upload a `code_source` snapshot that selects no files (for example when `root_path` is git-ignored) and explains the cause, and `--dry-run` runs the snapshot file selection and reports the file count. ([#7570](#7570))
 * Repeat the Jobs run link after live log streaming ends. ([#7571](#7571))
 * `air get -o json` includes run details including flat `accelerator_type` and `num_accelerators`. Use `--include-config` to include stored config, which may contain plaintext `env_variables` and parameters; downloads over 1 MiB are omitted, and config-dependent `max_retries` is otherwise null. ([#7624](#7624))
 * `air get` and the `air list` info pane no longer show `Max Retries 0` when the Jobs API leaves the task's retry limit out. For a run submitted with `air run`, the limit now comes from the run's uploaded config, shown as `3 (CLI default)` when the config doesn't set `max_retries`; it shows `N/A` when the limit can't be known (job runs, multi-container runs, legacy runs with no explicit value, or an unreadable config). ([#7623](#7623))
 * Reorganize `databricks air` help: subcommands are grouped by task, `air run` has examples, and YAML fields are listed under headings, with pool fields marked optional and `command`, `compute.num_accelerators`, and `compute.accelerator_type` correctly marked required. ([#7512](#7512))
 * `air run -h config` now lists the environment variables AI Runtime sets on every node (such as `MLFLOW_RUN_ID` and `CODE_SOURCE_PATH`), and `air logs --help` notes that `--download-to` cannot be combined with `--tail` or `--minutes`. ([#7655](#7655))
 * Support full workspace accelerator types in `air list` filters while preserving family matching, and reject invalid filter values. ([#7515](#7515))
 * `databricks air list --all-status` now warns when the AI Runtime run index is unavailable and it falls back to scanning Jobs runs. That fallback scan is capped at 250 runs, and a stuck index request fails after 15 seconds instead of retrying for minutes. `--all-status --filter user=<your email>` now uses the run index instead of scanning the workspace's full run history. ([#7619](#7619))
 * Return structured JSON errors for invalid `air list --filter` values when `-o json` is used. ([#7612](#7612))
 * `air list --all-status` no longer fails when a listed run has been purged. ([#7523](#7523))
 * **Behavior change:** `databricks air logs` now prints the logs available now and exits, also for an active run, matching `kubectl logs`, `docker logs` and `databricks apps logs`. Pass `--follow` (`-f`) to stream new lines until the run completes; `--tail N --follow` prints the last N lines first. `databricks air run --watch` still submits and streams. ([#7658](#7658))
 * `databricks air logs -o json` and `databricks air run --watch -o json` now emit a `WAITING` event, for example `{"type":"WAITING","message":"Waiting for accelerator compute capacity..."}`, when the reported waiting status changes. ([#7695](#7695))
 * `databricks air logs` on a run that ended in the last two minutes now waits for the log index to settle before printing, instead of sometimes dropping lines that were indexed late. ([#7663](#7663))
 * `air logs` and `air run --watch` now print the waiting status (for example `Waiting for accelerator compute capacity...`) as a line on stderr when stderr is not a terminal, instead of printing nothing until the run produces logs. ([#7657](#7657))
 * Route AIR log links to the MLflow Logs page instead of the hidden artifact path. ([#7491](#7491))
 * Invalid `compute.accelerator_type` errors and `air run -h config.compute` no longer list the CLI's built-in accelerator types, which could differ from the workspace's. ([#7520](#7520))
 * `databricks air --help` now notes that GPU pools are not required to run workloads. ([#7581](#7581))
 * `air run` now prints the job run URL with the `?o=<workspace-id>` suffix, matching `air get` and `air list`; `dashboard_url` in `air run -o json` and the `--watch` events gains it too. When `air list` cannot resolve the workspace ID, its links now omit `?o=` instead of ending in `?o=0`. ([#7573](#7573))
 * `databricks air run` now reports an unknown workload YAML key by its field path, suggests the closest valid field, and points to `-h config` instead of naming an internal Go type. ([#7574](#7574))
 * `air run` (including `--dry-run`) now warns before submitting when the `code_source` imports `torch` but PyTorch is not declared: the environment is the Standard environment (version 5 or later) with no `torch` dependency, so the run may fail after it waits for a GPU. ([#7656](#7656))

### Bundles

 * Add `isolation_mode` to the `catalogs` resource, so a catalog can be deployed as `OPEN` or `ISOLATED`. ([#7463](#7463))
 * `bundle validate` no longer warns when resource permissions and top-level or target permissions name the same principal. ([#7291](#7291))
 * Retry bundle file uploads, reads and deletes when the server does not respond within the HTTP timeout ("request timed out after ... of inactivity"). ([#7521](#7521))
 * Show how to recover when `bundle deploy` fails because an app's service principal does not exist or was deleted. ([#7533](#7533))
 * Retry transient server errors when creating Lakebase (`postgres_*`) resources in the direct deployment engine. ([#7557](#7557))
 * Fix `bundle plan` reporting the `spark_conf` and `custom_tags` the backend derives from `is_single_node: true` as drift on clusters. ([#7577](#7577))
 * Stop planning a spurious update for Lakeflow Connect pipelines whose `ingestion_definition.data_staging_options` is filled in by the backend. ([#7603](#7603))
 * Retry the root path check in `bundle destroy` on transient errors, so a dropped connection no longer fails the destroy. ([#7597](#7597))
 * Use the Lakebase branch when its create request times out but the branch was created, instead of failing the deploy, in the direct deployment engine. ([#7602](#7602))
 * Retry transient server errors when deleting resources in the direct deployment engine. ([#7708](#7708))
 * Use the Lakebase project when its create request times out but the project was created, instead of failing the deploy with "already exists", in the direct deployment engine. ([#7747](#7747))
 * Use the feature, model serving endpoint or dashboard that a create request created despite timing out, instead of failing the deploy, in the direct deployment engine. ([#7757](#7757))
 * Use the Lakebase role when its create request fails with a transient error but the role was created, instead of failing the deploy, in the direct deployment engine. ([#7767](#7767))
 * Use the MCP service, model service or model provider service that a create request created despite failing with a timeout, a 499 or a reset stream, instead of failing the deploy, in the direct deployment engine. Create recovery for features, serving endpoints and dashboards now also covers 499 and reset streams. ([#7782](#7782))
 * Retry app updates and deletes in the direct deployment engine when they fail with 400 BAD_REQUEST because the app compute is `UPDATING`, after waiting for the compute to settle. ([#7606](#7606))
 * Enable drift detection for `source_code_path` and `git_source` on apps. ([#6399](#6399))
 * `bundle destroy` and `bundle deploy` no longer fail with "internal error, missing in state" when the state holds an empty-ID entry left by an interrupted recreate in an older CLI. ([#7676](#7676))
 * Fix a feature whose `full_name` contains uppercase letters being recreated on every deploy. ([#7398](#7398))
 * Fix `bundle deployment bind` (and `bundle generate --bind`) failing with `no such node "resources.internal_immutable_snapshots.immutable"` when `workspace.sync_mode` is `immutable`, as reported in [#7306](#7306). ([#7329](#7329))
 * Retry creating the deployment lock (without `--force-lock`) when the request fails with a transient server error, times out ("request timed out after ... of inactivity") or loses its connection ("unexpected EOF"). ([#7594](#7594))
 * Fix a materialization set member whose `feature_name` contains uppercase letters failing to deploy, or recreating the set on every deploy. ([#7398](#7398))
 * Delete the old resource when its name follows a renamed schema. ([#7718](#7718))
 * Fix `pipelines dry-run` starting a real run when given a job KEY, and a crash in its shell completion. ([#7672](#7672))
 * Rename schemas in place; schemas and volumes follow a renamed catalog or schema. ([#7343](#7343))
 * Fix a panic when validating or deploying jobs with local libraries and serverless job clusters. ([#7725](#7725))
 * Fix `tgz` artifacts with `git` set producing an empty tarball or failing when the bundle is in a subdirectory of the git repository. ([#7674](#7674))
 * Apply `presets.trigger_pause_status` and the development mode pause default to `triggers[]` entries on jobs. ([#7450](#7450))

### Dependency Updates

 * Bump dependencies with known vulnerabilities. ([#7750](#7750))
 * Update the Go SDK to [v0.198.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.198.0). ([#7530](#7530))
 * Update the Go SDK to [v0.199.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.199.0). ([#7584](#7584))
 * Update the Go SDK to [v0.200.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.200.0). ([#7653](#7653))
 * Update the Go SDK to [v0.201.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.201.0). ([#7769](#7769))
@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
…ut create created (databricks#7757)

## Changes

When the create request for a feature, a model serving endpoint or a
dashboard fails with a transient error (5xx, 408, inactivity timeout),
the direct engine now looks for the resource before giving up, and
repeats the request while it is missing:

- **The resource exists and our request created it** (its create time is
no earlier than a minute before our first request): use it.
- **The resource exists but predates our first request:** fail with the
original error. It isn't taken over.
- **The resource is missing:** pause for the standard retry interval,
then repeat the create, up to `retry.MaxRetries` times. A repeat that
fails with "already exists" means an earlier request has landed in the
meantime; the next look finds that resource and uses it.
- A lookup that fails for another reason returns the original create
error, with the lookup error as a warning. A cancelled context is
returned as is.

How each type is looked up:

| type | name the bundle chooses | lookup | duplicate create (checked on
AWS) |
|---|---|---|---|
| features | `full_name` | `GET
/feature-engineering/features/{full_name}` | 409 `ALREADY_EXISTS` |
| model_serving_endpoints | `name` | `GET /serving-endpoints/{name}` |
400 `RESOURCE_ALREADY_EXISTS` |
| dashboards | `<parent_path>/<display_name>.lvdash.json` |
`workspace/get-status` → `resource_id` → `GET /lakeview/dashboards/{id}`
| 409 `ALREADY_EXISTS` |

The loop is a new generic `recoverCreate` in
`bundle/direct/dresources/create_recovery.go`, the same approach
[databricks#7602](databricks#7602) took for Lakebase
branches and [databricks#7747](databricks#7747)
proposes for projects.

`libs/testserver` now rejects a second feature with the same `full_name`
and a second dashboard at the same path, as the cloud does, and sets a
feature's `created_at`. Trashing or renaming a fake dashboard now moves
its workspace file, so the path check does not see stale entries.

Judgment calls:
- Alerts are left out. They hit this error twice in the last week, and
the lookup needs the workspace object id of the `.dbalert.json` file,
which the fake server models differently from the cloud. They can follow
the same pattern later.
- Using a recovered resource is logged at Info, not Warn, as in
[databricks#7602](databricks#7602): the deploy got the
resource it asked for, so cloud goldens stay the same when a deploy
recovers.
- `recoverCreate` is generic rather than Lakebase-specific; once both
PRs land, the Lakebase helper from
[databricks#7747](databricks#7747) could reuse it. The
vector search endpoint create now reuses its `isAlreadyExists` and
`createTimeSkew`. Postgres branches are not moved onto it here, to avoid
conflicting with [databricks#7747](databricks#7747).
- Recovery starts only when the first error is transient. An "already
exists" first error is returned as is. The SDK repeats a 504 itself only
when the message matches its transient patterns, and the 504s seen here
have an empty message, so it does not.
- After a repeat gets "already exists", the loop looks up the resource
again, up to three lookups spaced by the retry interval. If every one of
them reads a stale 404, the deploy fails with "already exists", as it
does today.
- A dashboard whose display name the workspace stores under a different
file name won't be found. That falls through to the same "already
exists" failure as before this change.

## Why

These create endpoints sometimes answer with `504 DEADLINE_EXCEEDED` and
an empty message, which the SDK does not retry. The deploy failed, and
dependent resources failed with it. A blind retry is not safe: if the
first request landed, the retry and every later deploy fail with
"already exists".

Dashboard:

| error | snippet |
|---|---|
|
[error/454](http://go/cli/dashboard/error/454?main=1&pr=1&end=2026-11-18T08:00:00Z&last=7d)
| `Error: cannot create resources.features.feata: (504
DEADLINE_EXCEEDED)` on `POST
[DATABRICKS_URL]/api/2.0/feature-engineering/features` |
|
[error/433](http://go/cli/dashboard/error/433?main=1&pr=1&end=2026-11-18T08:00:00Z&last=7d)
| `Error: cannot create resources.dashboards.[NAME]: (504
DEADLINE_EXCEEDED)` |
|
[error/494](http://go/cli/dashboard/error/494?main=1&pr=1&end=2026-11-18T08:00:00Z&last=7d)
| `Error: cannot create resources.model_serving_endpoints.my_endpoint:
(504 DEADLINE_EXCEEDED)` on `POST
[DATABRICKS_URL]/api/2.0/serving-endpoints` |

## Tests

- New unit tests in `bundle/direct/dresources/create_recovery_test.go`
drive `ResourceFeature.DoCreate` against a testserver with scripted
responses, covering every path of `recoverCreate`: created despite the
timeout, predates the request, no create time, missing then created, a
repeat that gets `ALREADY_EXISTS`, giving up after `retry.MaxRetries`, a
failed lookup, a non-transient repeat error, a non-transient first
error, and a cancelled context.
- New local acceptance tests
`bundle/resources/{features,model_serving_endpoints,dashboards}/create_504`,
each with two 504 scenarios: the resource was created (it is used) and
it was not (the create is repeated). The dashboard one also goes through
a stale first read, a repeat that gets `ALREADY_EXISTS`, and a second
look.
- All local `bundle/...` acceptance tests pass, except
`bundle/templates/pydabs/check-formatting`, which needs `ruff` (not
installed on my machine).
- `dashboards/create_504` unsets `MSYS_NO_PATHCONV` (set by
`dashboards/test.toml`) for `fault.py` and `print_requests.py`, as
`acceptance/script.prepare` does, so the test runs on Windows.
- On AWS: `features/lifecycle`, `dashboards/nested-folders` and
`model_serving_endpoints/basic` pass. The last commit asks CI to run
them on every cloud, and the integration run passed on all six
environments.

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
…icks#7767)

## Changes

When the request to create a postgres role fails with a transient error
(5xx, 408, inactivity timeout), the direct engine now looks for the role
before giving up, using the look-then-repeat loop branches
([databricks#7602](databricks#7602)) and projects
([databricks#7747](databricks#7747)) already use
(`recoverPostgresCreate`):

- **The role exists and our request created it** (by `create_time`,
allowing a minute of clock skew): use it, logged at Info.
- **The role exists but predates our first request:** fail with the
original error plus `role ... exists but was not created by this
deployment`. It isn't taken over.
- **The role is missing:** pause for the standard retry interval (with
the usual `retrying after ...` warning), then repeat the create, up to
`retry.MaxRetries` times. A repeat that fails with `ALREADY_EXISTS` goes
back to the lookup.
- **`replace_existing: true`:** the error is marked `retrySafe`, and
`DeploymentUnit.Create` repeats the create without reading the role, as
for branches.
- A read that fails returns the original create error, with the read
error as a warning.

Judgment calls:
- Roles have no ready state, so a found role is used as is, with no
wait.
- An inherited role on a child branch is not taken over by the
one-minute skew in the usual case. On AWS, a role inherited by a new
child branch reports its parent role's `create_time`, not the child
branch's. It falls inside the window only if the parent role was created
less than a minute before this create request, and the request also
fails with a transient error.

## Why

On every cloud, the role create request made right after a new branch is
created is answered with a 500 now and then. Nothing retried it, so the
deploy failed, and so did every database that depends on the role. This
happened 6 times in 7 days on AWS, Azure and GCP, on main and in PRs.

Dashboard:

| error | snippet |
|---|---|
|
[error/529](http://go/cli/dashboard/error/529?main=1&pr=1&end=2026-11-18T12:00:00Z&last=7d)
| `Error: cannot create resources.postgres_roles.owner: The request
could not be processed. Please try again later and contact Databricks if
the problem persists. [TraceId: [TRACE_ID]] (500 INTERNAL_ERROR)` on
`POST
[DATABRICKS_URL]/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/main/roles?role_id=app-owner`
|

Tests:
[postgres_databases/replace_existing/DMS=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/postgres_databases/replace_existing/DMS=?main=1&pr=1&end=2026-11-18T12:00:00Z&last=7d),
[postgres_databases/basic/DMS=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/postgres_databases/basic/DMS=?main=1&pr=1&end=2026-11-18T12:00:00Z&last=7d),
[postgres_databases/recreate/DMS=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/postgres_databases/recreate/DMS=?main=1&pr=1&end=2026-11-18T12:00:00Z&last=7d),
[postgres_projects/update_keeps_children/DMS=true](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/postgres_projects/update_keeps_children/DMS=true?main=1&pr=1&end=2026-11-18T12:00:00Z&last=7d).

## Tests

- New unit tests in `bundle/direct/dresources/postgres_role_test.go`
drive `DoCreate` against a testserver with scripted responses: the role
is found and used; missing, then a successful repeat; missing, then a
repeat that gets the API's 400 "role with that name already exists",
then the role is found; missing on every attempt; a role that predates
the request; a failed read; `replace_existing`; a non-transient error
(no read). All but the last fail without the change.
- New local acceptance test
`bundle/resources/postgres_roles/create_transient_500`: the role was
created despite the 500 and is used; it was not created and the create
is repeated; the read fails and the deploy fails with the original
error; `replace_existing` repeats the create without a read.
- The local `bundle/resources/postgres_*` acceptance tests and the
`bundle/direct/...` unit tests pass.
- The last commit asks CI to run `postgres_roles`,
`postgres_databases/basic` and `postgres_databases/replace_existing` on
every cloud.

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
…atabricks#7782)

## Changes

Create requests for `mcp_services`, `model_provider_services` and
`model_services` (all AI Gateway) fail now and then with errors that
leave it unknown whether the service was created. The direct engine now
looks the service up after such an error, the same way `recoverCreate`
does for features, serving endpoints and dashboards
([databricks#7757](databricks#7757)):

- **The service exists and was created by our request** (`create_time`
no earlier than a minute before the first request): use it.
- **It is missing:** pause and repeat the create, up to
`retry.MaxRetries` times. A repeat answered with 409 `ALREADY_EXISTS`
means an earlier request landed, and the next lookup finds it.

`recoverCreate` now also treats these errors as uncertain, besides the
5xx/408 that `retry.IsTransient` covers. This applies to every resource
that uses it:
- **499 `CANCELLED`.** The status is matched, not `apierr.ErrCancelled`,
because a mapped `error_code` takes precedence over the status.
- **HTTP/2 stream resets** (`stream error: stream ID N; ...`) that are
not API errors.

The lookup after a failed create retries these errors as well.

Other changes:
- `libs/testserver` now rejects a second MCP service, model service or
model provider service with the same name with 409 `ALREADY_EXISTS`, as
the cloud does (checked on AWS for model provider services), and sets
`create_time`.
- The retry warning no longer prints a double space for status codes
that have no status text (`retrying after 499 from ...`).

## Why

These errors hit the `*/lifecycle` tests on main. Each one is a
transient backend failure: the errors only come in 1–2 runs, and the AI
Gateway backend has no visible pattern. Without recovery, a create that
landed despite the error leaves a service that the next deploy fails to
create. I chose a CLI recovery over a backend ticket because these
errors are rare and transient. The PRs listed below took the same
approach.

Judgment calls:
- `model_services` has not hit these errors yet, but it uses the same
create API; added on review. MCP and model services use the same AI
Gateway backend. I assumed they answer a duplicate create with 409
`ALREADY_EXISTS` like model provider services; the probe ran only for
model provider services.
- The Lakebase recoveries
([databricks#7602](databricks#7602),
[databricks#7747](databricks#7747),
[databricks#7767](databricks#7767)) and vector search
endpoints still use `retry.IsTransient`. Moving the "uncertain create"
check into `libs/retry` for all of them can follow separately.

## Dashboard

| error | snippet |
|---|---|
|
[error/479](http://go/cli/dashboard/error/479?main=1\&pr=1\&end=2026-11-18T18:00:00Z\&last=7d)
| `cannot create resources.mcp_services.[NAME]: Storage timeout (504
DEADLINE_EXCEEDED)` |
|
[error/484](http://go/cli/dashboard/error/484?main=1\&pr=1\&end=2026-11-18T18:00:00Z\&last=7d)
| `cannot create resources.model_provider_services.[NAME]: (499
CANCELLED)` |
|
[error/3770](http://go/cli/dashboard/error/3770?main=1\&pr=1\&end=2026-11-18T18:00:00Z\&last=7d),
[error/3771](http://go/cli/dashboard/error/3771?main=1\&pr=1\&end=2026-11-18T18:00:00Z\&last=7d),
[error/3777](http://go/cli/dashboard/error/3777?main=1\&pr=1\&end=2026-11-18T18:00:00Z\&last=7d)
| model_provider_services: `response body: stream error: stream ID N;
INTERNAL_ERROR` |
|
[error/3775](http://go/cli/dashboard/error/3775?main=1\&pr=1\&end=2026-11-18T18:00:00Z\&last=7d)
| mcp_services: `response body: stream error: stream ID N;
INTERNAL_ERROR` |

## Tests

- New acceptance tests
`bundle/resources/{mcp_services,model_provider_services,model_services}/create_failure`,
covering:
  - a 504 after the service was created;
  - a 499 before it was created;
- a 504 after creation followed by a lookup 404, so the repeated create
gets 409 `ALREADY_EXISTS`;
- for model provider services, a 499 on both the create and the first
lookup.
- A unit test for `isUncertainCreateError`.
- Shared `sdkTime` helper for the create-time closures (also used by
features).
- The three `*/lifecycle` tests pass on AWS with this change. They are
named in the commit message for the PR's cloud run.

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
…abricks#7801)

## Changes

Lakebase catalog creates now recover from transient errors the way
postgres branches, projects and roles already do
([databricks#7602](databricks#7602),
[databricks#7747](databricks#7747),
[databricks#7767](databricks#7767)). Until now,
`database_catalogs` and `postgres_catalogs` failed the deploy on any
create error. If the catalog had actually been created, the next deploy
also failed, with "already exists".

- **postgres_catalogs**: on a transient error (5xx, 408, inactivity
timeout), `recoverPostgresCreate` looks the catalog up. It uses the
catalog if its `create_time` shows this deployment created it, and
repeats the create while the catalog is missing. This is the same code
path roles use.
- **database_catalogs**: these use the generic `recoverCreate`, as
features, serving endpoints and AI Gateway services do
([databricks#7757](databricks#7757),
[databricks#7782](databricks#7782)). The Database API
returns no create time, so the time check uses `created_at` of the Unity
Catalog catalog the database catalog registers. I checked on AWS that it
is there, with `catalog_type: MANAGED_ONLINE_CATALOG`. If the database
catalog exists but its UC catalog can't be found, the deploy fails with
"has no create time" rather than repeating the create.
- Both APIs answer a duplicate create with `400 CATALOG_ALREADY_EXISTS`
("Catalog '...' already exists"; checked on AWS).
`isPostgresAlreadyExists` already matches that response.
`isAlreadyExists` now matches it as well, so a repeated database catalog
create that lands after an earlier one triggers another lookup.

Testserver, to match the cloud:
- A database catalog create also registers the UC catalog with
`created_at`, and the delete removes it.
- A missing database catalog returns `404 CATALOG_DOES_NOT_EXIST`
instead of a plain-text 404. Without the error code the SDK doesn't
report it as missing.
- A duplicate database or postgres catalog returns `400
CATALOG_ALREADY_EXISTS`, not 409 or silently overwriting.

Judgment calls:
- postgres_catalogs gates recovery on `retry.IsTransient`, like its
postgres siblings, not on the wider `isUncertainCreateError` (499,
stream resets). Neither of those has been seen on Lakebase creates.
- A recovered postgres catalog is used as soon as GET finds it, as with
roles. The catalog has no readiness state to wait on.

## Why

These create errors fail cloud runs on main. The two dashboard errors
this PR fixes:

## Dashboard

| error | failure |
|---|---|
|
[error/2208](http://go/cli/dashboard/error/2208?main=1&pr=1&end=2026-11-19T10:00:00Z&last=30d)
| `Error: cannot create resources.database_catalogs.my_catalog: (504
DEADLINE_EXCEEDED)` (Azure, 4 failures since 2026-10-26) |
|
[error/3554](http://go/cli/dashboard/error/3554?main=1&pr=1&end=2026-11-19T10:00:00Z&last=14d)
| `Error: cannot create resources.postgres_catalogs.my_catalog: The
request could not be processed. Please try again later and contact
Databricks if the problem persists. [TraceId: [TRACE_ID]] (500
INTERNAL_ERROR)` (AWS, GCP) |

Tests:
[TestAccept/bundle/resources/synced_database_tables/basic/DMS=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/synced_database_tables/basic/DMS=?main=1&pr=1&end=2026-11-19T10:00:00Z&last=30d),
[TestAccept/bundle/resources/synced_database_tables/basic/DMS=true](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/synced_database_tables/basic/DMS=true?main=1&pr=1&end=2026-11-19T10:00:00Z&last=30d),
[TestAccept/bundle/resources/postgres_synced_tables/basic/DMS=true](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/postgres_synced_tables/basic/DMS=true?main=1&pr=1&end=2026-11-19T10:00:00Z&last=14d)

## Tests

- New acceptance tests use injected faults:
`database_catalogs/create_504` covers a catalog created despite a 504, a
catalog that was not created, a catalog that only shows up after the
repeat hits "already exists", and an unreadable catalog.
`postgres_catalogs/create_transient_500` covers created, not created
(504) and unreadable.
- Unit tests cover catalogs that predate the deploy (not taken over), a
missing create time, and a missing UC catalog.
- Ran the full local acceptance suite. Probed on AWS with `testme-aws`:
UC `created_at` on a database catalog, and the duplicate-create error
from both APIs.

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
…bricks#7838)

## Changes

Follow-up to [databricks#7771](databricks#7771). In
cloud runs the acceptance proxy now also drops the create-recovery
lookup when the CLI repeats a failed create.

When a Lakebase create (project, branch, role, catalog) is answered with
a transient error, the CLI reads the resource before it repeats the
create, to see whether the failed request created it anyway
(`recoverPostgresCreate`,
[databricks#7747](databricks#7747)). databricks#7771 dropped the
failed POST once it was repeated, but the lookup GET (answered with 404)
stayed in `out.requests.txt`, so the request goldens still flaked. databricks#7771
listed this case as not covered.

The recorder now remembers, for a POST answered with a transient error,
the resource paths it names (the request path plus one of its query
values, e.g. `POST /api/2.0/postgres/projects?project_id=p` names
`/api/2.0/postgres/projects/p`). A GET of such a path answered with 404
is dropped together with the failed attempt when the CLI repeats the
POST, and logged as `Dropped from out.requests.txt, a lookup before a
repeated create: GET <path>`.

Judgment calls:
- Only 404 lookups are dropped, and only once the create is repeated.
When the lookup finds the resource (the failed create did create it), no
repeat follows, and the failed POST plus the lookup stay recorded, as
before. That case records a different sequence (it also waits for the
default branch), so mirroring the normal flow there would mean dropping
requests that did happen. A found resource also ends the recovery: the
failed attempt no longer counts as pending, so a later identical create
(deploy, destroy, deploy in one script) keeps its POST. Before this
change that later POST would have dropped the first one.
- Lookups are matched by exact path (POST path + "/" + the value of an
`*_id` query parameter), so concurrent requests for other resources are
not touched. A more general rule ("drop every 404 GET between a failed
request and its repeat") would also catch unrelated parallel reads, so I
kept it to the resource the create names.
- Not covered here:
[error/3773](http://go/cli/dashboard/error/3773?main=1&pr=1&end=2026-11-20T08:00:00Z&last=3d)
("The request could not be processed. Please try again later ...") on
`postgres_branches/update_protected` and `postgres_databases/update`. It
comes from a plain `databricks postgres get-branch` / `get-database`
command that fails, not from a bundle retry, so it is not a recording
problem.

## Why

Dashboard:

| error | snippet |
|---|---|
|
[error/3788](http://go/cli/dashboard/error/3788?main=1&pr=1&end=2026-11-20T08:00:00Z&last=3d)
| `Warn: deploying resources.postgres_projects.my_project: retrying
after 500 Internal Server Error from POST /api/2.0/postgres/projects`,
then an extra `"method": "GET", "path":
"/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]"` before the POST
in `out.requests.json` |
|
[error/3667](http://go/cli/dashboard/error/3667?main=1&pr=1&end=2026-11-20T08:00:00Z&last=3d)
| `Warn: deploying resources.postgres_branches.main: retrying after 500
Internal Server Error from POST
/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches`, then an
extra `"method": "GET", "path":
"/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/main"`
in `out.requests.create.json` |

Tests:
[postgres_catalogs/basic/DMS=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/postgres_catalogs/basic/DMS=?main=1&pr=1&end=2026-11-20T08:00:00Z&last=3d),
[postgres_roles/basic/DMS=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/postgres_roles/basic/DMS=?main=1&pr=1&end=2026-11-20T08:00:00Z&last=3d),
[postgres_roles/update/DMS=](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/resources/postgres_roles/update/DMS=?main=1&pr=1&end=2026-11-20T08:00:00Z&last=3d).

All of these show `prepare_server.go: Dropped from out.requests.txt,
repeated after a transient error: POST ...`, so databricks#7771 handled the POST;
only the lookup was left.

## Tests

- `selftest/record_proxy_retry` (`Proxy = true`, same proxy as cloud
runs) gets a new case: a 500 on the project create, which the CLI
recovers from with a lookup (404) and a repeat. The recording has one
POST and no GET of the project; without the change it has the GET.
- A second new case: a 504 on the project create after the handler
created it (`fault.py --after-handler`). The lookup finds the project,
then the script destroys and deploys again in the same recording. Both
POSTs stay recorded; without the reset the second deploy dropped the
first POST.
- Local `selftest/` and `bundle/resources/postgres_*` acceptance tests
pass.
- The commit asks CI to run `postgres_roles` and
`postgres_catalogs/basic` on gcp and `postgres_roles/basic` on aws (they
only hit the path on a transient backend error, so this checks nothing
else changed).

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants