Skip to content

direct: use the Lakebase catalog a failed create request created - #7801

Merged
denik merged 2 commits into
mainfrom
denik/lakebase-catalog-create-recovery
Nov 19, 2026
Merged

denik merged 2 commits into
mainfrom
denik/lakebase-catalog-create-recovery

Conversation

@denik

@denik denik commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Changes

Lakebase catalog creates now recover from transient errors the way postgres branches, projects and roles already do (#7602, #7747, #7767). Until now, database_catalogs and postgres_catalogs failed the deploy on any create error. If the catalog had actually been created, the next deploy also failed, with "already exists".

  • postgres_catalogs: on a transient error (5xx, 408, inactivity timeout), recoverPostgresCreate looks the catalog up. It uses the catalog if its create_time shows this deployment created it, and repeats the create while the catalog is missing. This is the same code path roles use.
  • database_catalogs: these use the generic recoverCreate, as features, serving endpoints and AI Gateway services do (#7757, #7782). The Database API returns no create time, so the time check uses created_at of the Unity Catalog catalog the database catalog registers. I checked on AWS that it is there, with catalog_type: MANAGED_ONLINE_CATALOG. If the database catalog exists but its UC catalog can't be found, the deploy fails with "has no create time" rather than repeating the create.
  • Both APIs answer a duplicate create with 400 CATALOG_ALREADY_EXISTS ("Catalog '...' already exists"; checked on AWS). isPostgresAlreadyExists already matches that response. isAlreadyExists now matches it as well, so a repeated database catalog create that lands after an earlier one triggers another lookup.

Testserver, to match the cloud:

  • A database catalog create also registers the UC catalog with created_at, and the delete removes it.
  • A missing database catalog returns 404 CATALOG_DOES_NOT_EXIST instead of a plain-text 404. Without the error code the SDK doesn't report it as missing.
  • A duplicate database or postgres catalog returns 400 CATALOG_ALREADY_EXISTS, not 409 or silently overwriting.

Judgment calls:

  • postgres_catalogs gates recovery on retry.IsTransient, like its postgres siblings, not on the wider isUncertainCreateError (499, stream resets). Neither of those has been seen on Lakebase creates.
  • A recovered postgres catalog is used as soon as GET finds it, as with roles. The catalog has no readiness state to wait on.

Why

These create errors fail cloud runs on main. The two dashboard errors this PR fixes:

Dashboard

error failure
error/2208 Error: cannot create resources.database_catalogs.my_catalog: (504 DEADLINE_EXCEEDED) (Azure, 4 failures since 2026-10-26)
error/3554 Error: cannot create resources.postgres_catalogs.my_catalog: The request could not be processed. Please try again later and contact Databricks if the problem persists. [TraceId: [TRACE_ID]] (500 INTERNAL_ERROR) (AWS, GCP)

Tests: TestAccept/bundle/resources/synced_database_tables/basic/DMS=, TestAccept/bundle/resources/synced_database_tables/basic/DMS=true, TestAccept/bundle/resources/postgres_synced_tables/basic/DMS=true

Tests

  • New acceptance tests use injected faults: database_catalogs/create_504 covers a catalog created despite a 504, a catalog that was not created, a catalog that only shows up after the repeat hits "already exists", and an unreadable catalog. postgres_catalogs/create_transient_500 covers created, not created (504) and unreadable.
  • Unit tests cover catalogs that predate the deploy (not taken over), a missing create time, and a missing UC catalog.
  • Ran the full local acceptance suite. Probed on AWS with testme-aws: UC created_at on a database catalog, and the duplicate-create error from both APIs.

This pull request and its description were written by Isaac.

database_catalogs and postgres_catalogs creates now get the same recovery
as postgres branches, projects and roles: on a transient error, look the
catalog up, use it if this deployment created it, and repeat the create
while it is missing.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 19, 2026
cloud: TestAccept/bundle/resources/database_catalogs TestAccept/bundle/resources/postgres_catalogs TestAccept/bundle/resources/synced_database_tables/basic TestAccept/bundle/resources/postgres_synced_tables/basic

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the DABs DABs related issues label Nov 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Commit: a2d3ad0c1 | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

16 tests: regression=4 unreleased=0 coverage=12 failing=0
test branch main (ab7c938) latest (v1.27.0)
TestAccept/bundle/resources/database_catalogs/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/database_catalogs/basic/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/database_catalogs/create_504/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/database_catalogs/create_504/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/database_catalogs/recreate/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/database_catalogs/recreate/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_catalogs/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_catalogs/basic/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_catalogs/create_transient_500/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/postgres_catalogs/create_transient_500/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/postgres_catalogs/recreate/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_catalogs/recreate/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_synced_tables/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_synced_tables/basic/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/synced_database_tables/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/synced_database_tables/basic/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/database_catalogs/create_504/DMS=: branch ✅ | main (ab7c938) ❌ | latest (v1.27.0) ❌

main (ab7c938):

acceptance_test.go:1388: Diff:
        --- bundle/resources/database_catalogs/create_504/output.txt
        +++ /tmp/TestAcceptbundleresourcesdatabase_catalogscreate_504DMS=2901027787/002/output.txt
        @@ -3,54 +3,7 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created database_catalogs.my_catalog
         Created database_instances.my_instance
        -Files: 0 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py --get //database/catalogs //unity-catalog/catalogs
        -POST /api/2.0/database/catalogs
        -GET /api/2.0/database/catalogs/my_catalog
        -GET /api/2.1/unity-catalog/catalogs/my_catalog
        -
        -=== The create request times out and the catalog was not created: deploy repeats the create
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created database_instances.my_instance
        -Warn: deploying resources.database_catalogs.my_catalog: retrying after 504 Gateway Timeout from POST /api/2.0/database/catalogs
        -Created database_catalogs.my_catalog
        -Files: 0 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py --get //database/catalogs //unity-catalog/catalogs
        -POST /api/2.0/database/catalogs
        -GET /api/2.0/database/catalogs/my_catalog
        -POST /api/2.0/database/catalogs
        -
        -=== The create request times out, the catalog shows up only after the create is repeated: deploy uses the catalog
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created database_instances.my_instance
        -Warn: deploying resources.database_catalogs.my_catalog: retrying after 504 Gateway Timeout from POST /api/2.0/database/catalogs
        -Created database_catalogs.my_catalog
        -Files: 0 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py --get //database/catalogs //unity-catalog/catalogs
        -POST /api/2.0/database/catalogs
        -GET /api/2.0/database/catalogs/my_catalog
        -POST /api/2.0/database/catalogs
        -GET /api/2.0/database/catalogs/my_catalog
        -GET /api/2.1/unity-catalog/catalogs/my_catalog
        -
        -=== The catalog cannot be read after the create request times out: deploy fails with the create error
        -
        ->>> musterr [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created database_instances.my_instance
        -Warn: deploying resources.database_catalogs.my_catalog: checking whether database catalog my_catalog was created: Fault injected by test.
         Error: cannot create resources.database_catalogs.my_catalog: Fault injected by test. (504 DEADLINE_EXCEEDED)
         
         Endpoint: POST [DATABRICKS_URL]/api/2.0/database/catalogs
        @@ -60,6 +12,4 @@
         
         Files: 0 uploaded, 0 deleted
         
        ->>> print_requests.py --get //database/catalogs //unity-catalog/catalogs
        -POST /api/2.0/database/catalogs
        -GET /api/2.0/database/catalogs/my_catalog
        +Exit code: 1
        
    acceptance_test.go:1178: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json

latest (v1.27.0): same output as main (ab7c938)

TestAccept/bundle/resources/database_catalogs/create_504/DMS=true: branch ✅ | main (ab7c938) ❌ | latest (v1.27.0) ❌

main (ab7c938):

acceptance_test.go:1388: Diff:
        --- bundle/resources/database_catalogs/create_504/output.txt
        +++ /tmp/TestAcceptbundleresourcesdatabase_catalogscreate_504DMS=tru2895677248/002/output.txt
        @@ -3,54 +3,7 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created database_catalogs.my_catalog
         Created database_instances.my_instance
        -Files: 0 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py --get //database/catalogs //unity-catalog/catalogs
        -POST /api/2.0/database/catalogs
        -GET /api/2.0/database/catalogs/my_catalog
        -GET /api/2.1/unity-catalog/catalogs/my_catalog
        -
        -=== The create request times out and the catalog was not created: deploy repeats the create
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created database_instances.my_instance
        -Warn: deploying resources.database_catalogs.my_catalog: retrying after 504 Gateway Timeout from POST /api/2.0/database/catalogs
        -Created database_catalogs.my_catalog
        -Files: 0 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py --get //database/catalogs //unity-catalog/catalogs
        -POST /api/2.0/database/catalogs
        -GET /api/2.0/database/catalogs/my_catalog
        -POST /api/2.0/database/catalogs
        -
        -=== The create request times out, the catalog shows up only after the create is repeated: deploy uses the catalog
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created database_instances.my_instance
        -Warn: deploying resources.database_catalogs.my_catalog: retrying after 504 Gateway Timeout from POST /api/2.0/database/catalogs
        -Created database_catalogs.my_catalog
        -Files: 0 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py --get //database/catalogs //unity-catalog/catalogs
        -POST /api/2.0/database/catalogs
        -GET /api/2.0/database/catalogs/my_catalog
        -POST /api/2.0/database/catalogs
        -GET /api/2.0/database/catalogs/my_catalog
        -GET /api/2.1/unity-catalog/catalogs/my_catalog
        -
        -=== The catalog cannot be read after the create request times out: deploy fails with the create error
        -
        ->>> musterr [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created database_instances.my_instance
        -Warn: deploying resources.database_catalogs.my_catalog: checking whether database catalog my_catalog was created: Fault injected by test.
         Error: cannot create resources.database_catalogs.my_catalog: Fault injected by test. (504 DEADLINE_EXCEEDED)
         
         Endpoint: POST [DATABRICKS_URL]/api/2.0/database/catalogs
        @@ -60,6 +12,4 @@
         
         Files: 0 uploaded, 0 deleted
         
        ->>> print_requests.py --get //database/catalogs //unity-catalog/catalogs
        -POST /api/2.0/database/catalogs
        -GET /api/2.0/database/catalogs/my_catalog
        +Exit code: 1
        
    acceptance_test.go:1178: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
FAIL

latest (v1.27.0): same output as main (ab7c938)

TestAccept/bundle/resources/postgres_catalogs/create_transient_500/DMS=: branch ✅ | main (ab7c938) ❌ | latest (v1.27.0) ❌

main (ab7c938):

acceptance_test.go:1388: Diff:
        --- bundle/resources/postgres_catalogs/create_transient_500/output.txt
        +++ /tmp/TestAcceptbundleresourcespostgres_catalogscreate_transient_53006614606/002/output.txt
        @@ -3,37 +3,7 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created postgres_catalogs.my_catalog
         Created postgres_projects.my_project
        -Files: 0 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py --get //postgres/catalogs
        -POST /api/2.0/postgres/catalogs
        -GET /api/2.0/postgres/catalogs/lakebase_test
        -
        -=== The create request times out and the catalog was not created: deploy repeats the create
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created postgres_projects.my_project
        -Warn: deploying resources.postgres_catalogs.my_catalog: retrying after 504 Gateway Timeout from POST /api/2.0/postgres/catalogs
        -Created postgres_catalogs.my_catalog
        -Files: 0 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py --get //postgres/catalogs
        -POST /api/2.0/postgres/catalogs
        -GET /api/2.0/postgres/catalogs/lakebase_test
        -POST /api/2.0/postgres/catalogs
        -GET /api/2.0/postgres/catalogs/lakebase_test/operations/[UUID]
        -
        -=== The catalog cannot be read after the create request fails: deploy fails with the create error
        -
        ->>> musterr [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created postgres_projects.my_project
        -Warn: deploying resources.postgres_catalogs.my_catalog: checking whether catalogs/lakebase_test was created: Fault injected by test.
         Error: cannot create resources.postgres_catalogs.my_catalog: Fault injected by test. (500 INTERNAL_ERROR)
         
         Endpoint: POST [DATABRICKS_URL]/api/2.0/postgres/catalogs?catalog_id=lakebase_test
        @@ -43,6 +12,4 @@
         
         Files: 0 uploaded, 0 deleted
         
        ->>> print_requests.py --get //postgres/catalogs
        -POST /api/2.0/postgres/catalogs
        -GET /api/2.0/postgres/catalogs/lakebase_test
        +Exit code: 1

latest (v1.27.0): same output as main (ab7c938)

TestAccept/bundle/resources/postgres_catalogs/create_transient_500/DMS=true: branch ✅ | main (ab7c938) ❌ | latest (v1.27.0) ❌

main (ab7c938):

acceptance_test.go:1388: Diff:
        --- bundle/resources/postgres_catalogs/create_transient_500/output.txt
        +++ /tmp/TestAcceptbundleresourcespostgres_catalogscreate_transient_52081228248/002/output.txt
        @@ -3,37 +3,7 @@
         
         >>> [CLI] bundle deploy
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created postgres_catalogs.my_catalog
         Created postgres_projects.my_project
        -Files: 0 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py --get //postgres/catalogs
        -POST /api/2.0/postgres/catalogs
        -GET /api/2.0/postgres/catalogs/lakebase_test
        -
        -=== The create request times out and the catalog was not created: deploy repeats the create
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created postgres_projects.my_project
        -Warn: deploying resources.postgres_catalogs.my_catalog: retrying after 504 Gateway Timeout from POST /api/2.0/postgres/catalogs
        -Created postgres_catalogs.my_catalog
        -Files: 0 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py --get //postgres/catalogs
        -POST /api/2.0/postgres/catalogs
        -GET /api/2.0/postgres/catalogs/lakebase_test
        -POST /api/2.0/postgres/catalogs
        -GET /api/2.0/postgres/catalogs/lakebase_test/operations/[UUID]
        -
        -=== The catalog cannot be read after the create request fails: deploy fails with the create error
        -
        ->>> musterr [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Created postgres_projects.my_project
        -Warn: deploying resources.postgres_catalogs.my_catalog: checking whether catalogs/lakebase_test was created: Fault injected by test.
         Error: cannot create resources.postgres_catalogs.my_catalog: Fault injected by test. (500 INTERNAL_ERROR)
         
         Endpoint: POST [DATABRICKS_URL]/api/2.0/postgres/catalogs?catalog_id=lakebase_test
        @@ -43,6 +12,4 @@
         
         Files: 0 uploaded, 0 deleted
         
        ->>> print_requests.py --get //postgres/catalogs
        -POST /api/2.0/postgres/catalogs
        -GET /api/2.0/postgres/catalogs/lakebase_test
        +Exit code: 1

latest (v1.27.0): same output as main (ab7c938)

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: a2d3ad0

Run: 44517417407

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 3 282 23 15:30
💚​ aws-windows-latest-4core-16gb 3 284 21 9:09
💚​ azure linux-2core-8gb 3 281 23 11:19
💚​ azure-windows-latest-4core-16gb 3 283 21 8:32
💚​ gcp linux-2core-8gb 3 279 24 14:46
💚​ gcp-windows-latest-4core-16gb 3 281 22 12:45
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/basic 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/basic/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 13 slowest tests (at least 2 minutes):
duration env testname
3:47 gcp-windows-latest-4core-16gb TestAccept
3:26 aws linux-2core-8gb TestFilerRecursiveDelete/workspace_files_extensions
3:08 aws-windows-latest-4core-16gb TestAccept
2:49 azure-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
2:48 azure-windows-latest-4core-16gb TestAccept
2:39 aws linux-2core-8gb TestExportDir
2:29 aws linux-2core-8gb TestLock
2:23 azure-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_catalogs/recreate/DMS=true
2:16 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_catalogs/recreate/DMS=
2:13 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat
2:06 azure linux-2core-8gb TestFilerRecursiveDelete/workspace_files
2:00 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/postgres_catalogs/recreate/DMS=true
2:00 gcp-windows-latest-4core-16gb TestImportDirWithOverwriteFlag

@denik
denik marked this pull request as ready for review November 19, 2026 12:19
@denik
denik requested review from a team as code owners November 19, 2026 12:19

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me — no issues found.

@denik
denik added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit b2c0257 Nov 19, 2026
30 checks passed
@denik
denik deleted the denik/lakebase-catalog-create-recovery branch November 19, 2026 12:47
@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants