Skip to content

acc: drop create-recovery lookups from cloud request recordings - #7838

Merged
denik merged 2 commits into
mainfrom
denik/acc-drop-create-lookup
Nov 20, 2026
Merged

denik merged 2 commits into
mainfrom
denik/acc-drop-create-lookup

Conversation

@denik

@denik denik commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Follow-up to #7771. In cloud runs the acceptance proxy now also drops the create-recovery lookup when the CLI repeats a failed create.

When a Lakebase create (project, branch, role, catalog) is answered with a transient error, the CLI reads the resource before it repeats the create, to see whether the failed request created it anyway (recoverPostgresCreate, #7747). #7771 dropped the failed POST once it was repeated, but the lookup GET (answered with 404) stayed in out.requests.txt, so the request goldens still flaked. #7771 listed this case as not covered.

The recorder now remembers, for a POST answered with a transient error, the resource paths it names (the request path plus one of its query values, e.g. POST /api/2.0/postgres/projects?project_id=p names /api/2.0/postgres/projects/p). A GET of such a path answered with 404 is dropped together with the failed attempt when the CLI repeats the POST, and logged as Dropped from out.requests.txt, a lookup before a repeated create: GET <path>.

Judgment calls:

  • Only 404 lookups are dropped, and only once the create is repeated. When the lookup finds the resource (the failed create did create it), no repeat follows, and the failed POST plus the lookup stay recorded, as before. That case records a different sequence (it also waits for the default branch), so mirroring the normal flow there would mean dropping requests that did happen. A found resource also ends the recovery: the failed attempt no longer counts as pending, so a later identical create (deploy, destroy, deploy in one script) keeps its POST. Before this change that later POST would have dropped the first one.
  • Lookups are matched by exact path (POST path + "/" + the value of an *_id query parameter), so concurrent requests for other resources are not touched. A more general rule ("drop every 404 GET between a failed request and its repeat") would also catch unrelated parallel reads, so I kept it to the resource the create names.
  • Not covered here: error/3773 ("The request could not be processed. Please try again later ...") on postgres_branches/update_protected and postgres_databases/update. It comes from a plain databricks postgres get-branch / get-database command that fails, not from a bundle retry, so it is not a recording problem.

Why

Dashboard:

error snippet
error/3788 Warn: deploying resources.postgres_projects.my_project: retrying after 500 Internal Server Error from POST /api/2.0/postgres/projects, then an extra "method": "GET", "path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]" before the POST in out.requests.json
error/3667 Warn: deploying resources.postgres_branches.main: retrying after 500 Internal Server Error from POST /api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches, then an extra "method": "GET", "path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/main" in out.requests.create.json

Tests: postgres_catalogs/basic/DMS=, postgres_roles/basic/DMS=, postgres_roles/update/DMS=.

All of these show prepare_server.go: Dropped from out.requests.txt, repeated after a transient error: POST ..., so #7771 handled the POST; only the lookup was left.

Tests

  • selftest/record_proxy_retry (Proxy = true, same proxy as cloud runs) gets a new case: a 500 on the project create, which the CLI recovers from with a lookup (404) and a repeat. The recording has one POST and no GET of the project; without the change it has the GET.
  • A second new case: a 504 on the project create after the handler created it (fault.py --after-handler). The lookup finds the project, then the script destroys and deploys again in the same recording. Both POSTs stay recorded; without the reset the second deploy dropped the first POST.
  • Local selftest/ and bundle/resources/postgres_* acceptance tests pass.
  • The commit asks CI to run postgres_roles and postgres_catalogs/basic on gcp and postgres_roles/basic on aws (they only hit the path on a transient backend error, so this checks nothing else changed).

This pull request and its description were written by Isaac.

When a Lakebase create is answered with a transient error, the CLI reads the
resource (404) before it repeats the create. The proxy dropped the failed POST
but kept the lookup GET, so out.requests goldens still flaked on cloud.

gcp: TestAccept/bundle/resources/postgres_roles TestAccept/bundle/resources/postgres_catalogs/basic
aws: TestAccept/bundle/resources/postgres_roles/basic

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 20, 2026
@github-actions github-actions Bot added the test-only-fix PRs that update failing of flaky tests without changing CLI code. label Nov 20, 2026
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Commit: df963fe7e | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

19 tests: regression=0 unreleased=0 coverage=19 failing=0
test branch main (1ede87e) latest (v1.27.0)
TestAccept/bundle/resources/postgres_catalogs/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_catalogs/basic/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/basic/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/create_transient_500/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/create_transient_500/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/inherited-role-bind/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/inherited-role-bind/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/inherited-role-conflict/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/inherited-role-conflict/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/recreate-postgres-role/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/recreate-postgres-role/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/recreate/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/recreate/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/replace_existing/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/replace_existing/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/update/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_roles/update/DMS=true ✅ ✅ ✅
TestAccept/selftest/record_proxy_retry ✅ ✅ ✅

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: df963fe

Run: 44704586750

Env 🤯​MISS 💚​RECOVERED ✅​pass 🙈​skip Time
🤯​ aws linux-2core-8gb 8 1 275 20 3:18
🤯​ aws-windows-latest-4core-16gb 8 1 277 18 5:06
🤯​ azure linux-2core-8gb 9 1 271 20 2:56
🤯​ azure-windows-latest-4core-16gb 9 1 273 18 4:29
💚​ gcp linux-2core-8gb 1 290 23 4:13
💚​ gcp-windows-latest-4core-16gb 1 292 21 5:59
10 interesting tests: 9 MISS, 1 RECOVERED
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🤯​ TestAccept/bundle/resources/postgres_catalogs/basic 🤯​M 🤯​M 🤯​M 🤯​M ✅​p ✅​p
🤯​ TestAccept/bundle/resources/postgres_roles/basic ✅​p ✅​p 🤯​M 🤯​M ✅​p ✅​p
🤯​ TestAccept/bundle/resources/postgres_roles/create_transient_500 🤯​M 🤯​M 🤯​M 🤯​M 🙈​s 🙈​s
🤯​ TestAccept/bundle/resources/postgres_roles/inherited-role-bind 🤯​M 🤯​M 🤯​M 🤯​M 🙈​s 🙈​s
🤯​ TestAccept/bundle/resources/postgres_roles/inherited-role-conflict 🤯​M 🤯​M 🤯​M 🤯​M 🙈​s 🙈​s
🤯​ TestAccept/bundle/resources/postgres_roles/recreate 🤯​M 🤯​M 🤯​M 🤯​M ✅​p ✅​p
🤯​ TestAccept/bundle/resources/postgres_roles/recreate-postgres-role 🤯​M 🤯​M 🤯​M 🤯​M ✅​p ✅​p
🤯​ TestAccept/bundle/resources/postgres_roles/replace_existing 🤯​M 🤯​M 🤯​M 🤯​M ✅​p ✅​p
🤯​ TestAccept/bundle/resources/postgres_roles/update 🤯​M 🤯​M 🤯​M 🤯​M ✅​p ✅​p
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:32 aws-windows-latest-4core-16gb TestAccept
4:28 azure-windows-latest-4core-16gb TestAccept
4:01 gcp-windows-latest-4core-16gb TestAccept

Match lookups only on *_id query values, rewrite out.requests.txt once per
repeat, and stop treating reads as lookups once the create is repeated or
the resource is found, so a later identical create keeps its POST.

gcp: TestAccept/bundle/resources/postgres_roles TestAccept/bundle/resources/postgres_catalogs/basic
aws: TestAccept/bundle/resources/postgres_roles/basic

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik marked this pull request as ready for review November 20, 2026 09:07
@denik
denik requested a review from a team as a code owner November 20, 2026 09:07

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at df963fe. Each finding was verified against the code.

Not ready yet: part of the automated review did not complete.

This PR extends the acceptance-test request recorder to also drop the recovery GET lookup (answered with 404) that the CLI makes between a failed POST create and its retry, complementing the earlier #7771 which only dropped the failed POST itself. The logic is narrowly scoped to exact resource paths derived from _id query parameters, and resets properly when the lookup finds the resource (no retry follows). The new test cases in selftest/record_proxy_retry cover the 404-lookup-then-retry path and the found-resource-then-redeploy path. The change looks correct and well contained; no issues survived verification.

@denik
denik added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit f4a7fe5 Nov 20, 2026
29 checks passed
@denik
denik deleted the denik/acc-drop-create-lookup branch November 20, 2026 09:37
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

❌ Integration test report

Commit: f4a7fe5

Run: 44712434220

Env ❌​FAIL 🟨​KNOWN 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 27 1619 1420 131:53
💚​ aws-windows-latest-4core-16gb 27 1535 1448 143:36
❌​ azure linux-2core-8gb 6 13 14 1593 1428 125:19
🟨​ azure-windows-latest-4core-16gb 13 14 1515 1456 138:08
💚​ gcp linux-2core-8gb 27 1606 1426 126:46
💚​ gcp-windows-latest-4core-16gb 27 1522 1454 148:27
33 interesting tests: 14 RECOVERED, 13 KNOWN, 6 FAIL
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
🟨​ TestAccept 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
❌​ TestAccept/bundle/deploy/immutable/can-manage ✅​p ✅​p ❌​F ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/deploy/immutable/can-manage/DMS= ✅​p ✅​p ❌​F ✅​p ✅​p ✅​p
🟨​ TestAccept/bundle/deployment/bind/alert 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/deployment/bind/alert/DMS= 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/deployment/bind/alert/DMS=true 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
❌​ TestAccept/bundle/dms/stale-plan ✅​p ✅​p ❌​F ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/dms/stale-plan/DMS=true/READPLAN=1 ✅​p ✅​p ❌​F ✅​p ✅​p ✅​p
🟨​ TestAccept/bundle/generate/alert 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/generate/alert/DMS= 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/generate/alert/DMS=true 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/alerts/basic 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/alerts/basic/DMS= 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/alerts/basic/DMS=true 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/alerts/with_file 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/alerts/with_file/DMS= 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/alerts/with_file/DMS=true 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
❌​ TestAccept/bundle/resources/clusters/deploy/data_security_mode ✅​p ✅​p ❌​F ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS= ✅​p ✅​p ❌​F ✅​p ✅​p ✅​p
💚​ TestAccept/bundle/resources/postgres_synced_tables/basic 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/basic/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 50 slowest tests (at least 2 minutes):
duration env testname
12:09 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
11:59 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
11:43 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
11:36 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
11:12 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
10:56 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
10:52 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
10:21 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
9:36 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:32 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:14 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:03 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:02 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:49 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:40 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
8:31 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
8:31 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:28 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:28 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:13 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:59 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:41 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
7:27 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:20 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
7:14 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:05 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
6:51 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:39 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:36 gcp linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
6:29 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:28 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
6:23 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:22 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:21 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:18 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:08 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
6:08 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=
6:04 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
5:35 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:26 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:21 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
5:17 aws-windows-latest-4core-16gb TestAccept/pipelines/run/refresh-flags
5:14 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
5:12 gcp linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
5:06 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=
4:56 azure-windows-latest-4core-16gb TestAccept/pipelines/run/refresh-flags
4:43 gcp-windows-latest-4core-16gb TestAccept/pipelines/run/refresh-flags
4:41 aws-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=
4:37 azure linux-2core-8gb TestAccept/pipelines/run/refresh-flags
4:37 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/inline_config/DMS=true

@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

factory-generated test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants