Skip to content

acc: retry postgres get/list reads in Lakebase resource tests - #7844

Merged
denik merged 2 commits into
mainfrom
acc-postgres-get-retry
Nov 20, 2026
Merged

denik merged 2 commits into
mainfrom
acc-postgres-get-retry

Conversation

@denik

@denik denik commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Wrap the databricks postgres get-* / list-* reads in the Lakebase (postgres_*) acceptance scripts with the existing retry helper (acceptance/bin/retry.py), as the dashboards tests already do for lakeview get. Reads that are expected to fail (musterr, errcode) are left alone. Goldens change only by the retry prefix in the traced command line.

Why

On main, plain postgres get-branch / get-database calls in these scripts failed on GCP with a transient Lakebase 500:

error snippet
error/3773 Error: The request could not be processed. Please try again later and contact Databricks if the problem persists. [TraceId: [TRACE_ID]]

The SDK retries only 429 and 503 (plus some message patterns). Bundle code retries the other 5xx itself (libs/retry), but a plain CLI command run from the test script does not, so one 500 fails the whole test. The request recorder already drops a repeated failed attempt from out.requests.txt, so the retried read keeps request goldens stable.

Decisions:

  • Applied to every Lakebase script (74 reads in 41 scripts), not only the two that failed: each test has several such reads, and they all hit the same backend.
  • I did not change the CLI to retry GET 500s for every command. That would be a product behavior change (a failing GET would retry for up to the SDK's 5-minute retry timeout), so it's the owner's call. Happy to do it in a follow-up if wanted.
  • retry keeps its default of 5 attempts, 500ms apart.

Tests

  • Local acceptance: go test ./acceptance -run 'TestAccept/bundle/resources/postgres_|TestAccept/bundle/deployment/bind/postgres' -update; the only golden diffs are the retry prefixes.
  • Cloud: TestAccept/bundle/resources/postgres_databases/update/DMS=true passes on AWS locally (testme-aws). In this PR's cloud run, all Lakebase tests pass on AWS, Azure and GCP (Linux and Windows), including the two that failed on main.

Dashboard

This pull request and its description were written by Isaac.

gcp: TestAccept/bundle/resources/postgres_branches/update_protected TestAccept/bundle/resources/postgres_databases/update
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 20, 2026
@github-actions github-actions Bot added DABs DABs related issues test-only-fix PRs that update failing of flaky tests without changing CLI code. labels Nov 20, 2026
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 42406a945 | Workflow run

Regression test report

Selected 20 changed tests (limit=20, 21 not selected)

36 tests: regression=0 unreleased=0 coverage=36 failing=0
test branch main (054a0bf) latest (v1.27.0)
TestAccept/bundle/deployment/bind/postgres_data_api/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/basic/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/purge_on_delete/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/purge_on_delete/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/replace_existing/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/replace_existing/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/update_expire_time/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/update_expire_time/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/update_protected/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_branches/update_protected/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_catalogs/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_catalogs/basic/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_data_apis/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_data_apis/out-of-band/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_data_apis/recreate/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_data_apis/update/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_data_apis/url_reference/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_databases/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_databases/basic/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_databases/recreate/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_databases/recreate/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_databases/replace_existing/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_databases/replace_existing/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_databases/update/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_databases/update/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_endpoints/basic/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_endpoints/basic/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_endpoints/recreate/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_endpoints/recreate/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_endpoints/remove_suspend_timeout/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_endpoints/remove_suspend_timeout/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_endpoints/replace_existing/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_endpoints/replace_existing/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/recreate/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/postgres_projects/recreate/DMS=true ✅ ✅ ✅

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 42406a9

Run: 44730392429

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 3 378 21 11:39
💚​ aws-windows-latest-4core-16gb 3 380 19 17:51
💚​ azure linux-2core-8gb 3 367 26 9:53
💚​ azure-windows-latest-4core-16gb 3 369 24 13:54
💚​ gcp linux-2core-8gb 3 368 26 8:14
💚​ gcp-windows-latest-4core-16gb 3 370 24 12:40
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/basic 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/basic/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:02 aws-windows-latest-4core-16gb TestAccept
3:27 gcp-windows-latest-4core-16gb TestAccept
3:23 azure-windows-latest-4core-16gb TestAccept

@denik
denik marked this pull request as ready for review November 20, 2026 11:00
@denik
denik requested review from a team as code owners November 20, 2026 11:00

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at fd56237. Each finding was verified against the code.

Not ready yet: part of the automated review did not complete.

This PR wraps every $CLI postgres get-* / list-* read in the Lakebase acceptance test scripts with the existing retry helper, leaving expected-failure reads (musterr, errcode) untouched. The only golden-output changes are the retry prefix in the traced command lines. No production code is modified. The change is mechanical and consistent across all 41 scripts; I found no defects to flag.

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

No blocking or major issues found.

…etry for standalone factcheck runs

cloud: TestAccept/bundle/resources/postgres_projects/recreate
@denik
denik added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit ef8fc0b Nov 20, 2026
27 checks passed
@denik
denik deleted the acc-postgres-get-retry branch November 20, 2026 12:30
@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants