Skip to content

acc: suppress DMS history warnings caused by transport errors - #7748

Merged
denik merged 2 commits into
mainfrom
denik/acc-suppress-dms-transport-warnings
Nov 18, 2026
Merged

denik merged 2 commits into
mainfrom
denik/acc-suppress-dms-transport-warnings

Conversation

@denik

@denik denik commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Changes

acceptance/bundle/test.toml gets a new SuppressLines.warn_dms_transport pattern. It moves a deployment history warning to the test log when the cause is a transport error:

  • stream error: stream ID N; INTERNAL_ERROR; received from peer
  • unexpected EOF
  • context deadline exceeded (the best-effort drain deadline)

It covers the six best-effort warnings: Failed to record/complete deployment history, Could not enable/record/read/delete deployment history. bundle/dms/suppress-transport-warnings pins the pattern with lines it must suppress and lines it must keep.

Why

With DMS=true, the bundle suite records deployment history in every test and compares against the same goldens as the plain run. Recording is best effort, so the CLI only warns when a history call fails. When the server resets an HTTP/2 stream mid-response, that extra Warn: line fails an otherwise passing test. On 2026-11-17/18 this happened on every cloud, several tests per run.

Judgment calls:

  • Suppress only transport causes and the drain budget. context deadline exceeded is the OperationBuffer.Drain budget, which a slow service also hits; it is best effort too. An error the service returns, such as a validation error or an empty-message 5xx, is still compared. A blanket filter on these warnings would have hidden real CLI bugs like the operations.resource_key must have a known resource type regression. The dms_fault and dms tests assert their injected warnings, which don't match (indented, or Fault injected by test).
  • No CLI retry. The reset arrives after the response headers, so the server may already have applied the write. The history writes aren't idempotent: CreateDeployment assigns the id server-side, CreateVersion conflicts on a repeat, and UpdateOperation sends the sequence id from the previous write. libs/retry.IsTransientRead only matches *url.Error, and these errors come from the SDK's body read (response body: ...). Close dead HTTP/2 connections in the bundle workspace client #7723 adds HTTP/2 health-check pings for dead connections and does not change this.
  • Not covered here: the same stream reset on resource creates (secrets, model_provider_services) is a real deploy error and stays visible. So do the [CLI] api get .../resources failures in test scripts. The backend side is reported to the service owners.

Tests

  • A throwaway Go test checked the pattern (as decoded from the TOML) against the failure lines below, plus the lines it must keep (empty-message errors, validation errors, fault-injection lines, resource create errors).
  • Local TestAccept/bundle/{dms,dms_fault,resources/secrets,config-remote-sync} pass.
  • AWS: TestAccept/bundle/config-remote-sync/cli_defaults/DMS=true passes. The commit message runs three affected tests on every cloud.

Dashboard:

error snippet
error/3729 Warn: Failed to record deployment history: recording operation for resources.pipelines.[NAME]: response body: stream error: stream ID 51; INTERNAL_ERROR; received from peer
error/3731 Warn: Failed to complete deployment history version: response body: stream error: stream ID 63; INTERNAL_ERROR; received from peer
error/3732 Warn: Failed to record deployment history: recording operation for resources.postgres_roles.[NAME]: response body: stream error: stream ID 69; INTERNAL_ERROR; received from peer
error/3733 Warn: Failed to complete deployment history version: response body: stream error: stream ID 9429; INTERNAL_ERROR; received from peer
error/3013 Warn: Could not enable deployment history for this deploy: failed to create deployment: response body: stream error: stream ID 43; INTERNAL_ERROR; received from peer
error/3031 Warn: Failed to record deployment history: context deadline exceeded
error/3642 Warn: Failed to record deployment history: recording operation for resources.catalogs.[NAME]: Patch "[DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/<N>/operations/catalogs.catalog1": unexpected EOF

This pull request and its description were written by Isaac.

Deployment history recording is best effort. A dropped HTTP/2 stream, an
unexpected EOF or the drain deadline only produces a warning, but that warning
fails the DMS=true variant because it shares goldens with the plain run.
Suppress only those transport-caused warnings. Errors the service returns are
still compared.

cloud: TestAccept/bundle/config-remote-sync/cli_defaults/DMS=true TestAccept/bundle/resources/dashboards/delete-trashed-out-of-band/DMS=true TestAccept/bundle/resources/secrets/basic/DMS=true

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 18, 2026
@github-actions github-actions Bot added DABs DABs related issues test-only-fix PRs that update failing of flaky tests without changing CLI code. labels Nov 18, 2026
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 47a734d

Run: 44257839763

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 276 20 4:16
💚​ aws-windows-latest-4core-16gb 1 276 19 4:45
💚​ azure linux-2core-8gb 1 275 20 4:24
💚​ azure-windows-latest-4core-16gb 1 275 19 4:54
💚​ gcp linux-2core-8gb 1 276 20 4:02
💚​ gcp-windows-latest-4core-16gb 1 276 19 4:35
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:18 azure-windows-latest-4core-16gb TestAccept
4:06 aws-windows-latest-4core-16gb TestAccept
3:39 gcp-windows-latest-4core-16gb TestAccept

@denik
denik marked this pull request as ready for review November 18, 2026 05:52
@denik
denik requested review from a team as code owners November 18, 2026 05:52
@denik
denik requested review from Sankalp-Mittal, andrewnester, rclarey and shreyas-goenka and removed request for rclarey November 18, 2026 05:53

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me — no issues found.

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

No blocking or major issues found.

Comment thread acceptance/bundle/test.toml Resolved
Comment thread acceptance/bundle/test.toml Resolved
Comment thread acceptance/bundle/test.toml Resolved
Add "Could not delete deployment history" to the suppressed warnings:
deleteDeployment retries only IsTransient, so a stream reset during
destroy still produced the warning. Name the drain budget as the source of
"context deadline exceeded" in the comment. Add
bundle/dms/suppress-transport-warnings, which echoes lines the pattern
must suppress and lines it must keep.

cloud: TestAccept/bundle/config-remote-sync/cli_defaults/DMS=true TestAccept/bundle/resources/secrets/basic/DMS=true

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

Copy link
Copy Markdown
Contributor

Commit: 47a734de4 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

3 tests: regression=0 unreleased=0 coverage=3 failing=0
test branch main (a9494c0) latest (v1.26.0)
TestAccept/bundle/config-remote-sync/cli_defaults/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms/suppress-transport-warnings/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/secrets/basic/DMS=true ✅ ✅ ✅

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Follow-up review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

No new issues found.

@denik
denik added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit e76ef13 Nov 18, 2026
27 checks passed
@denik
denik deleted the denik/acc-suppress-dms-transport-warnings branch November 18, 2026 11:24
@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants