Skip to content

Make libs/telemetry upload tests tolerate runner stalls - #7788

Merged
denik merged 1 commit into
mainfrom
telemetry-test-stall-flakes
Nov 19, 2026
Merged

denik merged 1 commit into
mainfrom
telemetry-test-stall-flakes

Conversation

@denik

@denik denik commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Changes

Two libs/telemetry unit tests flaked on main on the macOS directdms test job. In both runs, every telemetry test ran slowly at the same moment (TestTelemetryUploadTimeoutBoundsTokenRefreshOnRetry took 8.66s instead of the usual 2.0s). That points to the test process stalling on the runner, not to a bug in Upload.

  • TestTelemetryUploadMaxRetries ran on the default 3s upload timeout. The stall made creating the API client alone take more than 3s, so Upload returned a timeout before it made any attempt. It now sets a 60s upload timeout, and so do three tests with the same weakness that test something other than the timeout: the other two retry tests, and TestTelemetryUploadHintOnlyWithUnresolvedConfig (1s hint-only budget).
  • TestTelemetryUploadTimeoutBoundsTokenAcquisition uses a 0.1s upload timeout and a 10s watchdog. A stall longer than 10s fires both at the same moment, and the watchdog can win the select. The watchdog is now 60s, and the identical one in TestSendTimeoutBoundsTokenAcquisition is too. A real hang (Upload blocking until the token source returns) still fails the test, just after 60s.

Test-only change; production code is untouched.

Why

Dashboard:

error failure
error/3757 Error message not equal: expected: "failed to upload telemetry logs after three attempts" actual : "failed to create API client: context deadline exceeded"
error/3758 logger_test.go:285: Upload did not return after its timeout / --- FAIL: TestTelemetryUploadTimeoutBoundsTokenAcquisition (10.00s)

Tests: libs/telemetry.TestTelemetryUploadMaxRetries, libs/telemetry.TestTelemetryUploadTimeoutBoundsTokenAcquisition

Tests

I reproduced the stall locally by pausing the test binary with SIGSTOP/SIGCONT:

  • Before: 5s pauses failed TestTelemetryUploadRetriesForStatusCodes/RetriesOnPartialSuccess in every iteration (failed to upload telemetry logs after three attempts, uploading telemetry logs timed out: failed in rate limiter: context deadline exceeded), and an 11s pause failed TestTelemetryUploadTimeoutBoundsTokenAcquisition with Upload did not return after its timeout.
  • After: the same pauses pass.
  • go test ./libs/telemetry/ passes, and so did 300 iterations under -race -cpu=1,4.

This pull request and its description were written by Isaac.

TestTelemetryUploadMaxRetries and TestTelemetryUploadTimeoutBoundsTokenAcquisition
failed on main during a process stall on the macOS runner: the default 3s
upload timeout expired while the client was still being created, and a 10s
watchdog fired at the same moment as a 0.1s upload deadline.

Tests that check something other than the timeout now use a 60s upload
timeout, and the hang watchdogs wait 60s.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 18, 2026
@github-actions github-actions Bot added the test-only-fix PRs that update failing of flaky tests without changing CLI code. label Nov 18, 2026
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 6568de6

Run: 44372341127

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 8:56
💚​ aws-windows-latest-4core-16gb 1 274 17 6:38
💚​ azure linux-2core-8gb 1 271 19 9:59
💚​ azure-windows-latest-4core-16gb 1 273 17 6:29
💚​ gcp linux-2core-8gb 1 272 19 9:27
💚​ gcp-windows-latest-4core-16gb 1 274 17 6:08
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 4 slowest tests (at least 2 minutes):
duration env testname
3:06 azure-windows-latest-4core-16gb TestAccept
3:06 aws-windows-latest-4core-16gb TestAccept
3:03 gcp-windows-latest-4core-16gb TestAccept
2:34 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat

@denik
denik marked this pull request as ready for review November 18, 2026 19:02
@denik
denik requested review from a team as code owners November 18, 2026 19:02
@denik
denik requested review from Sankalp-Mittal, andrewnester, maksim-zuev-db and rugpanov and removed request for rugpanov November 18, 2026 19:02

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me — no issues found.

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings from a full pass; I'll take a manual look at the telemetry changes before approving.

@denik
denik added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 8d7812a Nov 19, 2026
32 checks passed
@denik
denik deleted the telemetry-test-stall-flakes branch November 19, 2026 13:50
@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

factory-generated test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants