Repository navigation
Make libs/telemetry upload tests tolerate runner stalls - #7788
Merged
Merged
Conversation
TestTelemetryUploadMaxRetries and TestTelemetryUploadTimeoutBoundsTokenAcquisition failed on main during a process stall on the macOS runner: the default 3s upload timeout expired while the client was still being created, and a 10s watchdog fired at the same moment as a 0.1s upload deadline. Tests that check something other than the timeout now use a 60s upload timeout, and the hang watchdogs wait 60s. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: 6568de6
Top 4 slowest tests (at least 2 minutes):
|
denik
marked this pull request as ready for review
November 18, 2026 19:02
denik
requested review from
Sankalp-Mittal,
andrewnester,
maksim-zuev-db and
rugpanov
and removed request for
rugpanov
November 18, 2026 19:02
andrewnester
left a comment
Contributor
There was a problem hiding this comment.
No blocking findings from a full pass; I'll take a manual look at the telemetry changes before approving.
andrewnester
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Two
libs/telemetryunit tests flaked on main on the macOSdirectdmstest job. In both runs, every telemetry test ran slowly at the same moment (TestTelemetryUploadTimeoutBoundsTokenRefreshOnRetrytook 8.66s instead of the usual 2.0s). That points to the test process stalling on the runner, not to a bug inUpload.TestTelemetryUploadMaxRetriesran on the default 3s upload timeout. The stall made creating the API client alone take more than 3s, soUploadreturned a timeout before it made any attempt. It now sets a 60s upload timeout, and so do three tests with the same weakness that test something other than the timeout: the other two retry tests, andTestTelemetryUploadHintOnlyWithUnresolvedConfig(1s hint-only budget).TestTelemetryUploadTimeoutBoundsTokenAcquisitionuses a 0.1s upload timeout and a 10s watchdog. A stall longer than 10s fires both at the same moment, and the watchdog can win theselect. The watchdog is now 60s, and the identical one inTestSendTimeoutBoundsTokenAcquisitionis too. A real hang (Uploadblocking until the token source returns) still fails the test, just after 60s.Test-only change; production code is untouched.
Why
Dashboard:
Error message not equal: expected: "failed to upload telemetry logs after three attempts" actual : "failed to create API client: context deadline exceeded"logger_test.go:285: Upload did not return after its timeout/--- FAIL: TestTelemetryUploadTimeoutBoundsTokenAcquisition (10.00s)Tests: libs/telemetry.TestTelemetryUploadMaxRetries, libs/telemetry.TestTelemetryUploadTimeoutBoundsTokenAcquisition
Tests
I reproduced the stall locally by pausing the test binary with
SIGSTOP/SIGCONT:TestTelemetryUploadRetriesForStatusCodes/RetriesOnPartialSuccessin every iteration (failed to upload telemetry logs after three attempts,uploading telemetry logs timed out: failed in rate limiter: context deadline exceeded), and an 11s pause failedTestTelemetryUploadTimeoutBoundsTokenAcquisitionwithUpload did not return after its timeout.go test ./libs/telemetry/passes, and so did 300 iterations under-race -cpu=1,4.This pull request and its description were written by Isaac.