Repository navigation
Skip TestTelemetryEndpoint when /telemetry-ext stays rate limited - #7794
Merged
Merged
Conversation
The SDK retries 429 for its 5-minute retry timeout. When the shared test workspace keeps throttling /telemetry-ext for longer (traffic from other concurrent CI runs), the test fails with "Too Many Requests" after 300s. That says nothing about the endpoint accepting the logs, so skip instead. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: aaaeb6b
Top 3 slowest tests (at least 2 minutes):
|
denik
marked this pull request as ready for review
November 19, 2026 10:15
denik
requested review from
Sankalp-Mittal,
andrewnester and
maksim-zuev-db
and removed request for
maksim-zuev-db
November 19, 2026 10:15
andrewnester
approved these changes
Nov 19, 2026
Collaborator
❌ Integration test reportCommit: bc2ce9e
847 interesting tests: 813 MISS, 15 RECOVERED, 7 KNOWN, 6 PANIC, 6 FAIL
Top 50 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
TestTelemetryEndpointnow skips when/telemetry-extstill answers 429 after the SDK's retries.Why
The test fails a few times a week on AWS and Azure, on main and in PRs, after exactly 300s:
300s is the SDK's default retry timeout. The SDK retries 429, and once the timeout expires it returns the last
APIErrorwithout the "timed out" wrapper. So the endpoint kept throttling for the whole 5 minutes.The test sends one request, so it isn't the cause. In the failing azure job on 2026-11-18, no other test got a 429 during those 5 minutes. About 25 CLI integration runs (PR and nightly) were running against the same test workspaces in that window. Throttling on a shared workspace says nothing about whether the endpoint accepts these protos, which is what this test checks.
Decision: skip on 429. Retrying longer would just make the job wait more on a throttle that already lasted 5 minutes. Running this test with less parallelism wouldn't help either, since the load comes from other runs. The CLI itself is not affected: telemetry uploads are best-effort with a 1-3s timeout.
Tests
testme-aws -run TestTelemetryEndpoint: passes (0.79s).errors.AsType[*apierr.APIError]matches withStatusCode == 429, error text "Too Many Requests", same as in CI.Dashboard
error/1132, integration/libs/telemetry.TestTelemetryEndpoint: 10 failures in 30 days (aws, azure), all
Received unexpected error: Too Many Requestsafter ~300s.This pull request and its description were written by Isaac.