Skip to content

Report the pipelines test wait timeout when it fails a read before it is sent - #7820

Merged
denik merged 2 commits into
mainfrom
generr-wait-timeout
Nov 19, 2026
Merged

denik merged 2 commits into
mainfrom
generr-wait-timeout

Conversation

@denik

@denik denik commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Changes

In waitForTestUpdate (databricks pipelines test), a read that fails with context.Canceled now also carries the wait context's cause (errTestWaitTimeout). The read's own error stays wrapped. Other read errors are returned unchanged, even if the context was canceled by then. The command's caller still checks context.Cause as a fallback.

Why

TestWaitForTestUpdateTimeout failed once on main on Windows:

Target error should be in err chain:
expected: "test wait timed out"
in chain: "failed to fetch pipeline events: failed in rate limiter: context canceled"

The 20ms timeout can pass between two reads. Windows timers are coarse, so this happens there. A request that is already in flight gets the cancel cause from net/http. But the SDK rate limiter checks the context before sending, and it fails with a bare context canceled, which hides the timeout. The command itself reported this case correctly, because it checked context.Cause separately. The function didn't, so the test was flaky.

This is not a regression from #7800: that PR changed pipelines run and bundle run (cmd/pipelines/run.go, bundle/run/). It did not change pipelines test's wait loop (cmd/pipelines/test.go, test_events.go). The flaky test came with #7426.

Tests

New TestWaitReadError covers: a real error is kept, and a context error gets the cause and keeps its message. New TestWaitForTestUpdateTimeoutBeforeRead: the wait context is already canceled with the timeout cause, so the first read fails in the rate limiter. Without the fix it fails with the same message as on CI; with the fix it passes. The other TestWaitForTestUpdate* tests and the pipelines acceptance tests pass.

Decision: this does not change one existing behavior. If the timeout fires during the final events read after the update has already ended, the run is still reported as timed out. That is unchanged from main and rare, since it needs the timeout to hit that exact read.

Dashboard

error verbatim snippet
error/3790 Target error should be in err chain: expected "test wait timed out", in chain: "failed to fetch pipeline events: failed in rate limiter: context canceled"

Test: cmd/pipelines.TestWaitForTestUpdateTimeout

This pull request and its description were written by Isaac.

… is sent

The SDK rate limiter fails a request whose context is already canceled with a
bare "context canceled", so a timeout that passed between two reads surfaced
as "failed to fetch pipeline events: failed in rate limiter: context canceled"
instead of the wait timeout. Return the context's cause for a read that fails
after the wait was canceled.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 19, 2026
… check

Only add the wait's cause to a context-canceled read error, wrapping it so
the read's own message stays, and keep the caller's context.Cause check as a
fallback for any read path that does not go through waitReadError.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik marked this pull request as ready for review November 19, 2026 17:08
@denik
denik requested review from a team as code owners November 19, 2026 17:08

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me — no issues found.

@denik
denik added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit b8046e0 Nov 19, 2026
26 checks passed
@denik
denik deleted the generr-wait-timeout branch November 19, 2026 17:49
@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants