Repository navigation
aitools: fix flaky TestRecordKeepsConcurrentOptOut lock wait - #7802
Merged
Merged
Conversation
…er lock wait The test starts 21 writers that take the state lock one at a time. Each waits at most stateLockWait (1s) for it; on slow runners (Windows, ~180ms per locked read and rename) the last writers give up with 'held by another CLI process'. Make stateLockWait a var and raise it in this test. Co-authored-by: Isaac <no-reply@databricks.com>
denik
marked this pull request as ready for review
November 19, 2026 12:03
denik
requested review from
Sankalp-Mittal,
abdelrahmanomar1515 and
andrewnester
November 19, 2026 12:03
andrewnester
approved these changes
Nov 19, 2026
Collaborator
❌ Integration test reportCommit: 3e3e02f
1169 interesting tests: 1143 MISS, 15 RECOVERED, 6 FAIL, 5 PANIC
Top 50 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
TestRecordKeepsConcurrentOptOutstarts 21 writers (20Record+ 1SetOptedOut) that take the state lock one at a time. Each writer waits at moststateLockWait(1s) for the lock. On slow runners, mostly Windows, each locked read-and-rename takes long enough that the last writers time out withheld by another CLI process. The failing Windows attempt ran for 3.83s, about 180ms per writer. The test was added in #7702.This PR makes
stateLockWaita package var and raises it to a minute in this test. Production behavior is unchanged: the wait is still 1s, which is plenty for the few CLI processes that contend for the lock in real use.Decision: I kept the 20 concurrent
Recordcalls. Using fewer writers would make the flake rarer without fixing it, and it would make the test less likely to catch a lost opt-out.Why
Flaky on main and in unrelated PRs: 12 failures in 3 days, 9 of them on Windows.
Tests
tryLockso each holder slept 60ms. It failed every run before this change (5 "held by another CLI process" errors per run) and passed after.go test -race -count=3 ./libs/aitools/autoinstall/passes.Dashboard
This pull request and its description were written by Isaac.