Repository navigation
test(repo): the Ctrl-C test waits for the interrupt to reach the main thread, not for time - #875
antonwolfy wants to merge 3 commits into
Conversation
… thread, not for time (intel#871) The driver sent SIGINT from rep 1 and let both running cells go on at once. `_map_cells` cancels the queued cells only when the main thread handles the KeyboardInterrupt, so on a busy runner the worker that finished rep 2 took rep 3 (and 4, 5) from the queue, which have no barrier, and they finished: the test listed r1..r5 instead of r1, r2. The two running cells now wait, after the SIGINT, until `ThreadPoolExecutor.shutdown(cancel_futures=True)` has returned in the main thread, which is the call that cancels the queue. No worker is free before then, so the result no longer depends on how fast the main thread handles the signal. The assertions are unchanged. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s). CI at Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #875 +/- ##
=======================================
Coverage 95.79% 95.80%
=======================================
Files 429 429
Lines 56821 56821
Branches 9783 9783
=======================================
+ Hits 54433 54435 +2
+ Misses 1485 1484 -1
+ Partials 903 902 -1 🚀 New features to boost your workflow:
|
…intel#871) Settles the failed `Test Linux Wheels Python 3.11` check: its traceback shows the KeyboardInterrupt raised inside `ThreadPoolExecutor.submit`, while the main thread waited for a worker thread to start, and the child exited 1 instead of by SIGINT. The driver now wraps `ThreadPoolExecutor.map`, which returns only after every cell is submitted and its threads started, and the first cell sends SIGINT after that. The interrupt then always reaches the main thread in the wait for results, the case the test is about. If the submission never finishes, the child exits 3 and the test fails. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at c6813e0. Waiting for the cancel instead of for time is right, and shutdown(wait=False, ...) in _map_cells means the patched shutdown cannot deadlock the workers. One race is left, in the line comment: the interrupt can still arrive while the pool is starting the second worker, which is how this PR's own wheel job failed.
What I ran, at c6813e0:
make ci-local --offline: 22 passed, 1 failed, 20 skipped; the test suite passes, this test included. The failure isschema_compatagainst the PR base, which comes frommaindrift, not from this test-only PR.
Checked, no issues
- Patched
shutdown: the onlycancel_futures=Truecall isrunner.py:518, withwait=False, socancelledis set right after the queue is cancelled. Thewithexit calls it withoutcancel_futures, so that does not set it.
@vlad-perevezentsev (#875 (review)): Nothing to change. The one race it names is the same as the line comment above and is closed by 052829a. The |
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
vchamarthi
left a comment
There was a problem hiding this comment.
🟢 Ready to approve · reviewed e223876
Next: nothing required from @antonwolfy. 1 nit optional.
Verified on e223876: ✅ changed tests: 14 passed, pass on main too · ❌ full suite with main merged in fails at tests/test_fuzz_targets.py::test_a_bare_deep_reply_is_a_parse_error_not_a_crash[extract_json_array-text0], ci.yml:test / Run tests · ✅ merges onto main · 2 mutations tried
3 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Ctrl-c test | The 14 tests in test_repo_batch_lifecycle.py pass on the head, including the Ctrl-C test that pins exit code 130 or -SIGINT, KeyboardInterrupt on stderr, and exactly r1 and r2 finished |
tests on the head |
| Sigint after submission | Removing submitted.set() (line 285) fails the Ctrl-C test: the first cell waits its 30 s and the child exits 3, so the test pins the interrupt to follow submission |
broke the code, tests caught it |
| Merge | The PR merges onto current main with no conflict | merge onto main |
| if not submitted.wait(30): # an interrupt inside submit() is a slow thread start, not the Ctrl-C under test | ||
| os._exit(3) | ||
| signal.pthread_kill(threading.main_thread().ident, signal.SIGINT) | ||
| cancelled.wait(30) # no worker is free to take a queued cell until the main thread has cancelled them |
There was a problem hiding this comment.
🟡 Nit: Removing cancelled.set() leaves the Ctrl-C test green: the running cells then wait out their 30 s timeout.
Input: delete cancelled.set() (line 280). On the head the test still passes, 1 passed in 35 s, because the two running cells sit out the 30 s timeout on line 295 and then finish. Expected: a failure, since the release is meant to come from the queue being cancelled, not from the timer.
Fix, run on e223876: the tests now catch the broken variant; tests and lint are no worse.
| cancelled.wait(30) # no worker is free to take a queued cell until the main thread has cancelled them | |
| if not cancelled.wait(30): # no worker is free to take a queued cell until the main thread has cancelled them | |
| os._exit(4) |
Summary
tests/test_repo_batch_lifecycle.py::test_ctrl_c_makes_the_repo_run_process_exit_nonzerois flaky._map_cellscancels the queued cells only when the main thread handles the KeyboardInterrupt (pool.shutdown(wait=False, cancel_futures=True),repos/runner.py). The test driver sent SIGINT from rep 1 and let both running cells go on at once, so when the main thread handled the signal late, the worker that finished rep 2 took rep 3 (and 4, 5) from the queue, which have no barrier, and they finished. Onmain, with the driver's handler sleeping 1 s before it raises (a busy runner), the test fails withassert ['fix_add__gu...raw__m-1__r5'] == ['fix_add__gu...raw__m-1__r2']: all five cells finished.The two running cells now wait, after the SIGINT, until
ThreadPoolExecutor.shutdown(cancel_futures=True)has returned in the main thread, i.e. until the queue is cancelled. This is a stronger wait than "the handler has run": a worker could still pick up r3 between the handler and theexceptbranch. The assertions are unchanged.The same wait on time is in
test_ctrl_c_starts_no_queued_cell, tracked in #874.Closes #871
Changes
dokimi/tests/test_repo_batch_lifecycle.py(_CTRL_C_DRIVER): the driver wrapsThreadPoolExecutor.shutdownto set an event once acancel_futures=Truecall has returned, and the rep 1 and rep 2 cells wait on it (30 s) before they run.ThreadPoolExecutor.map, which returns once every cell is submitted and its threads started, and rep 1 sends SIGINT only after that (it exits 3 if that takes over 30 s). On the Python 3.11 wheel job the interrupt had landed insideThreadPoolExecutor.submit, while the main thread waited for a worker thread to start, and the child exited 1. An interrupt during submission is covered bytest_an_interrupt_during_submission_starts_no_queued_cell.dokimi/dokimi/, no changelog fragment (tests only).Risk
None for users: tests only. The test still checks the exit code is 130 or -SIGINT, that
KeyboardInterruptis in stderr, and that exactly r1 and r2 finished; the assertions and the timeout are unchanged.Testing
On a Linux review host, at
e223876(onmainea458d2):make ci-local: 34 passed, 0 failed, 14 skipped (steps needing a CI-only matrix, micromamba, semgrep, fuzz, deploy). The suite step: 22648 passed, 19 skipped.test_repo_batch_lifecycle.py, 150 copies of the file at once under-n 48on Python 3.13: 300 passed. A Python 3.11 CI run of this test failed with r3 finished (doneheld r1, r2 and r3); that does not reproduce here.main's test with the driver's SIGINT handler sleeping 1 s:1 failed, 1 passed, 12 deselected,assert [...r5] == [...r2].run_cellwith a handler that sleeps 3 s: fails in 6 of 8 runs on the same final assertion (Left contains 3 more items, first extra item: r3); in 2 runs it still passed, so it shows the wait is needed but does not catch the flake on every run.submit), the child's exit is -SIGINT before and after the change; the Python 3.11 exit 1 itself was not reproduced.submitted.set()removed from themapwrapper: the test fails (the child exits 3).Open questions
None.
Revert
A plain revert restores the old driver, which can fail on a slow runner. Nothing is stored.
Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)