Skip to content

test(repo): the Ctrl-C test waits for the interrupt to reach the main thread, not for time - #875

Open
antonwolfy wants to merge 3 commits into
intel:mainfrom
antonwolfy:test/ctrl-c-flaky
Open

antonwolfy wants to merge 3 commits into
intel:mainfrom
antonwolfy:test/ctrl-c-flaky

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

tests/test_repo_batch_lifecycle.py::test_ctrl_c_makes_the_repo_run_process_exit_nonzero is flaky. _map_cells cancels the queued cells only when the main thread handles the KeyboardInterrupt (pool.shutdown(wait=False, cancel_futures=True), repos/runner.py). The test driver sent SIGINT from rep 1 and let both running cells go on at once, so when the main thread handled the signal late, the worker that finished rep 2 took rep 3 (and 4, 5) from the queue, which have no barrier, and they finished. On main, with the driver's handler sleeping 1 s before it raises (a busy runner), the test fails with assert ['fix_add__gu...raw__m-1__r5'] == ['fix_add__gu...raw__m-1__r2']: all five cells finished.

The two running cells now wait, after the SIGINT, until ThreadPoolExecutor.shutdown(cancel_futures=True) has returned in the main thread, i.e. until the queue is cancelled. This is a stronger wait than "the handler has run": a worker could still pick up r3 between the handler and the except branch. The assertions are unchanged.

The same wait on time is in test_ctrl_c_starts_no_queued_cell, tracked in #874.

Closes #871

Changes

  • dokimi/tests/test_repo_batch_lifecycle.py (_CTRL_C_DRIVER): the driver wraps ThreadPoolExecutor.shutdown to set an event once a cancel_futures=True call has returned, and the rep 1 and rep 2 cells wait on it (30 s) before they run.
  • The same driver wraps ThreadPoolExecutor.map, which returns once every cell is submitted and its threads started, and rep 1 sends SIGINT only after that (it exits 3 if that takes over 30 s). On the Python 3.11 wheel job the interrupt had landed inside ThreadPoolExecutor.submit, while the main thread waited for a worker thread to start, and the child exited 1. An interrupt during submission is covered by test_an_interrupt_during_submission_starts_no_queued_cell.
  • No change to dokimi/dokimi/, no changelog fragment (tests only).

Risk

None for users: tests only. The test still checks the exit code is 130 or -SIGINT, that KeyboardInterrupt is in stderr, and that exactly r1 and r2 finished; the assertions and the timeout are unchanged.

Testing

On a Linux review host, at e223876 (on main ea458d2):

  • make ci-local: 34 passed, 0 failed, 14 skipped (steps needing a CI-only matrix, micromamba, semgrep, fuzz, deploy). The suite step: 22648 passed, 19 skipped.
  • The two Ctrl-C tests of test_repo_batch_lifecycle.py, 150 copies of the file at once under -n 48 on Python 3.13: 300 passed. A Python 3.11 CI run of this test failed with r3 finished (done held r1, r2 and r3); that does not reproduce here.
  • On main's test with the driver's SIGINT handler sleeping 1 s: 1 failed, 1 passed, 12 deselected, assert [...r5] == [...r2].
  • With the fix and the same slow handler: 8 of 8 runs pass.
  • Mutant, the wait removed from run_cell with a handler that sleeps 3 s: fails in 6 of 8 runs on the same final assertion (Left contains 3 more items, first extra item: r3); in 2 runs it still passed, so it shows the wait is needed but does not catch the flake on every run.
  • With a thread start that sleeps 3 s (the interrupt inside submit), the child's exit is -SIGINT before and after the change; the Python 3.11 exit 1 itself was not reproduced.
  • Mutant, submitted.set() removed from the map wrapper: the test fails (the child exits 3).
  • Not run: the other Python versions (3.11, 3.12, 3.14).

Open questions

None.

Revert

A plain revert restores the old driver, which can fail on a slow runner. Nothing is stored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

… thread, not for time (intel#871)

The driver sent SIGINT from rep 1 and let both running cells go on at once. `_map_cells` cancels the queued cells only when the main thread handles the KeyboardInterrupt, so on a busy runner the worker that finished rep 2 took rep 3 (and 4, 5) from the queue, which have no barrier, and they finished: the test listed r1..r5 instead of r1, r2.

The two running cells now wait, after the SIGINT, until `ThreadPoolExecutor.shutdown(cancel_futures=True)` has returned in the main thread, which is the call that cancels the queue. No worker is free before then, so the result no longer depends on how fast the main thread handles the signal. The assertions are unchanged.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s).

CI at e223876: 23 passing, 2 skipped.

Review details
  • Commit reviewed: e223876 against merge base ea458d2
  • Coverage: 1 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 50k tokens (36k from cache), 11s
  • Tool calls: 4 (2 file_read, 1 ci_status, 1 code_search)
  • dokimi/tests/test_repo_batch_lifecycle.py: 1 file

@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.80%. Comparing base (ea458d2) to head (e223876).
⚠️ Report is 3 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #875   +/-   ##
=======================================
  Coverage   95.79%   95.80%           
=======================================
  Files         429      429           
  Lines       56821    56821           
  Branches     9783     9783           
=======================================
+ Hits        54433    54435    +2     
+ Misses       1485     1484    -1     
+ Partials      903      902    -1     

see 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…intel#871)

Settles the failed `Test Linux Wheels Python 3.11` check: its traceback shows the KeyboardInterrupt raised inside `ThreadPoolExecutor.submit`, while the main thread waited for a worker thread to start, and the child exited 1 instead of by SIGINT.

The driver now wraps `ThreadPoolExecutor.map`, which returns only after every cell is submitted and its threads started, and the first cell sends SIGINT after that. The interrupt then always reaches the main thread in the wait for results, the case the test is about. If the submission never finishes, the child exits 3 and the test fails.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at c6813e0. Waiting for the cancel instead of for time is right, and shutdown(wait=False, ...) in _map_cells means the patched shutdown cannot deadlock the workers. One race is left, in the line comment: the interrupt can still arrive while the pool is starting the second worker, which is how this PR's own wheel job failed.

What I ran, at c6813e0:

  • make ci-local --offline: 22 passed, 1 failed, 20 skipped; the test suite passes, this test included. The failure is schema_compat against the PR base, which comes from main drift, not from this test-only PR.

Checked, no issues

  • Patched shutdown: the only cancel_futures=True call is runner.py:518, with wait=False, so cancelled is set right after the queue is cancelled. The with exit calls it without cancel_futures, so that does not set it.

@antonwolfy

Copy link
Copy Markdown
Contributor Author

(commented) Review at c6813e0. Waiting for the cancel instead of for time is right, and `shutdown(wait=Fals

@vlad-perevezentsev (#875 (review)): Nothing to change. The one race it names is the same as the line comment above and is closed by 052829a. The schema_compat failure in make ci-local it mentions is from main drift, which this test-only PR does not touch.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Ready to approve · reviewed e223876

Next: nothing required from @antonwolfy. 1 nit optional.

Verified on e223876: ✅ changed tests: 14 passed, pass on main too · ❌ full suite with main merged in fails at tests/test_fuzz_targets.py::test_a_bare_deep_reply_is_a_parse_error_not_a_crash[extract_json_array-text0], ci.yml:test / Run tests · ✅ merges onto main · 2 mutations tried

3 checks behind it, each with its evidence
Area What holds How
Ctrl-c test The 14 tests in test_repo_batch_lifecycle.py pass on the head, including the Ctrl-C test that pins exit code 130 or -SIGINT, KeyboardInterrupt on stderr, and exactly r1 and r2 finished tests on the head
Sigint after submission Removing submitted.set() (line 285) fails the Ctrl-C test: the first cell waits its 30 s and the child exits 3, so the test pins the interrupt to follow submission broke the code, tests caught it
Merge The PR merges onto current main with no conflict merge onto main

if not submitted.wait(30): # an interrupt inside submit() is a slow thread start, not the Ctrl-C under test
os._exit(3)
signal.pthread_kill(threading.main_thread().ident, signal.SIGINT)
cancelled.wait(30) # no worker is free to take a queued cell until the main thread has cancelled them

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: Removing cancelled.set() leaves the Ctrl-C test green: the running cells then wait out their 30 s timeout.

Input: delete cancelled.set() (line 280). On the head the test still passes, 1 passed in 35 s, because the two running cells sit out the 30 s timeout on line 295 and then finish. Expected: a failure, since the release is meant to come from the queue being cancelled, not from the timer.

Fix, run on e223876: the tests now catch the broken variant; tests and lint are no worse.

Suggested change
cancelled.wait(30) # no worker is free to take a queued cell until the main thread has cancelled them
if not cancelled.wait(30): # no worker is free to take a queued cell until the main thread has cancelled them
os._exit(4)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky on CI: tests/test_repo_batch_lifecycle.py::test_ctrl_c_makes_the_repo_run_process_exit_nonzero

5 participants