[evanh]

STREAM-2033

[linear-code]

STREAM-2033

[evanh]

$ cd clients/python && uv run pytest -q tests/worker/test_heartbeat.py tests/test_registry.py
35 passed in 0.56s

$ cd clients/python && uv run pytest -q
301 passed, 1 skipped
(a second run fails tests/test_metrics.py::test_track_memory_usage; it flakes the same way on main: "1 failed, 290 passed")

$ uv run mypy clients/python
Success: no issues found in 53 source files

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Heartbeat process ignores terminate signals

Medium Severity

The heartbeat child ignores SIGTERM, but shutdown and multiprocessing daemon cleanup both use terminate() (SIGTERM). That fallback is a no-op, so a child stuck in import_app or a gRPC call is not killed. Python then join()s that daemon on interpreter exit, which can hang worker shutdown until the child happens to finish.

Additional Locations (1)

• clients/python/src/taskbroker_client/worker/heartbeat.py#L162-L168

 

^{Reviewed by Cursor Bugbot for commit afcfb0a. Configure here.}

[evanh]

b7fa36a adds pull mode (heartbeats go to the broker host each task came from), starts the heartbeat process when the worker starts, and adds test_heartbeat to the integration tests and CI.

$ uv run python -m pytest integration_tests/integration_tests/test_heartbeat.py
1 passed in 19.67s

$ cd clients/python && uv run pytest -q
302 passed, 1 skipped, 1 failed (tests/test_metrics.py::test_track_memory_usage, flaky on main too)

$ uv run mypy clients/python
Success: no issues found in 53 source files

Manual run of the real heartbeat process (spawn, examples app, pull broker, 2s heartbeat):

after 8s: ('Processing', 0)
killed heartbeat process 11171
taskworker.worker.heartbeat.restarted
after kill + 6s: ('Processing', 0) new pid: 11219 alive: True
after untrack + 6s: ('Pending', 1)

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

There are 3 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit b7fa36a. Configure here.}

[cursor]

Pull worker heartbeats start too late

High Severity

TaskWorker.start kicks off the heartbeat process and then immediately fetches work. That process still has to import_app before it can renew anything, and the parent already has the app loaded so GetTask can succeed in the meantime. The broker starts the heartbeat clock at claim time, so with the default spawn context a slow import can miss the 10s minimum and reset tasks that are still running.

Additional Locations (2)

• clients/python/src/taskbroker_client/worker/heartbeat.py#L49-L50
• clients/python/src/taskbroker_client/worker/worker.py#L885-L886

 

^{Reviewed by Cursor Bugbot for commit b7fa36a. Configure here.}

[cursor]

Host heartbeats run one after another

Medium Severity

Pull-mode heartbeats send one host at a time, and each RPC can block for HEARTBEAT_RPC_TIMEOUT_SEC (5s). A hung broker delays renewals for every other host in the same loop. Two timeouts already exceed the 10s minimum heartbeat, so healthy brokers can miss deadlines and retry tasks that are still running.

Additional Locations (1)

• clients/python/src/taskbroker_client/worker/client.py#L43-L45

 

^{Reviewed by Cursor Bugbot for commit b7fa36a. Configure here.}