[evanh]

Right now when a shutdown on a worker is triggered, the remaining child processes will get killed after 5 seconds (roughly, they get killed sequentially). For any task that regularly takes more than 5 seconds, it is getting killed every time a shutdown happens on the pod (deploys, node balancing etc.)

This was done partly to ensure that all dropped/drained tasks were properly tracked, but it comes at the cost of dropped tasks. With Launchpad and Seer having more need of longer running tasks, this becomes a more serious issue.

Refactor the taskworker shutdown to wait as long as possible for tasks to drain. This comes at the cost of not necessarily reporting all dropped tasks but means that controlling how long to wait is with a single configuration: terminationGrace in k8s.

[evanh]

$ pytest tests/worker/test_worker.py -q -k "shutdown or spawn_children"
27 passed, 89 deselected in 4.66s

$ pytest tests -q
FAILED tests/test_metrics.py::test_track_memory_usage - assert 0 > 0
1 failed, 290 passed, 1 skipped

test_track_memory_usage fails the same way on main in a full run, and passes when run alone.