Repository navigation
Conversation
jamieQ
marked this pull request as ready for review
November 10, 2026 14:02
Member
|
One thought: if a child process gets hung for some reason after being shutdown, that process won't ever be replaced. It might be worth adding a new timeout or some failsafe to catch that case. |
Count retiring children toward concurrency until they exit and are reaped when wait_for_child_exit is enabled. Expose the option in both push and pull workers, preserving the existing default behavior.
Record when child exit permission is granted and emit the oldest exiting child age in seconds per pool and pod. Report in both recycling modes, with zero when no children are exiting, so prolonged exits can be monitored without a hardcoded threshold or termination policy.
jamieQ
force-pushed
the
fix/wait-for-child-exit
branch
from
November 11, 2026 15:35
378418f to
9ed5292
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Currently the
concurrencysetting for task workers does not provide a strict cap on the number of activations that may be executing concurrently. In particular, when a child worker requests to exit, it does so asynchronously. Due to the asynchrony, after requesting to exit from its parent, the child worker may still pick up new work. However, once in this state, the parent no longer counts the child toward theconcurrencylimit. Thus we can effectively have more work "in flight" than one might naively think possible.This behavior is sort of problematic for the launchpad service. Launchpad processes potentially-large native app binaries & resources, and its tasks use a fair amount of memory. It was provisioned under the assumption that setting
concurrencyto2meant at most 2 apps would be processed on a pod at a time. Due to the implementation details outlined above, however, sometimes more apps would be processed simultaneously, which occasionally seems to have led pods to OOM.Changes
Add a new
wait_for_child_exitconfiguration option to control child worker process recycling behavior. When this new value is set to true (defaults to false), exiting workers will still count toward theconcurrencylimit until they exit.