Skip to content

feat(taskworker): Count child startup as busy in occupancy - #844

Closed
evanh wants to merge 1 commit into
mainfrom
evanh/feat/occupancy-startup-busy
Closed

evanh wants to merge 1 commit into
mainfrom
evanh/feat/occupancy-startup-busy

Conversation

@evanh

@evanh evanh commented Nov 4, 2026

Copy link
Copy Markdown
Member

A warming child can't take work, but occupancy left it out of both busy time and the ceiling, so respawns shrank capacity without moving the metric.

🤖 Generated with Claude Code

A warming child can't take work, but occupancy left it out of both busy time and the ceiling, so respawns
shrank capacity without moving the metric. The parent now bills spawn-to-running as busy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@evanh
evanh requested a review from a team as a code owner November 4, 2026 19:29
@evanh

evanh commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Why this does not bill pod startup as busy (and can't feed a scale-up loop):

  • _should_publish_occupancy requires _received_task, which is only set in push_task after a task lands on the child queue (worker.py:1455). Until then the gauge is never set, for statsd or Prometheus.
  • A new pod holds gRPC health at NOT_SERVING until every child sends running (_await_children_warm, called from start). The broker does not route to it, so no task arrives while its children are pending.
  • By the time the gate opens, every child has already moved to reading its shared-memory slot, so the startup busy time is never published. A new pod first reports occupancy with all children warm and idle, which pulls the pool average down.

So the new busy time only shows up for respawns mid-life (max_task_count, crashes, deadline kills), when real capacity is lost.

One edge: if warmup hits warmup_timeout, the pod serves anyway with some children still pending, and once a task arrives those children read busy. That is real lost capacity, it only affects children slower than the timeout, and taskworker.worker.warmup_timeout already flags it. Startup would have to exceed the timeout on every new pod for it to compound.

Covered by test_emit_periodic_metrics_skips_occupancy_during_warmup and test_occupancy_withheld_until_first_task.

@evanh
evanh marked this pull request as draft November 4, 2026 19:32
@evanh

evanh commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

This is not actually needed, the problem is probably how Envoy is removing workers from the pool.

@evanh evanh closed this Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant