Skip to content

feat(metrics): account serial queue worker wall time by phase - #248

Open
iskakaushik wants to merge 1 commit into
mainfrom
kaushik/queue-worker-time-metrics
Open

iskakaushik wants to merge 1 commit into
mainfrom
kaushik/queue-worker-time-metrics

Conversation

@iskakaushik

@iskakaushik iskakaushik commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What

One counter family for where every serial worker spends its wall time.

walshadow_worker_seconds_total{worker,state}

  • worker: pump, queue, batcher, inserter, and during a bootstrap backup_tap, backup_walk
  • state: busy (own work), idle (waiting for input), blocked_in (waiting on a result it already requested), blocked_out (waiting to hand work downstream)

For pump, queue and batcher the states are disjoint and sum to about 1 s/s. inserter is summed over the pool and its busy includes the ClickHouse round trip, failed attempts included. backup_tap and backup_walk run once per backup part in parallel, and backup_tap busy includes its wait on backup_walk.

Pipeline rows and backup rows are published by different writers and merged, so a session that bootstrapped keeps both.

Breaking

Removed, now members of the family above:

removed now
walshadow_pump_queue_wait_seconds_total {worker="pump",state="blocked_out"}
walshadow_bootstrap_tap_seconds_total {worker="backup_tap",state="busy"}
walshadow_bootstrap_decode_seconds_total {worker="backup_walk",state="busy"}
walshadow_bootstrap_channel_block_seconds_total {worker="backup_walk",state="blocked_out"}

PromQL

# time budget per worker
sum by (worker, state) (rate(walshadow_worker_seconds_total[1m]))
# bottleneck: busy share near 1
sum by (worker) (rate(walshadow_worker_seconds_total{state="busy",worker=~"pump|queue|batcher"}[1m]))
# queue busy without TOAST fetches
sum(rate(walshadow_worker_seconds_total{worker="queue",state="busy"}[1m])) - sum(rate(walshadow_toast_value_fetch_seconds_total[1m]))

Tests

  • worker_clock_accounts_busy_and_idle, worker_clock_accounts_blocked_in
  • bootstrap_gauges_keep_pipeline_worker_rows
  • render_emits_plan_families_labelled

Also adds plans/observability.md, the metric naming plan the rest of the stack follows.

@iskakaushik
iskakaushik force-pushed the kaushik/queue-worker-time-metrics branch from 98d3902 to 250e704 Compare October 20, 2026 14:38
@harshil-goel
harshil-goel force-pushed the kaushik/queue-worker-time-metrics branch from 250e704 to 6001ea7 Compare November 10, 2026 18:13
@harshil-goel
harshil-goel marked this pull request as ready for review November 10, 2026 18:15
@harshil-goel
harshil-goel force-pushed the kaushik/queue-worker-time-metrics branch from 6001ea7 to ee26af8 Compare November 10, 2026 19:29
@harshil-goel
harshil-goel added this pull request to stack #313 November 10, 2026 19:39
@harshil-goel
harshil-goel force-pushed the kaushik/queue-worker-time-metrics branch 2 times, most recently from 0e6da93 to 14987ea Compare November 11, 2026 07:25
Comment thread plans/observability.md Outdated
## Open

- Debug exposition flag for the debug tier.
- `docker/grafana` dashboard: trim to a local-dev view or delete; the

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should be making this an in-tree reference on what metrics matter

@harshil-goel
harshil-goel force-pushed the kaushik/queue-worker-time-metrics branch 2 times, most recently from 1bcb709 to dfb1824 Compare November 11, 2026 08:58
@harshil-goel
harshil-goel removed this pull request from stack #313 November 11, 2026 09:23
@harshil-goel
harshil-goel added this pull request to stack #317 November 11, 2026 09:23
@harshil-goel
harshil-goel force-pushed the kaushik/queue-worker-time-metrics branch 3 times, most recently from b808a93 to f0867d3 Compare November 11, 2026 11:38
@harshil-goel
harshil-goel force-pushed the kaushik/queue-worker-time-metrics branch 2 times, most recently from 0f576dc to 8628f0f Compare November 18, 2026 05:11
@harshil-goel
harshil-goel force-pushed the kaushik/queue-worker-time-metrics branch from 8628f0f to bd16240 Compare November 18, 2026 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants