Skip to content

feat(shadow-stream): walsender connection, backfill and queue metrics - #253

Open
iskakaushik wants to merge 1 commit into
kaushik/source-mode-metricsfrom
kaushik/shadow-stream-metrics
Open

iskakaushik wants to merge 1 commit into
kaushik/source-mode-metricsfrom
kaushik/shadow-stream-metrics

Conversation

@iskakaushik

@iskakaushik iskakaushik commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What

Visibility into walshadow's walsender, the socket the shadow Postgres streams from.

metric meaning
walshadow_shadow_stream_sockets_total{event} socket lifecycle: accepted, closed_slow_client (dropped at the send queue cap), closed_by_client (clean hangup), closed_error (handshake, I/O, protocol failure or refused START_REPLICATION). Each socket end is counted once
walshadow_wal_bytes_total{flow="walsender_backfill"} WAL bytes replayed from the retained buffer to reconnecting clients
walshadow_buffer_bytes{buffer="walsender_send_queue"} largest unsent queue across connections, reconnect backfill left out, to read against the slow-client cap

A START_REPLICATION below the retained buffer is refused with the removed-segment SQLSTATE, so the walreceiver takes that range from restore_command.

Breaking

walshadow_shadow_stream_dropped_connections_total is removed. Use walshadow_shadow_stream_sockets_total{event="closed_slow_client"}.

PromQL

# reconnect churn by cause
sum by (event) (rate(walshadow_shadow_stream_sockets_total{event!="accepted"}[10m]))
# sockets alive
sum without (event) (walshadow_shadow_stream_sockets_total{event="accepted"}) - sum without (event) (walshadow_shadow_stream_sockets_total{event!="accepted"})
# reconnects served from the buffer
rate(walshadow_wal_bytes_total{flow="walsender_backfill"}[10m])
# slow-client pressure
walshadow_buffer_bytes{buffer="walsender_send_queue"}

Tests

  • backfill_is_counted_and_starts_below_the_buffer_are_refused
  • send_queue_bytes_is_the_largest_queue, send_queue_bytes_leaves_out_the_reconnect_backfill
  • socket_events_count_per_kind
  • unix_walsender_drops_a_slow_client asserts the closed_slow_client event

serprex
serprex previously approved these changes Nov 4, 2026
@harshil-goel
harshil-goel force-pushed the kaushik/shadow-stream-metrics branch 2 times, most recently from 12caab7 to 540bcea Compare November 10, 2026 19:30
@harshil-goel
harshil-goel changed the base branch from main to kaushik/source-mode-metrics November 10, 2026 19:38
@harshil-goel
harshil-goel added this pull request to stack #313 November 10, 2026 19:39
@harshil-goel
harshil-goel marked this pull request as ready for review November 10, 2026 19:39
@harshil-goel
harshil-goel force-pushed the kaushik/shadow-stream-metrics branch from 540bcea to 5d72bb6 Compare November 10, 2026 20:22
@harshil-goel
harshil-goel force-pushed the kaushik/shadow-stream-metrics branch from 5d72bb6 to cc9e4f7 Compare November 11, 2026 07:25
@harshil-goel
harshil-goel force-pushed the kaushik/shadow-stream-metrics branch from cc9e4f7 to 6699560 Compare November 11, 2026 07:48
@harshil-goel
harshil-goel force-pushed the kaushik/shadow-stream-metrics branch from 6699560 to 218bae8 Compare November 11, 2026 08:58
@harshil-goel
harshil-goel removed this pull request from stack #313 November 11, 2026 09:23
@harshil-goel
harshil-goel added this pull request to stack #317 November 11, 2026 09:23
@harshil-goel
harshil-goel force-pushed the kaushik/shadow-stream-metrics branch from 218bae8 to d0c7b8d Compare November 11, 2026 09:31
@harshil-goel
harshil-goel force-pushed the kaushik/shadow-stream-metrics branch from d0c7b8d to f4bdacf Compare November 11, 2026 09:45
@harshil-goel
harshil-goel force-pushed the kaushik/shadow-stream-metrics branch from f4bdacf to 9537f42 Compare November 11, 2026 11:38
@harshil-goel
harshil-goel force-pushed the kaushik/shadow-stream-metrics branch from 9537f42 to 86053fb Compare November 17, 2026 10:08
@harshil-goel
harshil-goel force-pushed the kaushik/shadow-stream-metrics branch from 86053fb to 316aee5 Compare November 18, 2026 05:11
@harshil-goel
harshil-goel force-pushed the kaushik/shadow-stream-metrics branch from 316aee5 to 4b52719 Compare November 18, 2026 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants