Skip to content

feat(metrics): source mode state and stream error counters - #250

Open
iskakaushik wants to merge 1 commit into
kaushik/queue-worker-time-metricsfrom
kaushik/source-mode-metrics
Open

iskakaushik wants to merge 1 commit into
kaushik/queue-worker-time-metricsfrom
kaushik/source-mode-metrics

Conversation

@iskakaushik

@iskakaushik iskakaushik commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What

What the pump is doing, and why the source stream was lost.

walshadow_source_mode{mode}: 1 for the current mode, 0 for the rest.

mode meaning
streaming on the source replication socket
archive catching up from archived WAL
reconnecting redialing a lost source
crossing crossing timelines, the wait at the switch included
paused paused by an operator, outranks archive and crossing
initial_load set by the bootstrap, later in the stack

walshadow_errors_total{component="source",kind}: one per lost stream.

kind meaning
wal_removed SQLSTATE 58P01, counted on the live stream, at start and on a redial
disconnect I/O error, peer close, or receive timeout
walsender_exit the source walsender ended the stream cleanly
other anything else

PromQL

# state timeline
walshadow_source_mode == 1
# stream losses by cause
sum by (kind) (increase(walshadow_errors_total{component="source"}[1h]))
# shadows not streaming
max by (ubi_postgres_wal_shadow_ubid) (walshadow_source_mode{mode=~"archive|reconnecting"}) == 1

Notes

  • Mode is sampled once per pump iteration, so a flap shorter than a scrape is not visible.
  • A failed redial that is not a removed-WAL error is not counted, only the loss that started it.

Tests

  • mode_precedence_and_error_classes
  • render_emits_source_mode_families

@harshil-goel
harshil-goel force-pushed the kaushik/source-mode-metrics branch from 9ad2147 to f33b60f Compare November 10, 2026 18:13
@harshil-goel
harshil-goel marked this pull request as ready for review November 10, 2026 18:15
@harshil-goel
harshil-goel force-pushed the kaushik/source-mode-metrics branch from f33b60f to 0a8fae2 Compare November 10, 2026 19:30
@harshil-goel
harshil-goel changed the base branch from main to kaushik/queue-worker-time-metrics November 10, 2026 19:38
@harshil-goel
harshil-goel added this pull request to stack #313 November 10, 2026 19:39
@harshil-goel
harshil-goel force-pushed the kaushik/source-mode-metrics branch from 0a8fae2 to 7ba9dfd Compare November 10, 2026 20:22
@harshil-goel
harshil-goel force-pushed the kaushik/source-mode-metrics branch from 7ba9dfd to 8cd7aba Compare November 11, 2026 07:25
@harshil-goel
harshil-goel force-pushed the kaushik/source-mode-metrics branch from 8cd7aba to 33a24d8 Compare November 11, 2026 07:47
@harshil-goel
harshil-goel force-pushed the kaushik/source-mode-metrics branch from 33a24d8 to c7d463b Compare November 11, 2026 08:58
@harshil-goel
harshil-goel removed this pull request from stack #313 November 11, 2026 09:23
@harshil-goel
harshil-goel added this pull request to stack #317 November 11, 2026 09:23
@harshil-goel
harshil-goel force-pushed the kaushik/source-mode-metrics branch from c7d463b to d8da496 Compare November 11, 2026 09:31
@harshil-goel
harshil-goel force-pushed the kaushik/source-mode-metrics branch from d8da496 to cf5bbbc Compare November 11, 2026 09:44
@harshil-goel
harshil-goel force-pushed the kaushik/source-mode-metrics branch from cf5bbbc to d05fa11 Compare November 11, 2026 11:38
@harshil-goel
harshil-goel force-pushed the kaushik/source-mode-metrics branch from d05fa11 to b5aa20e Compare November 17, 2026 10:08
@harshil-goel
harshil-goel force-pushed the kaushik/source-mode-metrics branch from b5aa20e to 120c3c7 Compare November 18, 2026 05:11
@harshil-goel
harshil-goel force-pushed the kaushik/source-mode-metrics branch from 120c3c7 to f0744a6 Compare November 18, 2026 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants