Skip to content

feat(metrics): source server WAL end and in-process source lag - #252

Open
iskakaushik wants to merge 1 commit into
kaushik/shadow-stream-metricsfrom
kaushik/source-lag-metrics
Open

iskakaushik wants to merge 1 commit into
kaushik/shadow-stream-metricsfrom
kaushik/source-lag-metrics

Conversation

@iskakaushik

@iskakaushik iskakaushik commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What

Every WAL position in one family, published in one status tick, so any two subtract without scrape skew.

walshadow_position_lsn{at}

at meaning
source_head newest server_wal_end heard on the source socket. Present only while the pump is streaming: absent during archive catch-up, while reconnecting, and for the first tick after a reconnect or endpoint swap
received source head floored at the filter dispatch frontier
filter last dispatched filtered byte
decoder_commit highest commit drained
ack ClickHouse watermark
shadow_replay shadow Postgres replay position
floor durable resume floor
pause_consumed, pause_received frontiers frozen by a pause
promotion_target_replay, promotion_target_receive target positions read while paused
timeline_switch fork of the last crossing

A member is absent while its position is unknown.

Breaking

Removed, now members of the family above: walshadow_source_received_lsn, walshadow_filter_lsn, walshadow_decoder_commit_lsn, walshadow_emitter_ack_lsn, walshadow_shadow_replay_lsn, walshadow_floor_lsn, walshadow_timeline_switch_lsn, walshadow_pause_consumed_lsn, walshadow_pause_received_lsn.

Subtraction across members needs ignoring(at).

PromQL

# backlog from the source head to the ClickHouse ack, no scrape skew
walshadow_position_lsn{at="source_head"} - ignoring(at) walshadow_position_lsn{at="ack"}
# same, still drawn during archive catch-up
(walshadow_position_lsn{at="source_head"} or ignoring(at) walshadow_position_lsn{at="received"}) - ignoring(at) walshadow_position_lsn{at="ack"}
# received but not yet decoded
walshadow_position_lsn{at="received"} - ignoring(at) walshadow_position_lsn{at="decoder_commit"}

The bundled Grafana dashboard under docker/grafana is updated to the new family.

Tests

  • render_omits_unknown_positions, source_server_wal_end_is_absent_until_heard
  • render_emits_shadow_apply_lag_lines and the serve test assert walshadow_position_lsn{at=...}

@harshil-goel
harshil-goel force-pushed the kaushik/source-lag-metrics branch from d4c6394 to 9ca4404 Compare November 10, 2026 18:13
@harshil-goel
harshil-goel marked this pull request as ready for review November 10, 2026 18:15
@harshil-goel
harshil-goel force-pushed the kaushik/source-lag-metrics branch from 9ca4404 to cfec66c Compare November 10, 2026 19:30
@harshil-goel
harshil-goel changed the base branch from main to kaushik/shadow-stream-metrics November 10, 2026 19:38
@harshil-goel
harshil-goel added this pull request to stack #313 November 10, 2026 19:39
@harshil-goel
harshil-goel force-pushed the kaushik/source-lag-metrics branch 2 times, most recently from 6e58712 to d9e8988 Compare November 10, 2026 21:31
@harshil-goel
harshil-goel force-pushed the kaushik/source-lag-metrics branch from d9e8988 to 0ef7c2f Compare November 11, 2026 07:26
@harshil-goel
harshil-goel force-pushed the kaushik/source-lag-metrics branch from 0ef7c2f to 01e172b Compare November 11, 2026 07:48
@harshil-goel
harshil-goel force-pushed the kaushik/source-lag-metrics branch from 01e172b to 6d0b6ca Compare November 11, 2026 08:58
@harshil-goel
harshil-goel removed this pull request from stack #313 November 11, 2026 09:23
@harshil-goel
harshil-goel added this pull request to stack #317 November 11, 2026 09:23
@harshil-goel
harshil-goel force-pushed the kaushik/source-lag-metrics branch from 6d0b6ca to 928f98c Compare November 11, 2026 09:32
@harshil-goel
harshil-goel force-pushed the kaushik/source-lag-metrics branch from 928f98c to 911e04b Compare November 11, 2026 09:45
@harshil-goel
harshil-goel force-pushed the kaushik/source-lag-metrics branch from 911e04b to d670aeb Compare November 11, 2026 11:38
@harshil-goel
harshil-goel force-pushed the kaushik/source-lag-metrics branch from d670aeb to 28dcd5f Compare November 17, 2026 10:08
@harshil-goel
harshil-goel force-pushed the kaushik/source-lag-metrics branch from 28dcd5f to c19797d Compare November 18, 2026 05:11
@harshil-goel
harshil-goel force-pushed the kaushik/source-lag-metrics branch from c19797d to 6ae75f3 Compare November 18, 2026 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant