Skip to content

feat(metrics): commit-time lag gauges for ack and source - #256

Open
iskakaushik wants to merge 1 commit into
kaushik/bridge-latency-histogramsfrom
kaushik/ack-commit-lag
Open

iskakaushik wants to merge 1 commit into
kaushik/bridge-latency-histogramsfrom
kaushik/ack-commit-lag

Conversation

@iskakaushik

@iskakaushik iskakaushik commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What

Replication lag in seconds, with no rate window.

walshadow_commit_timestamp_seconds{at}

at meaning
source commit time of the newest commit the pump has read from the source
ack commit time of the newest transaction ClickHouse has acknowledged. Advances when the ack position passes a commit and holds while the ack moves over idle WAL

Unix seconds. A member is absent until its first commit.

Commits replayed from the backup window or an object-store gap carry their commit time too, so the ack member moves through a long replay.

PromQL

# lag, correct while the source is idle
walshadow_commit_timestamp_seconds{at="source"} - ignoring(at) walshadow_commit_timestamp_seconds{at="ack"}
# age of the newest acknowledged commit
time() - walshadow_commit_timestamp_seconds{at="ack"}

Tests

  • commit_ts_advances_with_the_contiguous_frontier
  • commit_ts_keeps_the_newest_and_ignores_trailing
  • partial_slices_and_untimed_seqs_do_not_move_commit_ts
  • render_omits_commit_timestamps_until_the_first

@iskakaushik
iskakaushik force-pushed the kaushik/ack-commit-lag branch from 3d07545 to 05e2f1c Compare October 20, 2026 14:23
@iskakaushik
iskakaushik marked this pull request as ready for review October 20, 2026 15:50
@serprex
serprex force-pushed the kaushik/ack-commit-lag branch 3 times, most recently from 6d6a195 to dc5def9 Compare October 23, 2026 20:51
@serprex
serprex requested a review from harshil-goel October 23, 2026 21:25
serprex
serprex previously approved these changes Oct 23, 2026
@harshil-goel
harshil-goel force-pushed the kaushik/ack-commit-lag branch 2 times, most recently from 73f5d25 to 76f7d8e Compare November 10, 2026 19:30
@harshil-goel
harshil-goel changed the base branch from main to kaushik/bridge-latency-histograms November 10, 2026 19:38
@harshil-goel
harshil-goel added this pull request to stack #313 November 10, 2026 19:39
@harshil-goel
harshil-goel force-pushed the kaushik/ack-commit-lag branch from 76f7d8e to bd042fa Compare November 10, 2026 20:22
@harshil-goel
harshil-goel force-pushed the kaushik/ack-commit-lag branch from bd042fa to 2980dbe Compare November 10, 2026 21:31
@harshil-goel
harshil-goel force-pushed the kaushik/ack-commit-lag branch from 2980dbe to 3613af5 Compare November 11, 2026 07:26
@harshil-goel
harshil-goel force-pushed the kaushik/ack-commit-lag branch from 3613af5 to acf83fc Compare November 11, 2026 07:48
@harshil-goel
harshil-goel force-pushed the kaushik/ack-commit-lag branch from acf83fc to 4f38f0c Compare November 11, 2026 08:58
@harshil-goel
harshil-goel removed this pull request from stack #313 November 11, 2026 09:23
@harshil-goel
harshil-goel added this pull request to stack #317 November 11, 2026 09:23
@harshil-goel
harshil-goel force-pushed the kaushik/ack-commit-lag branch from 4f38f0c to 1543b42 Compare November 11, 2026 09:32
Carry the commit record's xact_time with the emitter ack frontier and
export it as walshadow_emitter_ack_commit_timestamp_seconds, plus the
newest drained commit as walshadow_source_last_commit_timestamp_seconds.
Both are Unix seconds, NaN until the first commit, so replication lag is
a plain subtraction with no PromQL rate window.
@harshil-goel
harshil-goel force-pushed the kaushik/ack-commit-lag branch from 232cd0e to c39e676 Compare November 18, 2026 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants