Skip to content

feat(metrics): bootstrap phase stages, live gate progress, hourly config log - #314

Open
harshil-goel wants to merge 1 commit into
kaushik/ack-commit-lagfrom
harshil/bootstrap-stage-metrics
Open

harshil-goel wants to merge 1 commit into
kaushik/ack-commit-lagfrom
harshil/bootstrap-stage-metrics

Conversation

@harshil-goel

@harshil-goel harshil-goel commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

What

Bootstrap phases as stages, live progress for each, and a config log.

  • Stage timers for every bootstrap phase: bootstrap, bootstrap_walk, bootstrap_window, shadow_replay, bootstrap_deferred, bootstrap_gate. Every walshadow_stage_* series carries title (operator name) and order (running order, spaced by 10).
  • walshadow_stage_progress{stage,title,order,measure}: work_total, work_done, rows. Values only rise, work_done lands on work_total when the stage completes, and nothing is published by a daemon that never bootstrapped. Units: backup parts for the walk, spool bytes for the large-value fill, deferred tuples for the visibility check.
  • walshadow_bootstrap_gate_tuples_total{verdict}: emitted, gated, deferred, pending, counted live. A deferred tuple is counted again under its final verdict.
  • walshadow_source_mode{mode="initial_load"} is set for the whole bootstrap.
  • Hourly config log on target walshadow::config: per source database one shadow config line, then one table config line per table. No secrets: endpoint is host:port/dbname.

bootstrap_window covers both ways the window is replayed: live beside the walk, or from the landed WAL after it. bootstrap_walk ends when the walk and its inserts drain, and is not started on a resumed attempt.

Breaking

walshadow_bootstrap_gate_emitted_total, _gated_total, _deferred_total are removed. Use walshadow_bootstrap_gate_tuples_total{verdict}.

PromQL

# active phase
max by (order, title) (walshadow_stage_active{stage=~"bootstrap.*|shadow_replay"}) == 1
# progress of the running phase
walshadow_stage_progress{measure="work_done"} / ignoring(measure) (walshadow_stage_progress{measure="work_total"} > 0)
# gate verdicts
sum by (verdict) (rate(walshadow_bootstrap_gate_tuples_total[1m]))

Tests

  • work_and_rows_are_absent_until_set, work_only_rises_and_lands_on_total_at_completion
  • lock_only_multixact_rows_emit_and_count asserts the live gate counters

Stacked on #256.

@harshil-goel
harshil-goel added this pull request to stack #313 November 10, 2026 19:42
@harshil-goel
harshil-goel force-pushed the harshil/bootstrap-stage-metrics branch from e52aec3 to 14dae62 Compare November 10, 2026 20:23
@harshil-goel
harshil-goel force-pushed the harshil/bootstrap-stage-metrics branch from 14dae62 to e22afb6 Compare November 10, 2026 21:31
@harshil-goel
harshil-goel force-pushed the harshil/bootstrap-stage-metrics branch from e22afb6 to faa2eec Compare November 11, 2026 07:26
@harshil-goel
harshil-goel force-pushed the harshil/bootstrap-stage-metrics branch from faa2eec to 2404a68 Compare November 11, 2026 07:48
@harshil-goel
harshil-goel force-pushed the harshil/bootstrap-stage-metrics branch from 2404a68 to 51cc8a3 Compare November 11, 2026 08:58
@harshil-goel
harshil-goel removed this pull request from stack #313 November 11, 2026 09:23
@harshil-goel
harshil-goel added this pull request to stack #317 November 11, 2026 09:23
@harshil-goel
harshil-goel force-pushed the harshil/bootstrap-stage-metrics branch from 51cc8a3 to d9b6ad6 Compare November 11, 2026 09:32
@harshil-goel
harshil-goel force-pushed the harshil/bootstrap-stage-metrics branch from d9b6ad6 to 13dcc8f Compare November 11, 2026 09:45
@harshil-goel
harshil-goel force-pushed the harshil/bootstrap-stage-metrics branch from 13dcc8f to f46da85 Compare November 11, 2026 11:38
@harshil-goel
harshil-goel force-pushed the harshil/bootstrap-stage-metrics branch 2 times, most recently from 10be5c7 to 8adf3f5 Compare November 17, 2026 16:15
@harshil-goel
harshil-goel force-pushed the harshil/bootstrap-stage-metrics branch from 8adf3f5 to e5f5037 Compare November 18, 2026 05:11
@harshil-goel
harshil-goel force-pushed the harshil/bootstrap-stage-metrics branch from e5f5037 to db3d37e Compare November 18, 2026 08:13
@harshil-goel
harshil-goel force-pushed the harshil/bootstrap-stage-metrics branch from db3d37e to 4ffd115 Compare November 18, 2026 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant