Skip to content

feat(metrics): histograms for bridge request latency and TOAST values per fetch - #255

Open
iskakaushik wants to merge 1 commit into
kaushik/source-lag-metricsfrom
kaushik/bridge-latency-histograms
Open

iskakaushik wants to merge 1 commit into
kaushik/source-lag-metricsfrom
kaushik/bridge-latency-histograms

Conversation

@iskakaushik

@iskakaushik iskakaushik commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What

Histograms where only means existed.

metric meaning
walshadow_request_duration_seconds{database,target="bridge",op} bridge request wall time, lock wait included. 13 buckets, 25 us doubling to 102.4 ms, plus +Inf
walshadow_toast_values_per_fetch values requested per chunk-store fetch, misses included. Buckets 1, 2, 4 to 256, plus +Inf

Existing counters are unchanged.

Implementation

Histogram<FIRST, N> in ops/metrics.rs: lock-free, N buckets starting at FIRST and doubling. The bounds are part of the type, so observe and snapshot cannot disagree. One relaxed add on the bucket and one on the sum per observation. Bootstrap and live bridges of one database fold into one series. A scrape racing an observation can see the count one ahead of the sum.

PromQL

# bridge p99 per op
histogram_quantile(0.99, sum by (op, le) (rate(walshadow_request_duration_seconds_bucket{target="bridge"}[5m])))
# share of bridge requests slower than 6.4 ms
1 - sum(rate(walshadow_request_duration_seconds_bucket{target="bridge",le="0.0064"}[5m]))
  / sum(rate(walshadow_request_duration_seconds_count{target="bridge"}[5m]))
# median values per TOAST fetch
histogram_quantile(0.5, sum by (le) (rate(walshadow_toast_values_per_fetch_bucket[5m])))
# share of single-value fetches
sum(rate(walshadow_toast_values_per_fetch_bucket{le="1.0"}[5m])) / sum(rate(walshadow_toast_values_per_fetch_count[5m]))

Tests

  • histograms_render_cumulative_buckets_per_series covers the boundary, overflow, merge, rendering, and the real le values 0.000025 and 0.0064
  • exposition_parses_as_openmetrics
  • db_series_folds_bootstrap_bridge_into_the_database_it_restored
  • resolver_mem_stores_and_fetches

@harshil-goel
harshil-goel force-pushed the kaushik/bridge-latency-histograms branch from 9cdcbec to ed58b73 Compare November 10, 2026 18:13
@harshil-goel
harshil-goel marked this pull request as ready for review November 10, 2026 18:15
@harshil-goel
harshil-goel force-pushed the kaushik/bridge-latency-histograms branch from ed58b73 to bf0fc26 Compare November 10, 2026 19:30
@harshil-goel
harshil-goel changed the base branch from main to kaushik/source-lag-metrics November 10, 2026 19:38
@harshil-goel
harshil-goel added this pull request to stack #313 November 10, 2026 19:39
@harshil-goel
harshil-goel force-pushed the kaushik/bridge-latency-histograms branch from bf0fc26 to 9d85078 Compare November 10, 2026 20:22
@harshil-goel
harshil-goel force-pushed the kaushik/bridge-latency-histograms branch from 9d85078 to 7451ef3 Compare November 10, 2026 21:31
@harshil-goel
harshil-goel force-pushed the kaushik/bridge-latency-histograms branch from 7451ef3 to fa0e229 Compare November 11, 2026 07:26
@harshil-goel
harshil-goel force-pushed the kaushik/bridge-latency-histograms branch from fa0e229 to 7607fcc Compare November 11, 2026 07:48
@harshil-goel
harshil-goel force-pushed the kaushik/bridge-latency-histograms branch from 7607fcc to 223d675 Compare November 11, 2026 08:58
@harshil-goel
harshil-goel removed this pull request from stack #313 November 11, 2026 09:23
@harshil-goel
harshil-goel added this pull request to stack #317 November 11, 2026 09:23
@harshil-goel
harshil-goel force-pushed the kaushik/bridge-latency-histograms branch from 223d675 to 8b0c303 Compare November 11, 2026 09:32
@harshil-goel
harshil-goel force-pushed the kaushik/bridge-latency-histograms branch from 8b0c303 to 838dca1 Compare November 11, 2026 09:45
@harshil-goel
harshil-goel force-pushed the kaushik/bridge-latency-histograms branch from 838dca1 to 3252222 Compare November 11, 2026 11:38
Comment thread src/ops/bridge.rs Outdated
/// Upper bounds of the bridge request-latency buckets: 25us doubling to
/// 102.4ms, which spans a local socket round trip through a worker stalled on
/// slow storage. Slower requests land in `+Inf`
pub const REQUEST_LATENCY_BOUNDS_NANOS: [u64; REQUEST_LATENCY_BUCKETS] = {

@serprex serprex Nov 11, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why have const array when this is basic function 25_000 << x

Makes me think we should support HistogramFn or there's something wrong with the Historgram concept

@harshil-goel
harshil-goel force-pushed the kaushik/bridge-latency-histograms branch from 3252222 to 0a7577d Compare November 17, 2026 10:08
@harshil-goel
harshil-goel force-pushed the kaushik/bridge-latency-histograms branch from 0a7577d to d8918f1 Compare November 18, 2026 05:11
@harshil-goel
harshil-goel force-pushed the kaushik/bridge-latency-histograms branch from d8918f1 to d5f95c0 Compare November 18, 2026 08:12
@harshil-goel
harshil-goel force-pushed the kaushik/bridge-latency-histograms branch from d5f95c0 to 0b039a8 Compare November 18, 2026 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants