Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
bbde2e8
prometheus: serve protobuf exposition with created timestamps
sathvik09 Nov 3, 2026
64da5c6
HISTORY: add v5.13.0 entry for protobuf exposition
sathvik09 Nov 3, 2026
04d508b
prometheus: say which scrapers put protobuf first by default
sathvik09 Nov 3, 2026
9ae25aa
prometheus: say the Accept ranking ignores formats the handler does n…
sathvik09 Nov 3, 2026
1ccab19
prometheus: build protobuf messages after releasing the store lock
sathvik09 Nov 3, 2026
c8f1cb3
prometheus: limit the same-series claim to Prometheus 3.x
sathvik09 Nov 3, 2026
ac502e6
prometheus: compare full series sets between protobuf and text
sathvik09 Nov 3, 2026
11e0021
prometheus: order colliding protobuf series and families deterministi…
sathvik09 Nov 3, 2026
20b23c4
prometheus: stop claiming the text format merges colliding families
sathvik09 Nov 3, 2026
8d11553
prometheus: give a zero-time first update a created time
sathvik09 Nov 3, 2026
e1b2131
prometheus: write the text le="+Inf" bucket from _count
sathvik09 Nov 3, 2026
470da00
prometheus: write protobuf families one at a time, and send Vary
sathvik09 Nov 3, 2026
1ca7a60
prometheus: test real Prometheus Accept headers and a concurrent scrape
sathvik09 Nov 3, 2026
0f58552
HISTORY: clarify which scrapers put protobuf first, and note the +Inf…
sathvik09 Nov 3, 2026
c6c5df4
HISTORY: rewrap
sathvik09 Nov 3, 2026
164519f
prometheus: fix misspelling flagged by golangci-lint
sathvik09 Nov 3, 2026
5fcbb1a
prometheus: break text sort ties so a histogram's series comes first
sathvik09 Nov 4, 2026
516b501
prometheus: keep a zero update time when falling back for the created…
sathvik09 Nov 4, 2026
4b983a7
prometheus: fix the Prometheus versions in the Accept header test
sathvik09 Nov 4, 2026
f522143
go.mod: go back to client_model v0.6.2
sathvik09 Nov 4, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions HISTORY.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,52 @@
# History

### v5.13.0

**The `prometheus` handler can serve the protobuf exposition, with created
timestamps.** `increase()` and `rate()` never count a counter's first sample:
it is taken as the starting point. A series that starts at 5 and stays there
shows an increase of 0, and a series that goes idle long enough for
`MetricTimeout` to delete it loses its first update again when it comes back.

The handler now records when each counter and histogram series started from
zero (its first update, or its first update after `MetricTimeout` deleted it)
and publishes it as `created_timestamp` in the protobuf exposition
(`application/vnd.google.protobuf; proto=io.prometheus.client.MetricFamily;
encoding=delimited`). Prometheus is renaming created timestamps to start
timestamps; the wire field is the same. A scraper with created (start)
timestamp zero ingestion enabled (Prometheus
`created-timestamp-zero-ingestion`, Alloy `start_timestamp_zero_ingestion`)
writes a zero at that time, so the first update is counted.

Protobuf is served only to a scraper that ranks it above the text format in
its `Accept` header. That is a scraper whose scrape protocols put
`PrometheusProto` first, or a Prometheus with `scrape_native_histograms` or
created-timestamp zero ingestion on, which puts it first by default: such a
Prometheus switches to protobuf when the application upgrades, with no change
to its configuration. Alloy puts it first by default only with
`scrape_native_histograms`; `start_timestamp_zero_ingestion` alone does not
change the order, so `scrape_protocols` has to list `PrometheusProto` first.
Every other scraper keeps getting the text format. Sample timestamps are still
left to the scraper. Gauges carry no created timestamp.

Both formats render the same metric and label names, so a Prometheus 3.x
scraper that switches format keeps writing to the same series. On Prometheus
2.x, histogram buckets with integer bounds start new series: the scraper
formats the bound of a protobuf bucket as `le="1.0"`, while the text format
writes `le="1"`, and only 3.x normalizes the text value to match.

The text format now writes a histogram's `le="+Inf"` bucket from `_count`
rather than from the total of the other buckets. The two differed after a NaN
observation, which lands in no bucket, or after a change to the registered
buckets, which resets them but not `_count`. Where a gauge and a histogram
series render to the same name and labels (a gauge `y_count` next to a
histogram `y`), the text format now always lists the histogram's first, as
the protobuf format does; their order used to change between scrapes, so the
value a scraper kept alternated between the two. Responses now carry
`Vary: Accept, Accept-Encoding`.

The module now depends on `github.com/prometheus/client_model`.

### v5.12.0

**`procstats` now reads CPU and memory limits on cgroup v2.** It only knew the
Expand Down
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ go 1.26.0

require (
github.com/mdlayher/taskstats v0.0.0-20241219020249-a291fa5f5a69
github.com/prometheus/client_model v0.6.2
github.com/segmentio/encoding v0.5.4
github.com/segmentio/fasthash v1.0.3
github.com/segmentio/vpcinfo v0.2.0
Expand Down
2 changes: 2 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ github.com/mdlayher/socket v0.7.0 h1:qVREPVwtUMg17pwvveQxpurq0PVisMxi3FGgpsorMYQ
github.com/mdlayher/socket v0.7.0/go.mod h1:f7iKql2EK/rfsWYDKofKjk2Ig7I+6HQWdMIdn1tO4A4=
github.com/mdlayher/taskstats v0.0.0-20241219020249-a291fa5f5a69 h1:A33BCEd5ClzJCnUcicB1VRuRrgikbUc8J/v1qrtd2gw=
github.com/mdlayher/taskstats v0.0.0-20241219020249-a291fa5f5a69/go.mod h1:0gPvvNyurVKWZFI09u1YSPCqhSnj6QwlwEijSrHXYag=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/segmentio/asm v1.2.1 h1:DTNbBqs57ioxAD4PrArqftgypG4/qNpXoJx8TVXxPR0=
github.com/segmentio/asm v1.2.1/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
github.com/segmentio/encoding v0.5.4 h1:OW1VRern8Nw6ITAtwSZ7Idrl3MXCFwXHPgqESYfvNt0=
Expand Down
24 changes: 23 additions & 1 deletion prometheus/handler.go
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,20 @@ func (h *Handler) ServeHTTP(res http.ResponseWriter, req *http.Request) {
}

w := io.Writer(res)
res.Header().Set("Content-Type", "text/plain; version=0.0.4")
protobuf := acceptsProtobuf(req.Header.Get("Accept"))

// The format follows Accept and the compression Accept-Encoding, so a
// cache must not serve one response in place of the other.
res.Header().Add("Vary", "Accept")
res.Header().Add("Vary", "Accept-Encoding")

// Text stays the default. Only the protobuf exposition carries created
// timestamps, and a scraper gets it only by asking for it first.
if protobuf {
res.Header().Set("Content-Type", protobufContentType)
} else {
res.Header().Set("Content-Type", "text/plain; version=0.0.4")
}

if acceptEncoding(req.Header.Get("Accept-Encoding"), "gzip") {
res.Header().Set("Content-Encoding", "gzip")
Expand All @@ -162,6 +175,15 @@ func (h *Handler) ServeHTTP(res http.ResponseWriter, req *http.Request) {
w = zw
}

// Write errors, here and from WriteStats and the deferred gzip Close,
// are ignored: the only cause is the connection failing, and by then
// the status and part of the body have been sent, so there is nothing
// left to report the error with. The scraper sees a response that ends
// early and records a failed scrape.
if protobuf {
_ = h.writeProtobuf(w)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor:

  • The error from writeProtobuf (and gzip Close via defer) is dropped. A partial body is silently served; at least worth a comment, or logging if the handler has a logger.
  • Responses vary by Accept now; consider Vary: Accept alongside the content type.
  • Memory: at 100k series, protobuf built ~155 MiB / 2.59M allocations per scrape vs ~126 MiB / 17 allocations for text in a local benchmark. Building and marshalling one family at a time would lower the peak.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 470da00:

  • Dropped errors: added a comment instead of logging. The only way the write (or the gzip Close) can fail is the connection failing. By then the status and part of the body have been sent, so there is nothing left to report the error with, and the scraper sees a response that ends early and records a failed scrape. The text path already behaves this way, and Handler has no logger. Adding one would be new public API for little gain.
  • Vary: the handler now sends Vary: Accept, Accept-Encoding. The gzip choice depends on a request header too and never had one. scrapeProtobuf asserts it.
  • Memory: families() now returns the sorted families with their copied series, and writeProtobuf builds each family's messages just before writing it, so only one family is held at a time. I did not reuse messages between families, so the allocation count stays at about 2.59M. New BenchmarkWriteProtobuf (build and marshal, 100k series, 10 runs, benchstat):
before after
peak heap, 2 families 111.7 MiB 88.3 MiB (−21%)
peak heap, 100 families 111.7 MiB 98.6 MiB (−12%)
time 287 / 210 ms 284 / 205 ms
allocs 2.59M 2.59M

Peak heap is HeapAlloc sampled at each write, above the level before the call. It includes garbage the GC has not collected yet, so read it as relative, not absolute.

return
}
h.WriteStats(w)
}

Expand Down
50 changes: 47 additions & 3 deletions prometheus/metric.go
Original file line number Diff line number Diff line change
Expand Up @@ -311,6 +311,12 @@ type metricState struct {
sum float64
count uint64
time time.Time

// created is when the series started counting from zero, at its first
// update. The protobuf exposition publishes it so that a scraper can write a zero
// sample at that time, which lets increase() count the first value. A
// state re-created after MetricTimeout gets a new one.
created time.Time
}

func newMetricState(labels labels) *metricState {
Expand All @@ -319,7 +325,7 @@ func newMetricState(labels labels) *metricState {
}
}

func (state *metricState) update(mtype metricType, value float64, time time.Time, buckets []stats.Value) {
func (state *metricState) update(mtype metricType, value float64, t time.Time, buckets []stats.Value) {
state.mutex.Lock()

switch mtype {
Expand All @@ -334,14 +340,29 @@ func (state *metricState) update(mtype metricType, value float64, time time.Time
// entry than the registry slice. Comparing against len(buckets) here
// would rebuild — and zero the counts — on every observation.
if len(state.buckets) != len(buckets)+1 {
// A rebuild zeroes the bucket counts, but not _sum and _count.
// Moving the created time forward would make a scraper write a
// zero under a _count that kept counting, so it stays put.
state.buckets = makeMetricBuckets(buckets, state.labels)
}
state.buckets.update(value)
state.sum += value
state.count++
}

state.time = time
if state.created.IsZero() {
// A zero created time means none has been set, so a measure with a
// zero time cannot set one: the next update would take its place,
// and the series would claim to start later than its first value.
// state.time keeps the zero, so expiry treats this update like any
// other zero-time update.
created := t
if created.IsZero() {
created = time.Now()
}
state.created = created
}
Comment on lines +353 to +364

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major: a zero-time first update leaves created too late.

If the first update has time.IsZero() (ReportAt with a zero time, or a direct HandleMeasures), created stays zero, and the next update sets it to that later time. Example: Add 5 at zero time, then Add 1 at now, publishes value 6, created=now. The scraper writes a 0 at now after it already holds a sample of 5, which reads as a reset and double-counts.

Today the engine's only zero-time measure is the go_version gauge, so user ReportAt callers are the exposure. Suggested fix: track first-seen with a bool, and fall back to time.Now() when the measure time is zero. Worth a test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8d11553. When the series has no created time yet and the measure time is zero, created falls back to time.Now(). That is never zero, so created can't fall back into the unset state, and no extra bool is needed. TestCreatedTimestampZeroTime adds 5 at zero time, then 1 a minute later, and asserts value 6 with created at the first update. It fails without the fix (created lands on the second update).

state.time = t
state.mutex.Unlock()
}

Expand Down Expand Up @@ -370,6 +391,13 @@ func (state *metricState) collect(metrics []metric, entry *metricEntry, name str
var cumulativeCount uint64
for _, bucket := range state.buckets {
cumulativeCount += bucket.count
if math.IsInf(bucket.limit, 1) {
// le="+Inf" is _count, as the scraper derives it from a
// protobuf exposition. The bucket total can fall short of it:
// a NaN lands in no bucket, and a rebuild zeroes the buckets
// but not _count.
cumulativeCount = state.count
}
metrics = append(metrics, metric{
mtype: entry.mtype,
scope: entry.scope,
Expand Down Expand Up @@ -524,6 +552,13 @@ func (metrics byNameAndLabels) Swap(i, j int) {
//
// Comparing the two parts in turn rather than the joined "scope_name" avoids
// building a string for every comparison in the sort.
//
// Two metrics can still tie on all three, a gauge "y_count" and the _count
// series of a histogram "y" for example. A scraper keeps the first of the two
// samples, and metrics come out of a map, so the tie is broken on the root
// name and then the type: otherwise the kept value would alternate between
// scrapes. The root name puts the histogram first, as the protobuf format
// does.
func (metrics byNameAndLabels) Less(i, j int) bool {
m1 := &metrics[i]
m2 := &metrics[j]
Expand All @@ -534,5 +569,14 @@ func (metrics byNameAndLabels) Less(i, j int) bool {
if m1.name != m2.name {
return m1.name < m2.name
}
return m1.labels.less(m2.labels)
if m1.labels.less(m2.labels) {
return true
}
if m2.labels.less(m1.labels) {
return false
}
if r1, r2 := m1.rootName(), m2.rootName(); r1 != r2 {
return r1 < r2
}
return m1.mtype < m2.mtype
}
Loading
Loading