Repository navigation
prometheus: serve protobuf exposition with created timestamps #240
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
bbde2e8
64da5c6
04d508b
9ae25aa
1ccab19
c8f1cb3
ac502e6
11e0021
20b23c4
8d11553
e1b2131
470da00
1ca7a60
0f58552
c6c5df4
164519f
5fcbb1a
516b501
4b983a7
f522143
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -311,6 +311,12 @@ type metricState struct { | |
| sum float64 | ||
| count uint64 | ||
| time time.Time | ||
|
|
||
| // created is when the series started counting from zero, at its first | ||
| // update. The protobuf exposition publishes it so that a scraper can write a zero | ||
| // sample at that time, which lets increase() count the first value. A | ||
| // state re-created after MetricTimeout gets a new one. | ||
| created time.Time | ||
| } | ||
|
|
||
| func newMetricState(labels labels) *metricState { | ||
|
|
@@ -319,7 +325,7 @@ func newMetricState(labels labels) *metricState { | |
| } | ||
| } | ||
|
|
||
| func (state *metricState) update(mtype metricType, value float64, time time.Time, buckets []stats.Value) { | ||
| func (state *metricState) update(mtype metricType, value float64, t time.Time, buckets []stats.Value) { | ||
| state.mutex.Lock() | ||
|
|
||
| switch mtype { | ||
|
|
@@ -334,14 +340,29 @@ func (state *metricState) update(mtype metricType, value float64, time time.Time | |
| // entry than the registry slice. Comparing against len(buckets) here | ||
| // would rebuild — and zero the counts — on every observation. | ||
| if len(state.buckets) != len(buckets)+1 { | ||
| // A rebuild zeroes the bucket counts, but not _sum and _count. | ||
| // Moving the created time forward would make a scraper write a | ||
| // zero under a _count that kept counting, so it stays put. | ||
| state.buckets = makeMetricBuckets(buckets, state.labels) | ||
| } | ||
| state.buckets.update(value) | ||
| state.sum += value | ||
| state.count++ | ||
| } | ||
|
|
||
| state.time = time | ||
| if state.created.IsZero() { | ||
| // A zero created time means none has been set, so a measure with a | ||
| // zero time cannot set one: the next update would take its place, | ||
| // and the series would claim to start later than its first value. | ||
| // state.time keeps the zero, so expiry treats this update like any | ||
| // other zero-time update. | ||
| created := t | ||
| if created.IsZero() { | ||
| created = time.Now() | ||
| } | ||
| state.created = created | ||
| } | ||
|
Comment on lines
+353
to
+364
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Major: a zero-time first update leaves If the first update has Today the engine's only zero-time measure is the
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in 8d11553. When the series has no created time yet and the measure time is zero, |
||
| state.time = t | ||
| state.mutex.Unlock() | ||
| } | ||
|
|
||
|
|
@@ -370,6 +391,13 @@ func (state *metricState) collect(metrics []metric, entry *metricEntry, name str | |
| var cumulativeCount uint64 | ||
| for _, bucket := range state.buckets { | ||
| cumulativeCount += bucket.count | ||
| if math.IsInf(bucket.limit, 1) { | ||
| // le="+Inf" is _count, as the scraper derives it from a | ||
| // protobuf exposition. The bucket total can fall short of it: | ||
| // a NaN lands in no bucket, and a rebuild zeroes the buckets | ||
| // but not _count. | ||
| cumulativeCount = state.count | ||
| } | ||
| metrics = append(metrics, metric{ | ||
| mtype: entry.mtype, | ||
| scope: entry.scope, | ||
|
|
@@ -524,6 +552,13 @@ func (metrics byNameAndLabels) Swap(i, j int) { | |
| // | ||
| // Comparing the two parts in turn rather than the joined "scope_name" avoids | ||
| // building a string for every comparison in the sort. | ||
| // | ||
| // Two metrics can still tie on all three, a gauge "y_count" and the _count | ||
| // series of a histogram "y" for example. A scraper keeps the first of the two | ||
| // samples, and metrics come out of a map, so the tie is broken on the root | ||
| // name and then the type: otherwise the kept value would alternate between | ||
| // scrapes. The root name puts the histogram first, as the protobuf format | ||
| // does. | ||
| func (metrics byNameAndLabels) Less(i, j int) bool { | ||
| m1 := &metrics[i] | ||
| m2 := &metrics[j] | ||
|
|
@@ -534,5 +569,14 @@ func (metrics byNameAndLabels) Less(i, j int) bool { | |
| if m1.name != m2.name { | ||
| return m1.name < m2.name | ||
| } | ||
| return m1.labels.less(m2.labels) | ||
| if m1.labels.less(m2.labels) { | ||
| return true | ||
| } | ||
| if m2.labels.less(m1.labels) { | ||
| return false | ||
| } | ||
| if r1, r2 := m1.rootName(), m2.rootName(); r1 != r2 { | ||
| return r1 < r2 | ||
| } | ||
| return m1.mtype < m2.mtype | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor:
writeProtobuf(and gzipClosevia defer) is dropped. A partial body is silently served; at least worth a comment, or logging if the handler has a logger.Acceptnow; considerVary: Acceptalongside the content type.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed in 470da00:
Close) can fail is the connection failing. By then the status and part of the body have been sent, so there is nothing left to report the error with, and the scraper sees a response that ends early and records a failed scrape. The text path already behaves this way, andHandlerhas no logger. Adding one would be new public API for little gain.Vary: Accept, Accept-Encoding. The gzip choice depends on a request header too and never had one.scrapeProtobufasserts it.families()now returns the sorted families with their copied series, andwriteProtobufbuilds each family's messages just before writing it, so only one family is held at a time. I did not reuse messages between families, so the allocation count stays at about 2.59M. NewBenchmarkWriteProtobuf(build and marshal, 100k series, 10 runs, benchstat):Peak heap is
HeapAllocsampled at each write, above the level before the call. It includes garbage the GC has not collected yet, so read it as relative, not absolute.