From 448f648e7ab33d8570cb78d6922160444394f156 Mon Sep 17 00:00:00 2001 From: Lukasz Marchewka Date: Wed, 4 Nov 2026 14:41:50 +0100 Subject: [PATCH 1/2] Add Service Availability StackPack - Create endpoint components and optional service grouping from Blackbox metrics. - Monitor availability using probe_success and scraper up metrics. - Add optional certificate warnings below 14 days and Critical alerts below two days. - Add menu entries, overview and highlight pages, and certificate charts. - Document AppCo Blackbox Exporter, OTel scraper and Probe configuration. --- stackpacks/service-availability/README.md | 107 ++++++ .../icons/availability.svg | 3 + .../service-availability/icons/endpoint.svg | 5 + .../service-availability/icons/service.svg | 6 + .../certificate-expiring.md.hbs | 14 + .../endpoint-unavailable.md.hbs | 15 + .../service-unavailable.md.hbs | 11 + .../resources/configuration.md | 332 ++++++++++++++++++ .../resources/deprovisioning.md | 3 + .../service-availability/resources/error.md | 6 + .../service-availability/resources/logo.svg | 3 + .../resources/overview.md | 25 ++ .../resources/provisioning.md | 3 + .../resources/waitingfordata.md | 6 + .../settings/component-mappings/endpoints.sty | 35 ++ .../settings/component-mappings/services.sty | 36 ++ .../settings/main-menu.sty | 7 + .../certificate-expiration-critical.sty | 23 ++ .../monitors/certificate-expiration.sty | 23 ++ .../monitors/endpoint-availability.sty | 23 ++ .../monitors/service-availability.sty | 23 ++ .../settings/presentations/endpoints.sty | 175 +++++++++ .../settings/presentations/services.sty | 102 ++++++ .../relation-mappings/service-endpoints.sty | 31 ++ .../service-availability/stackpack.yaml | 16 + 25 files changed, 1033 insertions(+) create mode 100644 stackpacks/service-availability/README.md create mode 100644 stackpacks/service-availability/icons/availability.svg create mode 100644 stackpacks/service-availability/icons/endpoint.svg create mode 100644 stackpacks/service-availability/icons/service.svg create mode 100644 stackpacks/service-availability/includes/remediation-hints/certificate-expiring.md.hbs create mode 100644 stackpacks/service-availability/includes/remediation-hints/endpoint-unavailable.md.hbs create mode 100644 stackpacks/service-availability/includes/remediation-hints/service-unavailable.md.hbs create mode 100644 stackpacks/service-availability/resources/configuration.md create mode 100644 stackpacks/service-availability/resources/deprovisioning.md create mode 100644 stackpacks/service-availability/resources/error.md create mode 100644 stackpacks/service-availability/resources/logo.svg create mode 100644 stackpacks/service-availability/resources/overview.md create mode 100644 stackpacks/service-availability/resources/provisioning.md create mode 100644 stackpacks/service-availability/resources/waitingfordata.md create mode 100644 stackpacks/service-availability/settings/component-mappings/endpoints.sty create mode 100644 stackpacks/service-availability/settings/component-mappings/services.sty create mode 100644 stackpacks/service-availability/settings/main-menu.sty create mode 100644 stackpacks/service-availability/settings/monitors/certificate-expiration-critical.sty create mode 100644 stackpacks/service-availability/settings/monitors/certificate-expiration.sty create mode 100644 stackpacks/service-availability/settings/monitors/endpoint-availability.sty create mode 100644 stackpacks/service-availability/settings/monitors/service-availability.sty create mode 100644 stackpacks/service-availability/settings/presentations/endpoints.sty create mode 100644 stackpacks/service-availability/settings/presentations/services.sty create mode 100644 stackpacks/service-availability/settings/relation-mappings/service-endpoints.sty create mode 100644 stackpacks/service-availability/stackpack.yaml diff --git a/stackpacks/service-availability/README.md b/stackpacks/service-availability/README.md new file mode 100644 index 0000000..8690005 --- /dev/null +++ b/stackpacks/service-availability/README.md @@ -0,0 +1,107 @@ +# Service Availability + +A declarative StackPack 2.0 integration for HTTP and HTTPS checks collected by +Prometheus Blackbox Exporter through the SUSE Observability OTel Prometheus +scraper. See [setup instructions](resources/configuration.md). + +The pack creates one `http endpoint` component per URL and an availability +monitor. The optional `availability_service` label groups multiple URLs under an +`availability service` component with its own availability monitor. +**Service Availability → Endpoints** and **Services** provide overviews and +highlight pages. TLS certificate validity is displayed when available. Optional +certificate monitors warn below 14 days and report Critical below two days. +Certificate expiration is shown as a duration in the overview. The time-series +chart shows whole days with a vertical axis starting at zero. An orange dotted +line marks the default 14-day warning threshold; a red dotted line marks the +two-day critical threshold. + +## Telemetry contract + +| Input | Purpose | +| --- | --- | +| `probe_success` | Creates endpoint topology and reports whether the endpoint check succeeded. | +| `up` | Creates endpoint topology even when the first scrape fails, and makes availability Critical when a scrape fails. | +| `probe_http_status_code` | Displays the received HTTP status. | +| `probe_duration_seconds` | Displays total probe duration. | +| `probe_ssl_earliest_cert_expiry` | Optional Unix timestamp of the earliest certificate expiration. | +| Datapoint label `service_availability="true"` | Opts a target into this integration. | +| Datapoint label `endpoint` | The complete target URL, including its path and query. | +| Datapoint label `availability_service` | Optional nonempty identity grouping endpoints into a service. | +| Datapoint label `monitor_certificate_expiry="true"` | Opts targets into expiration checks when TLS certificate data is reported, including HTTP redirects to HTTPS. | + +These metric and attribute names are the Blackbox/Prometheus contract, not +OpenTelemetry semantic conventions. The Probe example preserves target identity +in `endpoint`, independently of the Prometheus receiver's conversion of `job` +and `instance` to OTel resource attributes. + +The Probe example uses `http_2xx`, which follows redirects and requires a final +2xx response. The optional `http_available` module implements the below-400 +policy without following redirects. Both supply the same metrics; availability +uses the selected module's `probe_success` result. For an existing exporter, +check additional module constraints, such as requiring HTTPS. + +Topology identifier: `urn:service-availability:endpoint/`. +The monitor constructs the same identifier from its `endpoint` result label. +The same URL checked by multiple Probes has one component; the monitor uses the +minimum success value, so any failing observation makes it critical. Different +paths or queries are separate endpoints. + +Service identifier: `urn:service-availability:service/`. +The service mapping and monitor share this identity, and a hierarchical relation +links the service to every endpoint. Use environment-qualified service identities +when the same service name should represent separate installations. + +Availability monitors run every minute against the latest samples, with no +additional five-minute failure delay. Any failing observation makes its endpoint +critical; any failing check labeled for a service makes that service critical. +The two optional certificate monitors run every five minutes and report +Deviating below 14 days remaining and Critical below two days. They use reported +TLS certificate data, including HTTP targets that redirect to HTTPS. Plain HTTP +targets without TLS, missing timestamps and zero timestamps produce no +certificate result. + +Components and relations expire after 15 minutes without matching metrics. +The service monitor aggregates observed samples; it cannot detect a missing URL +without an inventory of expected targets. A disappearing URL can reduce the +displayed endpoint count. Missing telemetry is not evidence of successful checks. +Runtime topology, health and UI behavior require validation on an instance. + +## Development + +```bash +sts stackpack package -d stackpacks/service-availability +sts stackpack validate -d stackpacks/service-availability +``` + +The repository CI discovers this directory automatically. For offline validation, +use the `VALIDATOR_IMAGE` selected by `.github/workflows/stackpacks-ci.yml`: + +```bash +bash scripts/ci/validate_stackpack.sh stackpacks/service-availability +``` + +The manifest follows the StackPack SDK's `YamlReader` in +`StackVista/stackpacks`. Setup instructions use `configurationUrls` because the +SDK ignores the legacy `detailedOverviewUrl` field for schema 2.0. The same +instructions remain available after installation. + +On an explicitly selected development instance, test installation with: + +```bash +sts stackpack test-deploy -d stackpacks/service-availability --yes +``` + +With `http_2xx`, verify that a 200 is healthy and redirects succeed only when +their final response is 2xx. With `http_available`, verify that a 302 is healthy +without following the redirect. Verify that a 404 or 500, connection errors and +TLS validation errors are critical with both modules. Check that a failed target +still creates its component, duplicate URLs merge, and non-opted-in probes do +not create components. Group two URLs under one service and verify that either +URL failing makes that service critical. Check that unlabeled URLs remain +standalone endpoints. + +For certificate warnings, verify certificates inside and outside the 14-day +window, opt-out targets, HTTP targets, and missing or zero timestamps. Valid +certificates inside the warning window should keep availability clear while +the certificate monitor warns. Inspect both overviews, the highlights and the +service-to-endpoint relations. diff --git a/stackpacks/service-availability/icons/availability.svg b/stackpacks/service-availability/icons/availability.svg new file mode 100644 index 0000000..9d6982b --- /dev/null +++ b/stackpacks/service-availability/icons/availability.svg @@ -0,0 +1,3 @@ + + + diff --git a/stackpacks/service-availability/icons/endpoint.svg b/stackpacks/service-availability/icons/endpoint.svg new file mode 100644 index 0000000..57cdef0 --- /dev/null +++ b/stackpacks/service-availability/icons/endpoint.svg @@ -0,0 +1,5 @@ + + + + + diff --git a/stackpacks/service-availability/icons/service.svg b/stackpacks/service-availability/icons/service.svg new file mode 100644 index 0000000..aab5ee5 --- /dev/null +++ b/stackpacks/service-availability/icons/service.svg @@ -0,0 +1,6 @@ + + + + + + diff --git a/stackpacks/service-availability/includes/remediation-hints/certificate-expiring.md.hbs b/stackpacks/service-availability/includes/remediation-hints/certificate-expiring.md.hbs new file mode 100644 index 0000000..fd465ac --- /dev/null +++ b/stackpacks/service-availability/includes/remediation-hints/certificate-expiring.md.hbs @@ -0,0 +1,14 @@ +The earliest certificate expiration reported for **{{ labels.endpoint }}** is +less than the configured expiration threshold away. The default warning is +14 days; the critical window is two days. Prioritize renewal when Critical. + +1. Inspect the certificate served from the Blackbox Exporter's network, including + intermediate certificates in its chain. +2. Check your certificate issuer and renewal automation. Renew the certificate + and confirm that the endpoint serves the renewed chain. +3. Keep normal TLS verification enabled. An expired or invalid certificate also + fails the endpoint's availability check. + +Certificate expiration checks are selected by `monitor_certificate_expiry="true"` +on Probe target labels. Change the monitor's threshold in seconds to adjust the +warning or critical window. diff --git a/stackpacks/service-availability/includes/remediation-hints/endpoint-unavailable.md.hbs b/stackpacks/service-availability/includes/remediation-hints/endpoint-unavailable.md.hbs new file mode 100644 index 0000000..96e0724 --- /dev/null +++ b/stackpacks/service-availability/includes/remediation-hints/endpoint-unavailable.md.hbs @@ -0,0 +1,15 @@ +Check the endpoint **{{ labels.endpoint }}** from the Blackbox Exporter's network. + +1. Inspect `probe_http_status_code` and the exporter's probe output. With + `http_2xx`, the final response must be 2xx. With `http_available`, responses + below 400 are accepted without following redirects. +2. If no response was received, check DNS, routing, network policies, timeouts and + the endpoint's workload. +3. For HTTPS, check the certificate chain, hostname and validity. Expired or + untrusted certificates fail the probe. +4. Confirm that the Probe's module exists in the exporter configuration and + that the exporter is running and reachable from the OTel scraper. + Check `up` for this endpoint: zero means the scrape failed, even if + `probe_success` is absent or still shows an older successful check. + +If multiple Probes check this URL, any failing probe makes the endpoint critical. diff --git a/stackpacks/service-availability/includes/remediation-hints/service-unavailable.md.hbs b/stackpacks/service-availability/includes/remediation-hints/service-unavailable.md.hbs new file mode 100644 index 0000000..a72c58f --- /dev/null +++ b/stackpacks/service-availability/includes/remediation-hints/service-unavailable.md.hbs @@ -0,0 +1,11 @@ +At least one endpoint of **{{ labels.availability_service }}** failed its check +or scrape. + +Open the service highlight and inspect its related endpoints. Check the failing +endpoint's health check, HTTP response code and probe duration, then investigate +its application, network path or TLS certificate. Check the endpoint's `up` +metric and scraper logs when its Blackbox metrics are missing. + +Every labeled URL is required for service availability. If a URL is an optional +dependency, monitor it separately without the service's `availability_service` +label. diff --git a/stackpacks/service-availability/resources/configuration.md b/stackpacks/service-availability/resources/configuration.md new file mode 100644 index 0000000..296a51f --- /dev/null +++ b/stackpacks/service-availability/resources/configuration.md @@ -0,0 +1,332 @@ +# Configure service availability checks + +The data flow is: + +```text +Probe → Target Allocator → OTel Prometheus scraper → Blackbox Exporter + ↓ ↓ + SUSE Observability HTTP(S) endpoint +``` + +The Target Allocator discovers the Probe and gives the scraper its scrape job. +The scraper calls Blackbox Exporter's `/probe` endpoint with the target URL. +Blackbox Exporter performs the check and returns metrics, which the scraper +forwards to SUSE Observability. This StackPack turns those metrics into endpoint +components and health checks. It also uses the scraper's `up` metric to detect +failed scrapes, including exporter connection errors, scrape timeouts and unknown +modules. + +## Install Blackbox Exporter + +Use the **Prometheus Blackbox Exporter** chart from **SUSE Application Collection +(AppCo)**, available at +[SUSE Application Collection](https://apps.rancher.io/). Follow Application +Collection's access instructions to authenticate Helm and create its image pull +secret in the `monitoring` namespace. + +Save the following as `blackbox-values.yaml`. Set `appco-credentials` to the name +of your Application Collection image pull secret. The Probe example uses +`http_2xx`, which follows redirects and requires a final 2xx response. The +optional `http_available` module accepts responses below 400 without following +redirects. + +```yaml +global: + imagePullSecrets: + - appco-credentials +config: + modules: + http_2xx: + prober: http + timeout: 5s + http: + follow_redirects: true + http_available: + prober: http + timeout: 5s + http: + follow_redirects: false + valid_status_codes: + [100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, + 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, + 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, + 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, + 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, + 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, + 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, + 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, + 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, + 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, + 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, + 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, + 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, + 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, + 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, + 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, + 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, + 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, + 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, + 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, + 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, + 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, + 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, + 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, + 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399] +``` + +Blackbox defaults to accepting 200–299 when no accepted status codes are listed. +The `http_available` module lists 100–399 explicitly to implement the below-400 +policy. The modules above support HTTP and HTTPS and keep normal TLS certificate +and hostname verification enabled. Do not enable `insecure_skip_verify` when +certificate validation is part of the availability check. + +Install through your deployment tooling, or use Helm: + +```bash +helm upgrade --install prometheus-blackbox-exporter \ + oci://dp.apps.rancher.io/charts/prometheus-blackbox-exporter \ + --namespace monitoring --create-namespace \ + --values blackbox-values.yaml +``` + +For an existing exporter, use its `http_2xx` module if it meets your requirements. +Check its configuration: existing modules may add constraints such as +`fail_if_not_ssl: true`, which requires HTTPS. Merge any additional modules into +the current configuration rather than replacing modules used by other checks. +Use a chart version approved for your environment. + +To use the below-400 policy, add `http_available` and set the Probe's +`spec.module` to `http_available`. Both modules supply the same metrics, so the +StackPack mappings and monitors work with either. The selected module determines +which responses are healthy. + +## Enable Probe discovery + +Merge these overrides into your existing SUSE Observability agent installation: + +```yaml +otel: + enabled: true + prometheusScraping: + monitorCrds: + enabled: true +``` + +Prometheus scraping is enabled by default. If your existing values explicitly +disable it, also set `otel.prometheusScraping.enabled: true`. +The chart's default Probe selector matches `observability.suse.com/agent: scrape`, +as used in the example below, and its default namespace selector includes all +namespaces. If your installation overrides these selectors, use matching Probe +labels and include the `monitoring` namespace. + +`monitorCrds.enabled: true` installs the ServiceMonitor, PodMonitor and Probe +CRDs. Omit this override when those CRDs are already managed by another chart +or deployment tool. + +The scraper must reach the exporter, and the exporter must reach each endpoint, including DNS +and HTTPS connections. An internal endpoint is supported when it is reachable +from the exporter's network. + +## Create a Probe + +Create this resource through your Kubernetes deployment tooling. +**Replace all uppercase `REPLACE_WITH_...` values and the example URLs before creating the Probe.** Keep the label +name `availability_service`; replace its value with your own service identity. +Multiple targets can be listed under `static`; their full URLs identify separate +endpoint components. The example groups both URLs under one service and enables +optional certificate expiration warnings. + +`spec.jobName` can be any descriptive scrape job name; `service-availability` +was only an example. The StackPack does not select targets by job name. +It selects targets using `service_availability: "true"` and identifies each +endpoint by its `endpoint` label. The optional `availability_service` label +provides the service identity, independently of `jobName`. + +```yaml +apiVersion: monitoring.coreos.com/v1 +kind: Probe +metadata: + name: REPLACE_WITH_YOUR_PROBE_NAME + namespace: monitoring + labels: + observability.suse.com/agent: scrape +spec: + jobName: REPLACE_WITH_YOUR_JOB_NAME + interval: 60s + scrapeTimeout: 10s + module: http_2xx + prober: + url: prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115 + scheme: http + path: /probe + targets: + staticConfig: + static: + - https://example.com/health + - https://example.com/api/ready + labels: + service_availability: "true" + availability_service: REPLACE_WITH_YOUR_SERVICE_NAME + monitor_certificate_expiry: "true" + relabelingConfigs: + - sourceLabels: [__param_target] + targetLabel: endpoint +``` + +| Field | What to change | +| --- | --- | +| `metadata.name` | Replace `REPLACE_WITH_YOUR_PROBE_NAME` with a valid lowercase Kubernetes resource name, such as `checkout-health`. | +| `spec.jobName` | Replace `REPLACE_WITH_YOUR_JOB_NAME` with your scrape job name. | +| `targets.staticConfig.static` | Replace `https://example.com/health` and `https://example.com/api/ready` with your complete HTTP or HTTPS URLs, including the correct paths. Add or remove URLs as needed. | +| `targets.staticConfig.labels.availability_service` | Replace `REPLACE_WITH_YOUR_SERVICE_NAME` with a service identity such as `production/my-service`. Remove this label if you do not want service grouping. | +| `metadata.namespace` and `spec.prober.url` | Adjust if your Probe or exporter uses a different namespace, or the exporter has a different service name. | +| `targets.staticConfig.labels.monitor_certificate_expiry` | Keep `"true"` to enable certificate checks; set `"false"` or remove the label to disable them. | + +If the exporter uses a different release name or namespace, adjust `prober.url`. +The relabeling copies the complete target URL into the `endpoint` metric label. +Keep this relabeling and the `service_availability` label: they are the contract +used by the topology mapping, monitor and presentation. For an existing Probe, +add both to its `targets.staticConfig`. Set `spec.module` to `http_2xx`, or to +`http_available` after configuring that optional module in the exporter. +The interval can be changed; keep it well below the 15-minute topology expiry. + +Do not put credentials in a target URL or its query string: that URL is stored +as a metric label and shown in the UI. Configure authentication separately in +the exporter's HTTP module when an endpoint requires it. + +## Monitor several URLs for one service + +Set `availability_service` to the same nonempty value on the targets of each +Probe belonging to a service. One Probe can list several URLs, as above. Separate +Probes can use the same service label when URLs need different modules, scrape +intervals, or certificate warning settings. + +The StackPack creates a service component and links it to each endpoint. Open +**Service Availability → Services**, then select a service to inspect its related +endpoints. The service availability monitor becomes critical when any of its +currently observed endpoint checks or scrapes fails. Each URL also retains its own +availability monitor and highlight page. + +Use a unique service identity across your monitored environments, for example +`production/checkout` and `staging/checkout`, if they should be separate services. +Omit `availability_service` to monitor URLs individually without creating a +service component. Every URL assigned to a service contributes to availability; +monitor optional dependencies separately. + +The service monitor uses available `probe_success` and `up` samples, rather than +an inventory of expected URLs. A failed scrape reports `up=0`, so its URL remains +in the monitored endpoint count and makes availability Critical. If the scraper +stops emitting both metrics entirely for a URL, its absence is not automatically +counted as a failure. Services and relations expire 15 minutes after their last +matching metric. + +## Optional certificate expiration warnings + +Blackbox reports `probe_ssl_earliest_cert_expiry` when a TLS certificate is +available, including HTTP targets that redirect to HTTPS. Both certificate +monitors use the reported certificate data regardless of the target's initial +HTTP or HTTPS scheme. Plain HTTP checks without TLS have no certificate result. +The **Certificate expires in** overview column shows the remaining validity +as a duration until the earliest expiration in the +certificate chain. The chart in the **TLS certificate** highlight section shows +whole days, rounded down, so less than one day remaining displays as zero. +The time-series chart uses the days unit with zero decimal places and a vertical +axis starting at zero, representing expiration. An orange dotted line marks +the 14-day warning window; a red dotted line marks the two-day critical window. +The UI labels these lines **Threshold 1** and **Threshold 2**; custom threshold +names are not supported. If you change a monitor threshold, update its chart +reference to match. Checks without certificate data show no value, rather +than zero. + +To enable both certificate monitors, add `monitor_certificate_expiry: "true"` +to the Probe's `targets.staticConfig.labels`. Omit the label or set it to +`"false"` to disable certificate checks for those targets. No separate exporter +module or StackPack is required. + +Both monitors evaluate every five minutes using the exact remaining duration, +not the chart's rounded days. **Endpoint certificate expiring** reports +**Deviating** below 14 days. **Endpoint certificate expiration imminent** reports +**Critical** below two days. Below two days both checks fail, and the endpoint's +combined health is Critical. At exactly two days only the warning fails; at +exactly 14 days neither certificate check fails. + +Adjust each threshold in the Monitors UI if needed. Values are in seconds: +14 days is `1209600`, and two days is `172800`. + +An expired or otherwise invalid certificate still fails the availability probe, +even when certificate warnings are disabled. When TLS negotiation fails, +Blackbox may not report an expiration timestamp; the availability monitor then +reports the failed check. Missing or zero expiration values are excluded from +both certificate monitors. + +## Install the StackPack and verify + +Install **Service Availability** from the StackPacks page after making its +package available to your instance. No installation parameters or additional +topology StackPacks are needed. + +In the Metrics Explorer, run: + +```promql +probe_success{service_availability="true"} +``` + +Each result should have an `endpoint` label and a value of `1` for healthy or `0` +for unhealthy. Also run: + +```promql +up{service_availability="true"} +``` + +Each scheduled target should have an `endpoint` label. A value of `1` means the +scrape succeeded; `0` means it failed. A successful scrape can still report +`probe_success=0` when the application is unhealthy. Both metrics contribute to +the endpoint and service availability monitors: any zero makes them Critical. +Check that `probe_http_status_code` and `probe_duration_seconds` +also have these labels. +For grouped endpoints, check that `availability_service` has your service name. +For certificate warnings, check that `probe_ssl_earliest_cert_expiry` has the +`endpoint` and `monitor_certificate_expiry` labels. + +Open **Service Availability → Endpoints**. Each URL has a component. A failed +check or scrape makes its availability monitor critical on the next evaluation; a +successful check clears that monitor. Certificate checks can still keep the +endpoint Deviating or Critical. The overview shows health, HTTP status and probe +duration. The highlight shows health and metric history, including certificate validity when +reported. **Service Availability → Services** lists grouped services and their +monitored endpoint counts. + +If the same URL is checked by multiple Probes, any failing observation makes +its single component critical. Missing telemetry is not converted to success. +An unreachable exporter, an unknown module, or a failed scrape can produce +`up=0` without `probe_success`. These collection failures now make the endpoint +Critical and still create its component, including on the first scrape. +If the collector stops sending both metrics, an independent expected-target +inventory or collection-health monitor is needed to detect the missing data. +If a Probe is removed or its data stops arriving, its component expires +15 minutes after the last matching metric. + +For a direct diagnostic, port-forward the exporter service: + +```bash +kubectl -n monitoring port-forward svc/prometheus-blackbox-exporter 9115:9115 +``` + +Then, in another terminal: + +```bash +curl --get 'http://localhost:9115/probe' \ + --data-urlencode 'module=http_2xx' \ + --data-urlencode 'target=https://example.com/health' \ + --data-urlencode 'debug=true' +``` + +Use the same module as your Probe. An HTTP 400 response with `Unknown module` +means the exporter has no module with that name; configure it or select an +existing module. A completed probe returns metrics even when the endpoint is +unhealthy. + +If the check succeeds here but metrics are missing in Observability, inspect the +Probe selector, Target Allocator, scraper logs and exporter connectivity. +If the endpoint returns an HTTP error or a connection/TLS failure, investigate +the application and the exporter's network path. diff --git a/stackpacks/service-availability/resources/deprovisioning.md b/stackpacks/service-availability/resources/deprovisioning.md new file mode 100644 index 0000000..cc04f07 --- /dev/null +++ b/stackpacks/service-availability/resources/deprovisioning.md @@ -0,0 +1,3 @@ +The Service Availability integration settings are being removed. Blackbox +Exporter and Probe resources are managed separately and are not uninstalled by +this StackPack. diff --git a/stackpacks/service-availability/resources/error.md b/stackpacks/service-availability/resources/error.md new file mode 100644 index 0000000..41dc8e5 --- /dev/null +++ b/stackpacks/service-availability/resources/error.md @@ -0,0 +1,6 @@ +The Service Availability StackPack could not be provisioned. Inspect the +provisioning error and verify that the platform supports StackPacks 2.0. + +If provisioning succeeds but endpoints are missing, inspect `probe_success` and `up` +metrics and confirm the `service_availability` and `endpoint` labels described +in the setup instructions. diff --git a/stackpacks/service-availability/resources/logo.svg b/stackpacks/service-availability/resources/logo.svg new file mode 100644 index 0000000..9d6982b --- /dev/null +++ b/stackpacks/service-availability/resources/logo.svg @@ -0,0 +1,3 @@ + + + diff --git a/stackpacks/service-availability/resources/overview.md b/stackpacks/service-availability/resources/overview.md new file mode 100644 index 0000000..8e29041 --- /dev/null +++ b/stackpacks/service-availability/resources/overview.md @@ -0,0 +1,25 @@ +# Service Availability + +Monitor HTTP and HTTPS endpoints with Prometheus Blackbox Exporter and the SUSE +Observability agent's OpenTelemetry Prometheus scraper. + +Each endpoint URL becomes a component with a health state. Open **Service +Availability → Endpoints** to see endpoint health, HTTP response codes and probe +duration. Select an endpoint to inspect its health checks and metric history. +HTTPS endpoints also show remaining TLS certificate validity as a duration in +the overview and as whole days in a time-series chart when available. + +Use the optional `availability_service` Probe target label to group several URLs +under one service. Open **Service Availability → Services** to see their combined +availability and navigate to the individual endpoints. + +The Probe example uses `http_2xx`: it follows redirects and requires a final 2xx +response. An optional `http_available` module accepts responses below 400 without +following redirects. Availability follows the selected module's rules; +connection failures and invalid or expired TLS certificates fail both modules. + +The exporter and Probe resources are installed separately. Topology is created +directly from metrics; no Kubernetes or OpenTelemetry StackPack installation is +required. Certificate expiration checks are optional: opt in with the +`monitor_certificate_expiry` Probe target label to warn below 14 days and report +Critical below two days. diff --git a/stackpacks/service-availability/resources/provisioning.md b/stackpacks/service-availability/resources/provisioning.md new file mode 100644 index 0000000..5e348f4 --- /dev/null +++ b/stackpacks/service-availability/resources/provisioning.md @@ -0,0 +1,3 @@ +The endpoint mapping, availability monitor and presentation are being installed. +Configure Blackbox Exporter and your Probe resources as described in the setup +instructions. diff --git a/stackpacks/service-availability/resources/waitingfordata.md b/stackpacks/service-availability/resources/waitingfordata.md new file mode 100644 index 0000000..efb6e54 --- /dev/null +++ b/stackpacks/service-availability/resources/waitingfordata.md @@ -0,0 +1,6 @@ +Waiting for `probe_success` or `up` metrics with `service_availability="true"` and an +`endpoint` label containing the HTTP or HTTPS target URL. + +Check that the Probe has `observability.suse.com/agent: scrape`, the OTel +Prometheus scraper and Target Allocator are enabled, and the Blackbox Exporter +service is reachable. diff --git a/stackpacks/service-availability/settings/component-mappings/endpoints.sty b/stackpacks/service-availability/settings/component-mappings/endpoints.sty new file mode 100644 index 0000000..5376f0c --- /dev/null +++ b/stackpacks/service-availability/settings/component-mappings/endpoints.sty @@ -0,0 +1,35 @@ +nodes: + - _type: OtelComponentMapping + name: HTTP endpoint + description: Creates one endpoint component from opted-in Blackbox probe metrics. + identifier: urn:stackpack:service-availability:otel-component-mapping:endpoint + input: + signal: ["METRICS"] + resource: + scope: + metric: + condition: "metric.name in ['probe_success', 'up']" + datapoint: + action: "'CREATE'" + condition: >- + 'service_availability' in datapoint.attributes && + datapoint.attributes['service_availability'] == 'true' && + 'endpoint' in datapoint.attributes && + datapoint.attributes['endpoint'].matches('^https?://.+') + vars: + - name: endpoint + value: "datapoint.attributes['endpoint']" + output: + identifier: "'urn:service-availability:endpoint/' + vars.endpoint" + name: vars.endpoint + typeName: "'http endpoint'" + required: + configuration: "{'url': vars.endpoint}" + tags: + - source: "'service-availability'" + target: stackpack + - source: vars.endpoint + target: endpoint.url + expireAfter: 900000 + rank: + specificity: 100 diff --git a/stackpacks/service-availability/settings/component-mappings/services.sty b/stackpacks/service-availability/settings/component-mappings/services.sty new file mode 100644 index 0000000..46fd2cf --- /dev/null +++ b/stackpacks/service-availability/settings/component-mappings/services.sty @@ -0,0 +1,36 @@ +nodes: + - _type: OtelComponentMapping + name: Availability service + description: Groups HTTP endpoints using the optional availability_service metric label. + identifier: urn:stackpack:service-availability:otel-component-mapping:service + input: + signal: ["METRICS"] + resource: + scope: + metric: + condition: "metric.name in ['probe_success', 'up']" + datapoint: + action: "'CREATE'" + condition: >- + 'service_availability' in datapoint.attributes && + datapoint.attributes['service_availability'] == 'true' && + 'endpoint' in datapoint.attributes && + datapoint.attributes['endpoint'].matches('^https?://.+') && + 'availability_service' in datapoint.attributes && + datapoint.attributes['availability_service'] != '' + vars: + - name: service + value: "datapoint.attributes['availability_service']" + output: + identifier: "'urn:service-availability:service/' + vars.service" + name: vars.service + typeName: "'availability service'" + required: + tags: + - source: "'service-availability'" + target: stackpack + - source: vars.service + target: availability.service + expireAfter: 900000 + rank: + specificity: 100 diff --git a/stackpacks/service-availability/settings/main-menu.sty b/stackpacks/service-availability/settings/main-menu.sty new file mode 100644 index 0000000..0fdba65 --- /dev/null +++ b/stackpacks/service-availability/settings/main-menu.sty @@ -0,0 +1,7 @@ +nodes: + - _type: MainMenuGroup + name: Service Availability + identifier: urn:stackpack:service-availability:main-menu-group:service-availability + iconbase64: !icon availability.svg + defaultOpen: true + items: [] diff --git a/stackpacks/service-availability/settings/monitors/certificate-expiration-critical.sty b/stackpacks/service-availability/settings/monitors/certificate-expiration-critical.sty new file mode 100644 index 0000000..c1fefd4 --- /dev/null +++ b/stackpacks/service-availability/settings/monitors/certificate-expiration-critical.sty @@ -0,0 +1,23 @@ +nodes: + - _type: Monitor + name: Endpoint certificate expiration imminent + description: Reports Critical when an opted-in endpoint's reported TLS certificate has less than two days remaining. + identifier: urn:stackpack:service-availability:monitor:certificate-expiration-imminent + function: urn:stackpack:common:monitor-function:threshold + arguments: + comparator: LT + threshold: 172800.0 + failureState: CRITICAL + metric: + query: >- + min by (endpoint) ( + probe_ssl_earliest_cert_expiry{service_availability="true",monitor_certificate_expiry="true",endpoint=~"https?://.+"} > 0 + ) - time() + aliasTemplate: Certificate validity ${endpoint} + unit: s + titleTemplate: Certificate expiration is imminent + urnTemplate: urn:service-availability:endpoint/${endpoint} + intervalSeconds: 300 + status: ENABLED + tags: ["availability", "certificates", "blackbox"] + remediationHint: !include remediation-hints/certificate-expiring.md.hbs diff --git a/stackpacks/service-availability/settings/monitors/certificate-expiration.sty b/stackpacks/service-availability/settings/monitors/certificate-expiration.sty new file mode 100644 index 0000000..d9e2a6b --- /dev/null +++ b/stackpacks/service-availability/settings/monitors/certificate-expiration.sty @@ -0,0 +1,23 @@ +nodes: + - _type: Monitor + name: Endpoint certificate expiring + description: Warns when an opted-in endpoint's reported TLS certificate has less than 14 days remaining. + identifier: urn:stackpack:service-availability:monitor:certificate-expiring + function: urn:stackpack:common:monitor-function:threshold + arguments: + comparator: LT + threshold: 1209600.0 + failureState: DEVIATING + metric: + query: >- + min by (endpoint) ( + probe_ssl_earliest_cert_expiry{service_availability="true",monitor_certificate_expiry="true",endpoint=~"https?://.+"} > 0 + ) - time() + aliasTemplate: Certificate validity ${endpoint} + unit: s + titleTemplate: Certificate is expiring + urnTemplate: urn:service-availability:endpoint/${endpoint} + intervalSeconds: 300 + status: ENABLED + tags: ["availability", "certificates", "blackbox"] + remediationHint: !include remediation-hints/certificate-expiring.md.hbs diff --git a/stackpacks/service-availability/settings/monitors/endpoint-availability.sty b/stackpacks/service-availability/settings/monitors/endpoint-availability.sty new file mode 100644 index 0000000..23509a1 --- /dev/null +++ b/stackpacks/service-availability/settings/monitors/endpoint-availability.sty @@ -0,0 +1,23 @@ +nodes: + - _type: Monitor + name: Endpoint unavailable + description: Reports failed endpoint checks or scrapes, including connection, TLS validation and exporter errors. + identifier: urn:stackpack:service-availability:monitor:endpoint-unavailable + function: urn:stackpack:common:monitor-function:threshold + arguments: + comparator: LT + threshold: 1.0 + failureState: CRITICAL + metric: + query: >- + min by (endpoint) ( + {__name__=~"probe_success|up",service_availability="true",endpoint=~"https?://.+"} + ) + aliasTemplate: Availability ${endpoint} + unit: short + titleTemplate: Endpoint is unavailable + urnTemplate: urn:service-availability:endpoint/${endpoint} + intervalSeconds: 60 + status: ENABLED + tags: ["availability", "http", "blackbox"] + remediationHint: !include remediation-hints/endpoint-unavailable.md.hbs diff --git a/stackpacks/service-availability/settings/monitors/service-availability.sty b/stackpacks/service-availability/settings/monitors/service-availability.sty new file mode 100644 index 0000000..1c94054 --- /dev/null +++ b/stackpacks/service-availability/settings/monitors/service-availability.sty @@ -0,0 +1,23 @@ +nodes: + - _type: Monitor + name: Service endpoint unavailable + description: Reports a service as unavailable when any of its labeled endpoint checks or scrapes fails. + identifier: urn:stackpack:service-availability:monitor:service-unavailable + function: urn:stackpack:common:monitor-function:threshold + arguments: + comparator: LT + threshold: 1.0 + failureState: CRITICAL + metric: + query: >- + min by (availability_service) ( + {__name__=~"probe_success|up",service_availability="true",availability_service!="",endpoint=~"https?://.+"} + ) + aliasTemplate: Availability ${availability_service} + unit: short + titleTemplate: An endpoint of ${availability_service} is unavailable + urnTemplate: urn:service-availability:service/${availability_service} + intervalSeconds: 60 + status: ENABLED + tags: ["availability", "service", "blackbox"] + remediationHint: !include remediation-hints/service-unavailable.md.hbs diff --git a/stackpacks/service-availability/settings/presentations/endpoints.sty b/stackpacks/service-availability/settings/presentations/endpoints.sty new file mode 100644 index 0000000..1a9a952 --- /dev/null +++ b/stackpacks/service-availability/settings/presentations/endpoints.sty @@ -0,0 +1,175 @@ +nodes: + - _type: ComponentPresentation + name: HTTP endpoints + description: Availability, HTTP response status, probe duration and optional TLS certificate validity. + identifier: urn:stackpack:service-availability:component-presentation:endpoint + binding: + _type: ComponentPresentationQueryBinding + query: 'label = "stackpack:service-availability" and type = "http endpoint"' + rank: + specificity: 10 + presentation: + icon: !icon endpoint.svg + overview: + name: + singular: endpoint + plural: endpoints + title: Endpoints + mainMenu: + group: Service Availability + order: 100 + columns: + - columnId: endpoint_health + title: Health + projection: + _type: HealthProjection + value: healthState + - columnId: endpoint_name + title: Endpoint + projection: + _type: ComponentLinkProjection + name: name + identifier: "identifiers[0]" + - columnId: endpoint_http_status + title: HTTP status + projection: + _type: MetricProjection + query: >- + max by (endpoint) (probe_http_status_code{service_availability="true",endpoint="${tags.singleValue('endpoint.url')}"}) + unit: short + decimalPlaces: 0 + sparkline: false + metricId: endpoint-http-status + - columnId: endpoint_probe_duration + title: Probe duration + projection: + _type: MetricProjection + query: >- + max by (endpoint) (probe_duration_seconds{service_availability="true",endpoint="${tags.singleValue('endpoint.url')}"}) + unit: s + decimalPlaces: 3 + sparkline: true + metricId: endpoint-probe-duration + - columnId: endpoint_certificate_validity + title: Certificate expires in + projection: + _type: MetricProjection + query: >- + min by (endpoint) (probe_ssl_earliest_cert_expiry{service_availability="true",endpoint="${tags.singleValue('endpoint.url')}"} > 0) - time() + unit: s + decimalPlaces: 0 + sparkline: false + metricId: endpoint-certificate-validity + sort: + - columnId: endpoint_health + - columnId: endpoint_name + highlight: + title: HTTP endpoint + fields: + - fieldId: health + title: Health + order: 100 + projection: + _type: HealthProjection + value: healthState + - fieldId: url + title: URL + order: 90 + projection: + _type: TextProjection + value: "tags.singleValue('endpoint.url')" + provisioning: + topologySourceSelector: "source.mapping.identifier == 'urn:stackpack:service-availability:otel-component-mapping:endpoint'" + showConfiguration: true + showStatus: false + relatedResources: + - resourceId: endpoint_services + title: Services + order: 100 + topologyQuery: '(withNeighborsOf(direction = "up", components = (identifier = "${identifiers[0]}"), levels = "1")) and type = "availability service"' + presentationIdentifier: urn:stackpack:service-availability:component-presentation:service + metrics: + - sectionId: availability + title: Availability + order: 100 + metrics: + - metricId: endpoint-probe-success + order: 100 + - metricId: endpoint-http-status + order: 90 + - metricId: endpoint-probe-duration + order: 80 + - sectionId: certificates + title: TLS certificate + order: 90 + metrics: + - metricId: endpoint-certificate-validity + order: 100 + metricPerspective: + tabs: + - tabId: availability + title: Availability + order: 100 + sections: + - sectionId: availability + title: Availability + order: 100 + metrics: + - metricId: endpoint-probe-success + - metricId: endpoint-http-status + - metricId: endpoint-probe-duration + - sectionId: certificates + title: TLS certificate + order: 90 + metrics: + - metricId: endpoint-certificate-validity + metrics: + - metricId: endpoint-probe-success + name: Probe success + description: One means the available check and scrape results succeeded; zero means a check or scrape failed. + metricQueries: + - expression: >- + min({__name__=~"probe_success|up",service_availability="true",endpoint="${tags.singleValue('endpoint.url')}"}) + alias: Probe success + chart: + _type: TimeSeriesChart + unit: short + - metricId: endpoint-http-status + name: HTTP status + description: The HTTP response code; zero indicates no HTTP response was received. + metricQueries: + - expression: >- + probe_http_status_code{service_availability="true",endpoint="${tags.singleValue('endpoint.url')}"} + alias: HTTP status ${service_name} + chart: + _type: TimeSeriesChart + unit: short + - metricId: endpoint-probe-duration + name: Probe duration + description: Total time spent performing the probe. + metricQueries: + - expression: >- + probe_duration_seconds{service_availability="true",endpoint="${tags.singleValue('endpoint.url')}"} + alias: Probe duration ${service_name} + chart: + _type: TimeSeriesChart + unit: s + - metricId: endpoint-certificate-validity + name: Certificate expires in (days) + description: Whole days until the earliest TLS certificate expiration; absent when no certificate is reported. + metricQueries: + - expression: >- + floor((min by (endpoint) (probe_ssl_earliest_cert_expiry{service_availability="true",endpoint="${tags.singleValue('endpoint.url')}"} > 0) - time()) / 86400) + alias: Days remaining + chart: + _type: TimeSeriesChart + unit: d + decimals: 0 + min: 0 + thresholds: + mode: absolute + steps: + - value: 14 + color: "#FF7F00" + - value: 2 + color: "#E00000" diff --git a/stackpacks/service-availability/settings/presentations/services.sty b/stackpacks/service-availability/settings/presentations/services.sty new file mode 100644 index 0000000..5436702 --- /dev/null +++ b/stackpacks/service-availability/settings/presentations/services.sty @@ -0,0 +1,102 @@ +nodes: + - _type: ComponentPresentation + name: Availability services + description: Service availability and the HTTP endpoints that contribute to it. + identifier: urn:stackpack:service-availability:component-presentation:service + binding: + _type: ComponentPresentationQueryBinding + query: 'label = "stackpack:service-availability" and type = "availability service"' + rank: + specificity: 10 + presentation: + icon: !icon service.svg + overview: + name: + singular: service + plural: services + title: Services + mainMenu: + group: Service Availability + order: 90 + columns: + - columnId: availability_service_health + title: Health + projection: + _type: HealthProjection + value: healthState + - columnId: availability_service_name + title: Service + projection: + _type: ComponentLinkProjection + name: name + identifier: "identifiers[0]" + - columnId: availability_service_endpoints + title: Monitored endpoints + projection: + _type: MetricProjection + query: >- + count by (availability_service) (count by (availability_service, endpoint) ({__name__=~"probe_success|up",service_availability="true",availability_service="${tags.singleValue('availability.service')}",endpoint=~"https?://.+"})) + unit: short + decimalPlaces: 0 + sparkline: false + metricId: service-endpoint-count + sort: + - columnId: availability_service_health + - columnId: availability_service_name + highlight: + title: Availability service + fields: + - fieldId: health + title: Health + order: 100 + projection: + _type: HealthProjection + value: healthState + relatedResources: + - resourceId: service_endpoints + title: Endpoints + order: 100 + topologyQuery: '(withNeighborsOf(direction = "down", components = (identifier = "${identifiers[0]}"), levels = "1")) and type = "http endpoint"' + presentationIdentifier: urn:stackpack:service-availability:component-presentation:endpoint + metrics: + - sectionId: availability + title: Availability + order: 100 + metrics: + - metricId: service-probe-success + order: 100 + - metricId: service-endpoint-count + order: 90 + metricPerspective: + tabs: + - tabId: availability + title: Availability + order: 100 + sections: + - sectionId: availability + title: Availability + order: 100 + metrics: + - metricId: service-probe-success + - metricId: service-endpoint-count + metrics: + - metricId: service-probe-success + name: Service probe success + description: One when all available service check and scrape results succeed; zero when any fails. + metricQueries: + - expression: >- + min({__name__=~"probe_success|up",service_availability="true",availability_service="${tags.singleValue('availability.service')}",endpoint=~"https?://.+"}) + alias: Service availability + chart: + _type: TimeSeriesChart + unit: short + - metricId: service-endpoint-count + name: Monitored endpoints + description: Number of distinct endpoint URLs with current check or scrape samples for this service, including failed scrapes. + metricQueries: + - expression: >- + count by (availability_service) (count by (availability_service, endpoint) ({__name__=~"probe_success|up",service_availability="true",availability_service="${tags.singleValue('availability.service')}",endpoint=~"https?://.+"})) + alias: Monitored endpoints + chart: + _type: TimeSeriesChart + unit: short diff --git a/stackpacks/service-availability/settings/relation-mappings/service-endpoints.sty b/stackpacks/service-availability/settings/relation-mappings/service-endpoints.sty new file mode 100644 index 0000000..85747b7 --- /dev/null +++ b/stackpacks/service-availability/settings/relation-mappings/service-endpoints.sty @@ -0,0 +1,31 @@ +nodes: + - _type: OtelRelationMapping + name: Service endpoint + description: Links a service to each URL labeled with its availability_service identity. + identifier: urn:stackpack:service-availability:otel-relation-mapping:service-endpoint + input: + signal: ["METRICS"] + resource: + scope: + metric: + condition: "metric.name in ['probe_success', 'up']" + datapoint: + action: "'CREATE'" + condition: >- + 'service_availability' in datapoint.attributes && + datapoint.attributes['service_availability'] == 'true' && + 'endpoint' in datapoint.attributes && + datapoint.attributes['endpoint'].matches('^https?://.+') && + 'availability_service' in datapoint.attributes && + datapoint.attributes['availability_service'] != '' + vars: + - name: service + value: "datapoint.attributes['availability_service']" + - name: endpoint + value: "datapoint.attributes['endpoint']" + output: + sourceId: "'urn:service-availability:service/' + vars.service" + targetId: "'urn:service-availability:endpoint/' + vars.endpoint" + typeName: "'has endpoint'" + dependencyType: "'HIERARCHICAL'" + expireAfter: 900000 diff --git a/stackpacks/service-availability/stackpack.yaml b/stackpacks/service-availability/stackpack.yaml new file mode 100644 index 0000000..9c33021 --- /dev/null +++ b/stackpacks/service-availability/stackpack.yaml @@ -0,0 +1,16 @@ +name: service-availability +version: "0.0.1" +schemaVersion: "2.0" +displayName: Service Availability +categories: ["Open Telemetry"] +logoUrl: !resource logo.svg +overviewUrl: !resource overview.md +configurationUrls: + NOT_INSTALLED: !resource configuration.md + PROVISIONING: !resource provisioning.md + WAITING_FOR_DATA: !resource waitingfordata.md + INSTALLED: !resource configuration.md + DEPROVISIONING: !resource deprovisioning.md + ERROR: !resource error.md +provision: + companionStackPacks: [] From 3917c7512d4b56ddd970e4b09f348ef810c8b1d4 Mon Sep 17 00:00:00 2001 From: Lukasz Marchewka Date: Tue, 10 Nov 2026 08:29:05 +0100 Subject: [PATCH 2/2] Simplify Service Availability documentation Use the standard http_2xx example and remove redundant installation and development instructions. Shorten certificate guidance and align the overview and remediation hints. Use the Endpoints and Services pages for verification, with metric queries and exporter diagnostics in a concise troubleshooting section. --- stackpacks/service-availability/README.md | 60 +---- .../endpoint-unavailable.md.hbs | 3 +- .../resources/configuration.md | 225 ++++++------------ .../resources/overview.md | 5 +- 4 files changed, 81 insertions(+), 212 deletions(-) diff --git a/stackpacks/service-availability/README.md b/stackpacks/service-availability/README.md index 8690005..d4f0e96 100644 --- a/stackpacks/service-availability/README.md +++ b/stackpacks/service-availability/README.md @@ -1,8 +1,8 @@ # Service Availability -A declarative StackPack 2.0 integration for HTTP and HTTPS checks collected by -Prometheus Blackbox Exporter through the SUSE Observability OTel Prometheus -scraper. See [setup instructions](resources/configuration.md). +Monitor HTTP and HTTPS endpoints with Prometheus Blackbox Exporter and the SUSE +Observability agent's OpenTelemetry Prometheus scraper. See the +[setup instructions](resources/configuration.md). The pack creates one `http endpoint` component per URL and an availability monitor. The optional `availability_service` label groups multiple URLs under an @@ -10,10 +10,6 @@ monitor. The optional `availability_service` label groups multiple URLs under an **Service Availability → Endpoints** and **Services** provide overviews and highlight pages. TLS certificate validity is displayed when available. Optional certificate monitors warn below 14 days and report Critical below two days. -Certificate expiration is shown as a duration in the overview. The time-series -chart shows whole days with a vertical axis starting at zero. An orange dotted -line marks the default 14-day warning threshold; a red dotted line marks the -two-day critical threshold. ## Telemetry contract @@ -35,10 +31,9 @@ in `endpoint`, independently of the Prometheus receiver's conversion of `job` and `instance` to OTel resource attributes. The Probe example uses `http_2xx`, which follows redirects and requires a final -2xx response. The optional `http_available` module implements the below-400 -policy without following redirects. Both supply the same metrics; availability -uses the selected module's `probe_success` result. For an existing exporter, -check additional module constraints, such as requiring HTTPS. +2xx response. Availability uses the selected module's `probe_success` result. +For an existing exporter, check additional module constraints, such as requiring +HTTPS. Topology identifier: `urn:service-availability:endpoint/`. The monitor constructs the same identifier from its `endpoint` result label. @@ -63,45 +58,4 @@ certificate result. Components and relations expire after 15 minutes without matching metrics. The service monitor aggregates observed samples; it cannot detect a missing URL without an inventory of expected targets. A disappearing URL can reduce the -displayed endpoint count. Missing telemetry is not evidence of successful checks. -Runtime topology, health and UI behavior require validation on an instance. - -## Development - -```bash -sts stackpack package -d stackpacks/service-availability -sts stackpack validate -d stackpacks/service-availability -``` - -The repository CI discovers this directory automatically. For offline validation, -use the `VALIDATOR_IMAGE` selected by `.github/workflows/stackpacks-ci.yml`: - -```bash -bash scripts/ci/validate_stackpack.sh stackpacks/service-availability -``` - -The manifest follows the StackPack SDK's `YamlReader` in -`StackVista/stackpacks`. Setup instructions use `configurationUrls` because the -SDK ignores the legacy `detailedOverviewUrl` field for schema 2.0. The same -instructions remain available after installation. - -On an explicitly selected development instance, test installation with: - -```bash -sts stackpack test-deploy -d stackpacks/service-availability --yes -``` - -With `http_2xx`, verify that a 200 is healthy and redirects succeed only when -their final response is 2xx. With `http_available`, verify that a 302 is healthy -without following the redirect. Verify that a 404 or 500, connection errors and -TLS validation errors are critical with both modules. Check that a failed target -still creates its component, duplicate URLs merge, and non-opted-in probes do -not create components. Group two URLs under one service and verify that either -URL failing makes that service critical. Check that unlabeled URLs remain -standalone endpoints. - -For certificate warnings, verify certificates inside and outside the 14-day -window, opt-out targets, HTTP targets, and missing or zero timestamps. Valid -certificates inside the warning window should keep availability clear while -the certificate monitor warns. Inspect both overviews, the highlights and the -service-to-endpoint relations. +displayed endpoint count. Missing telemetry is not evidence of successful checks. \ No newline at end of file diff --git a/stackpacks/service-availability/includes/remediation-hints/endpoint-unavailable.md.hbs b/stackpacks/service-availability/includes/remediation-hints/endpoint-unavailable.md.hbs index 96e0724..e143472 100644 --- a/stackpacks/service-availability/includes/remediation-hints/endpoint-unavailable.md.hbs +++ b/stackpacks/service-availability/includes/remediation-hints/endpoint-unavailable.md.hbs @@ -1,8 +1,7 @@ Check the endpoint **{{ labels.endpoint }}** from the Blackbox Exporter's network. 1. Inspect `probe_http_status_code` and the exporter's probe output. With - `http_2xx`, the final response must be 2xx. With `http_available`, responses - below 400 are accepted without following redirects. + `http_2xx`, the final response after redirects must be 2xx. 2. If no response was received, check DNS, routing, network policies, timeouts and the endpoint's workload. 3. For HTTPS, check the certificate chain, hostname and validity. Expired or diff --git a/stackpacks/service-availability/resources/configuration.md b/stackpacks/service-availability/resources/configuration.md index 296a51f..e826f49 100644 --- a/stackpacks/service-availability/resources/configuration.md +++ b/stackpacks/service-availability/resources/configuration.md @@ -25,10 +25,9 @@ Collection's access instructions to authenticate Helm and create its image pull secret in the `monitoring` namespace. Save the following as `blackbox-values.yaml`. Set `appco-credentials` to the name -of your Application Collection image pull secret. The Probe example uses -`http_2xx`, which follows redirects and requires a final 2xx response. The -optional `http_available` module accepts responses below 400 without following -redirects. +of your Application Collection image pull secret. The `http_2xx` module follows +redirects and considers the endpoint healthy when the final response is 2xx. +It supports HTTP and HTTPS and verifies TLS certificates. ```yaml global: @@ -40,46 +39,11 @@ config: prober: http timeout: 5s http: + valid_http_versions: ["HTTP/1.0", "HTTP/1.1", "HTTP/2.0"] follow_redirects: true - http_available: - prober: http - timeout: 5s - http: - follow_redirects: false - valid_status_codes: - [100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, - 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, - 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, - 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, - 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, - 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, - 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, - 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, - 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, - 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, - 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, - 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, - 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, - 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, - 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, - 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, - 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, - 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, - 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, - 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, - 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, - 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, - 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, - 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, - 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399] + fail_if_not_ssl: true ``` -Blackbox defaults to accepting 200–299 when no accepted status codes are listed. -The `http_available` module lists 100–399 explicitly to implement the below-400 -policy. The modules above support HTTP and HTTPS and keep normal TLS certificate -and hostname verification enabled. Do not enable `insecure_skip_verify` when -certificate validation is part of the availability check. - Install through your deployment tooling, or use Helm: ```bash @@ -89,16 +53,9 @@ helm upgrade --install prometheus-blackbox-exporter \ --values blackbox-values.yaml ``` -For an existing exporter, use its `http_2xx` module if it meets your requirements. -Check its configuration: existing modules may add constraints such as -`fail_if_not_ssl: true`, which requires HTTPS. Merge any additional modules into -the current configuration rather than replacing modules used by other checks. -Use a chart version approved for your environment. - -To use the below-400 policy, add `http_available` and set the Probe's -`spec.module` to `http_available`. Both modules supply the same metrics, so the -StackPack mappings and monitors work with either. The selected module determines -which responses are healthy. +If you already have an exporter, reuse its `http_2xx` module after checking its +settings. For example, `fail_if_not_ssl: true` requires HTTPS. Preserve modules +used by existing checks when updating its configuration. ## Enable Probe discovery @@ -136,12 +93,6 @@ Multiple targets can be listed under `static`; their full URLs identify separate endpoint components. The example groups both URLs under one service and enables optional certificate expiration warnings. -`spec.jobName` can be any descriptive scrape job name; `service-availability` -was only an example. The StackPack does not select targets by job name. -It selects targets using `service_availability: "true"` and identifies each -endpoint by its `endpoint` label. The optional `availability_service` label -provides the service identity, independently of `jobName`. - ```yaml apiVersion: monitoring.coreos.com/v1 kind: Probe @@ -176,21 +127,19 @@ spec: | Field | What to change | | --- | --- | | `metadata.name` | Replace `REPLACE_WITH_YOUR_PROBE_NAME` with a valid lowercase Kubernetes resource name, such as `checkout-health`. | -| `spec.jobName` | Replace `REPLACE_WITH_YOUR_JOB_NAME` with your scrape job name. | +| `spec.jobName` | Replace `REPLACE_WITH_YOUR_JOB_NAME` with any descriptive scrape job name. | | `targets.staticConfig.static` | Replace `https://example.com/health` and `https://example.com/api/ready` with your complete HTTP or HTTPS URLs, including the correct paths. Add or remove URLs as needed. | | `targets.staticConfig.labels.availability_service` | Replace `REPLACE_WITH_YOUR_SERVICE_NAME` with a service identity such as `production/my-service`. Remove this label if you do not want service grouping. | | `metadata.namespace` and `spec.prober.url` | Adjust if your Probe or exporter uses a different namespace, or the exporter has a different service name. | | `targets.staticConfig.labels.monitor_certificate_expiry` | Keep `"true"` to enable certificate checks; set `"false"` or remove the label to disable them. | -If the exporter uses a different release name or namespace, adjust `prober.url`. The relabeling copies the complete target URL into the `endpoint` metric label. -Keep this relabeling and the `service_availability` label: they are the contract -used by the topology mapping, monitor and presentation. For an existing Probe, -add both to its `targets.staticConfig`. Set `spec.module` to `http_2xx`, or to -`http_available` after configuring that optional module in the exporter. +Keep this relabeling and the `service_availability: "true"` label so the +StackPack can discover your endpoints. For an existing Probe, add both to its +`targets.staticConfig` and select the exporter's `http_2xx` module. The interval can be changed; keep it well below the 15-minute topology expiry. -Do not put credentials in a target URL or its query string: that URL is stored +**Do not put credentials in a target URL or its query string**: that URL is stored as a metric label and shown in the UI. Configure authentication separately in the exporter's HTTP module when an endpoint requires it. @@ -222,111 +171,79 @@ matching metric. ## Optional certificate expiration warnings -Blackbox reports `probe_ssl_earliest_cert_expiry` when a TLS certificate is -available, including HTTP targets that redirect to HTTPS. Both certificate -monitors use the reported certificate data regardless of the target's initial -HTTP or HTTPS scheme. Plain HTTP checks without TLS have no certificate result. -The **Certificate expires in** overview column shows the remaining validity -as a duration until the earliest expiration in the -certificate chain. The chart in the **TLS certificate** highlight section shows -whole days, rounded down, so less than one day remaining displays as zero. -The time-series chart uses the days unit with zero decimal places and a vertical -axis starting at zero, representing expiration. An orange dotted line marks -the 14-day warning window; a red dotted line marks the two-day critical window. -The UI labels these lines **Threshold 1** and **Threshold 2**; custom threshold -names are not supported. If you change a monitor threshold, update its chart -reference to match. Checks without certificate data show no value, rather -than zero. - To enable both certificate monitors, add `monitor_certificate_expiry: "true"` to the Probe's `targets.staticConfig.labels`. Omit the label or set it to -`"false"` to disable certificate checks for those targets. No separate exporter -module or StackPack is required. +`"false"` to disable certificate checks for those targets. -Both monitors evaluate every five minutes using the exact remaining duration, -not the chart's rounded days. **Endpoint certificate expiring** reports +Both monitors evaluate every five minutes. **Endpoint certificate expiring** reports **Deviating** below 14 days. **Endpoint certificate expiration imminent** reports -**Critical** below two days. Below two days both checks fail, and the endpoint's -combined health is Critical. At exactly two days only the warning fails; at -exactly 14 days neither certificate check fails. +**Critical** below two days. -Adjust each threshold in the Monitors UI if needed. Values are in seconds: -14 days is `1209600`, and two days is `172800`. +The **Certificate expires in** overview column shows the remaining validity of +the certificate chain. The **TLS certificate** highlight chart shows whole days, +rounded down, with lines marking the warning and critical thresholds. Certificate +data is also available for HTTP endpoints that redirect to HTTPS. -An expired or otherwise invalid certificate still fails the availability probe, -even when certificate warnings are disabled. When TLS negotiation fails, -Blackbox may not report an expiration timestamp; the availability monitor then -reports the failed check. Missing or zero expiration values are excluded from -both certificate monitors. +To change the thresholds, edit the monitors in the Monitors UI. Values are in +seconds: 14 days is `1209600`, and two days is `172800`. Update the chart's +threshold lines to match. -## Install the StackPack and verify +An expired or invalid certificate fails the availability check even when +certificate warnings are disabled. Checks without certificate data show no +expiration value and do not trigger the certificate monitors. -Install **Service Availability** from the StackPacks page after making its -package available to your instance. No installation parameters or additional -topology StackPacks are needed. +## Check your endpoints -In the Metrics Explorer, run: +Open **Service Availability → Endpoints** to check that your URLs appear with +their health, HTTP response codes and probe durations. Select an endpoint to +inspect its health checks and metric history. A failed check or scrape makes +its availability monitor Critical on the next evaluation. -```promql -probe_success{service_availability="true"} -``` +Open **Service Availability → Services** to see grouped services and their +endpoint counts. Select a service to inspect its related endpoints. -Each result should have an `endpoint` label and a value of `1` for healthy or `0` -for unhealthy. Also run: +## Troubleshooting -```promql -up{service_availability="true"} -``` +1. **Endpoints or services are missing.** In the Metrics Explorer, run: -Each scheduled target should have an `endpoint` label. A value of `1` means the -scrape succeeded; `0` means it failed. A successful scrape can still report -`probe_success=0` when the application is unhealthy. Both metrics contribute to -the endpoint and service availability monitors: any zero makes them Critical. -Check that `probe_http_status_code` and `probe_duration_seconds` -also have these labels. -For grouped endpoints, check that `availability_service` has your service name. -For certificate warnings, check that `probe_ssl_earliest_cert_expiry` has the -`endpoint` and `monitor_certificate_expiry` labels. - -Open **Service Availability → Endpoints**. Each URL has a component. A failed -check or scrape makes its availability monitor critical on the next evaluation; a -successful check clears that monitor. Certificate checks can still keep the -endpoint Deviating or Critical. The overview shows health, HTTP status and probe -duration. The highlight shows health and metric history, including certificate validity when -reported. **Service Availability → Services** lists grouped services and their -monitored endpoint counts. - -If the same URL is checked by multiple Probes, any failing observation makes -its single component critical. Missing telemetry is not converted to success. -An unreachable exporter, an unknown module, or a failed scrape can produce -`up=0` without `probe_success`. These collection failures now make the endpoint -Critical and still create its component, including on the first scrape. -If the collector stops sending both metrics, an independent expected-target -inventory or collection-health monitor is needed to detect the missing data. -If a Probe is removed or its data stops arriving, its component expires -15 minutes after the last matching metric. - -For a direct diagnostic, port-forward the exporter service: + ```promql + {__name__=~"probe_success|up",service_availability="true"} + ``` -```bash -kubectl -n monitoring port-forward svc/prometheus-blackbox-exporter 9115:9115 -``` + Each result needs an `endpoint` label containing the full HTTP or HTTPS URL. + For service grouping, it also needs a nonempty `availability_service` label. + If labels are missing, check the Probe's target labels and relabeling. + If no metrics arrive, check Probe discovery, the Target Allocator and scraper + logs. Components expire after 15 minutes without data. -Then, in another terminal: +2. **An endpoint is Critical.** Check its availability monitor: + `probe_success=0` means the endpoint check failed; `up=0` means the scraper + could not collect the probe result. Check the endpoint's application, network + path and TLS certificate for a failed probe. For a failed scrape, check the + exporter connection and module name. If availability is healthy, check + whether a certificate expiration monitor is reporting the problem. -```bash -curl --get 'http://localhost:9115/probe' \ - --data-urlencode 'module=http_2xx' \ - --data-urlencode 'target=https://example.com/health' \ - --data-urlencode 'debug=true' -``` +3. **Test the exporter directly.** Port-forward its service: + + ```bash + kubectl -n monitoring port-forward svc/prometheus-blackbox-exporter 9115:9115 + ``` + + In another terminal, replace the target URL with your endpoint and use the + same module as your Probe: + + ```bash + curl --get 'http://localhost:9115/probe' \ + --data-urlencode 'module=http_2xx' \ + --data-urlencode 'target=https://example.com/health' \ + --data-urlencode 'debug=true' + ``` -Use the same module as your Probe. An HTTP 400 response with `Unknown module` -means the exporter has no module with that name; configure it or select an -existing module. A completed probe returns metrics even when the endpoint is -unhealthy. + `Unknown module` means the exporter has no module with that name; configure + it or select an existing module. A completed probe returns metrics even when + the endpoint is unhealthy. Use the debug output to identify HTTP, connection + or TLS failures. -If the check succeeds here but metrics are missing in Observability, inspect the -Probe selector, Target Allocator, scraper logs and exporter connectivity. -If the endpoint returns an HTTP error or a connection/TLS failure, investigate -the application and the exporter's network path. + If this check succeeds but metrics are missing in SUSE Observability, check + the Probe selector, Target Allocator, scraper logs and connectivity from the + scraper to the exporter. diff --git a/stackpacks/service-availability/resources/overview.md b/stackpacks/service-availability/resources/overview.md index 8e29041..d33a865 100644 --- a/stackpacks/service-availability/resources/overview.md +++ b/stackpacks/service-availability/resources/overview.md @@ -14,9 +14,8 @@ under one service. Open **Service Availability → Services** to see their combi availability and navigate to the individual endpoints. The Probe example uses `http_2xx`: it follows redirects and requires a final 2xx -response. An optional `http_available` module accepts responses below 400 without -following redirects. Availability follows the selected module's rules; -connection failures and invalid or expired TLS certificates fail both modules. +response. Connection failures and invalid or expired TLS certificates also fail +the availability check. The exporter and Probe resources are installed separately. Topology is created directly from metrics; no Kubernetes or OpenTelemetry StackPack installation is