diff --git a/.agents/skills/create-stackpack/SKILL.md b/.agents/skills/create-stackpack/SKILL.md index c6a72a4..db59b93 100644 --- a/.agents/skills/create-stackpack/SKILL.md +++ b/.agents/skills/create-stackpack/SKILL.md @@ -13,10 +13,11 @@ Use the current declarative scaffold and adapt it to that integration. Read the official [creation guide](https://documentation.suse.com/cloudnative/suse-observability/latest/en/setup/custom-integrations/create.html) before starting. Follow its reference links for the settings being changed. Use the [StackPack reference](https://documentation.suse.com/cloudnative/suse-observability/latest/en/setup/custom-integrations/stackpacks.html) for project structure, versioning, and CLI commands. -Use [contrib-stackpacks](https://github.com/StackVista/contrib-stackpacks/tree/main/stackpacks) for working examples: +Use [core-stackpacks](https://github.com/StackVista/core-stackpacks/tree/main/stackpacks) for working examples: - `open-telemetry`: component and relation mappings, presentations, and monitors. - `otel-k8s-crd`: topology from Kubernetes custom-resource events. +- `service-availability`: endpoint and service topology, Blackbox availability monitors, and optional TLS certificate checks. - `suse-observability`: presentations and monitors for existing platform components. Read only the relevant example files. Prefer current schemas over copying an entire integration. diff --git a/README.md b/README.md index a94c634..744f75d 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,7 @@ The SUSE Observability team maintains these StackPacks: - `open-telemetry` - `otel-k8s-crd` +- [`service-availability`](stackpacks/service-availability/README.md) — HTTP and HTTPS endpoint availability, service grouping, and optional TLS certificate expiration checks. - `suse-observability` The product includes the core bundle by default. diff --git a/stackpacks/service-availability/README.md b/stackpacks/service-availability/README.md new file mode 100644 index 0000000..d4f0e96 --- /dev/null +++ b/stackpacks/service-availability/README.md @@ -0,0 +1,61 @@ +# Service Availability + +Monitor HTTP and HTTPS endpoints with Prometheus Blackbox Exporter and the SUSE +Observability agent's OpenTelemetry Prometheus scraper. See the +[setup instructions](resources/configuration.md). + +The pack creates one `http endpoint` component per URL and an availability +monitor. The optional `availability_service` label groups multiple URLs under an +`availability service` component with its own availability monitor. +**Service Availability → Endpoints** and **Services** provide overviews and +highlight pages. TLS certificate validity is displayed when available. Optional +certificate monitors warn below 14 days and report Critical below two days. + +## Telemetry contract + +| Input | Purpose | +| --- | --- | +| `probe_success` | Creates endpoint topology and reports whether the endpoint check succeeded. | +| `up` | Creates endpoint topology even when the first scrape fails, and makes availability Critical when a scrape fails. | +| `probe_http_status_code` | Displays the received HTTP status. | +| `probe_duration_seconds` | Displays total probe duration. | +| `probe_ssl_earliest_cert_expiry` | Optional Unix timestamp of the earliest certificate expiration. | +| Datapoint label `service_availability="true"` | Opts a target into this integration. | +| Datapoint label `endpoint` | The complete target URL, including its path and query. | +| Datapoint label `availability_service` | Optional nonempty identity grouping endpoints into a service. | +| Datapoint label `monitor_certificate_expiry="true"` | Opts targets into expiration checks when TLS certificate data is reported, including HTTP redirects to HTTPS. | + +These metric and attribute names are the Blackbox/Prometheus contract, not +OpenTelemetry semantic conventions. The Probe example preserves target identity +in `endpoint`, independently of the Prometheus receiver's conversion of `job` +and `instance` to OTel resource attributes. + +The Probe example uses `http_2xx`, which follows redirects and requires a final +2xx response. Availability uses the selected module's `probe_success` result. +For an existing exporter, check additional module constraints, such as requiring +HTTPS. + +Topology identifier: `urn:service-availability:endpoint/`. +The monitor constructs the same identifier from its `endpoint` result label. +The same URL checked by multiple Probes has one component; the monitor uses the +minimum success value, so any failing observation makes it critical. Different +paths or queries are separate endpoints. + +Service identifier: `urn:service-availability:service/`. +The service mapping and monitor share this identity, and a hierarchical relation +links the service to every endpoint. Use environment-qualified service identities +when the same service name should represent separate installations. + +Availability monitors run every minute against the latest samples, with no +additional five-minute failure delay. Any failing observation makes its endpoint +critical; any failing check labeled for a service makes that service critical. +The two optional certificate monitors run every five minutes and report +Deviating below 14 days remaining and Critical below two days. They use reported +TLS certificate data, including HTTP targets that redirect to HTTPS. Plain HTTP +targets without TLS, missing timestamps and zero timestamps produce no +certificate result. + +Components and relations expire after 15 minutes without matching metrics. +The service monitor aggregates observed samples; it cannot detect a missing URL +without an inventory of expected targets. A disappearing URL can reduce the +displayed endpoint count. Missing telemetry is not evidence of successful checks. \ No newline at end of file diff --git a/stackpacks/service-availability/icons/availability.svg b/stackpacks/service-availability/icons/availability.svg new file mode 100644 index 0000000..9d6982b --- /dev/null +++ b/stackpacks/service-availability/icons/availability.svg @@ -0,0 +1,3 @@ + + + diff --git a/stackpacks/service-availability/icons/endpoint.svg b/stackpacks/service-availability/icons/endpoint.svg new file mode 100644 index 0000000..57cdef0 --- /dev/null +++ b/stackpacks/service-availability/icons/endpoint.svg @@ -0,0 +1,5 @@ + + + + + diff --git a/stackpacks/service-availability/icons/service.svg b/stackpacks/service-availability/icons/service.svg new file mode 100644 index 0000000..aab5ee5 --- /dev/null +++ b/stackpacks/service-availability/icons/service.svg @@ -0,0 +1,6 @@ + + + + + + diff --git a/stackpacks/service-availability/includes/remediation-hints/certificate-expiring.md.hbs b/stackpacks/service-availability/includes/remediation-hints/certificate-expiring.md.hbs new file mode 100644 index 0000000..fd465ac --- /dev/null +++ b/stackpacks/service-availability/includes/remediation-hints/certificate-expiring.md.hbs @@ -0,0 +1,14 @@ +The earliest certificate expiration reported for **{{ labels.endpoint }}** is +less than the configured expiration threshold away. The default warning is +14 days; the critical window is two days. Prioritize renewal when Critical. + +1. Inspect the certificate served from the Blackbox Exporter's network, including + intermediate certificates in its chain. +2. Check your certificate issuer and renewal automation. Renew the certificate + and confirm that the endpoint serves the renewed chain. +3. Keep normal TLS verification enabled. An expired or invalid certificate also + fails the endpoint's availability check. + +Certificate expiration checks are selected by `monitor_certificate_expiry="true"` +on Probe target labels. Change the monitor's threshold in seconds to adjust the +warning or critical window. diff --git a/stackpacks/service-availability/includes/remediation-hints/endpoint-unavailable.md.hbs b/stackpacks/service-availability/includes/remediation-hints/endpoint-unavailable.md.hbs new file mode 100644 index 0000000..e143472 --- /dev/null +++ b/stackpacks/service-availability/includes/remediation-hints/endpoint-unavailable.md.hbs @@ -0,0 +1,14 @@ +Check the endpoint **{{ labels.endpoint }}** from the Blackbox Exporter's network. + +1. Inspect `probe_http_status_code` and the exporter's probe output. With + `http_2xx`, the final response after redirects must be 2xx. +2. If no response was received, check DNS, routing, network policies, timeouts and + the endpoint's workload. +3. For HTTPS, check the certificate chain, hostname and validity. Expired or + untrusted certificates fail the probe. +4. Confirm that the Probe's module exists in the exporter configuration and + that the exporter is running and reachable from the OTel scraper. + Check `up` for this endpoint: zero means the scrape failed, even if + `probe_success` is absent or still shows an older successful check. + +If multiple Probes check this URL, any failing probe makes the endpoint critical. diff --git a/stackpacks/service-availability/includes/remediation-hints/service-unavailable.md.hbs b/stackpacks/service-availability/includes/remediation-hints/service-unavailable.md.hbs new file mode 100644 index 0000000..a72c58f --- /dev/null +++ b/stackpacks/service-availability/includes/remediation-hints/service-unavailable.md.hbs @@ -0,0 +1,11 @@ +At least one endpoint of **{{ labels.availability_service }}** failed its check +or scrape. + +Open the service highlight and inspect its related endpoints. Check the failing +endpoint's health check, HTTP response code and probe duration, then investigate +its application, network path or TLS certificate. Check the endpoint's `up` +metric and scraper logs when its Blackbox metrics are missing. + +Every labeled URL is required for service availability. If a URL is an optional +dependency, monitor it separately without the service's `availability_service` +label. diff --git a/stackpacks/service-availability/resources/configuration.md b/stackpacks/service-availability/resources/configuration.md new file mode 100644 index 0000000..e826f49 --- /dev/null +++ b/stackpacks/service-availability/resources/configuration.md @@ -0,0 +1,249 @@ +# Configure service availability checks + +The data flow is: + +```text +Probe → Target Allocator → OTel Prometheus scraper → Blackbox Exporter + ↓ ↓ + SUSE Observability HTTP(S) endpoint +``` + +The Target Allocator discovers the Probe and gives the scraper its scrape job. +The scraper calls Blackbox Exporter's `/probe` endpoint with the target URL. +Blackbox Exporter performs the check and returns metrics, which the scraper +forwards to SUSE Observability. This StackPack turns those metrics into endpoint +components and health checks. It also uses the scraper's `up` metric to detect +failed scrapes, including exporter connection errors, scrape timeouts and unknown +modules. + +## Install Blackbox Exporter + +Use the **Prometheus Blackbox Exporter** chart from **SUSE Application Collection +(AppCo)**, available at +[SUSE Application Collection](https://apps.rancher.io/). Follow Application +Collection's access instructions to authenticate Helm and create its image pull +secret in the `monitoring` namespace. + +Save the following as `blackbox-values.yaml`. Set `appco-credentials` to the name +of your Application Collection image pull secret. The `http_2xx` module follows +redirects and considers the endpoint healthy when the final response is 2xx. +It supports HTTP and HTTPS and verifies TLS certificates. + +```yaml +global: + imagePullSecrets: + - appco-credentials +config: + modules: + http_2xx: + prober: http + timeout: 5s + http: + valid_http_versions: ["HTTP/1.0", "HTTP/1.1", "HTTP/2.0"] + follow_redirects: true + fail_if_not_ssl: true +``` + +Install through your deployment tooling, or use Helm: + +```bash +helm upgrade --install prometheus-blackbox-exporter \ + oci://dp.apps.rancher.io/charts/prometheus-blackbox-exporter \ + --namespace monitoring --create-namespace \ + --values blackbox-values.yaml +``` + +If you already have an exporter, reuse its `http_2xx` module after checking its +settings. For example, `fail_if_not_ssl: true` requires HTTPS. Preserve modules +used by existing checks when updating its configuration. + +## Enable Probe discovery + +Merge these overrides into your existing SUSE Observability agent installation: + +```yaml +otel: + enabled: true + prometheusScraping: + monitorCrds: + enabled: true +``` + +Prometheus scraping is enabled by default. If your existing values explicitly +disable it, also set `otel.prometheusScraping.enabled: true`. +The chart's default Probe selector matches `observability.suse.com/agent: scrape`, +as used in the example below, and its default namespace selector includes all +namespaces. If your installation overrides these selectors, use matching Probe +labels and include the `monitoring` namespace. + +`monitorCrds.enabled: true` installs the ServiceMonitor, PodMonitor and Probe +CRDs. Omit this override when those CRDs are already managed by another chart +or deployment tool. + +The scraper must reach the exporter, and the exporter must reach each endpoint, including DNS +and HTTPS connections. An internal endpoint is supported when it is reachable +from the exporter's network. + +## Create a Probe + +Create this resource through your Kubernetes deployment tooling. +**Replace all uppercase `REPLACE_WITH_...` values and the example URLs before creating the Probe.** Keep the label +name `availability_service`; replace its value with your own service identity. +Multiple targets can be listed under `static`; their full URLs identify separate +endpoint components. The example groups both URLs under one service and enables +optional certificate expiration warnings. + +```yaml +apiVersion: monitoring.coreos.com/v1 +kind: Probe +metadata: + name: REPLACE_WITH_YOUR_PROBE_NAME + namespace: monitoring + labels: + observability.suse.com/agent: scrape +spec: + jobName: REPLACE_WITH_YOUR_JOB_NAME + interval: 60s + scrapeTimeout: 10s + module: http_2xx + prober: + url: prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115 + scheme: http + path: /probe + targets: + staticConfig: + static: + - https://example.com/health + - https://example.com/api/ready + labels: + service_availability: "true" + availability_service: REPLACE_WITH_YOUR_SERVICE_NAME + monitor_certificate_expiry: "true" + relabelingConfigs: + - sourceLabels: [__param_target] + targetLabel: endpoint +``` + +| Field | What to change | +| --- | --- | +| `metadata.name` | Replace `REPLACE_WITH_YOUR_PROBE_NAME` with a valid lowercase Kubernetes resource name, such as `checkout-health`. | +| `spec.jobName` | Replace `REPLACE_WITH_YOUR_JOB_NAME` with any descriptive scrape job name. | +| `targets.staticConfig.static` | Replace `https://example.com/health` and `https://example.com/api/ready` with your complete HTTP or HTTPS URLs, including the correct paths. Add or remove URLs as needed. | +| `targets.staticConfig.labels.availability_service` | Replace `REPLACE_WITH_YOUR_SERVICE_NAME` with a service identity such as `production/my-service`. Remove this label if you do not want service grouping. | +| `metadata.namespace` and `spec.prober.url` | Adjust if your Probe or exporter uses a different namespace, or the exporter has a different service name. | +| `targets.staticConfig.labels.monitor_certificate_expiry` | Keep `"true"` to enable certificate checks; set `"false"` or remove the label to disable them. | + +The relabeling copies the complete target URL into the `endpoint` metric label. +Keep this relabeling and the `service_availability: "true"` label so the +StackPack can discover your endpoints. For an existing Probe, add both to its +`targets.staticConfig` and select the exporter's `http_2xx` module. +The interval can be changed; keep it well below the 15-minute topology expiry. + +**Do not put credentials in a target URL or its query string**: that URL is stored +as a metric label and shown in the UI. Configure authentication separately in +the exporter's HTTP module when an endpoint requires it. + +## Monitor several URLs for one service + +Set `availability_service` to the same nonempty value on the targets of each +Probe belonging to a service. One Probe can list several URLs, as above. Separate +Probes can use the same service label when URLs need different modules, scrape +intervals, or certificate warning settings. + +The StackPack creates a service component and links it to each endpoint. Open +**Service Availability → Services**, then select a service to inspect its related +endpoints. The service availability monitor becomes critical when any of its +currently observed endpoint checks or scrapes fails. Each URL also retains its own +availability monitor and highlight page. + +Use a unique service identity across your monitored environments, for example +`production/checkout` and `staging/checkout`, if they should be separate services. +Omit `availability_service` to monitor URLs individually without creating a +service component. Every URL assigned to a service contributes to availability; +monitor optional dependencies separately. + +The service monitor uses available `probe_success` and `up` samples, rather than +an inventory of expected URLs. A failed scrape reports `up=0`, so its URL remains +in the monitored endpoint count and makes availability Critical. If the scraper +stops emitting both metrics entirely for a URL, its absence is not automatically +counted as a failure. Services and relations expire 15 minutes after their last +matching metric. + +## Optional certificate expiration warnings + +To enable both certificate monitors, add `monitor_certificate_expiry: "true"` +to the Probe's `targets.staticConfig.labels`. Omit the label or set it to +`"false"` to disable certificate checks for those targets. + +Both monitors evaluate every five minutes. **Endpoint certificate expiring** reports +**Deviating** below 14 days. **Endpoint certificate expiration imminent** reports +**Critical** below two days. + +The **Certificate expires in** overview column shows the remaining validity of +the certificate chain. The **TLS certificate** highlight chart shows whole days, +rounded down, with lines marking the warning and critical thresholds. Certificate +data is also available for HTTP endpoints that redirect to HTTPS. + +To change the thresholds, edit the monitors in the Monitors UI. Values are in +seconds: 14 days is `1209600`, and two days is `172800`. Update the chart's +threshold lines to match. + +An expired or invalid certificate fails the availability check even when +certificate warnings are disabled. Checks without certificate data show no +expiration value and do not trigger the certificate monitors. + +## Check your endpoints + +Open **Service Availability → Endpoints** to check that your URLs appear with +their health, HTTP response codes and probe durations. Select an endpoint to +inspect its health checks and metric history. A failed check or scrape makes +its availability monitor Critical on the next evaluation. + +Open **Service Availability → Services** to see grouped services and their +endpoint counts. Select a service to inspect its related endpoints. + +## Troubleshooting + +1. **Endpoints or services are missing.** In the Metrics Explorer, run: + + ```promql + {__name__=~"probe_success|up",service_availability="true"} + ``` + + Each result needs an `endpoint` label containing the full HTTP or HTTPS URL. + For service grouping, it also needs a nonempty `availability_service` label. + If labels are missing, check the Probe's target labels and relabeling. + If no metrics arrive, check Probe discovery, the Target Allocator and scraper + logs. Components expire after 15 minutes without data. + +2. **An endpoint is Critical.** Check its availability monitor: + `probe_success=0` means the endpoint check failed; `up=0` means the scraper + could not collect the probe result. Check the endpoint's application, network + path and TLS certificate for a failed probe. For a failed scrape, check the + exporter connection and module name. If availability is healthy, check + whether a certificate expiration monitor is reporting the problem. + +3. **Test the exporter directly.** Port-forward its service: + + ```bash + kubectl -n monitoring port-forward svc/prometheus-blackbox-exporter 9115:9115 + ``` + + In another terminal, replace the target URL with your endpoint and use the + same module as your Probe: + + ```bash + curl --get 'http://localhost:9115/probe' \ + --data-urlencode 'module=http_2xx' \ + --data-urlencode 'target=https://example.com/health' \ + --data-urlencode 'debug=true' + ``` + + `Unknown module` means the exporter has no module with that name; configure + it or select an existing module. A completed probe returns metrics even when + the endpoint is unhealthy. Use the debug output to identify HTTP, connection + or TLS failures. + + If this check succeeds but metrics are missing in SUSE Observability, check + the Probe selector, Target Allocator, scraper logs and connectivity from the + scraper to the exporter. diff --git a/stackpacks/service-availability/resources/deprovisioning.md b/stackpacks/service-availability/resources/deprovisioning.md new file mode 100644 index 0000000..cc04f07 --- /dev/null +++ b/stackpacks/service-availability/resources/deprovisioning.md @@ -0,0 +1,3 @@ +The Service Availability integration settings are being removed. Blackbox +Exporter and Probe resources are managed separately and are not uninstalled by +this StackPack. diff --git a/stackpacks/service-availability/resources/error.md b/stackpacks/service-availability/resources/error.md new file mode 100644 index 0000000..41dc8e5 --- /dev/null +++ b/stackpacks/service-availability/resources/error.md @@ -0,0 +1,6 @@ +The Service Availability StackPack could not be provisioned. Inspect the +provisioning error and verify that the platform supports StackPacks 2.0. + +If provisioning succeeds but endpoints are missing, inspect `probe_success` and `up` +metrics and confirm the `service_availability` and `endpoint` labels described +in the setup instructions. diff --git a/stackpacks/service-availability/resources/logo.svg b/stackpacks/service-availability/resources/logo.svg new file mode 100644 index 0000000..9d6982b --- /dev/null +++ b/stackpacks/service-availability/resources/logo.svg @@ -0,0 +1,3 @@ + + + diff --git a/stackpacks/service-availability/resources/overview.md b/stackpacks/service-availability/resources/overview.md new file mode 100644 index 0000000..d33a865 --- /dev/null +++ b/stackpacks/service-availability/resources/overview.md @@ -0,0 +1,24 @@ +# Service Availability + +Monitor HTTP and HTTPS endpoints with Prometheus Blackbox Exporter and the SUSE +Observability agent's OpenTelemetry Prometheus scraper. + +Each endpoint URL becomes a component with a health state. Open **Service +Availability → Endpoints** to see endpoint health, HTTP response codes and probe +duration. Select an endpoint to inspect its health checks and metric history. +HTTPS endpoints also show remaining TLS certificate validity as a duration in +the overview and as whole days in a time-series chart when available. + +Use the optional `availability_service` Probe target label to group several URLs +under one service. Open **Service Availability → Services** to see their combined +availability and navigate to the individual endpoints. + +The Probe example uses `http_2xx`: it follows redirects and requires a final 2xx +response. Connection failures and invalid or expired TLS certificates also fail +the availability check. + +The exporter and Probe resources are installed separately. Topology is created +directly from metrics; no Kubernetes or OpenTelemetry StackPack installation is +required. Certificate expiration checks are optional: opt in with the +`monitor_certificate_expiry` Probe target label to warn below 14 days and report +Critical below two days. diff --git a/stackpacks/service-availability/resources/provisioning.md b/stackpacks/service-availability/resources/provisioning.md new file mode 100644 index 0000000..5e348f4 --- /dev/null +++ b/stackpacks/service-availability/resources/provisioning.md @@ -0,0 +1,3 @@ +The endpoint mapping, availability monitor and presentation are being installed. +Configure Blackbox Exporter and your Probe resources as described in the setup +instructions. diff --git a/stackpacks/service-availability/resources/waitingfordata.md b/stackpacks/service-availability/resources/waitingfordata.md new file mode 100644 index 0000000..efb6e54 --- /dev/null +++ b/stackpacks/service-availability/resources/waitingfordata.md @@ -0,0 +1,6 @@ +Waiting for `probe_success` or `up` metrics with `service_availability="true"` and an +`endpoint` label containing the HTTP or HTTPS target URL. + +Check that the Probe has `observability.suse.com/agent: scrape`, the OTel +Prometheus scraper and Target Allocator are enabled, and the Blackbox Exporter +service is reachable. diff --git a/stackpacks/service-availability/settings/component-mappings/endpoints.sty b/stackpacks/service-availability/settings/component-mappings/endpoints.sty new file mode 100644 index 0000000..5376f0c --- /dev/null +++ b/stackpacks/service-availability/settings/component-mappings/endpoints.sty @@ -0,0 +1,35 @@ +nodes: + - _type: OtelComponentMapping + name: HTTP endpoint + description: Creates one endpoint component from opted-in Blackbox probe metrics. + identifier: urn:stackpack:service-availability:otel-component-mapping:endpoint + input: + signal: ["METRICS"] + resource: + scope: + metric: + condition: "metric.name in ['probe_success', 'up']" + datapoint: + action: "'CREATE'" + condition: >- + 'service_availability' in datapoint.attributes && + datapoint.attributes['service_availability'] == 'true' && + 'endpoint' in datapoint.attributes && + datapoint.attributes['endpoint'].matches('^https?://.+') + vars: + - name: endpoint + value: "datapoint.attributes['endpoint']" + output: + identifier: "'urn:service-availability:endpoint/' + vars.endpoint" + name: vars.endpoint + typeName: "'http endpoint'" + required: + configuration: "{'url': vars.endpoint}" + tags: + - source: "'service-availability'" + target: stackpack + - source: vars.endpoint + target: endpoint.url + expireAfter: 900000 + rank: + specificity: 100 diff --git a/stackpacks/service-availability/settings/component-mappings/services.sty b/stackpacks/service-availability/settings/component-mappings/services.sty new file mode 100644 index 0000000..46fd2cf --- /dev/null +++ b/stackpacks/service-availability/settings/component-mappings/services.sty @@ -0,0 +1,36 @@ +nodes: + - _type: OtelComponentMapping + name: Availability service + description: Groups HTTP endpoints using the optional availability_service metric label. + identifier: urn:stackpack:service-availability:otel-component-mapping:service + input: + signal: ["METRICS"] + resource: + scope: + metric: + condition: "metric.name in ['probe_success', 'up']" + datapoint: + action: "'CREATE'" + condition: >- + 'service_availability' in datapoint.attributes && + datapoint.attributes['service_availability'] == 'true' && + 'endpoint' in datapoint.attributes && + datapoint.attributes['endpoint'].matches('^https?://.+') && + 'availability_service' in datapoint.attributes && + datapoint.attributes['availability_service'] != '' + vars: + - name: service + value: "datapoint.attributes['availability_service']" + output: + identifier: "'urn:service-availability:service/' + vars.service" + name: vars.service + typeName: "'availability service'" + required: + tags: + - source: "'service-availability'" + target: stackpack + - source: vars.service + target: availability.service + expireAfter: 900000 + rank: + specificity: 100 diff --git a/stackpacks/service-availability/settings/main-menu.sty b/stackpacks/service-availability/settings/main-menu.sty new file mode 100644 index 0000000..0fdba65 --- /dev/null +++ b/stackpacks/service-availability/settings/main-menu.sty @@ -0,0 +1,7 @@ +nodes: + - _type: MainMenuGroup + name: Service Availability + identifier: urn:stackpack:service-availability:main-menu-group:service-availability + iconbase64: !icon availability.svg + defaultOpen: true + items: [] diff --git a/stackpacks/service-availability/settings/monitors/certificate-expiration-critical.sty b/stackpacks/service-availability/settings/monitors/certificate-expiration-critical.sty new file mode 100644 index 0000000..c1fefd4 --- /dev/null +++ b/stackpacks/service-availability/settings/monitors/certificate-expiration-critical.sty @@ -0,0 +1,23 @@ +nodes: + - _type: Monitor + name: Endpoint certificate expiration imminent + description: Reports Critical when an opted-in endpoint's reported TLS certificate has less than two days remaining. + identifier: urn:stackpack:service-availability:monitor:certificate-expiration-imminent + function: urn:stackpack:common:monitor-function:threshold + arguments: + comparator: LT + threshold: 172800.0 + failureState: CRITICAL + metric: + query: >- + min by (endpoint) ( + probe_ssl_earliest_cert_expiry{service_availability="true",monitor_certificate_expiry="true",endpoint=~"https?://.+"} > 0 + ) - time() + aliasTemplate: Certificate validity ${endpoint} + unit: s + titleTemplate: Certificate expiration is imminent + urnTemplate: urn:service-availability:endpoint/${endpoint} + intervalSeconds: 300 + status: ENABLED + tags: ["availability", "certificates", "blackbox"] + remediationHint: !include remediation-hints/certificate-expiring.md.hbs diff --git a/stackpacks/service-availability/settings/monitors/certificate-expiration.sty b/stackpacks/service-availability/settings/monitors/certificate-expiration.sty new file mode 100644 index 0000000..d9e2a6b --- /dev/null +++ b/stackpacks/service-availability/settings/monitors/certificate-expiration.sty @@ -0,0 +1,23 @@ +nodes: + - _type: Monitor + name: Endpoint certificate expiring + description: Warns when an opted-in endpoint's reported TLS certificate has less than 14 days remaining. + identifier: urn:stackpack:service-availability:monitor:certificate-expiring + function: urn:stackpack:common:monitor-function:threshold + arguments: + comparator: LT + threshold: 1209600.0 + failureState: DEVIATING + metric: + query: >- + min by (endpoint) ( + probe_ssl_earliest_cert_expiry{service_availability="true",monitor_certificate_expiry="true",endpoint=~"https?://.+"} > 0 + ) - time() + aliasTemplate: Certificate validity ${endpoint} + unit: s + titleTemplate: Certificate is expiring + urnTemplate: urn:service-availability:endpoint/${endpoint} + intervalSeconds: 300 + status: ENABLED + tags: ["availability", "certificates", "blackbox"] + remediationHint: !include remediation-hints/certificate-expiring.md.hbs diff --git a/stackpacks/service-availability/settings/monitors/endpoint-availability.sty b/stackpacks/service-availability/settings/monitors/endpoint-availability.sty new file mode 100644 index 0000000..23509a1 --- /dev/null +++ b/stackpacks/service-availability/settings/monitors/endpoint-availability.sty @@ -0,0 +1,23 @@ +nodes: + - _type: Monitor + name: Endpoint unavailable + description: Reports failed endpoint checks or scrapes, including connection, TLS validation and exporter errors. + identifier: urn:stackpack:service-availability:monitor:endpoint-unavailable + function: urn:stackpack:common:monitor-function:threshold + arguments: + comparator: LT + threshold: 1.0 + failureState: CRITICAL + metric: + query: >- + min by (endpoint) ( + {__name__=~"probe_success|up",service_availability="true",endpoint=~"https?://.+"} + ) + aliasTemplate: Availability ${endpoint} + unit: short + titleTemplate: Endpoint is unavailable + urnTemplate: urn:service-availability:endpoint/${endpoint} + intervalSeconds: 60 + status: ENABLED + tags: ["availability", "http", "blackbox"] + remediationHint: !include remediation-hints/endpoint-unavailable.md.hbs diff --git a/stackpacks/service-availability/settings/monitors/service-availability.sty b/stackpacks/service-availability/settings/monitors/service-availability.sty new file mode 100644 index 0000000..1c94054 --- /dev/null +++ b/stackpacks/service-availability/settings/monitors/service-availability.sty @@ -0,0 +1,23 @@ +nodes: + - _type: Monitor + name: Service endpoint unavailable + description: Reports a service as unavailable when any of its labeled endpoint checks or scrapes fails. + identifier: urn:stackpack:service-availability:monitor:service-unavailable + function: urn:stackpack:common:monitor-function:threshold + arguments: + comparator: LT + threshold: 1.0 + failureState: CRITICAL + metric: + query: >- + min by (availability_service) ( + {__name__=~"probe_success|up",service_availability="true",availability_service!="",endpoint=~"https?://.+"} + ) + aliasTemplate: Availability ${availability_service} + unit: short + titleTemplate: An endpoint of ${availability_service} is unavailable + urnTemplate: urn:service-availability:service/${availability_service} + intervalSeconds: 60 + status: ENABLED + tags: ["availability", "service", "blackbox"] + remediationHint: !include remediation-hints/service-unavailable.md.hbs diff --git a/stackpacks/service-availability/settings/presentations/endpoints.sty b/stackpacks/service-availability/settings/presentations/endpoints.sty new file mode 100644 index 0000000..1a9a952 --- /dev/null +++ b/stackpacks/service-availability/settings/presentations/endpoints.sty @@ -0,0 +1,175 @@ +nodes: + - _type: ComponentPresentation + name: HTTP endpoints + description: Availability, HTTP response status, probe duration and optional TLS certificate validity. + identifier: urn:stackpack:service-availability:component-presentation:endpoint + binding: + _type: ComponentPresentationQueryBinding + query: 'label = "stackpack:service-availability" and type = "http endpoint"' + rank: + specificity: 10 + presentation: + icon: !icon endpoint.svg + overview: + name: + singular: endpoint + plural: endpoints + title: Endpoints + mainMenu: + group: Service Availability + order: 100 + columns: + - columnId: endpoint_health + title: Health + projection: + _type: HealthProjection + value: healthState + - columnId: endpoint_name + title: Endpoint + projection: + _type: ComponentLinkProjection + name: name + identifier: "identifiers[0]" + - columnId: endpoint_http_status + title: HTTP status + projection: + _type: MetricProjection + query: >- + max by (endpoint) (probe_http_status_code{service_availability="true",endpoint="${tags.singleValue('endpoint.url')}"}) + unit: short + decimalPlaces: 0 + sparkline: false + metricId: endpoint-http-status + - columnId: endpoint_probe_duration + title: Probe duration + projection: + _type: MetricProjection + query: >- + max by (endpoint) (probe_duration_seconds{service_availability="true",endpoint="${tags.singleValue('endpoint.url')}"}) + unit: s + decimalPlaces: 3 + sparkline: true + metricId: endpoint-probe-duration + - columnId: endpoint_certificate_validity + title: Certificate expires in + projection: + _type: MetricProjection + query: >- + min by (endpoint) (probe_ssl_earliest_cert_expiry{service_availability="true",endpoint="${tags.singleValue('endpoint.url')}"} > 0) - time() + unit: s + decimalPlaces: 0 + sparkline: false + metricId: endpoint-certificate-validity + sort: + - columnId: endpoint_health + - columnId: endpoint_name + highlight: + title: HTTP endpoint + fields: + - fieldId: health + title: Health + order: 100 + projection: + _type: HealthProjection + value: healthState + - fieldId: url + title: URL + order: 90 + projection: + _type: TextProjection + value: "tags.singleValue('endpoint.url')" + provisioning: + topologySourceSelector: "source.mapping.identifier == 'urn:stackpack:service-availability:otel-component-mapping:endpoint'" + showConfiguration: true + showStatus: false + relatedResources: + - resourceId: endpoint_services + title: Services + order: 100 + topologyQuery: '(withNeighborsOf(direction = "up", components = (identifier = "${identifiers[0]}"), levels = "1")) and type = "availability service"' + presentationIdentifier: urn:stackpack:service-availability:component-presentation:service + metrics: + - sectionId: availability + title: Availability + order: 100 + metrics: + - metricId: endpoint-probe-success + order: 100 + - metricId: endpoint-http-status + order: 90 + - metricId: endpoint-probe-duration + order: 80 + - sectionId: certificates + title: TLS certificate + order: 90 + metrics: + - metricId: endpoint-certificate-validity + order: 100 + metricPerspective: + tabs: + - tabId: availability + title: Availability + order: 100 + sections: + - sectionId: availability + title: Availability + order: 100 + metrics: + - metricId: endpoint-probe-success + - metricId: endpoint-http-status + - metricId: endpoint-probe-duration + - sectionId: certificates + title: TLS certificate + order: 90 + metrics: + - metricId: endpoint-certificate-validity + metrics: + - metricId: endpoint-probe-success + name: Probe success + description: One means the available check and scrape results succeeded; zero means a check or scrape failed. + metricQueries: + - expression: >- + min({__name__=~"probe_success|up",service_availability="true",endpoint="${tags.singleValue('endpoint.url')}"}) + alias: Probe success + chart: + _type: TimeSeriesChart + unit: short + - metricId: endpoint-http-status + name: HTTP status + description: The HTTP response code; zero indicates no HTTP response was received. + metricQueries: + - expression: >- + probe_http_status_code{service_availability="true",endpoint="${tags.singleValue('endpoint.url')}"} + alias: HTTP status ${service_name} + chart: + _type: TimeSeriesChart + unit: short + - metricId: endpoint-probe-duration + name: Probe duration + description: Total time spent performing the probe. + metricQueries: + - expression: >- + probe_duration_seconds{service_availability="true",endpoint="${tags.singleValue('endpoint.url')}"} + alias: Probe duration ${service_name} + chart: + _type: TimeSeriesChart + unit: s + - metricId: endpoint-certificate-validity + name: Certificate expires in (days) + description: Whole days until the earliest TLS certificate expiration; absent when no certificate is reported. + metricQueries: + - expression: >- + floor((min by (endpoint) (probe_ssl_earliest_cert_expiry{service_availability="true",endpoint="${tags.singleValue('endpoint.url')}"} > 0) - time()) / 86400) + alias: Days remaining + chart: + _type: TimeSeriesChart + unit: d + decimals: 0 + min: 0 + thresholds: + mode: absolute + steps: + - value: 14 + color: "#FF7F00" + - value: 2 + color: "#E00000" diff --git a/stackpacks/service-availability/settings/presentations/services.sty b/stackpacks/service-availability/settings/presentations/services.sty new file mode 100644 index 0000000..5436702 --- /dev/null +++ b/stackpacks/service-availability/settings/presentations/services.sty @@ -0,0 +1,102 @@ +nodes: + - _type: ComponentPresentation + name: Availability services + description: Service availability and the HTTP endpoints that contribute to it. + identifier: urn:stackpack:service-availability:component-presentation:service + binding: + _type: ComponentPresentationQueryBinding + query: 'label = "stackpack:service-availability" and type = "availability service"' + rank: + specificity: 10 + presentation: + icon: !icon service.svg + overview: + name: + singular: service + plural: services + title: Services + mainMenu: + group: Service Availability + order: 90 + columns: + - columnId: availability_service_health + title: Health + projection: + _type: HealthProjection + value: healthState + - columnId: availability_service_name + title: Service + projection: + _type: ComponentLinkProjection + name: name + identifier: "identifiers[0]" + - columnId: availability_service_endpoints + title: Monitored endpoints + projection: + _type: MetricProjection + query: >- + count by (availability_service) (count by (availability_service, endpoint) ({__name__=~"probe_success|up",service_availability="true",availability_service="${tags.singleValue('availability.service')}",endpoint=~"https?://.+"})) + unit: short + decimalPlaces: 0 + sparkline: false + metricId: service-endpoint-count + sort: + - columnId: availability_service_health + - columnId: availability_service_name + highlight: + title: Availability service + fields: + - fieldId: health + title: Health + order: 100 + projection: + _type: HealthProjection + value: healthState + relatedResources: + - resourceId: service_endpoints + title: Endpoints + order: 100 + topologyQuery: '(withNeighborsOf(direction = "down", components = (identifier = "${identifiers[0]}"), levels = "1")) and type = "http endpoint"' + presentationIdentifier: urn:stackpack:service-availability:component-presentation:endpoint + metrics: + - sectionId: availability + title: Availability + order: 100 + metrics: + - metricId: service-probe-success + order: 100 + - metricId: service-endpoint-count + order: 90 + metricPerspective: + tabs: + - tabId: availability + title: Availability + order: 100 + sections: + - sectionId: availability + title: Availability + order: 100 + metrics: + - metricId: service-probe-success + - metricId: service-endpoint-count + metrics: + - metricId: service-probe-success + name: Service probe success + description: One when all available service check and scrape results succeed; zero when any fails. + metricQueries: + - expression: >- + min({__name__=~"probe_success|up",service_availability="true",availability_service="${tags.singleValue('availability.service')}",endpoint=~"https?://.+"}) + alias: Service availability + chart: + _type: TimeSeriesChart + unit: short + - metricId: service-endpoint-count + name: Monitored endpoints + description: Number of distinct endpoint URLs with current check or scrape samples for this service, including failed scrapes. + metricQueries: + - expression: >- + count by (availability_service) (count by (availability_service, endpoint) ({__name__=~"probe_success|up",service_availability="true",availability_service="${tags.singleValue('availability.service')}",endpoint=~"https?://.+"})) + alias: Monitored endpoints + chart: + _type: TimeSeriesChart + unit: short diff --git a/stackpacks/service-availability/settings/relation-mappings/service-endpoints.sty b/stackpacks/service-availability/settings/relation-mappings/service-endpoints.sty new file mode 100644 index 0000000..85747b7 --- /dev/null +++ b/stackpacks/service-availability/settings/relation-mappings/service-endpoints.sty @@ -0,0 +1,31 @@ +nodes: + - _type: OtelRelationMapping + name: Service endpoint + description: Links a service to each URL labeled with its availability_service identity. + identifier: urn:stackpack:service-availability:otel-relation-mapping:service-endpoint + input: + signal: ["METRICS"] + resource: + scope: + metric: + condition: "metric.name in ['probe_success', 'up']" + datapoint: + action: "'CREATE'" + condition: >- + 'service_availability' in datapoint.attributes && + datapoint.attributes['service_availability'] == 'true' && + 'endpoint' in datapoint.attributes && + datapoint.attributes['endpoint'].matches('^https?://.+') && + 'availability_service' in datapoint.attributes && + datapoint.attributes['availability_service'] != '' + vars: + - name: service + value: "datapoint.attributes['availability_service']" + - name: endpoint + value: "datapoint.attributes['endpoint']" + output: + sourceId: "'urn:service-availability:service/' + vars.service" + targetId: "'urn:service-availability:endpoint/' + vars.endpoint" + typeName: "'has endpoint'" + dependencyType: "'HIERARCHICAL'" + expireAfter: 900000 diff --git a/stackpacks/service-availability/stackpack.yaml b/stackpacks/service-availability/stackpack.yaml new file mode 100644 index 0000000..9c33021 --- /dev/null +++ b/stackpacks/service-availability/stackpack.yaml @@ -0,0 +1,16 @@ +name: service-availability +version: "0.0.1" +schemaVersion: "2.0" +displayName: Service Availability +categories: ["Open Telemetry"] +logoUrl: !resource logo.svg +overviewUrl: !resource overview.md +configurationUrls: + NOT_INSTALLED: !resource configuration.md + PROVISIONING: !resource provisioning.md + WAITING_FOR_DATA: !resource waitingfordata.md + INSTALLED: !resource configuration.md + DEPROVISIONING: !resource deprovisioning.md + ERROR: !resource error.md +provision: + companionStackPacks: []