Stage the Coroot cluster heartbeat #100

🤖 Generated by the Agentic Engineer

Summary

Stage the reference platform's hardened cluster dead-man heartbeat in the explicit Coroot profiles before retiring kube-prometheus-stack.

Motivation

The Coroot profiles now replace Loki, Alloy, OpenCost, UI access, and incident delivery, but they still inherit kube-prometheus-stack because its Watchdog alert is the only cluster-down heartbeat. Removing that stack now would silently lose cluster-death detection. The reference platform instead runs a dedicated cluster-heartbeat CronJob from the dependent Coroot apps payload, after the infrastructure policy exclusion reports ready.

This slice separates that direct cluster-liveness signal from the Prometheus-to-Alertmanager Watchdog check without changing either default provider profile. Local and unconfigured Coroot renders remain inert through an invalid fallback.

Design correction (2026-07-22)

Exact-head review disproved the original proposal to reuse alertmanager_heartbeat_url. If Watchdog and the CronJob reset one monitor, the CronJob hides a broken alert pipeline; if Watchdog is disabled, the pipeline loses its only end-to-end dead-man check. Review also exposed an upgrade-order gap when the CronJob reconciled before the infrastructure layer's policy exclusion. The safe design uses two distinct external checks, keeps Watchdog on the existing Alertmanager URL, and stages the CronJob from the dependent apps layer only after infrastructure reports ready.

Hypothesis and success signal

If both opt-in Coroot apps profiles render one hardened heartbeat on a distinct external check after their infrastructure dependency is ready, a later metrics-stack retirement can remove kube-prometheus-stack without losing the direct cluster dead-man signal or prematurely weakening alert-pipeline coverage.

Success is a structural and rendered contract: default Docker/Hetzner payloads remain unchanged; local-coroot and prod-coroot controller payloads contain no heartbeat; their dependent apps payloads each render exactly one; the only new egress is exact hc-ping.com:443; the two monitor check identities remain distinct; and every existing incident-delivery assertion remains green.

Proposed solution

• Add the portable cluster-heartbeat CronJob to a shared apps base consumed only by the Coroot apps profiles.
• Require the apps Flux Kustomization to wait for infrastructure, which owns the generated-policy exclusion.
• Keep Watchdog enabled on alertmanager_heartbeat_url; use a separate optional cluster_heartbeat_url for the CronJob.
• Require canonical lowercase-UUID hc-ping.com URLs and reject reused check identities during bootstrap, matching the fail-closed network and signal-separation policies.
• Substitute the heartbeat URL into a Kubernetes Secret, expose it through a Secret-backed environment variable, and feed curl through stdin configuration rather than runtime arguments.
• Extend the observability validator with rendered, ordering, hardening, signal-separation, and negative-boundary assertions.
• Document that kube-prometheus-stack stays transitional until Flux-native reconciliation alerts are staged in a later child.

Acceptance criteria

• Default Docker and Hetzner renders contain no cluster heartbeat and keep the current kube-prometheus-stack payload unchanged.
• Both explicit Coroot controller payloads contain no heartbeat; both dependent apps payloads render exactly one five-minute cluster-heartbeat using the separate cluster_heartbeat_url substitution and inert fallback.
• Both Coroot apps Flux Kustomizations depend exactly on infrastructure with readiness waiting enabled.
• Watchdog stays enabled on the existing alertmanager_heartbeat_url in every profile; the CronJob never reuses that monitor.
• Bootstrap accepts an unset CronJob URL, otherwise requires canonical lowercase-UUID hc-ping.com URLs, rejects reused check identities and URL aliases, and encrypts the optional value into the production cluster variables Secret.
• The rendered bearer URL is held in a Kubernetes Secret and reaches curl through stdin configuration, not the PodSpec or curl's runtime argument vector.
• The CronJob is non-overlapping, deadline/history bounded, digest-pinned, non-root, read-only, token-free, and drops all capabilities.
• Coroot egress permits exactly hc-ping.com:443 for the heartbeat; no second policy, wildcard, or world selector may match the workload.
• Production incident delivery remains exactly once and local profiles remain free of real integrations.
• Mutation controls reject controller-layer staging, a missing apps dependency, a shared monitor identity, URL aliases, disabled Watchdog, broadened bootstrap hosts, PodSpec credential exposure, missing heartbeat, schedule/substitution/hardening drift, an additive matching policy, missing or broadened egress, and heartbeat leakage into defaults.
• Naming, observability, policy, local KSail, and production KSail validation pass.
• Operator docs explain the distinct checks, optional Coroot input, reconciliation order, and temporary kube-prometheus-stack retention.

Rough size

S

Part of #65