Skip to content

feat(observability): retire kube-prometheus-stack like the reference platform #387

Description

@devantler

Problem

The reference platform runs no kube-prometheus-stack. Its alerting moved to Coroot inspections, Flux notification-controller Alerts and off-cluster heartbeats (platform docs/dr/alerting.md, "Changed from the old stack"). The template's Coroot profile still keeps kube-prometheus-stack, and three things depend on it:

  • the platform-critical PrometheusRule (k8s/bases/infrastructure/alerts/, kept as FluxHelmReleaseNotReady by retain-helm-release-alert.yaml);
  • the Alertmanager Watchdog dead-man path (ALERTMANAGER_HEARTBEAT_URL in bootstrap.yaml);
  • the ServiceMonitor, PodMonitor and PrometheusRule CRDs that several chart values enable.

Proposed change, after the default-path sub-issue lands

  • Move HelmRelease failure coverage to the Flux-native path. Add HelmRelease to the production platform-reconciliation Alert's eventSources, then drop the PrometheusRule and k8s/bases/infrastructure/alerts/, removing it from .github/manifest-naming.yaml cr-directories.
  • Retire the Watchdog heartbeat in favour of the Coroot cluster-heartbeat, including the bootstrap secret handling, validate-bootstrap-workflow.sh and the docs.
  • Turn off every chart's serviceMonitor, podMonitor and prometheusRule values that depend on the Prometheus operator CRDs. Drop the allowances for the monitoring namespace from the CiliumNetworkPolicies and security exceptions.
  • Delete k8s/bases/infrastructure/controllers/kube-prometheus-stack, the hetzner store-tsdb-on-hcloud patch and the remove-kps-ui component.
  • Remove the kube-prometheus-stack entry from the allow-list in scripts/tests/test-drain-safety-policies.sh, which feat(policy): adopt drain-safe PDB and replica band policies #383 adds.

Acceptance criteria

  • No rendered payload contains a monitoring.coreos.com kind, the monitoring namespace, or a chart value that enables the Prometheus operator CRDs.
  • Production keeps a notification for failed Kustomizations and failed HelmReleases, and keeps one off-cluster dead-man heartbeat.
  • Every gate in AGENTS.md → Validation passes.

Part of #65.

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Fields

    Priority

    None yet

    Start date

    None yet

    Target date

    None yet

    Effort

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions