Sync observability stack to Coroot (+ audit-log-forwarder) like the reference platform #65

Activity

1. devantler

   added

   enhancement

   on Jul 13, 2026
2. devantler

   added a parent issue on Jul 18, 2026

   • roadmap: platform-template direction & themes (2026 H1) #24
3. devantler

   added this to 🌊 Project Boardon Jul 18, 2026
4. devantler

   moved this to 📥 Backlog in 🌊 Project Boardon Jul 18, 2026
5. devantler

   added the

   Feature

   issue type on Jul 18, 2026
6. devantler

   added a sub-issue on Jul 18, 2026

   • Stage OpenCost retirement behind the Coroot option #81
7. devantler commented on Jul 18, 2026

   

   devantler

   on Jul 18, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Daily AI Engineer

   Progress: #82 merged as b7d144ad1314ec9bec360f1dc7580d7729509c4e,
   closing child #81. The explicit Docker/Hetzner Coroot fixtures now model complete
   OpenCost retirement across workload objects, auth-proxy routing, network policy,
   ReferenceGrant, and security exemptions while every real provider render remains
   byte-for-byte unchanged by default.

   The direction changed after exact-head review prompted read-only live validation:
   the reference Platform's OpenCost returns non-zero requested-resource costs but
   zero CPU/RAM usage for all 34 namespace allocations because Coroot Prometheus
   lacks the standard node/cAdvisor usage series. That live defect is tracked in
   devantler-tech/platform#2689; it must not be copied into this template.

   The child passed 86-kustomization / 258-file validation for both local and prod,
   negative controls for future references and array drift, exact-head CI, Codex
   review, and post-merge TODO/CodeQL workflows. No cluster was mutated. Parent #65
   remains open for real-provider activation and the remaining rollout slices.
8. devantler commented on Jul 18, 2026

   

   devantler

   on Jul 18, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Daily AI Engineer

   Delivered the first production-profile deduplication slice in #91 (closing #90): the Hetzner Coroot controller payload no longer renders the legacy Alloy audit collector, while default profiles, Docker Coroot, and the main Alloy pod-log shipper remain unchanged. The replacement audit forwarder is now guarded end to end—file receiver → logs pipeline → Coroot OTLP endpoint with the expected API-key header—and three negative controls cover disconnection, endpoint drift, and auth-header drift. All CI and exact-head Codex review gates passed before merge.
9. devantler

   added sub-issues on Jul 20, 2026

   • Expose selectable Coroot UI through the SSO boundary #92
   • feat(observability): retire Loki and Alloy from Coroot profiles #96
10. devantler commented on Jul 20, 2026

    

    devantler

    on Jul 20, 2026

    ContributorAuthor

    More actions

    🤖 Generated by the Daily AI Engineer (Codex instance)

    Delivery child #96 is complete via #97 (8f53f316466684619f748d1d67b10d57c8f1f09c). Both selectable Coroot profiles now retire Loki, the pod-log and audit Alloy collectors, their dead Grafana datasource, and the now-unconsumed Grafana chart source, while default Docker and Hetzner controller renders remain byte-for-byte unchanged.

    The focused gate covers both providers and rejected four ablations: either provider dropping the component, default-stack drift, and retention of the dead source. Both KSail validations covered 97 kustomizations / 258 files; exact-head CI and Codex re-review were green after its Grafana-source finding was fixed. Parent #65 remains open for the separate kube-prometheus-stack metrics/alerting migration.
11. devantler

    added sub-issues on Jul 22, 2026

    • feat(observability): preserve production incident notifications in Coroot profiles #98
    • Stage the Coroot cluster heartbeat #100
    • feat(observability): restore Flux reconciliation alerts in Coroot profile #109
12. devantler

    removed a parent issue on Sep 13, 2026

    • roadmap: platform-template direction & themes (2026 H1) #24
13. devantler

    added a parent issue on Sep 13, 2026

    • roadmap: platform-template direction & themes (2026 H2) #175
14. devantler

    added a sub-issue on Sep 27, 2026

    • Retire the Grafana, Prometheus and Alertmanager UIs from the Coroot profiles #202
15. provider-upjet-github

    removed

    enhancement

    on Oct 25, 2026
16. devantler

    added sub-issues on Nov 4, 2026

    • feat(observability): make Coroot the default and delete the Loki, Alloy and OpenCost bases #386
    • feat(observability): retire kube-prometheus-stack like the reference platform #387
17. devantler commented on Nov 4, 2026

    

    devantler

    on Nov 4, 2026

    ContributorAuthor

    More actions

    🤖 Generated by the Agentic Engineer (interactive Claude Code session)

    Status: the outcome is not met yet, so this stays open. All seven earlier sub-issues are closed, but they built Coroot as an opt-in profile (clusters/local-coroot, clusters/prod-coroot). On main at 98ff4b1, the default cluster paths still render Loki, Alloy, OpenCost and the full kube-prometheus-stack. Even the Coroot profile still runs kube-prometheus-stack, because the platform-critical PrometheusRule, the Alertmanager Watchdog heartbeat and the ServiceMonitor CRDs depend on it. The reference platform runs none of these.

    I split the remaining work into two sub-issues, in order:

    1. feat(observability): make Coroot the default and delete the Loki, Alloy and OpenCost bases #386: make Coroot the default path, delete the Loki, Alloy and OpenCost bases, and keep the *-coroot paths as aliases for instances that opted in.
    2. feat(observability): retire kube-prometheus-stack like the reference platform #387: retire kube-prometheus-stack the way the platform did. HelmRelease failures move to the Flux Alert, the Watchdog gives way to cluster-heartbeat, and the chart ServiceMonitors and the monitoring namespace allowances go.

    The two can't be merged into one small change. The 1,400-line validate-observability-option.sh, the heartbeat and SSO tests, the bootstrap workflow and the alerting docs all encode the current opt-in boundary, so each step needs its own reviewed PR.

    🤖 Generated with Claude Code
18. devantler commented on Nov 10, 2026

    

    devantler

    on Nov 10, 2026

    ContributorAuthor

    More actions

    🤖 Generated by the Agentic Engineer (interactive Claude Code session)

    Status: all nine sub-issues are closed; one part of the outcome is still unproven, so this stays open.

    Delivered today:

    • feat(observability): make Coroot the default and delete the Loki, Alloy and OpenCost bases #386 (merged in feat(observability)!: make Coroot the default and delete Loki, Alloy and OpenCost #419): Coroot is the default on both providers, and the Loki, Alloy and OpenCost bases are deleted. The -coroot cluster paths remain as aliases.
    • feat(observability): retire kube-prometheus-stack like the reference platform #387 (merged in feat(observability)!: retire kube-prometheus-stack #425): kube-prometheus-stack is retired. Failed releases are reported by the Flux Alert, the cluster heartbeat is the one off-cluster dead-man signal, and no payload carries a Prometheus-operator resource or the monitoring namespace.

    On main at 0f2c89f the default paths render none of the retired stack, which was the drift this issue names.

    What is not proven: that bootstrap still converges on both providers. Everything above was checked on rendered payloads: both ksail … workload validate runs, the observability and alert-coverage validators and the bootstrap workflow contract. No cluster has taken the change. This repository never runs one, and the session that did this work had no container runtime or Hetzner credentials.

    To close: one local Docker cluster and one Hetzner cluster created from main reach Ready on every Flux layer with Coroot serving its UI behind SSO. An instance's system test after its next template sync covers the local half.

    Two things follow from the work and are tracked separately: #423 (alerts with no replacement: backup freshness, certificate expiry, autoscaler failures), and a one-line edit only a maintainer may make, removing the stale alertmanager_heartbeat_url placeholder from the protected production secret template.

    🤖 Generated with Claude Code