feat(observability): preserve production incident notifications in Coroot profiles #98

🤖 Generated by the Daily AI Engineer (Codex instance)

Evidence and audience

The selectable Coroot profiles already replace the legacy cost and log paths, but production alert delivery is still owned entirely by the transitional kube-prometheus-stack. Operators who finish that migration would lose the existing webhook path unless Coroot's incident notifications are wired before the old stack is removed.

The reference Platform has proven the generic path: Coroot Community Edition can send incident payloads through its webhook integration, using the existing per-cluster alertmanager_webhook_url and the authenticated Coroot UI as the link base. Local/CI should remain quiet.

Hypothesis and success signal

If the production Coroot profile reuses the existing webhook declaratively while the Docker profile stays notification-free, a generated platform can retire the legacy metrics stack in a later slice without creating an alert-delivery gap or adding new setup.

Success is a static provider render proving exactly one production webhook integration with JSON-safe incident output, no Docker integration, and byte-identical default provider payloads. The next retirement slice can then remove kube-prometheus-stack without first inventing another notification channel.

Smallest useful change

Add a Hetzner Coroot-profile patch that augments the existing project and API-key contract with an incident-only webhook integration. Keep the old metrics stack in place for this slice; this prepares the replacement before removal.

Acceptance criteria

• The prod-coroot render contains exactly one Coroot webhook integration using ${alertmanager_webhook_url} and https://observability.${domain}.
• The integration sends incident and resolved-incident payloads with JSON-safe templates; noisy inspection alerts remain in the Coroot UI rather than being fanned out.
• The existing ${cluster_name} project and coroot-api-key contract are preserved exactly.
• The local-coroot render has no notification integration, so local/CI remains quiet.
• Default Docker and Hetzner renders remain byte-identical and Coroot stays default-off.
• The observability validation gate starts with a failing control for the missing integration and includes negative controls for provider drift, project/index drift, webhook drift, and local notification leakage.
• The templating guide documents that the production Coroot option reuses the existing webhook and that kube-prometheus-stack retirement remains a separate follow-up.

Rough size

S

Part of #65.