Repository navigation
Default OTLP endpoint points at an addon namespace that usually does not exist #1258
Description
Default OTLP endpoint points at an addon namespace that usually does not exist
manifests/ate-install/ate-otel-config.yaml defaults to http://opentelemetry-collector.gke-managed-otel.svc.cluster.local:4317. On any cluster without GKE's managed OTel addon enabled - the default gke-managed-otel namespace does not exist, so every component fails to resolve the collector once a minute, forever, and telemetry is silently dead. It presents as a networking fault rather than a missing dependency.
The installer rewrites the ConfigMap, so patching it after install is silently reverted. The override has to go in at install time via ATE_OTLP_ENDPOINT, which hack/install-ate.sh already honours.
Ask: default to no exporter, or make the install refuse when the endpoint's namespace is absent.
Activity
JeffLuoo commentedon Aug 27, 2026
We should make Observability a must-have, despite which OTel collector users use, managed one or customized. I prefer "make the install refuse when the endpoint's namespace is absent", WDYT? Da Huang (@git286)
git286 commentedon Aug 27, 2026
yeah, I think we should have some preflight check in the install script and explicitly tell users what's wrong and what they should do, instead of silently failure
BenTheElder commentedon Aug 28, 2026
really? If the user doesn't want it, why is this a hard requirement? I could imagine not wanting to collect otel because storing it will add up with a lot of actors and you may already be doing something else at the actor level.
Our components shouldn't even assume that the install scripts are in use.
We need to split out installing for GKE versus installing generally. install-ate.sh is currently primarily for GKE but quasi-pretending to be generic. Instead what we have here must become generic. vendor neutrality is actually a requirement for e.g. CNCF sandbox were we to go that route.
Installing for GKE can expect the GKE namespace, the components need to be well behaved without it though.
JeffLuoo commentedon Aug 28, 2026
We don't have actor level metrics as we also discussed in other threads, but cardinality will grow with ate namespace or template name, which the system metric use as attributes.
Thanks for your feedback. It makes sense that we should make monitoring stack optional but recommended. Hence, we can add a flag in the script to make it configurable, and if it's configured, we ask users to provide an OTLP endpoint. If the endpoint is not provided, the script warns or return error.