[Bug]: Chart leaves OTLP export on when no endpoint is set, so components push to localhost:4317 #76

What happened?

With the chart defaults (otel.endpoint: "", otel.traces.enabled: true, otel.metrics.enabled: true), substrate.otel.env renders neither an endpoint nor an OTEL_<SIGNAL>_EXPORTER for ate-api-server, atelet and ate-controller. The Go components then use the SDK default, push metrics and traces to localhost:4317, and log a connection refused about once a minute per pod.

k8s-credential-provider does the same. Its template has no env block, so no values setting can turn it off.

The values comment says "A signal exports only when it is enabled and resolves to an endpoint", and the helper already gates OTEL_LOGS_EXPORTER=otlp on an endpoint because "otlp without one retries localhost:4317". Traces and metrics have no such gate.

A default kagent install (kagent 1.0.0-alpha13) ends up in this state, because the CLI passes no otel values to this chart.

Expected Behavior

When a signal is enabled but no endpoint resolves, the chart renders OTEL_<SIGNAL>_EXPORTER=none, as it already does when the signal is disabled. k8s-credential-provider gets the same env.

Steps to Reproduce

1. helm template t charts/substrate -n ate-system on main (f499a14) with default values.
2. ate-api-server, atelet, ate-controller and k8s-credential-provider have no OTEL_EXPORTER_OTLP_ENDPOINT and no OTEL_*_EXPORTER.
3. Install with the defaults and read the logs of any of these pods.

Sandbox Runtime

Both / Runtime Agnostic

Agent Substrate Version / Commit SHA

Chart 0.5.0-alpha2 as installed by kagent 1.0.0-alpha13. The render in step 1 is from main at f499a14.

Kubernetes Version & Environment

Talos v1.13.11, Kubernetes v1.36.5, Cilium 1.20.2, on-prem

Host OS & Architecture

Linux 6.18 (x86_64)

Relevant Logs and Diagnostic Output

# ate-api-server, atelet, ate-controller (each pod, about once a minute)
{"level":"WARN","msg":"OpenTelemetry SDK error","err":"failed to upload metrics: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused\""}
{"level":"WARN","msg":"OpenTelemetry SDK error","err":"traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:4317: connect: connection refused\""}

# k8s-credential-provider (about once a minute)
{"level":"INFO","msg":"failed to upload metrics: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused\""}

Additional Context

Setting otel.metrics.enabled=false and otel.traces.enabled=false stops it for the three components that include substrate.otel.env. Both errors went to zero on our cluster after that. There is no values workaround for k8s-credential-provider.

The values comment "traces.enabled takes effect on the router; Go components do not read it yet" looks outdated. InitTracing in serverboot checks OTEL_TRACES_EXPORTER in 0.5.0-alpha2, and the trace export errors above stopped once it was set to none.

The ateom worker pods show the same errors, but that comes from atecontroller propagating nothing when its endpoint is empty, which agent-substrate#2189 and agent-substrate#2426 address. This issue is only about the chart.

This change to both templates fixes it here. With the defaults, all four components render OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER and OTEL_LOGS_EXPORTER set to none. With otel.endpoint set, the output for the three existing components is unchanged.

--- a/charts/substrate/templates/_helpers.tpl
+++ b/charts/substrate/templates/_helpers.tpl
@@ -123,7 +123,9 @@ Usage:
 {{- range $signal := list "traces" "metrics" "logs" }}
 {{- $cfg := index $otel $signal }}
-{{- if not $cfg.enabled }}
-{{- /* Request no OTLP export with the SDK exporter name "none".
-       Actor events still go to stdout. */}}
+{{- if not (include "substrate.otel.signalEndpoint" (list $signal $)) }}
+{{- /* Disabled, or enabled with no endpoint to send to: request no OTLP
+       export with the SDK exporter name "none". Without it the SDK pushes
+       traces and metrics to its default, localhost:4317, where nothing
+       listens. Actor events still go to stdout. */}}
 - name: OTEL_{{ upper $signal }}_EXPORTER
   value: none
--- a/charts/substrate/templates/k8s-credential-provider.yaml
+++ b/charts/substrate/templates/k8s-credential-provider.yaml
@@ -90,4 +90,8 @@ spec:
         - "--injector-identity=spiffe://cluster.local/ns/{{ .Release.Namespace }}/sa/{{ include "substrate.fullname" (list "atenet-egress" .) }}"
         - "--log-level=info"
+{{- with include "substrate.otel.env" . }}
+        env:
+{{- . | trim | nindent 8 }}
+{{- end }}
         ports:
         - name: grpc

I can open a PR with this if it looks right.

Confirmation

• I have searched existing issues and verified that this is not a duplicate.
• I have verified that this issue occurs on the latest commit on main.