Skip to content

cli: trace each command and send traceparent on its RPCs - #562

Open
JeffLuoo wants to merge 1 commit into
google:mainfrom
JeffLuoo:cli-trace-propagation
Open

JeffLuoo wants to merge 1 commit into
google:mainfrom
JeffLuoo:cli-trace-propagation

Conversation

@JeffLuoo

@JeffLuoo JeffLuoo commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

ax-server cannot join a slow or failed command to the client that issued it, because the CLI sends no trace context. With --trace, an ax command starts a sampled root span, prints Trace ID: <id> to stderr, and sends a W3C traceparent on every AX RPC. ax-server (#557) and Substrate sample ParentBased, so they record the whole request.

  • Without --trace, the CLI creates no tracer and sends no trace context, so ax-server's own sampler decides for CLI traffic, as with kubectl ate.
  • RPCs are client spans under the command span, and resolving the ax-server address (which may start a port-forward) is a child span. CLI spans are exported over OTLP only when OTEL_EXPORTER_OTLP_ENDPOINT is set; OTEL_TRACES_EXPORTER and the resource follow ax-server: set up OpenTelemetry tracing, metrics, events, and JSON logs #557's rules.
  • ax ssh returns the remote exit code instead of calling os.Exit, so the command span ends and the port-forward is cleaned up.

Pins otel v1.45.0 and otelgrpc v0.70.0, as #557 does. It does not depend on #557; the two overlap only in go.mod/go.sum. Server spans join these traces once ax-server adds the otelgrpc server handler.

Refs #13, #452.

Testing

  • go test -race ./..., go vet ./..., and go mod tidy are clean. New tests in cmd/ax/trace_test.go cover the header with and without --trace, span parents, resource attributes, export and SDK warnings, streaming RPCs, the exporter rules, and exit codes.

ax-server cannot join a slow or failed command to the client that issued
it, because the CLI sends no trace context. With --trace, a command now
starts a sampled root span, prints its trace ID to stderr, and sends a
W3C traceparent on every AX RPC, so ax-server and Substrate, which sample
ParentBased, record the whole request.

- Without --trace the CLI creates no tracer and sends no trace context,
  so ax-server keeps the sampling decision for CLI traffic.
- Each RPC is a client span under the command span, and resolving the
  ax-server address (which may start a port-forward) is a child span.
  Only traceparent is propagated, never baggage.
- CLI spans are exported over OTLP only when an OTLP endpoint is set.
  OTEL_TRACES_EXPORTER follows ax-server's rules, and the resource reads
  OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES. Export waits at most
  2 s and warns on failure; OpenTelemetry errors print as warnings.
- ax ssh returns the remote exit code instead of exiting the process,
  so the command span ends and its port-forward is cleaned up.
@JeffLuoo
JeffLuoo requested review from erain and rakyll November 9, 2026 19:37
@JeffLuoo
JeffLuoo force-pushed the cli-trace-propagation branch from 099112e to 80cae1b Compare November 9, 2026 19:45

@erain erain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Verified at 80cae1b: go build ./cmd/..., go vet ./..., gofmt, go mod tidy, and go test -race -count=3 ./cmd/ax/... all pass cleanly.

  • Without --trace, activeTrace stays nil, creating no TracerProvider and sending no traceparent header so ax-server makes its own sampling decision.
  • With --trace, startCommandTrace starts an AlwaysSample root span (ax <cmd>), prints Trace ID: <id> to stderr, wraps tunnel.EnsureServerURL in a resolve ax-server address child span, and configures otelgrpc.NewClientHandler with propagation.TraceContext{} only (no baggage) and metricnoop.NewMeterProvider().
  • envSpanExporter and resolveExporters match #557's exporter and resource rules while only exporting CLI spans when an OTLP endpoint is set.
  • Returning exitCodeError from runSSH cleanly runs defer cleanup() and flushes the command span before exiting with the remote exit code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants