diff --git a/README.md b/README.md index 994c07a..097886c 100644 --- a/README.md +++ b/README.md @@ -201,6 +201,7 @@ Dashboard: **Agent Traces** (`agent-traces`); Tempo span-metrics carry `llm_mode | OpenClaw gateway | `diagnostics.otel` (unchanged) | `openclaw-gateway` | `transform/phoenix_openclaw` (agent id + channel from the system prompt's `Runtime:` line; drops the duplicate `openclaw.content.*` copy; **privacy:** agent `mira` spans lose all message content in Phoenix, tagged `privacy.content_dropped=mira`) | `diagnostics.otel.traces=false` | | Digital twin (Cloudflare Worker, public ask endpoint) | hand-rolled OTLP/JSON tracer in the Worker (`src/telemetry.js`), exported after the response over Cloudflare Tunnel + Access to the collector's **`otlp/twin`** receiver (container port 4320, not host-published, `bearertokenauth/twin`, token `secrets/twin_otel_token` from `.env` `TWIN_OTEL_BEARER_TOKEN`; `filter/twin` drops any service not named `*-digital-twin`). Dashboard `digital-twin-traces` | `digital-twin` | none (the Worker sets OpenInference itself) | Worker var `TRACING="off"`; Phoenix drops bot rejects (`method_not_allowed`, `forbidden_origin`) via `tail_sampling/twin_phoenix`, Tempo keeps them | | SpecStride | `lib/ralph_otel_spans.py`: prompt on the attempt/critic span (`input.value` <= `SPECSTRIDE_OTEL_PROMPT_MAX`, `specstride.prompt.{path,sha256,bytes}`), critic reply as `output.value`; exports `TRACEPARENT` so the claude/dsh child nests inside the run | `specstride` (children too) | `transform/phoenix_kind` | `SPECSTRIDE_OTEL_TRACES=false` | +| agentic-netops intent tier (kind cluster, 4 LangGraph agents) | each agent's `ioa-observe-sdk` POSTs OTLP/HTTP **once** to the tier-owned collector (`agent-otel-collector`, ns `agentic-netops-agents`), which fans out to ClickHouse, feature-001 and this collector via an `otlphttp/phoenix` exporter (`AOS_OTLP_HTTP_ENDPOINT`, default `http://172.17.0.1:4318` from inside kind). Spans are OpenInference-NATIVE (ioa-observe-sdk is a rebranded Traceloop SDK): `ChatLiteLLM.chat` spans carry `gen_ai.prompt/completion` content + `gen_ai.usage.*`/`llm.usage.*` tokens; decorator spans are `.agent` with `agent.name`; graph runs are `LangGraph.workflow` + `*.task`; A2A round-trips propagate W3C context so a request is one trace | `intent-supervisor`, `intent-mapper`, `intent-allocator`, `intent-deployer` (one per `service.name`) | `transform/phoenix_agentic_netops` (LLM kind on `ChatLiteLLM.chat` spans — content + tokens already native via `gen_ai.*`; the literal `llm.model_name=unknown` from the oauth2/ shim is fixed from `ioa_observe.association.properties.ls_model_name`; AGENT kind on `.agent` + `LangGraph.workflow` spans; A2A round-trips arrive as single traces filed under the supervisor's project) | remove `otlphttp/phoenix` from the traces pipeline exporters in `deploy/agents/telemetry.yaml` (agentic-netops) and re-apply | Nesting: Claude Code honours an inbound `TRACEPARENT` env in `-p`/SDK sessions (`parent.source=env` on `claude_code.interaction`); dsh honours it and forwards `traceparent` to LiteLLM, which honours @@ -209,7 +210,13 @@ caller's project; Tempo keeps each service name. Known limits: Claude Code traci smoke test fails when `claude_code.llm_request` spans stop mapping to LLM); detailed content (`new_context`, `response.model_output`, `tool_input`) exists only in `-p`/SDK sessions, and the system prompt is a 500-char preview + hash on the span (full text once per hash in Loki, -event `system_prompt`); OpenClaw strips session ids by design, so its traces have no `session.id`. +event `system_prompt`); OpenClaw strips session ids by design, so its traces have no `session.id`; +the agentic-netops intent tier's source-side redaction (agent-tier policy T333-T335 in +`agents/common/llm.py`) is currently inert — the SDK's context scoping keeps those +`agentic-netops.*` attributes off exported spans — so its Phoenix spans carry FULL +prompt/completion content (consistent with this stack's content-capture-on policy); the +literal `llm.model_name=unknown` is corrected to the resolved model by +`transform/phoenix_agentic_netops`. ## Docs diff --git a/bin/gen-dashboards.py b/bin/gen-dashboards.py index 8464779..52579f4 100644 --- a/bin/gen-dashboards.py +++ b/bin/gen-dashboards.py @@ -278,42 +278,6 @@ def write(d, folder="."): ] write(dashboard("OpenClaw Agents — Cost, Latency & Traces", "agents", ag, ["agents", "llm"]), "ai-agents") -# ---------------- Claude Code (by model + agent/subagent) ---------------- -# claude_code.* OTel metrics -> Prometheus. token_usage labels: model, type -# (input/output/cacheRead/cacheCreation), query_source (main/subagent), session_id. -# Detailed per-request EVENTS -> Loki (logs panel below). Metrics are session-driven. -_id = itertools.count(1) -# Claude Code's OTLP *metrics* are unreliable for real usage: short sessions exit before the 10s -# metric flush, and the counters are ephemeral (they vanish ~5min after a session ends). The per-request -# `api_request` LOG events are exported immediately and PERSIST in Loki — so the consumption panels are -# driven from Loki. Fields on api_request: cost_usd, input_tokens, output_tokens, cache_read_tokens, -# cache_creation_tokens, model, duration_ms. Subagents come from subagent_completed (agent_type, model, -# total_tokens). All totals below are over the dashboard time range ($__range) — true persistent history. -CC = '{service_name="claude-code"} | event_name=`api_request`' -cc = [ - lstat("Cost (range)", f'sum(sum_over_time({CC} | unwrap cost_usd [$__range]))', "currencyUSD", 6, 4, 0, 0), - lstat("Output tokens", f'sum(sum_over_time({CC} | unwrap output_tokens [$__range]))', "short", 6, 4, 6, 0), - lstat("Cache-read tokens", f'sum(sum_over_time({CC} | unwrap cache_read_tokens [$__range]))', "short", 6, 4, 12, 0), - lstat("Requests", f'sum(count_over_time({CC} [$__range]))', "short", 6, 4, 18, 0), - lts("Cost by MODEL", [(f'sum by (model)(sum_over_time({CC} | unwrap cost_usd [$__interval]))', "{{model}}")], "currencyUSD", 12, 8, 0, 4, stack=True), - lts("Tokens by TYPE (in / out / cacheRead / cacheCreation)", [ - (f'sum(sum_over_time({CC} | unwrap input_tokens [$__interval]))', "input"), - (f'sum(sum_over_time({CC} | unwrap output_tokens [$__interval]))', "output"), - (f'sum(sum_over_time({CC} | unwrap cache_read_tokens [$__interval]))', "cacheRead"), - (f'sum(sum_over_time({CC} | unwrap cache_creation_tokens [$__interval]))', "cacheCreation")], "short", 12, 8, 12, 4, stack=True), - lts("Output tokens by MODEL", [(f'sum by (model)(sum_over_time({CC} | unwrap output_tokens [$__interval]))', "{{model}}")], "short", 12, 8, 0, 12, stack=True), - lts("Requests by MODEL", [(f'sum by (model)(count_over_time({CC} [$__interval]))', "{{model}}")], "short", 12, 8, 12, 12, stack=True), - lbar("Cache-read tokens by MODEL (range)", f'sum by (model)(sum_over_time({CC} | unwrap cache_read_tokens [$__range]))', "short", 12, 8, 0, 20), - lts("Request latency p95 (by model)", [(f'quantile_over_time(0.95, {CC} | unwrap duration_ms [$__interval]) by (model)', "{{model}}")], "ms", 12, 8, 12, 20), - # ----- Named subagents (from Loki subagent_completed events: agent_type, model, total_tokens) ----- - lbar("Subagent runs by type", 'sum by (agent_type) (count_over_time({service_name="claude-code"} | event_name=`subagent_completed` [$__range]))', "short", 12, 8, 0, 28), - lbar("Subagent tokens by type", 'sum by (agent_type) (sum_over_time({service_name="claude-code"} | event_name=`subagent_completed` | unwrap total_tokens [$__range]))', "short", 12, 8, 12, 28), - lts("Subagent tokens over time (by type)", [('sum by (agent_type) (sum_over_time({service_name="claude-code"} | event_name=`subagent_completed` | unwrap total_tokens [$__interval]))', "{{agent_type}}")], "short", 12, 8, 0, 36, stack=True), - lts("Subagent activity (by model)", [('sum by (model) (count_over_time({service_name="claude-code"} | event_name=`subagent_completed` [$__interval]))', "{{model}}")], "short", 12, 8, 12, 36, stack=True), - logs("Recent Claude Code events (Loki — request/tool/subagent: model, tokens, agent_type)", '{service_name="claude-code"}', 24, 11, 0, 44), -] -write(dashboard("Claude Code", "claude-code", cc, ["ai", "claude-code"]), "ai-agents") - # ---------------- LLM Cost & Consumption (unified Claude Code + LiteLLM/OpenClaw) ---------------- _id = itertools.count(1) # Two cost sources: Claude Code (persistent Loki api_request events) + OpenClaw/LiteLLM (persistent @@ -474,83 +438,6 @@ def write(d, folder="."): write(dashboard("SpecStride Runs & Traces", "specstride-traces", spec, ["ai", "specstride", "traces", "ralph"], refresh="30s", templating=[dict(var_query("task", 'label_values({service_name="ralph"}, task)', ds=LOKI), allValue=".*")]), "ai-agents") -# ---------------- Agent Mix (12.1 frontier-fading tree) ---------------- -# The fading dashboard. ONE event stream (job="agentops", from agent-pack `agentops emit`) feeds -# every panel here; the SAME events also drive the kanban board (two sinks, one stream — they can -# never disagree). Loki stream labels (low-cardinality): job="agentops", event, task. Everything -# else is logfmt in the line: model, agent, verdict, rung, attempts, tokens_in, tokens_out, -# wall_ms, dial_state, degraded, reason, contract_id, card, adhoc, task_class. -# Events: dispatch (task handed to a leaf), verdict (pass|fail|escalated), escalation (kicked up -# the ladder), adhoc (bebop ask — a question, no card). LOCAL model = qwen* ; FRONTIER = the rest -# (claude-*, gpt-*). GPU-degraded-by-reason is the ONE Prometheus panel: metric -# agentops_gpu_degraded{reason=...} from the S3/H1 watchdog — reason="cpu-fallback" (H1) shows as -# its own series so a silent CPU-fallback is visually distinct from a device-health degrade (S3). -_id = itertools.count(1) -AOVERD = '{job="agentops", event="verdict", task=~"$task"}' -AODISP = '{job="agentops", event="dispatch", task=~"$task"}' -AOESC = '{job="agentops", event="escalation", task=~"$task"}' -LOCAL_RE = 'qwen.*' # local worker; everything else is frontier -MNOTE = ( - "**Frontier-fading agent tree (12.1).** opus plans/verifies, the local qwen worker executes, and " - "trust is earned per task-class. Every panel is the persistent Loki event stream " - "(`job=\"agentops\"`, one event per dispatch/verdict/escalation) over the dashboard range — the " - "**same** events that drive the kanban board. **Local** = `qwen*`; **Frontier** = `claude-*` / " - "`gpt-*`. The goal of fading is the two numbers up top moving in opposite directions: local task " - "share **up**, frontier tokens (cost proxy) **down**, without local-first pass-rate dropping below " - "the promotion bar (0.85). GPU-degraded is split **by reason** so a silent CPU-fallback (H1, " - "`reason=cpu-fallback`) reads distinctly from a device-health degrade (S3)." -) -mix = [ - text("About this dashboard", MNOTE, 24, 4, 0, 0), - # ----- headline stats over the range ----- - lstat("Tasks completed", f'sum(count_over_time({AOVERD} [$__range]))', "short", 6, 4, 0, 4), - lstat("Local task share", - f'sum(count_over_time({AOVERD} | logfmt | model=~`{LOCAL_RE}` [$__range])) ' - f'/ sum(count_over_time({AOVERD} | logfmt [$__range]))', - "percentunit", 6, 4, 6, 4, - thresholds=[{"color": "red", "value": None}, {"color": "orange", "value": 0.5}, {"color": "green", "value": 0.8}]), - lstat("Escalation rate", - f'sum(count_over_time({AOESC} [$__range])) ' - f'/ sum(count_over_time({AODISP} [$__range]))', - "percentunit", 6, 4, 12, 4, - thresholds=[{"color": "green", "value": None}, {"color": "orange", "value": 0.2}, {"color": "red", "value": 0.4}]), - stat("GPU degraded now", 'max(agentops_gpu_degraded) or vector(0)', "short", 6, 4, 18, 4, - thresholds=[{"color": "green", "value": None}, {"color": "red", "value": 1}]), - # ----- frontier share: tokens + task counts, local vs frontier ----- - lts("Tokens: local vs frontier (cost proxy)", [ - (f'sum(sum_over_time({AOVERD} | logfmt | model=~`{LOCAL_RE}` | unwrap tokens_out [$__interval]))', "local (qwen)"), - (f'sum(sum_over_time({AOVERD} | logfmt | model!~`{LOCAL_RE}` | unwrap tokens_out [$__interval]))', "frontier (claude/gpt)")], - "short", 12, 8, 0, 8, stack=True), - lts("Task mix: local vs frontier (count)", [ - (f'sum(count_over_time({AOVERD} | logfmt | model=~`{LOCAL_RE}` [$__interval]))', "local (qwen)"), - (f'sum(count_over_time({AOVERD} | logfmt | model!~`{LOCAL_RE}` [$__interval]))', "frontier (claude/gpt)")], - "short", 12, 8, 12, 8, stack=True), - # ----- the fading metrics, BY task-class ----- - lbar("Local-first pass rate by class (promote ≥ 0.85)", - f'sum by (task)(count_over_time({AOVERD} | logfmt | model=~`{LOCAL_RE}` | verdict=`pass` | attempts=`1` [$__range])) ' - f'/ sum by (task)(count_over_time({AOVERD} | logfmt | model=~`{LOCAL_RE}` [$__range]))', - "percentunit", 12, 8, 0, 16, legend="{{task}}"), - lbar("Escalation rate by class", - f'sum by (task)(count_over_time({AOESC} [$__range])) ' - f'/ sum by (task)(count_over_time({AODISP} [$__range]))', - "percentunit", 12, 8, 12, 16, legend="{{task}}"), - # ----- dial position + cost proxy breakdown ----- - lbar("Tasks by dial position (fading state)", - f'sum by (dial_state)(count_over_time({AOVERD} | logfmt [$__range]))', - "short", 12, 8, 0, 24, legend="{{dial_state}}"), - lbar("Frontier tokens by model (range) — cost proxy", - f'sum by (model)(sum_over_time({AOVERD} | logfmt | model!~`{LOCAL_RE}` | unwrap tokens_out [$__range]))', - "short", 12, 8, 12, 24, legend="{{model}}"), - # ----- GPU degraded windows, SPLIT BY reason (H1: cpu-fallback distinct from S3 device-health) ----- - ts("GPU degraded windows (by reason) — cpu-fallback = H1, others = S3 device-health", - [('agentops_gpu_degraded', "{{reason}}")], "short", 24, 7, 0, 32), - # ----- raw event stream ----- - logs("Recent agent-pack events (dispatch / verdict / escalation / adhoc)", - '{job="agentops", task=~"$task"}', 24, 11, 0, 39), -] -write(dashboard("Agent Mix — Frontier Fading (12.1)", "agent-mix", mix, ["ai", "agents", "12.1", "fading"], - templating=[var_query("task", 'label_values({job="agentops"}, task)', ds=LOKI)]), "ai-agents") - # ---------------- Containers ---------------- _id = itertools.count(1) cn = [ diff --git a/bot/grafy_bot.py b/bot/grafy_bot.py index 02e09e4..8ded131 100644 --- a/bot/grafy_bot.py +++ b/bot/grafy_bot.py @@ -29,42 +29,43 @@ # dashboard aliases -> (uid, slug, title) DASH = { + # --- Infrastructure --- "host": ("host", "host", "Host / System"), "system": ("host", "host", "Host / System"), "hypervisor": ("host", "host", "Host / System"), - "accelerators": ("accelerators", "accelerators", "Accelerators (iGPU+NPU)"), - "acc": ("accelerators", "accelerators", "Accelerators (iGPU+NPU)"), - "gpu": ("accelerators", "accelerators", "Accelerators (iGPU+NPU)"), - "npu": ("accelerators", "accelerators", "Accelerators (iGPU+NPU)"), + "accelerators": ("accelerators", "accelerators", "Accelerators (NVIDIA + iGPU + NPU)"), + "acc": ("accelerators", "accelerators", "Accelerators (NVIDIA + iGPU + NPU)"), + "gpu": ("accelerators", "accelerators", "Accelerators (NVIDIA + iGPU + NPU)"), + "npu": ("accelerators", "accelerators", "Accelerators (NVIDIA + iGPU + NPU)"), "network": ("network", "network", "Network & Connectivity"), "net": ("network", "network", "Network & Connectivity"), "containers": ("containers", "containers", "Containers"), "docker": ("containers", "containers", "Containers"), - "agents": ("agents", "agents", "OpenClaw Agents"), - "agent": ("agents", "agents", "OpenClaw Agents"), - "openclaw": ("agents", "agents", "OpenClaw Agents"), - "claude-code": ("claude-code", "claude-code", "Claude Code"), - "claude": ("claude-code", "claude-code", "Claude Code"), - "cc": ("claude-code", "claude-code", "Claude Code"), - "llm-cost": ("llm-cost", "llm-cost", "LLM Cost & Consumption"), - "cost": ("llm-cost", "llm-cost", "LLM Cost & Consumption"), - "llm": ("llm-cost", "llm-cost", "LLM Cost & Consumption"), - "spend": ("llm-cost", "llm-cost", "LLM Cost & Consumption"), - "rag": ("rag", "rag", "RAG / Memory"), - "memory": ("rag", "rag", "RAG / Memory"), - "qmd": ("rag", "rag", "RAG / Memory"), - "vectors": ("qmd-vector-space", "qmd-vector-space", "QMD Vector Space"), - "vector": ("qmd-vector-space", "qmd-vector-space", "QMD Vector Space"), - "vector-space": ("qmd-vector-space", "qmd-vector-space", "QMD Vector Space"), - "vectorspace": ("qmd-vector-space", "qmd-vector-space", "QMD Vector Space"), - "space": ("qmd-vector-space", "qmd-vector-space", "QMD Vector Space"), - "embeddings": ("qmd-vector-space", "qmd-vector-space", "QMD Vector Space"), - "pca": ("qmd-vector-space", "qmd-vector-space", "QMD Vector Space"), - # Composite: Claude Code vs pi - "composite": ("agents-composite", "agents-composite", "Agents — Claude Code vs pi"), - "agents-composite": ("agents-composite", "agents-composite", "Agents — Claude Code vs pi"), - "both": ("agents-composite", "agents-composite", "Agents — Claude Code vs pi"), - "claude-vs-pi": ("agents-composite", "agents-composite", "Agents — Claude Code vs pi"), + # --- AI & Agents --- + "agents": ("agents", "agents", "OpenClaw Agents — Cost, Latency & Traces"), + "agent": ("agents", "agents", "OpenClaw Agents — Cost, Latency & Traces"), + "openclaw": ("agents", "agents", "OpenClaw Agents — Cost, Latency & Traces"), + "llm-cost": ("llm-cost", "llm-cost", "LLM Cost & Consumption (Unified)"), + "cost": ("llm-cost", "llm-cost", "LLM Cost & Consumption (Unified)"), + "llm": ("llm-cost", "llm-cost", "LLM Cost & Consumption (Unified)"), + "spend": ("llm-cost", "llm-cost", "LLM Cost & Consumption (Unified)"), + "litellm-models": ("litellm-models", "litellm-models", "Models — LiteLLM Inventory"), + "models": ("litellm-models", "litellm-models", "Models — LiteLLM Inventory"), + "rag": ("rag", "rag", "RAG / Memory (QMD)"), + "memory": ("rag", "rag", "RAG / Memory (QMD)"), + "qmd": ("rag", "rag", "RAG / Memory (QMD)"), + "vectors": ("qmd-vector-space", "qmd-vector-space", "QMD Vector Space — memory corpus + benchmark queries"), + "vector": ("qmd-vector-space", "qmd-vector-space", "QMD Vector Space — memory corpus + benchmark queries"), + "vector-space": ("qmd-vector-space", "qmd-vector-space", "QMD Vector Space — memory corpus + benchmark queries"), + "vectorspace": ("qmd-vector-space", "qmd-vector-space", "QMD Vector Space — memory corpus + benchmark queries"), + "space": ("qmd-vector-space", "qmd-vector-space", "QMD Vector Space — memory corpus + benchmark queries"), + "embeddings": ("qmd-vector-space", "qmd-vector-space", "QMD Vector Space — memory corpus + benchmark queries"), + "pca": ("qmd-vector-space", "qmd-vector-space", "QMD Vector Space — memory corpus + benchmark queries"), + # Harness comparison (formerly "Claude Code vs pi") + "composite": ("agents-composite", "agents-composite", "Agents — Harness comparison (Claude Code / pi / dsh / prime)"), + "agents-composite": ("agents-composite", "agents-composite", "Agents — Harness comparison (Claude Code / pi / dsh / prime)"), + "both": ("agents-composite", "agents-composite", "Agents — Harness comparison (Claude Code / pi / dsh / prime)"), + "claude-vs-pi": ("agents-composite", "agents-composite", "Agents — Harness comparison (Claude Code / pi / dsh / prime)"), # Ralph autonomous loops (Loki-backed) "ralph": ("ralph-loops", "ralph-loops", "Ralph Loops (multi-harness)"), "ralph-loops": ("ralph-loops", "ralph-loops", "Ralph Loops (multi-harness)"), @@ -74,10 +75,14 @@ "inference": ("inference-llama", "inference-llama", "LLM Inference — llama.cpp + MTP (RTX 3090)"), "llama": ("inference-llama", "inference-llama", "LLM Inference — llama.cpp + MTP (RTX 3090)"), "mtp": ("inference-llama", "inference-llama", "LLM Inference — llama.cpp + MTP (RTX 3090)"), - # Agent mix — frontier fading (12.1) - "agent-mix": ("agent-mix", "agent-mix", "Agent Mix — Frontier Fading (12.1)"), - "mix": ("agent-mix", "agent-mix", "Agent Mix — Frontier Fading (12.1)"), - "frontier-fading": ("agent-mix", "agent-mix", "Agent Mix — Frontier Fading (12.1)"), + # Traces + "agent-traces": ("agent-traces", "agent-traces", "Agent Traces (Claude Code, dsh, LiteLLM)"), + "digital-twin": ("digital-twin-traces", "digital-twin-traces", "Digital Twin — requests, guards & LLM (traces)"), + "twin": ("digital-twin-traces", "digital-twin-traces", "Digital Twin — requests, guards & LLM (traces)"), + "sovereign-trace": ("sovereign-trace", "sovereign-trace", "Request trace"), + "sovereign": ("sovereign-trace", "sovereign-trace", "Request trace"), + "specstride": ("specstride-traces", "specstride-traces", "SpecStride Runs & Traces"), + "specstride-traces": ("specstride-traces", "specstride-traces", "SpecStride Runs & Traces"), } # Dashboards whose full render is unusable headless (e.g. a WebGL scatter3d panel that the @@ -132,13 +137,6 @@ def mb(v): return f"{float(v)/1e6:.0f} MB" ("Cache hit", 'sum(rate(litellm_input_cached_tokens_metric_total[30m]))/clamp_min(sum(rate(litellm_input_tokens_metric_total[30m])),1)*100', pct), ("Agents", 'openclaw_agents_total', num), ], - "claude-code": [ - ("CC tokens 24h", 'sum(increase(claude_code_token_usage_tokens_total[24h]))', num), - ("CC cost 24h", 'sum(increase(claude_code_cost_usage_USD_total[24h]))', usd), - ("Sessions 24h", 'sum(increase(claude_code_session_count_total[24h]))', num), - ("Models", 'count(count by(model)(claude_code_token_usage_tokens_total))', num), - ("Cache hit", 'sum(rate(claude_code_token_usage_tokens_total{type="cacheRead"}[30m]))/clamp_min(sum(rate(claude_code_token_usage_tokens_total{type=~"input|cacheRead|cacheCreation"}[30m])),1)*100', pct), - ], "llm-cost": [ ("Total spend 24h", '(sum(increase(claude_code_cost_usage_USD_total[24h])) or vector(0))+(sum(increase(litellm_spend_metric_total[24h])) or vector(0))', usd), ("Claude Code 24h", 'sum(increase(claude_code_cost_usage_USD_total[24h])) or vector(0)', usd), @@ -331,7 +329,7 @@ def send_photo(chat, png, caption): "/values <dashboard> [range] — values only\n" "/alerts — firing alerts\n" "/list — dashboards\n\n" - "dashboards: host, accelerators, network, containers, agents, claude-code, llm-cost, rag, vectors, ralph, inference\n" + "dashboards: host, accelerators, network, containers, agents, llm-cost, models, rag, vectors, composite, ralph, inference, agent-traces, twin, sovereign, specstride\n" "any dashboard in Grafana works too — use its uid (see /list)\n" "range: 1h (default 3h), 6h, 24h, 7d\n" "e.g. /graph ralph 24h" diff --git a/grafana/dashboards/archive/agent-mix.json b/grafana/dashboards/archive/agent-mix.json deleted file mode 100644 index c620d3a..0000000 --- a/grafana/dashboards/archive/agent-mix.json +++ /dev/null @@ -1,679 +0,0 @@ -{ - "uid": "agent-mix", - "title": "Agent Mix \u2014 Frontier Fading (12.1)", - "tags": [ - "ai", - "agents", - "12.1", - "fading" - ], - "schemaVersion": 39, - "version": 1, - "editable": true, - "refresh": "30s", - "time": { - "from": "now-6h", - "to": "now" - }, - "timezone": "browser", - "panels": [ - { - "id": 1, - "type": "text", - "title": "About this dashboard", - "gridPos": { - "h": 7, - "w": 24, - "x": 0, - "y": 0 - }, - "options": { - "mode": "markdown", - "content": "**Frontier-fading agent tree (12.1).** opus plans/verifies, the local qwen worker executes, and trust is earned per task-class. Every panel is the persistent Loki event stream (`job=\"agentops\"`, one event per dispatch/verdict/escalation) over the dashboard range \u2014 the **same** events that drive the kanban board. **Local** = `qwen*`; **Frontier** = `claude-*` / `gpt-*`. The goal of fading is the two numbers up top moving in opposite directions: local task share **up**, frontier tokens (cost proxy) **down**, without local-first pass-rate dropping below the promotion bar (0.85). GPU-degraded is split **by reason** so a silent CPU-fallback (H1, `reason=cpu-fallback`) reads distinctly from a device-health degrade (S3).\n\n---\n\n**Empty is not broken (verified 2026-08-27).** Every query here targets `job=\"agentops\"`, written by `agent-pack/bin/agentops-ship.py` (`dispatch | escalation | verdict | adhoc | dispatch_start`). That vocabulary is correct and the shipper is installed \u2014 but no governed agent-pack run has executed inside Loki's 7-day retention (the only `agentops` line in the window is a `test_governed_fleet_probe` from 2026-08-26). So these panels read *No data* because the fleet has been idle on this path, **not** because the queries are wrong. Run a governed card and they populate. If they are still empty after a run, the shipper is not reaching Loki \u2014 check `agentops-ship.py --loki`." - } - }, - { - "id": 2, - "type": "stat", - "title": "Tasks completed", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 4, - "w": 6, - "x": 0, - "y": 7 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - } - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "area", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(count_over_time({job=\"agentops\", event=\"verdict\", task=~\"$task\"} [$__range]))", - "queryType": "instant" - } - ] - }, - { - "id": 3, - "type": "stat", - "title": "Local task share", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 4, - "w": 6, - "x": 6, - "y": 7 - }, - "fieldConfig": { - "defaults": { - "unit": "percentunit", - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "red", - "value": null - }, - { - "color": "orange", - "value": 0.5 - }, - { - "color": "green", - "value": 0.8 - } - ] - } - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "area", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(count_over_time({job=\"agentops\", event=\"verdict\", task=~\"$task\"} | logfmt | model=~`qwen.*` [$__range])) / sum(count_over_time({job=\"agentops\", event=\"verdict\", task=~\"$task\"} | logfmt [$__range]))", - "queryType": "instant" - } - ] - }, - { - "id": 4, - "type": "stat", - "title": "Escalation rate", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 4, - "w": 6, - "x": 12, - "y": 7 - }, - "fieldConfig": { - "defaults": { - "unit": "percentunit", - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "orange", - "value": 0.2 - }, - { - "color": "red", - "value": 0.4 - } - ] - } - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "area", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(count_over_time({job=\"agentops\", event=\"escalation\", task=~\"$task\"} [$__range])) / sum(count_over_time({job=\"agentops\", event=\"dispatch\", task=~\"$task\"} [$__range]))", - "queryType": "instant" - } - ] - }, - { - "id": 5, - "type": "stat", - "title": "GPU degraded now", - "datasource": { - "type": "prometheus", - "uid": "prometheus" - }, - "gridPos": { - "h": 4, - "w": 6, - "x": 18, - "y": 7 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 1 - } - ] - } - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "area", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "prometheus", - "uid": "prometheus" - }, - "expr": "max(agentops_gpu_degraded) or vector(0)", - "legendFormat": "" - } - ] - }, - { - "id": 6, - "type": "timeseries", - "title": "Tokens: local vs frontier (cost proxy)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 11 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "custom": { - "drawStyle": "line", - "fillOpacity": 40, - "stacking": { - "mode": "normal" - } - } - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "calcs": [ - "lastNotNull", - "max" - ] - }, - "tooltip": { - "mode": "multi" - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "(sum(sum_over_time({job=\"agentops\", event=\"verdict\", task=~\"$task\"} | logfmt | model=~`qwen.*` | unwrap tokens_out [$__interval])) or sum(sum_over_time({job=\"agentops\", event=\"verdict\", task=~\"$task\"} | logfmt | model=~`qwen.*` | unwrap output_tokens [$__interval])))", - "legendFormat": "local (qwen)", - "queryType": "range" - }, - { - "refId": "B", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(sum_over_time({job=\"agentops\", event=\"verdict\", task=~\"$task\"} | logfmt | model!~`qwen.*` | unwrap tokens_out [$__interval]))", - "legendFormat": "frontier (claude/gpt)", - "queryType": "range" - } - ] - }, - { - "id": 7, - "type": "timeseries", - "title": "Task mix: local vs frontier (count)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 11 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "custom": { - "drawStyle": "line", - "fillOpacity": 40, - "stacking": { - "mode": "normal" - } - } - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "calcs": [ - "lastNotNull", - "max" - ] - }, - "tooltip": { - "mode": "multi" - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(count_over_time({job=\"agentops\", event=\"verdict\", task=~\"$task\"} | logfmt | model=~`qwen.*` [$__interval]))", - "legendFormat": "local (qwen)", - "queryType": "range" - }, - { - "refId": "B", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(count_over_time({job=\"agentops\", event=\"verdict\", task=~\"$task\"} | logfmt | model!~`qwen.*` [$__interval]))", - "legendFormat": "frontier (claude/gpt)", - "queryType": "range" - } - ] - }, - { - "id": 8, - "type": "bargauge", - "title": "Local-first pass rate by class (promote \u2265 0.85)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 19 - }, - "fieldConfig": { - "defaults": { - "unit": "percentunit", - "color": { - "mode": "palette-classic" - } - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum by (task)(count_over_time({job=\"agentops\", event=\"verdict\", task=~\"$task\"} | logfmt | model=~`qwen.*` | verdict=`pass` | attempts=`1` [$__range])) / sum by (task)(count_over_time({job=\"agentops\", event=\"verdict\", task=~\"$task\"} | logfmt | model=~`qwen.*` [$__range]))", - "queryType": "instant", - "legendFormat": "{{task}}" - } - ] - }, - { - "id": 9, - "type": "bargauge", - "title": "Escalation rate by class", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 19 - }, - "fieldConfig": { - "defaults": { - "unit": "percentunit", - "color": { - "mode": "palette-classic" - } - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum by (task)(count_over_time({job=\"agentops\", event=\"escalation\", task=~\"$task\"} [$__range])) / sum by (task)(count_over_time({job=\"agentops\", event=\"dispatch\", task=~\"$task\"} [$__range]))", - "queryType": "instant", - "legendFormat": "{{task}}" - } - ] - }, - { - "id": 10, - "type": "bargauge", - "title": "Tasks by dial position (fading state)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 27 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "color": { - "mode": "palette-classic" - } - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum by (dial_state)(count_over_time({job=\"agentops\", event=\"verdict\", task=~\"$task\"} | logfmt [$__range]))", - "queryType": "instant", - "legendFormat": "{{dial_state}}" - } - ] - }, - { - "id": 11, - "type": "bargauge", - "title": "Frontier tokens by model (range) \u2014 cost proxy", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 27 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "color": { - "mode": "palette-classic" - } - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum by (model)(sum_over_time({job=\"agentops\", event=\"verdict\", task=~\"$task\"} | logfmt | model!~`qwen.*` | unwrap tokens_out [$__range]))", - "queryType": "instant", - "legendFormat": "{{model}}" - } - ] - }, - { - "id": 12, - "type": "timeseries", - "title": "GPU degraded windows (by reason) \u2014 cpu-fallback = H1, others = S3 device-health", - "datasource": { - "type": "prometheus", - "uid": "prometheus" - }, - "gridPos": { - "h": 7, - "w": 24, - "x": 0, - "y": 35 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "custom": { - "drawStyle": "line", - "fillOpacity": 18, - "stacking": { - "mode": "none" - } - } - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "calcs": [ - "lastNotNull", - "max" - ] - }, - "tooltip": { - "mode": "multi" - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "prometheus", - "uid": "prometheus" - }, - "expr": "agentops_gpu_degraded", - "legendFormat": "{{reason}}" - } - ] - }, - { - "id": 13, - "type": "logs", - "title": "Recent agent-pack events (dispatch / verdict / escalation / adhoc)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 11, - "w": 24, - "x": 0, - "y": 42 - }, - "options": { - "showTime": true, - "wrapLogMessage": true, - "dedupStrategy": "none", - "sortOrder": "Descending", - "enableLogDetails": true - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "{job=\"agentops\", task=~\"$task\"}", - "queryType": "range" - } - ] - } - ], - "templating": { - "list": [ - { - "name": "task", - "type": "query", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "query": "label_values({job=\"agentops\"}, task)", - "includeAll": true, - "multi": true, - "refresh": 2, - "current": { - "text": "All", - "value": "$__all" - } - } - ] - } -} \ No newline at end of file diff --git a/grafana/dashboards/archive/claude-code.json b/grafana/dashboards/archive/claude-code.json deleted file mode 100644 index 9a73526..0000000 --- a/grafana/dashboards/archive/claude-code.json +++ /dev/null @@ -1,804 +0,0 @@ -{ - "uid": "claude-code", - "title": "Claude Code (Archived)", - "tags": [ - "ai", - "claude-code" - ], - "schemaVersion": 39, - "version": 2, - "editable": true, - "refresh": "30s", - "time": { - "from": "now-6h", - "to": "now" - }, - "timezone": "browser", - "panels": [ - { - "id": 1, - "type": "stat", - "title": "Cost (range)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 4, - "w": 6, - "x": 0, - "y": 0 - }, - "fieldConfig": { - "defaults": { - "unit": "currencyUSD", - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - } - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "area", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(sum_over_time({service_name=\"claude-code\"} | event_name=`api_request` | cost_usd != `0` | unwrap cost_usd [$__range]))", - "queryType": "instant" - } - ] - }, - { - "id": 2, - "type": "stat", - "title": "Output tokens", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 4, - "w": 6, - "x": 6, - "y": 0 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - } - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "area", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(sum_over_time({service_name=\"claude-code\"} | event_name=`api_request` | unwrap output_tokens [$__range]))", - "queryType": "instant" - } - ] - }, - { - "id": 3, - "type": "stat", - "title": "Cache-read tokens", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 4, - "w": 6, - "x": 12, - "y": 0 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - } - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "area", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(sum_over_time({service_name=\"claude-code\"} | event_name=`api_request` | unwrap cache_read_tokens [$__range]))", - "queryType": "instant" - } - ] - }, - { - "id": 4, - "type": "stat", - "title": "Requests", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 4, - "w": 6, - "x": 18, - "y": 0 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - } - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "area", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(count_over_time({service_name=\"claude-code\"} | event_name=`api_request` [$__range]))", - "queryType": "instant" - } - ] - }, - { - "id": 5, - "type": "timeseries", - "title": "Cost by MODEL", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 4 - }, - "fieldConfig": { - "defaults": { - "unit": "currencyUSD", - "custom": { - "drawStyle": "line", - "fillOpacity": 40, - "stacking": { - "mode": "normal" - } - } - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "calcs": [ - "lastNotNull", - "max" - ] - }, - "tooltip": { - "mode": "multi" - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum by (model)(sum_over_time({service_name=\"claude-code\"} | event_name=`api_request` | unwrap cost_usd [$__interval]))", - "legendFormat": "{{model}}", - "queryType": "range" - } - ] - }, - { - "id": 6, - "type": "timeseries", - "title": "Tokens by TYPE (in / out / cacheRead / cacheCreation)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 4 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "custom": { - "drawStyle": "line", - "fillOpacity": 40, - "stacking": { - "mode": "normal" - } - } - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "calcs": [ - "lastNotNull", - "max" - ] - }, - "tooltip": { - "mode": "multi" - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(sum_over_time({service_name=\"claude-code\"} | event_name=`api_request` | unwrap input_tokens [$__interval]))", - "legendFormat": "input", - "queryType": "range" - }, - { - "refId": "B", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(sum_over_time({service_name=\"claude-code\"} | event_name=`api_request` | unwrap output_tokens [$__interval]))", - "legendFormat": "output", - "queryType": "range" - }, - { - "refId": "C", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(sum_over_time({service_name=\"claude-code\"} | event_name=`api_request` | unwrap cache_read_tokens [$__interval]))", - "legendFormat": "cacheRead", - "queryType": "range" - }, - { - "refId": "D", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(sum_over_time({service_name=\"claude-code\"} | event_name=`api_request` | unwrap cache_creation_tokens [$__interval]))", - "legendFormat": "cacheCreation", - "queryType": "range" - } - ] - }, - { - "id": 7, - "type": "timeseries", - "title": "Output tokens by MODEL", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 12 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "custom": { - "drawStyle": "line", - "fillOpacity": 40, - "stacking": { - "mode": "normal" - } - } - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "calcs": [ - "lastNotNull", - "max" - ] - }, - "tooltip": { - "mode": "multi" - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum by (model)(sum_over_time({service_name=\"claude-code\"} | event_name=`api_request` | unwrap output_tokens [$__interval]))", - "legendFormat": "{{model}}", - "queryType": "range" - } - ] - }, - { - "id": 8, - "type": "timeseries", - "title": "Requests by MODEL", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 12 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "custom": { - "drawStyle": "line", - "fillOpacity": 40, - "stacking": { - "mode": "normal" - } - } - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "calcs": [ - "lastNotNull", - "max" - ] - }, - "tooltip": { - "mode": "multi" - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum by (model)(count_over_time({service_name=\"claude-code\"} | event_name=`api_request` [$__interval]))", - "legendFormat": "{{model}}", - "queryType": "range" - } - ] - }, - { - "id": 9, - "type": "bargauge", - "title": "Cache-read tokens by MODEL (range)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 20 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "color": { - "mode": "palette-classic" - } - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum by (model)(sum_over_time({service_name=\"claude-code\"} | event_name=`api_request` | unwrap cache_read_tokens [$__range]))", - "queryType": "instant", - "legendFormat": "{{agent_type}}" - } - ] - }, - { - "id": 10, - "type": "timeseries", - "title": "Request latency p95 (by model)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 20 - }, - "fieldConfig": { - "defaults": { - "unit": "ms", - "custom": { - "drawStyle": "line", - "fillOpacity": 18, - "stacking": { - "mode": "none" - } - } - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "calcs": [ - "lastNotNull", - "max" - ] - }, - "tooltip": { - "mode": "multi" - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "quantile_over_time(0.95, {service_name=\"claude-code\"} | event_name=`api_request` | unwrap duration_ms [$__interval]) by (model)", - "legendFormat": "{{model}}", - "queryType": "range" - } - ] - }, - { - "id": 11, - "type": "bargauge", - "title": "Subagent runs by type", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 28 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "color": { - "mode": "palette-classic" - } - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum by (agent_type) (count_over_time({service_name=\"claude-code\"} | event_name=`subagent_completed` [$__range]))", - "queryType": "instant", - "legendFormat": "{{agent_type}}" - } - ] - }, - { - "id": 12, - "type": "bargauge", - "title": "Subagent tokens by type", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 28 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "color": { - "mode": "palette-classic" - } - }, - "overrides": [] - }, - "options": { - "displayMode": "gradient", - "orientation": "horizontal", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum by (agent_type) (sum_over_time({service_name=\"claude-code\"} | event_name=`subagent_completed` | unwrap total_tokens [$__range]))", - "queryType": "instant", - "legendFormat": "{{agent_type}}" - } - ] - }, - { - "id": 13, - "type": "timeseries", - "title": "Subagent tokens over time (by type)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 36 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "custom": { - "drawStyle": "line", - "fillOpacity": 40, - "stacking": { - "mode": "normal" - } - } - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "calcs": [ - "lastNotNull", - "max" - ] - }, - "tooltip": { - "mode": "multi" - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum by (agent_type) (sum_over_time({service_name=\"claude-code\"} | event_name=`subagent_completed` | unwrap total_tokens [$__interval]))", - "legendFormat": "{{agent_type}}", - "queryType": "range" - } - ] - }, - { - "id": 14, - "type": "timeseries", - "title": "Subagent activity (by model)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 36 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "custom": { - "drawStyle": "line", - "fillOpacity": 40, - "stacking": { - "mode": "normal" - } - } - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "bottom", - "calcs": [ - "lastNotNull", - "max" - ] - }, - "tooltip": { - "mode": "multi" - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum by (model) (count_over_time({service_name=\"claude-code\"} | event_name=`subagent_completed` [$__interval]))", - "legendFormat": "{{model}}", - "queryType": "range" - } - ] - }, - { - "id": 15, - "type": "logs", - "title": "Recent Claude Code events (Loki \u2014 request/tool/subagent: model, tokens, agent_type)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 11, - "w": 24, - "x": 0, - "y": 44 - }, - "options": { - "showTime": true, - "wrapLogMessage": true, - "dedupStrategy": "none", - "sortOrder": "Descending", - "enableLogDetails": true - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "{service_name=\"claude-code\"}", - "queryType": "range" - } - ] - } - ], - "templating": { - "list": [] - }, - "description": "Archived \u2014 superseded by AI & Agents / Agents (Composite), which runs the same panels scoped by service_name so pi and ralph appear alongside claude-code. Verified 2026-08-27: every query here still returns data. Kept for the claude-code-only view." -} \ No newline at end of file diff --git a/grafana/dashboards/archive/governed-run-lifecycle.json b/grafana/dashboards/archive/governed-run-lifecycle.json deleted file mode 100644 index b973eda..0000000 --- a/grafana/dashboards/archive/governed-run-lifecycle.json +++ /dev/null @@ -1,367 +0,0 @@ -{ - "uid": "governed-run-lifecycle", - "title": "Governed Run Lifecycle (15.1)", - "tags": [ - "governed", - "15.1", - "agentops", - "trace" - ], - "timezone": "browser", - "schemaVersion": 39, - "version": 1, - "refresh": "30s", - "time": { - "from": "now-24h", - "to": "now" - }, - "templating": { - "list": [ - { - "name": "trace_id", - "label": "trace_id", - "type": "textbox", - "query": "", - "current": { - "text": "", - "value": "" - }, - "description": "The W3C trace_id that keys the whole run across ClickHouse spans, Loki events and the Postgres audit chain. Paste the demo run's trace_id." - }, - { - "name": "run_id", - "label": "run_id", - "type": "textbox", - "query": "", - "current": { - "text": "", - "value": "" - }, - "description": "The controller run_id (== AGENTPACK_CONTRACT_ID). Filters the same events by contract_id when trace_id is not to hand." - } - ] - }, - "panels": [ - { - "id": 1, - "type": "text", - "title": "About", - "gridPos": { - "h": 7, - "w": 24, - "x": 0, - "y": 0 - }, - "options": { - "mode": "markdown", - "content": "**Governed run lifecycle**, keyed on a single `trace_id` / `run_id`. Every governed hop (`agentops emit`) carries the controller-minted `trace_id`+`contract_id`, so filtering `{job=\"agentops\"}` by them replays the whole chain from one store. The SAME `trace_id` keys the ClickHouse `spans_v1` and the same `run_id` keys the Postgres `audit_events` chain (see RUNBOOK.md \u00a7\"Read a trace across the three stores\"). Set the `trace_id` variable (top-left) to the demo run's id.\n\n---\n\n**Empty is not broken (verified 2026-08-27).** Every query here targets `job=\"agentops\"`, written by `agent-pack/bin/agentops-ship.py` (`dispatch | escalation | verdict | adhoc | dispatch_start`). That vocabulary is correct and the shipper is installed \u2014 but no governed agent-pack run has executed inside Loki's 7-day retention (the only `agentops` line in the window is a `test_governed_fleet_probe` from 2026-08-26). So these panels read *No data* because the fleet has been idle on this path, **not** because the queries are wrong. Run a governed card and they populate. If they are still empty after a run, the shipper is not reaching Loki \u2014 check `agentops-ship.py --loki`." - } - }, - { - "id": 2, - "type": "stat", - "title": "Governed events (this trace)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 4, - "w": 6, - "x": 0, - "y": 7 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "red", - "value": null - }, - { - "color": "green", - "value": 1 - } - ] - } - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "none", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(count_over_time({job=\"agentops\"} |= \"$trace_id\" [$__range]))", - "queryType": "instant" - } - ] - }, - { - "id": 3, - "type": "stat", - "title": "PMO gate decisions", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 4, - "w": 6, - "x": 6, - "y": 7 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "color": { - "mode": "fixed", - "fixedColor": "blue" - } - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "none", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(count_over_time({job=\"agentops\", event=~\"pmo_gate|sdd_dispatch\"} |= \"$trace_id\" [$__range]))", - "queryType": "instant" - } - ] - }, - { - "id": 4, - "type": "stat", - "title": "Wiggum events relayed", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 4, - "w": 6, - "x": 12, - "y": 7 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "color": { - "mode": "fixed", - "fixedColor": "purple" - } - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "none", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(count_over_time({job=\"agentops\", event=\"wiggum_event\"} |= \"$trace_id\" [$__range]))", - "queryType": "instant" - } - ] - }, - { - "id": 5, - "type": "stat", - "title": "Final verdict", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 4, - "w": 6, - "x": 18, - "y": 7 - }, - "fieldConfig": { - "defaults": { - "unit": "short", - "color": { - "mode": "fixed", - "fixedColor": "green" - } - }, - "overrides": [] - }, - "options": { - "colorMode": "value", - "graphMode": "none", - "reduceOptions": { - "calcs": [ - "lastNotNull" - ] - }, - "textMode": "value" - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum(count_over_time({job=\"agentops\", event=~\"sdd_verdict|sdd_card_verdict\"} |= \"$trace_id\" [$__range]))", - "queryType": "instant" - } - ] - }, - { - "id": 6, - "type": "timeseries", - "title": "Lifecycle events over time (by event type)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 24, - "x": 0, - "y": 11 - }, - "fieldConfig": { - "defaults": { - "custom": { - "drawStyle": "bars", - "lineWidth": 1, - "fillOpacity": 60, - "stacking": { - "mode": "normal" - } - } - }, - "overrides": [] - }, - "options": { - "legend": { - "displayMode": "table", - "placement": "right", - "calcs": [ - "sum" - ] - } - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "sum by (event) (count_over_time({job=\"agentops\"} |= \"$trace_id\" [1m]))", - "queryType": "range", - "legendFormat": "{{event}}" - } - ] - }, - { - "id": 7, - "type": "logs", - "title": "Governed chain \u2014 raw events for this trace (card -> packet -> PMO gate -> spec -> Wiggum -> verdict)", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 12, - "w": 24, - "x": 0, - "y": 19 - }, - "options": { - "showTime": true, - "sortOrder": "Ascending", - "wrapLogMessage": true, - "prettifyLogMessage": false, - "dedupStrategy": "none" - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "{job=\"agentops\"} |= \"$trace_id\"", - "queryType": "range" - } - ] - }, - { - "id": 8, - "type": "logs", - "title": "By run_id (contract_id) \u2014 same chain, alternate key", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "h": 8, - "w": 24, - "x": 0, - "y": 31 - }, - "options": { - "showTime": true, - "sortOrder": "Ascending", - "wrapLogMessage": true - }, - "targets": [ - { - "refId": "A", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "expr": "{job=\"agentops\"} |= \"$run_id\"", - "queryType": "range" - } - ] - } - ] -} \ No newline at end of file diff --git a/grafana/dashboards/archive/models-inventory-infinity.json b/grafana/dashboards/archive/models-inventory-infinity.json deleted file mode 100644 index e47c88b..0000000 --- a/grafana/dashboards/archive/models-inventory-infinity.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "uid": "litellm-models-archive", - "title": "Models — LiteLLM Inventory (Infinity archived)", - "schemaVersion": 39, - "version": 1, - "editable": false, - "panels": [ - {"type": "text", "id": 1, "gridPos": {"h": 6, "w": 24, "x": 0, "y": 0}, - "options": {"mode": "markdown", "content": "This dashboard previously depended on the yesoreyeram-infinity-datasource plugin to call /v1/models directly. The plugin is removed to resolve a Grafana 11 SystemJS error. Use the active 'Models — LiteLLM Inventory' dashboard which reads Prometheus litellm_model_info instead."}} - ] -} diff --git a/grafana/provisioning/dashboards/dashboards.yml b/grafana/provisioning/dashboards/dashboards.yml index f58602a..fb450ef 100644 --- a/grafana/provisioning/dashboards/dashboards.yml +++ b/grafana/provisioning/dashboards/dashboards.yml @@ -22,14 +22,3 @@ providers: options: path: /var/lib/grafana/dashboards/ai-agents foldersFromFilesStructure: false - - - name: archive - orgId: 1 - folder: "Archive" - type: file - disableDeletion: false - updateIntervalSeconds: 30 - allowUiUpdates: false - options: - path: /var/lib/grafana/dashboards/archive - foldersFromFilesStructure: false diff --git a/otel-collector/config.yaml b/otel-collector/config.yaml index 128ec0f..4db3905 100644 --- a/otel-collector/config.yaml +++ b/otel-collector/config.yaml @@ -186,7 +186,66 @@ processors: - 'set(attributes["privacy.content_dropped"], "mira") where IsMatch(attributes["llm.input_messages.0.message.content"], "Runtime: agent=mira\\b")' - 'set(attributes["privacy.content_dropped"], "mira") where IsMatch(attributes["input.value"], "Runtime: agent=mira\\b")' - delete_matching_keys(attributes, "^(input\\.value|output\\.value|llm\\.(input|output)_messages\\..*)$") where attributes["privacy.content_dropped"] == "mira" - # Digital twin, Phoenix only: scanners/crawlers are ~99% of twin requests (GETs -> + # agentic-netops intent tier (the agentic-netops repo, kind cluster): four agent + # workloads (service.name intent-supervisor / intent-mapper / intent-allocator / + # intent-deployer), each emitting OTLP once to its tier-owned collector + # (research.md Decision 8 "one emission, one fan-out point"), which forwards + # here. Native shape (verified live 2026-10-11): ioa-observe-sdk is a rebranded + # Traceloop/OpenInference SDK, so spans are OpenInference-NATIVE — + # * ChatLiteLLM.chat spans (supervisor only; mapper/allocator/deployer are + # deterministic) carry gen_ai.prompt.N.content / gen_ai.completion.N.content, + # gen_ai.usage.{prompt,completion}_tokens, llm.usage.total_tokens, + # llm.model_name (often the literal "unknown" when the model is configured + # via the oauth2/ shim — the resolved model sits in + # ioa_observe.association.properties.ls_model_name), session.id; + # * decorator spans are named ".agent" and carry agent.name natively; + # * graph runs: "LangGraph.workflow" spans with traceloop.span.kind=workflow, + # plus traceloop.span.kind=task node spans; + # * A2A round-trips (supervisor -> worker) propagate W3C context, so a + # request's full trace (client + remote server + remote agent spans) + # lands in ONE trace, filed under the project of its first span. + # NOTE: agents/common/llm.py stamps agentic-netops.{model.id,tokens.*,prompt. + # redacted,response.redacted} on the span current at call time, but the SDK's + # context scoping means those attributes have not reached any exported span to + # date (T333-T335 stamping is inert for the same reason) — the rules below keep + # mapping them, so if/when the SDK context is fixed the data flows without + # collector changes. Mapping applied here: LLM kind on model spans (Phoenix's + # kind column), the "unknown" model fixed from ls_model_name (intentional + # overwrite — the one exception, like transform/phoenix_litellm's kind rule), + # AGENT kind on ".agent" spans and on workflow spans. Copy, never + # overwrite (`== nil`) except where noted. Phoenix pipeline only — + # Tempo/ClickHouse keep the native attributes unchanged. + transform/phoenix_agentic_netops: + error_mode: ignore + trace_statements: + - context: span + conditions: + - IsMatch(resource.attributes["service.name"], "^intent-(supervisor|mapper|allocator|deployer)$") + statements: + # LLM steps: the OpenInference langchain instrumentation does not set + # openinference.span.kind, so Phoenix's kind column reads CHAIN. Real + # spans carry NO llm.model_name key at all (only gen_ai.request.model + # = "unknown" + llm.usage.total_tokens), so match on any of the three. + - set(attributes["openinference.span.kind"], "LLM") where attributes["openinference.span.kind"] == nil and (attributes["llm.model_name"] != nil or attributes["gen_ai.request.model"] != nil or attributes["gen_ai.usage.prompt_tokens"] != nil) + # Fix the literal "unknown" model (oauth2/ shim hides the real name): + # the resolved model is a LangChain association property. + - set(attributes["llm.model_name"], attributes["ioa_observe.association.properties.ls_model_name"]) where (attributes["llm.model_name"] == nil or attributes["llm.model_name"] == "unknown") and attributes["ioa_observe.association.properties.ls_model_name"] != nil + - set(attributes["gen_ai.request.model"], attributes["ioa_observe.association.properties.ls_model_name"]) where attributes["gen_ai.request.model"] == "unknown" and attributes["ioa_observe.association.properties.ls_model_name"] != nil + - set(attributes["gen_ai.response.model"], attributes["ioa_observe.association.properties.ls_model_name"]) where attributes["gen_ai.response.model"] == "unknown" and attributes["ioa_observe.association.properties.ls_model_name"] != nil + # Agent decorator spans and graph runs render as agent steps. + - set(attributes["openinference.span.kind"], "AGENT") where attributes["openinference.span.kind"] == nil and IsMatch(name, "\\.agent$") + - set(attributes["agent.name"], attributes["agent_id"]) where attributes["agent.name"] == nil and attributes["agent_id"] != nil + - set(attributes["openinference.span.kind"], "AGENT") where attributes["openinference.span.kind"] == nil and attributes["traceloop.span.kind"] == "workflow" + # Future-proofing: llm.py span stamping (inert today, see note above). + - set(attributes["llm.model_name"], attributes["agentic-netops.model.id"]) where attributes["llm.model_name"] == nil and attributes["agentic-netops.model.id"] != nil + - set(attributes["llm.token_count.prompt"], attributes["agentic-netops.tokens.prompt"]) where attributes["llm.token_count.prompt"] == nil and attributes["agentic-netops.tokens.prompt"] != nil + - set(attributes["llm.token_count.completion"], attributes["agentic-netops.tokens.completion"]) where attributes["llm.token_count.completion"] == nil and attributes["agentic-netops.tokens.completion"] != nil + - set(attributes["llm.token_count.total"], attributes["agentic-netops.tokens.total"]) where attributes["llm.token_count.total"] == nil and attributes["agentic-netops.tokens.total"] != nil + - set(attributes["input.value"], attributes["agentic-netops.prompt.redacted"]) where attributes["input.value"] == nil and attributes["agentic-netops.prompt.redacted"] != nil + - set(attributes["output.value"], attributes["agentic-netops.response.redacted"]) where attributes["output.value"] == nil and attributes["agentic-netops.response.redacted"] != nil + - set(attributes["input.mime_type"], "text/plain") where attributes["input.mime_type"] == nil and attributes["agentic-netops.prompt.redacted"] != nil + - set(attributes["output.mime_type"], "text/plain") where attributes["output.mime_type"] == nil and attributes["agentic-netops.response.redacted"] != nil +# Digital twin, Phoenix only: scanners/crawlers are ~99% of twin requests (GETs -> # method_not_allowed, foreign POSTs -> forbidden_origin) and would bury real conversations. # Drop those WHOLE traces here (the outcome sits on the root span only, so a per-span filter # can't). Tempo + ClickHouse keep them, so the digital-twin-traces dashboard still counts them. @@ -287,7 +346,7 @@ service: # Separate pipeline = its own processors (project mapping) and its own exporter queue. traces/phoenix: receivers: [otlp] - processors: [memory_limiter, resource/phoenix_project, transform/phoenix_claude, transform/phoenix_openclaw, transform/phoenix_litellm, transform/phoenix_kind, batch] + processors: [memory_limiter, resource/phoenix_project, transform/phoenix_claude, transform/phoenix_openclaw, transform/phoenix_litellm, transform/phoenix_agentic_netops, transform/phoenix_kind, batch] exporters: [otlphttp/phoenix] # Digital twin via the tunnel receiver: same backends as every other producer. The Worker sets # openinference.span.kind itself, so the Phoenix transforms (all `== nil` guarded) keep them.