From 109a55163e7cefdee7a1185a8533fa9abb81639e Mon Sep 17 00:00:00 2001 From: Aswin Prakash Thiyagarajan Date: Thu, 17 Sep 2026 16:00:38 -0400 Subject: [PATCH 1/2] Vendor the nine shipped Opik skills from opik-mcp; retire agent-ops The plugin carried two hand-copied skills (`opik`, `agent-ops`) that fell behind their source. opik-mcp is now the single source of truth for the Opik skill pack (OPIK-7471) and ships nine skills, so the plugin vendors those instead, pinned. - scripts/sync-shared-skills.sh: pinned to opik-mcp@0baa5ae (main after #191), SHARED = the nine published skills, evals/ excluded like the pack. - .github/workflows/skills-drift.yml: re-vendors at the pin on PRs and main and fails on drift. - skills/: agent-ops removed (folded into opik/references upstream); opik refreshed; opik-compare, -diagnose, -evaluate, -explain, -instrument, -online-eval, -optimize, -test added. skills/SHARED.md explains the rule. - commands/instrument.md loads `opik` + `opik-instrument` (agent-ops is gone). - README / CONTRIBUTING: skills section and tree; plugin.json 0.3.1 -> 0.4.0. Supersedes #30. opik-verify follows once opik-mcp#193 merges (bump CANON_REF). Co-Authored-By: Claude Fable 5.1 --- .claude-plugin/plugin.json | 4 +- .github/workflows/skills-drift.yml | 26 + CONTRIBUTING.md | 5 +- README.md | 33 +- commands/instrument.md | 2 +- scripts/sync-shared-skills.sh | 36 ++ skills/SHARED.md | 14 + skills/agent-ops/SKILL.md | 106 ---- skills/opik-compare/SKILL.md | 172 ++++++ skills/opik-diagnose/SKILL.md | 188 ++++++ skills/opik-evaluate/SKILL.md | 149 +++++ .../references/error-analysis.md | 160 +++++ skills/opik-evaluate/references/eval-audit.md | 163 +++++ .../opik-evaluate/references/evaluate-rag.md | 110 ++++ .../references/generate-synthetic-data.md | 129 ++++ .../references/validate-evaluator.md | 194 ++++++ .../references/write-judge-prompt.md | 155 +++++ skills/opik-explain/SKILL.md | 122 ++++ skills/opik-instrument/SKILL.md | 135 +++++ skills/opik-online-eval/SKILL.md | 136 +++++ skills/opik-optimize/SKILL.md | 142 +++++ skills/opik-test/SKILL.md | 131 ++++ skills/opik/SKILL.md | 228 +++++-- .../references/agent-patterns.md | 5 + skills/opik/references/best-practices.md | 145 +++++ .../references/evaluation-datasets.md} | 238 ++++---- .../opik/references/evaluation-test-suites.md | 572 ++++++++++++++++++ skills/opik/references/integrations.md | 32 +- skills/opik/references/observability.md | 393 ++---------- .../references/production.md | 37 +- skills/opik/references/tracing-python.md | 192 ++++-- skills/opik/references/tracing-typescript.md | 119 +++- 32 files changed, 3568 insertions(+), 705 deletions(-) create mode 100644 .github/workflows/skills-drift.yml create mode 100755 scripts/sync-shared-skills.sh create mode 100644 skills/SHARED.md delete mode 100644 skills/agent-ops/SKILL.md create mode 100644 skills/opik-compare/SKILL.md create mode 100644 skills/opik-diagnose/SKILL.md create mode 100644 skills/opik-evaluate/SKILL.md create mode 100644 skills/opik-evaluate/references/error-analysis.md create mode 100644 skills/opik-evaluate/references/eval-audit.md create mode 100644 skills/opik-evaluate/references/evaluate-rag.md create mode 100644 skills/opik-evaluate/references/generate-synthetic-data.md create mode 100644 skills/opik-evaluate/references/validate-evaluator.md create mode 100644 skills/opik-evaluate/references/write-judge-prompt.md create mode 100644 skills/opik-explain/SKILL.md create mode 100644 skills/opik-instrument/SKILL.md create mode 100644 skills/opik-online-eval/SKILL.md create mode 100644 skills/opik-optimize/SKILL.md create mode 100644 skills/opik-test/SKILL.md rename skills/{agent-ops => opik}/references/agent-patterns.md (99%) create mode 100644 skills/opik/references/best-practices.md rename skills/{agent-ops/references/evaluation.md => opik/references/evaluation-datasets.md} (61%) create mode 100644 skills/opik/references/evaluation-test-suites.md rename skills/{agent-ops => opik}/references/production.md (90%) diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 95df712..f9a60a9 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "opik", - "version": "0.3.1", - "description": "LLM observability tooling for agent development and Claude Code", + "version": "0.4.0", + "description": "Opik tracing for Claude Code sessions, plus the Opik agent skills: instrument, diagnose, explain, test, compare, evaluate, online-eval, optimize", "author": { "name": "Comet ML", "url": "https://comet.com" diff --git a/.github/workflows/skills-drift.yml b/.github/workflows/skills-drift.yml new file mode 100644 index 0000000..cd51ef6 --- /dev/null +++ b/.github/workflows/skills-drift.yml @@ -0,0 +1,26 @@ +name: skills-drift + +# Fails if the vendored skills under skills/ have drifted from their source of truth +# (comet-ml/opik-mcp at the CANON_REF pinned in scripts/sync-shared-skills.sh). +# Fix: edit upstream, merge, bump CANON_REF, re-run the script, commit. + +on: + pull_request: + paths: ["skills/**", "scripts/sync-shared-skills.sh", ".github/workflows/skills-drift.yml"] + push: + branches: [main] + +jobs: + drift: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Re-vendor at the pinned ref and check for drift + run: | + bash scripts/sync-shared-skills.sh + if ! git diff --quiet -- skills || [ -n "$(git status --porcelain -- skills)" ]; then + echo "::error::Vendored skills differ from comet-ml/opik-mcp at the pinned CANON_REF." + git --no-pager status --porcelain -- skills + exit 1 + fi + echo "no drift" diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d08c485..baeafe8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -56,8 +56,7 @@ opik-claude-code-plugin/ │ └── opik-logger-* # Compiled binaries (darwin/linux, amd64/arm64) ├── src/ │ └── *.go # Go source for the tracing logger -├── skills/ -│ └── agent-ops/ # LLM observability skill + references +├── skills/ # vendored from comet-ml/opik-mcp — do not hand-edit (skills/SHARED.md) ├── agents/ │ └── agent-reviewer.md # Agent code review agent ├── commands/ @@ -69,7 +68,7 @@ opik-claude-code-plugin/ ## Contributing Skills -Skills live in `skills/` as directories containing a `SKILL.md` file and optional `references/` directory. +Skills live in `skills/` as directories containing a `SKILL.md` file and optional `references/` directory. **They are vendored from [`comet-ml/opik-mcp`](https://github.com/comet-ml/opik-mcp)** — propose skill changes there, then bump `CANON_REF` in `scripts/sync-shared-skills.sh` and re-run it here. ``` skills/ diff --git a/README.md b/README.md index 5a23531..483a699 100644 --- a/README.md +++ b/README.md @@ -175,17 +175,23 @@ The command ensures tracing starts at your entry point (critical for replay capa ## Skills -### `/opik:agent-ops` - LLM Observability Knowledge - -Comprehensive guidance on: -- Opik setup and configuration -- Tracing with Python/TypeScript SDKs -- 80+ framework integrations (LangChain, CrewAI, OpenAI, Anthropic, etc.) -- Evaluation with 41 built-in metrics -- Production monitoring, guardrails, and debugging - -## Agents - +### Opik skills (vendored from `comet-ml/opik-mcp`) + +The plugin ships the Opik skill pack — the same skills published as [`opik-skills`](https://github.com/comet-ml/opik-skills). Claude loads them by task: + +| Skill | Use for | +| --- | --- | +| `opik` | SDK reference — span types, integrations, threads, the prompt library | +| `opik-instrument` | "add opik tracing", "instrument my code" — and verify a real trace lands | +| `opik-diagnose` | "what is broken in production", "which traces need attention" | +| `opik-explain` | "why did this trace fail", "debug this trace" | +| `opik-test` | "turn this trace into a test", "make sure this doesn't happen again" | +| `opik-compare` | "did my fix work", "which cases regressed" | +| `opik-evaluate` | "evaluate my agent", "build an eval", "write an LLM judge" | +| `opik-online-eval` | "score production traces", "take this judge live" | +| `opik-optimize` | "optimize this prompt", "run the prompt optimizer" | + +They are vendored at a pinned `opik-mcp` commit (see `skills/SHARED.md`); edit them upstream, not here. ### `agent-reviewer` Reviews agent code for: @@ -210,8 +216,9 @@ opik-claude-code-plugin/ │ └── opik-logger-* # Compiled binaries (darwin/linux, amd64/arm64) ├── src/ │ └── *.go # Go source code -├── skills/ -│ └── agent-ops/ # Observability skill + references +├── skills/ # Opik skill pack, vendored from comet-ml/opik-mcp (see skills/SHARED.md) +│ ├── opik/ +│ ├── opik-instrument/ … opik-optimize/ ├── agents/ │ └── agent-reviewer.md # Agent review agent ├── commands/ diff --git a/commands/instrument.md b/commands/instrument.md index d760ba3..1f15370 100644 --- a/commands/instrument.md +++ b/commands/instrument.md @@ -23,7 +23,7 @@ Add tracing to the user's code so their LLM application is observable in Opik. Use the Skill tool to load BOTH of these skills before doing anything else: 1. **`opik`** — Opik SDK reference: all integrations, tracing patterns, span types, code snippets -2. **`agent-ops`** — Agent architecture patterns, evaluation, what to trace and why +2. **`opik-instrument`** — the end-to-end routine: detect the framework, add the minimum tracing, run a safe path, verify a real trace lands Load them both now. Do not proceed until both are loaded. diff --git a/scripts/sync-shared-skills.sh b/scripts/sync-shared-skills.sh new file mode 100755 index 0000000..9de2ef8 --- /dev/null +++ b/scripts/sync-shared-skills.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# Vendor the shared Opik skills from their source of truth (OPIK-7471). +# +# The skills under skills/ are OWNED by comet-ml/opik-mcp (src/opik_mcp/skills) and +# published from there as the opik-skills pack. Do not hand-edit them here — edit the +# source, get it merged, then bump CANON_REF and re-run this script. The drift check +# in .github/workflows/skills-drift.yml fails a pull request whose vendored copy +# differs from the pinned source. +# +# Usage: bash scripts/sync-shared-skills.sh # at the pinned ref +# CANON_REF= bash scripts/sync-shared-skills.sh +set -euo pipefail + +CANON_REPO="${CANON_REPO:-https://github.com/comet-ml/opik-mcp.git}" +CANON_REF="${CANON_REF:-0baa5aecf0224701473efd968be4aa5a166ba99f}" # opik-mcp main after #191 (nine skills) +SRC="src/opik_mcp/skills" +DEST="skills" +# Every skill the pack ships. Order matches the published index. +SHARED=(opik opik-compare opik-diagnose opik-evaluate opik-explain opik-instrument opik-online-eval opik-optimize opik-test) + +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT +git clone --quiet "$CANON_REPO" "$tmp" +git -C "$tmp" checkout --quiet "$CANON_REF" + +for s in "${SHARED[@]}"; do + if [ ! -d "$tmp/$SRC/$s" ]; then + echo "skip '$s' — not in opik-mcp@$CANON_REF" >&2 + continue + fi + rm -rf "${DEST:?}/$s" + # Same exclusions as the published pack: evals/ is development tooling. + rsync -a --exclude evals --exclude __pycache__ "$tmp/$SRC/$s/" "$DEST/$s/" + echo "synced $s" +done +echo "Done. Vendored ${#SHARED[@]} skills from opik-mcp@$CANON_REF" diff --git a/skills/SHARED.md b/skills/SHARED.md new file mode 100644 index 0000000..125d8d6 --- /dev/null +++ b/skills/SHARED.md @@ -0,0 +1,14 @@ +# These skills are vendored — do not hand-edit them + +Everything under `skills/` is **vendored from the source of truth**, +[`comet-ml/opik-mcp`](https://github.com/comet-ml/opik-mcp) (`src/opik_mcp/skills`, +OPIK-7471) — the same files published as the `opik-skills` pack. Edit them there, get +the change merged, then bump `CANON_REF` in `scripts/sync-shared-skills.sh` and re-run: + +```bash +bash scripts/sync-shared-skills.sh +``` + +`.github/workflows/skills-drift.yml` fails a pull request whose vendored copy differs +from the pinned source. This repo still owns everything else: `commands/`, `agents/`, +`hooks/`, and the session logger. diff --git a/skills/agent-ops/SKILL.md b/skills/agent-ops/SKILL.md deleted file mode 100644 index a88cde6..0000000 --- a/skills/agent-ops/SKILL.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -name: agent-ops -description: This skill should be used when the user asks about agent architecture, evaluation, metrics, production monitoring, debugging agents, or best practices for building reliable AI agents. Use for questions like "evaluate my agent", "set up production monitoring", "add guardrails", "detect hallucinations", "agent anti-patterns", "compare experiments", "create evaluation dataset". ---- - -# Agent Operations: Build, Evaluate, and Monitor AI Agents - -This skill covers the agent lifecycle beyond basic tracing: architecture patterns, evaluation, metrics, and production monitoring. All examples use Opik for observability — for SDK details (tracing, integrations, span types), load the `opik` skill. - -## The Agent Lifecycle - -1. **Instrument** — Add Opik tracing to make your agent's behavior visible (see `opik` skill) -2. **Evaluate** — Measure performance with datasets, metrics, and experiments -3. **Monitor** — Track quality, cost, and reliability in production -4. **Optimize** — Improve based on data from evaluation and production traces - -## Agent Architecture Patterns - -Trace every component of your agent with appropriate span types: - -```python -import opik - -@opik.track(name="research_agent") -def agent(query: str) -> str: - plan = plan_action(query) # general span - results = execute_tool(plan) # tool span - return generate_response(results) # llm span - -@opik.track(type="tool") -def execute_tool(action: dict) -> str: - return search_web(action["query"]) - -@opik.track(type="llm") -def generate_response(context: str) -> str: - return llm_call(context) -``` - -### What to Trace - -| Component | Span Type | Key Data | -|-----------|-----------|----------| -| Planning | `general` | Reasoning steps, decisions | -| Tool calls | `tool` | Tool name, parameters, results | -| LLM calls | `llm` | Prompt, response, tokens | -| Retrieval | `tool` | Query, documents | -| Validation | `guardrail` | Check results, pass/fail | - -## Evaluation - -Evaluate agents at multiple levels — end-to-end and per-component: - -```python -from opik.evaluation import evaluate -from opik.evaluation.metrics import AnswerRelevance, Hallucination, AgentTaskCompletion - -results = evaluate( - experiment_name="agent-v2", - dataset=dataset, - task=lambda item: {"output": agent(item["input"])}, - scoring_metrics=[ - AnswerRelevance(), - Hallucination(), - AgentTaskCompletion(), - ] -) -``` - -### Built-in Agent Metrics - -| Metric | What It Measures | -|--------|-----------------| -| `AgentTaskCompletion` | Did the agent fulfill its task? | -| `AgentToolCorrectness` | Were tools used correctly? | -| `TrajectoryAccuracy` | Did actions match expected sequence? | -| `AnswerRelevance` | Does the answer address the question? | -| `Hallucination` | Are there unsupported claims? | - -### 41 Total Built-in Metrics - -Heuristic (Equals, Contains, BLEU, ROUGE, BERTScore, IsJson, etc.), LLM-as-Judge (AnswerRelevance, Hallucination, Usefulness, GEval, etc.), RAG (ContextPrecision, ContextRecall, Faithfulness), and conversation metrics. See `references/evaluation.md` for the full list. - -## Production Monitoring - -- **Dashboards** — Visualize quality, cost, latency, and error trends -- **Online evaluation** — Automatically score production traces with LLM-as-Judge -- **Alerts** — Get notified when metrics deviate (quality drops, cost spikes, error rates) -- **Guardrails** — PII detection, topic validation, custom safety checks -- **Opik Assist** — AI-powered root cause analysis for failed traces - -## Common Anti-Patterns - -| Category | Anti-Pattern | -|----------|-------------| -| Reliability | Unbounded loops, retry storms, silent failures | -| Security | Prompt injection, privilege escalation, data leakage | -| Observability | Late tracing (missing input), orphaned spans | -| Tools | Tool loops, hallucinated tools, parameter errors | - -## Detailed References - -| Topic | Reference File | -|-------|----------------| -| Agent architecture, reliability, security patterns | `references/agent-patterns.md` | -| Evaluation datasets, experiments, all 41 metrics | `references/evaluation.md` | -| Production dashboards, alerts, guardrails, cost tracking | `references/production.md` | diff --git a/skills/opik-compare/SKILL.md b/skills/opik-compare/SKILL.md new file mode 100644 index 0000000..4f6c7fa --- /dev/null +++ b/skills/opik-compare/SKILL.md @@ -0,0 +1,172 @@ +--- +name: opik-compare +description: Run a candidate against the baseline over an Opik test suite and read the numbers back — which cases broke, which got fixed, the per-metric deltas, worst rows, and whether the two runs are comparable — with the Opik compare-view link. Runs via the SDK; reads results via the MCP when connected. Does not issue a ship/no-ship verdict. Use for "did my fix work", "compare against the baseline", "run the regression suite", "why did quality drop", "which cases regressed", "compare these two experiments". Not for live production triage (use diagnose), building an evaluation from scratch (use evaluate), or capturing a single case (use test). +compatibility: Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires a Python or TypeScript project with Opik configured and a test suite (from the test or evaluate skill) or two existing experiments. Install the `opik` skill alongside this one — it holds the shared test-suite and experiment references; without it, this skill falls back to the public docs. +allowed-tools: + - Read + - Grep + - Glob + - Bash + - Write +metadata: + last_updated: "2026-09-15" + source_commit: "2.0.0" + argument-hint: "[suite name, or two experiment ids/names]" +--- + +# Compare — Candidate vs Baseline Over a Suite + +**Definition of done:** two experiments on the same suite — a **baseline** and a **candidate** — read back item by item, with the **per-metric deltas**, the cases that went **pass → fail** (regressions) and **fail → pass** (fixes), the worst rows, a note on whether the runs are comparable, and the **Opik compare-view link**. If only one run exists, the done state is "baseline created — rerun after the change". If the suite can't be run or read, stop at the **first** genuine blocker and return **exactly one** next step. Aggregate scores alone are not a comparison; a verdict is not this skill's job. + +Operate: **run the candidate the same way the baseline was run, read both back from Opik rather than from the run's console output, name the specific cases that changed — and change no application code.** The only file this skill writes is a throwaway runner outside the repo. + +## Inputs + +The entry point is `/opik-compare ` (run the suite now as the candidate, compare against the latest prior run), `/opik-compare ` (read two existing runs, no new run), or `/opik-compare` right after `/opik-test` (that suite). Infer the rest; treat these as **optional overrides**: + +- suite (default: `-regressions`, or the one `/opik-test` just wrote to) · baseline (default: the most recent earlier experiment on the suite) · candidate name (default: `candidate-`) · judge model for assertions (default: the suite's / the baseline's) · runs per item (default: the suite's policy). + +Ask only at a genuine, non-inferable blocker (see **Blockers**). + +## Activation — the only in-scope work + +### 1. Resolve the suite and the baseline +```python +import opik +client = opik.Opik() + +suite = client.get_test_suite(name="", project_name="") +prior = client.get_test_suite_experiments(name="", project_name="") # newest first is not guaranteed — sort by created_at yourself +``` +Baseline = the most recent prior experiment on this suite, unless the user names one. **No prior experiment → this run *is* the baseline** (step 3 still runs; status `baseline_created`). Two explicit experiments → skip step 3, go to step 4. + +Confirm Opik is reachable: if `~/.opik.config` exists or `OPIK_API_KEY` is set, use it. Otherwise → **Blocker**. + +### 2. Build the task adapter (code candidates) +The suite's items say what to call: each item `description` written by `/opik-test` ends in `Entrypoint: `. Grep the repo for that function, import it, and wrap it: + +```python +def task(item: dict) -> dict: + return {"input": item["input"], "output": str(entrypoint(item["input"]))} +``` +Write the runner as a **temp file outside the repo** (or a scratch path the user names) — never into the codebase, never committed. It needs the app's provider credentials; if they're absent → **Blocker** (the app can't answer the items). Never point the runner at a production entrypoint that writes, sends, or spends. + +**Prompt candidates** (the change is a prompt version, not code): there is no adapter — run server-side instead: +```python +client.rest_client.experiments.execute_experiment( + dataset_name=suite.name, dataset_id=suite.id, + prompts=[{"model": "", "messages": [...], "configs": {}, "prompt_versions": [{"id": ""}]}], + project_name="", +) # 202 Accepted; one experiment per prompt variant, processed asynchronously — poll get_experiment_by_id until items fill +``` + +### 3. Run the candidate +Same suite, same version, same judge model, same runs-per-item as the baseline — vary **only** the thing under test. +```python +result = opik.run_tests( + test_suite=suite, # or suite.get_version_view("") to pin + task=task, + experiment_name="candidate-", + experiment_tags=["compare", ""], + model="", +) +candidate_id = result.experiment_id # result.experiment_url is the single-run link + +# GUARD: a missing judge credential does NOT raise — every item comes back failed with +# scoring_failed=True and a "Missing credentials" reason, and the experiment is still created. +# Treat that as a Blocker, not as a regression; do not compare against that run. +judge_failed = [ + r for ir in result.item_results.values() for t in ir.test_results + for r in t.score_results if getattr(r, "scoring_failed", False) +] +``` +Do not read scores off `result` and stop — step 4 reads both runs from Opik so baseline and candidate go through the same path. + +### 4. Read both runs back (SDK-first, MCP when connected) +```python +base = client.get_experiment_by_id("") +cand = client.get_experiment_by_id(candidate_id) +b_items = {i.dataset_item_id: i for i in base.get_items()} +c_items = {i.dataset_item_id: i for i in cand.get_items()} +# each item: dataset_item_data, evaluation_task_output, feedback_scores [{name, value, reason}], +# assertion_results [{value, passed, reason}], trace_id + +# Side by side, one call, both experiments: +rows = client.get_experiments_client().find_experiment_items_for_dataset( + suite.name, experiment_ids=["", candidate_id], project_name="" +) +``` +`get_experiment_by_name` is deprecated — resolve names with `get_experiments_by_name` and pick by id. When the hosted MCP is connected, `list('experiment', name=…)` already renders each experiment's per-metric averages and `read('experiment', id)` gives one run's summary — use them to find and sanity-check the two runs; the item-level join above stays on the SDK. + +### 5. Answer the questions, in this order +1. **Comparable?** Same suite version (`dataset_version` / item count), same judge model, same runs-per-item. If not, say so first — the deltas below are then indicative, not measured. +2. **Which cases broke** — `passed` in baseline, not in candidate. Each with its input (one line), the failing assertion, its `reason`, and the candidate trace link. +3. **Which cases got fixed** — the reverse. +4. **Net effect per metric** — mean of each `feedback_scores` name in both runs, and the pass rate; report `baseline → candidate (delta)`. +5. **Worst rows** — lowest candidate scores; are they the same rows as the baseline's worst? +6. **App or judge?** A flip whose `reason` cites the output is the app; a flip on an unchanged output with a wavering `reason` is the judge — flag it, and suggest `runs_per_item: 3` for that item rather than trusting one run. +7. **Cost vs quality** — `total_estimated_cost` / duration on the candidate traces vs baseline, when the traces carry them. +8. **Flaky cases** — items that flip across repeated runs of the *same* code (the suite's execution policy exposes `runs_passed`/`runs_total`). +9. **Trend** — when more than two runs exist, the pass rate across the last few, so a one-step delta has context. + +Report what the numbers say. **Do not decide ship or hold** — that decision has its own policy and is a later step. + +### 6. Link and hand off +Build the compare link with **both** ids — a URL-encoded JSON array — so the user lands on the side-by-side view: +```python +import json, urllib.parse +ids = urllib.parse.quote(json.dumps(["", candidate_id])) +compare_url = f"{ui_base}/{workspace}/experiments/{suite.id}/compare?experiments={ids}" +``` +`ui_base` is the Opik UI origin (the configured URL minus `/api`; `result.experiment_url` shows the exact host and workspace to reuse). Then one next step (see **Output**). + +### 7. Record (only when asked) +If the user wants the finding kept beside the data: a comment on a regressed case's trace via `client.rest_client.traces.add_trace_comment(trace_id, text=…)`, or a human score beside the judge's via `client.log_traces_feedback_scores([...])`. Never by default. + +## Blockers + +Stop at the **earliest** blocker and return **exactly one** next step: +- "Run `opik configure`, then rerun `/opik-compare `." +- "No test suite named `` in project `` — run `/opik-test ` to create one, or name the suite." +- "The suite's items don't say which function to call — pass the entrypoint (`/opik-compare --entrypoint answer`) and I'll build the runner." +- "The runner needs a provider credential — set `OPENAI_API_KEY` (or the relevant key) and rerun." +- "The assertion judge has no credential (every item reports `scoring_failed`, 'Missing credentials') — set the judge's provider key and rerun; the run `` is not a comparable candidate." +- "Experiment `` has no items yet (server-side run still processing) — rerun in a minute." + +## Output + +**User-facing:** a short human message — comparability note (if any), regressions first (case, assertion, why), then fixes, then the per-metric table (`baseline → candidate (delta)`), the compare link, and the single next step. No verdict. Not a raw dump of every item. + +**Underneath** (for composition / evals), one shape: +- `status`: `compared` | `baseline_created` | `blocked` +- `suite`: `name`, `id`, `version` +- `baseline` / `candidate`: `experiment_id`, `name`, `url`, `items`, `pass_rate` +- `comparable`: `true|false`, `note` +- `deltas`: list of `{metric, baseline, candidate, delta}` (pass rate included) +- `regressions`: list of `{dataset_item_id, input, assertion, reason, trace_url}` +- `fixes`: same shape +- `attribution`: `{app: [...ids], judge: [...ids]}` when a flip could be attributed +- `compare_url` +- `next_step`: exactly one +- `verdict`: **never present** + +Invariants: `compared` carries both experiments, a non-empty `deltas`, and a `compare_url` with both ids; `baseline_created` carries `baseline` only and a `next_step` of "rerun after the change"; `blocked` carries exactly one `next_step`; no path writes into the codebase; no path emits a ship/hold decision. + +## Examples + +**Fix verified.** `/opik-compare support-bot-regressions` after fixing `retrieve()`. Baseline = yesterday's run (3/5 passed). Runner built from `Entrypoint: answer`; candidate 5/5. Read back: 2 fixes (the refund and shipping items), 0 regressions, `pass_rate 0.60 → 1.00 (+0.40)`, same suite version, same judge. → **`compared`**; next step = "run `/opik-online-eval` to watch the refund assertion in production" or "merge — the decision is yours". + +**Regression surfaced.** Candidate fixes the refund case but the "does not give legal advice" item flips to fail; its `reason` quotes the new output offering legal steps. Attribution: app. → **`compared`** with one regression named first, the delta table after; next step = "`/opik-explain ` for the legal-advice item". + +**First run.** No prior experiments on the suite. Run once; report 3/5 with the failing items. → **`baseline_created`**: "This is the baseline — rerun `/opik-compare` after your change." + +**Two existing runs, no code.** `/opik-compare nightly-2026-09-14 nightly-2026-09-15`. Resolve by `get_experiments_by_name`, join items, report. → **`compared`** (no runner built). + +## Anti-patterns +Reading scores from the run's console output instead of from Opik; comparing runs on different suite versions or judge models without saying so; reporting only aggregates ("accuracy 0.82 → 0.85") without naming the cases that flipped; writing the runner into the repo or committing it; changing app code to make the suite pass; trusting a single run on a flaky item; **issuing a ship/no-ship verdict** (out of scope by design); using deprecated `get_experiment_by_name`; building the compare link with one id. + +## References + +Test-suite, experiment, and metric detail live in the `opik` skill, installed beside this one. Read the files directly — paths are relative to this file: `../opik/references/evaluation-test-suites.md` (`run_tests`, `TestSuiteResult`, execution policies, versions, `get_test_suite_experiments`), `../opik/references/evaluation-datasets.md` (`evaluate()`, experiments, OQL), `../opik/references/production.md` (cost and latency fields on traces). If your host lays skills out differently, locate the `opik` skill's `references/` directory. + +If the `opik` skill isn't installed, say so in the report and use rather than working from memory. diff --git a/skills/opik-diagnose/SKILL.md b/skills/opik-diagnose/SKILL.md new file mode 100644 index 0000000..6adf61f --- /dev/null +++ b/skills/opik-diagnose/SKILL.md @@ -0,0 +1,188 @@ +--- +name: opik-diagnose +description: Surface the Opik traces worth a developer's attention, ranked by signal — Diagnostics issues first, then errors, failed tool calls, latency, regressions, and low online-eval scores. With the Opik MCP connected it lists the project's agent_insights_issue entities, offers to turn Diagnostics on when the project has it switched off, then fills the gaps with list (filters, sort, a time window); without the MCP it reads the same via the SDK (agent_insights and search_traces), so it works with no MCP. Returns a ranked shortlist, each item ready to hand to the explain skill. Use for "what is broken in production", "which traces need attention", "find failing or slow traces", "which tool calls are failing", "triage my agent". Not for offline experiment results (use evaluate or compare) and not for root-causing one trace (use explain). +compatibility: Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires a Python or TypeScript project with Opik configured and a project that has traces. Install the `opik` skill alongside this one — it holds the shared SDK and observability references; without it, this skill falls back to the public docs. +allowed-tools: + - Read + - Grep + - Glob + - Bash +metadata: + last_updated: "2026-09-09" + source_commit: "2.0.0" + argument-hint: "[optional: project name, or what to look for]" +--- + +# Diagnose — Surface the Traces Worth Attention + +**Definition of done:** a **ranked shortlist** of the online/production traces (and Diagnostics issues) worth attention, each carrying the **signal that flagged it** and its **trace id**, scoped to a project and a recent window, and ready to hand to `/opik-explain`. "Worth attention" means errored, slow, regressed, or low online-eval score — not a dump of every trace, and never offline experiment results. If the project can't be read, stop at the first genuine blocker and return one next step. + +Operate: **rank by real signal over live data, surface the few things worth a look, hand the top one to `/opik-explain` — and change no code.** This skill is read-only by design. + +## Inputs + +The entry point is `/opik-diagnose` (the current project), `/opik-diagnose `, or `/opik-diagnose ` (e.g. "slow traces", "errors today"). Infer the rest; treat these as **optional overrides**: + +- project (default: inferred from config/repo) · window (default: recent) · signal focus (default: all — errors, latency, regressions, low scores) · shortlist size (default: a handful). + +Ask only at a genuine, non-inferable blocker (see **Blockers**). + +## Activation — the only in-scope work + +### 1. Resolve scope +Project (from config/repo) + a recent window. Confirm Opik is reachable: if `~/.opik.config` exists or `OPIK_API_KEY` is set, use it. Otherwise → **Blocker** ("run `opik configure`, then rerun"). + +### 2. Read the project once, for the shape of the week + +**With the Opik MCP connected**, `read('project', )` costs one call +and answers three things the rest of this skill would otherwise guess: whether +anything is wrong at all (error rate and latency against the previous 7 days — +a flat week is worth saying so and stopping), which score names the project +actually records (the ones worth filtering on later; guessing a name returns an +empty result that reads like good news), and what the project contains. + +A rate reported as `null` means no traces in the window, not a healthy zero. +If the window is quiet, widen it with `since="30d"` before concluding anything. + +### 3. Start from Diagnostics issues +Opik's Diagnostics already groups a project's recurring failures into ranked issues, each with a severity, occurrence counts, a cause, a suggested fix and example traces. Read that list first — it is the answer to "what is broken" the UI already computed, so do not rebuild it from raw traces. + +**With the Opik MCP connected**, the `agent_insights_issue` entity is the primary path: + +```text +list('agent_insights_issue', project_name='') # open issues, ranked as the Diagnostics page ranks them +read('agent_insights_issue', '', project_name='') +# → {issue: {name, cause, suggested_fix, severity, status, …}, example_trace_ids: [...], details: [...], +# url: '', trace_url_template: '<…/logs?trace={trace_id}>'} +``` + +Each open issue becomes one shortlist item with `signal=diagnostics`: `trace_id` comes from the first `example_trace_ids` entry (read the top few issues to get them) and `trace_url` from `trace_url_template` with that id filled in; mention the issue's `url` so the user can open the Diagnostics page itself. `why` carries the issue name, severity, `latest_count` and the cause. The link fields are absent when the server cannot name the UI base or workspace — fall back to the trace link template from the session instructions then (step 6). Keep the list's order — it is the Diagnostics page's ranking (most recently seen first, then most occurrences). Counts are all-time to match the UI; pass `since` (e.g. `"7d"`) to narrow to the window. + +**An empty list is not an all-clear.** It says which case it is, and what to do about it. Act on the sentence you get: + +| The list says | Do this | +| --- | --- | +| no open issues, last scan `