diff --git a/ADR.md b/ADR.md
index 055a1ce..8349f71 100644
--- a/ADR.md
+++ b/ADR.md
@@ -145,3 +145,23 @@ depends on two things that could change:
- **Baseline headroom.** At 90% top-1 there is little left to win. A larger,
harder or real incident corpus — especially real runbooks written by the
team that wrote the code — would be the test that could change this decision.
+
+## ADR-8: Stack traces in commit ranking — on whenever the alert has one
+
+**Context.** The alert was one line of error text; Sentinel never used the
+stack trace a real error tracker attaches. A trace says where an error
+surfaced, which is often not where it was introduced, so it could just as
+easily mislead the ranker as help it.
+
+**Decision.** An alert may carry an optional `traceback`. Ranking gets the
+newest 4000 characters of it, and each commit that touches a file named in a
+frame is marked (frames carry deploy paths, so files match on a path-segment
+suffix). The prompt says frames show where the error surfaced, not
+necessarily its cause. Without a trace the prompt is byte-identical to the
+baseline. No flag: having a trace is what switches it on.
+
+**Measured result** (METRICS.md, 2026-10-11): 95.8% top-1 against baselines
+of 90.3% and 91.7%. The gain comes from one case that went from 0/9 to 3/3;
+cases whose trace named only decoy files improved rather than regressed.
+These traces were written to go through decoys wherever the raising call
+was ambiguous, so they are harder than typical traces.
diff --git a/METRICS.md b/METRICS.md
index 5e85b33..d4e1eee 100644
--- a/METRICS.md
+++ b/METRICS.md
@@ -227,3 +227,60 @@ false alarms** (in-sample it was 9/13 and 9/131), so it holds. On Haiku 4.5 it
flags 7/14 wrong but 22/58 right picks, and Haiku's raw confidence is identical
for right and wrong picks (0.93 / 0.93): the rule is calibrated for Sonnet 5
and should be re-checked if the model changes.
+
+## Stack traces in commit ranking (2026-10-11)
+
+**Question:** does sending the alert's stack trace to the ranker, and marking
+commits that touch a file named in a frame, improve top-1?
+
+**Answer: yes, by one robust case flip; everything else ties.** Traces are
+used whenever an alert carries one (no flag).
+
+- **Inputs:** eval set v2 unchanged; 20/24 cases gained a trace
+ (`TRACEBACK_SET_VERSION = "v1"`), rendered against the post-commit fixture
+ repo so app frames carry real line numbers. The 4 metric alerts (timeouts,
+ OOM kill, idle queue) carry none, and their prompts are identical to
+ baseline.
+- **Traces were built against the treatment.** Where the raising call is
+ ambiguous (a leak surfaces at whichever `getconn` or `open` runs next), the
+ frames follow a path *not* through the culprit's lines. The culprit's file
+ appears in 13/20 traces; the other 7 name only decoy files or no app files.
+- **Control:** the two existing Sonnet 5 baselines. The baseline prompt is
+ byte-identical to the one they used (golden-prompt test), so no new
+ baseline run was needed. Same model, `claude-sonnet-5`, 3 trials per case.
+
+Reproduce: `python -m eval.report eval/results/baseline_v2_sonnet5_rerun_2026-10-07.json eval/results/traceback_v2_sonnet5_2026-10-11.json`
+
+| Metric | Baseline 10-04 | Baseline 10-07 | Traceback 10-11 |
+|---|---|---|---|
+| Top-1 | 90.3% (65/72) | 91.7% (66/72) | **95.8% (69/72)** |
+| Top-3 | 100% | 100% | 100% |
+| MRR | 0.947 | 0.958 | 0.975 |
+| LLM failures | 0/72 | 0/72 | 0/72 |
+| Ranking latency, median | 5.8s | 6.5s | 6.3s |
+| Input tokens per call, mean | n/a | 3066 | 3352 (+9%) |
+
+**Where the gain comes from.** `httpx_028_drops_proxies` went from **0/3 in
+every earlier Sonnet run (0/9 across baseline, RAG and re-run) to 3/3**. Its
+trace names only the decoy (`payments.py`, where `httpx.Client(proxies=...)`
+raises at import), yet seeing the call fail at construction pointed the model
+at the dependency bump. The other paired differences are single-trial swings
+already seen between the two baselines (`since_filter_naive_cutoff`,
+`page_size_zero_division`, `export_tempfile_never_closed`).
+
+| Group (trials) | Baseline 10-04 | Baseline 10-07 | Traceback |
+|---|---|---|---|
+| Culprit file in trace (39) | 38/39 | 38/39 | 39/39 |
+| Trace misses culprit file (21) | 15/21 | 16/21 | 18/21 |
+| No trace (12) | 12/12 | 12/12 | 12/12 |
+
+A trace that pointed at a decoy did not pull the ranking towards it: that
+group improved too. `export_tempfile_never_closed`, whose trace points at the
+audit-log decoy, stayed at 0/3, where baseline had already put it (0/3 and
+1/3).
+
+**Limitation:** a 4–5 point gain on 24 cases rests on one case. Read it as
+"traces help and did not hurt here", not as a precise effect size.
+
+**Truncation check:** none of the 216 baseline/RAG/re-run trials hit the
+3000-character diff limit, so truncation did not cause any of the misses.
diff --git a/README.md b/README.md
index 6054570..f0f4e45 100644
--- a/README.md
+++ b/README.md
@@ -85,6 +85,7 @@ Every hop reads and writes one auditable incident row conforming to
- [METRICS.md](METRICS.md) — measured end-to-end timings, methodology, and per-run results
- Measured diagnostic accuracy: faulty commit ranked #1 in 90.3% of 72 trials across 24 injected-fault cases with decoy commits ([METRICS.md, Phase 9](METRICS.md#phase-9--diagnostic-accuracy-and-runbook-context-in-ranking))
+- With the alert's stack trace in the ranking prompt: 95.8% of 72 trials, against 90.3% and 91.7% without, the gain coming from one case that went from 0/9 to 3/3 ([METRICS.md](METRICS.md#stack-traces-in-commit-ranking-2026-10-11))
- [ADR.md](ADR.md) — why explicit state loops over LangChain, polling over WebSockets, Chroma over Pinecone, FastAPI over Flask, tool-call JSON over free-form parsing, sandbox over live infra
## Sandbox reference
diff --git a/contract/pipeline_schema.json b/contract/pipeline_schema.json
index 074c2b6..cc7f361 100644
--- a/contract/pipeline_schema.json
+++ b/contract/pipeline_schema.json
@@ -13,7 +13,8 @@
"source": { "type": "string" },
"alert_name": { "type": "string" },
"timestamp": { "type": "string", "format": "date-time" },
- "error_signature": { "type": "string" }
+ "error_signature": { "type": "string" },
+ "traceback": { "type": "string", "description": "Optional stack trace; fed to commit ranking when present" }
}
},
"diagnostics": {
diff --git a/core/services/git_client.py b/core/services/git_client.py
index 83d850c..de55d74 100644
--- a/core/services/git_client.py
+++ b/core/services/git_client.py
@@ -1,3 +1,4 @@
+import re
import subprocess
from datetime import datetime, timedelta
@@ -41,6 +42,8 @@ def get_recent_diffs(repo_path: str, alert_ts: str, window_minutes: int = 60) ->
"author": author,
"timestamp": ts,
"subject": subject,
+ # from the full patch, before truncation, so traceback overlap sees every file
+ "files": re.findall(r"^diff --git a/.* b/(.*)$", diff.stdout, flags=re.M),
"diff": diff.stdout[:3000], # ponytail: truncated for LLM context; raise if diffs are large
}
)
diff --git a/core/services/llm_analyzer.py b/core/services/llm_analyzer.py
index edc8fdb..c9acfa9 100644
--- a/core/services/llm_analyzer.py
+++ b/core/services/llm_analyzer.py
@@ -1,4 +1,5 @@
import os
+import re
import anthropic
@@ -27,6 +28,28 @@ def _runbooks_block(runbooks: list[dict] | None) -> str:
)
+# ponytail: keep the newest frames only; recursion traces run to thousands of lines
+_TRACEBACK_MAX_CHARS = 4000
+
+
+def _traceback_block(traceback: str | None) -> str:
+ """'' without a traceback, so an alert that carries none ranks with the baseline prompt."""
+ if not traceback:
+ return ""
+ return (
+ "STACK TRACE (from the alert; frames show where the error surfaced, which is not necessarily "
+ "where it was introduced):\n"
+ f"\n{traceback[-_TRACEBACK_MAX_CHARS:]}\n\n\n"
+ )
+
+
+def files_in_traceback(commit_files: list[str], traceback: str | None) -> list[str]:
+ """Commit files that appear as a frame in the traceback. Frames carry deploy paths
+ (/srv/app/x.py), commits carry repo paths (x.py), so match on a path-segment suffix."""
+ frames = {p.replace("\\", "/") for p in re.findall(r'File "([^"]+)", line \d+', traceback or "")}
+ return [f for f in commit_files if any(p == f or p.endswith("/" + f) for p in frames)]
+
+
class LLMUnavailable(Exception):
"""LLM output could not be produced: timeout/network, malformed JSON, or refusal."""
@@ -61,9 +84,16 @@ def rank_suspect_commits(diffs: list[dict], alert: dict, runbooks: list[dict] |
if not diffs:
return []
+ traceback = alert.get("traceback")
+
+ def _overlap_line(c: dict) -> str:
+ hits = files_in_traceback(c.get("files", []), traceback)
+ return f"Touches files in the stack trace: {', '.join(hits)}\n" if hits else ""
+
commits_block = "\n\n".join(
f"COMMIT {c['commit_hash']} | {c['author']} | {c['timestamp']}\n"
f"Subject: {c['subject']}\n"
+ f"{_overlap_line(c)}"
f"Diff:\n{c['diff']}"
for c in diffs
)
@@ -74,6 +104,7 @@ def rank_suspect_commits(diffs: list[dict], alert: dict, runbooks: list[dict] |
f" name: {alert['alert_name']}\n"
f" error: {alert['error_signature']}\n"
f" time: {alert['timestamp']}\n\n"
+ f"{_traceback_block(traceback)}"
f"{_runbooks_block(runbooks)}"
f"RECENT COMMITS:\n{commits_block}\n\n"
f"Use rank_commits to return every commit with a confidence_score (0=unrelated, 1=certain cause) "
diff --git a/core/tests/test_traceback_ranking.py b/core/tests/test_traceback_ranking.py
new file mode 100644
index 0000000..0c468b5
--- /dev/null
+++ b/core/tests/test_traceback_ranking.py
@@ -0,0 +1,68 @@
+# ponytail: traceback context must not move the baseline prompt; overlap must match path segments only
+from types import SimpleNamespace as NS
+from unittest.mock import patch
+
+from core.services import llm_analyzer
+from core.tests.test_rag_ranking import _ALERT, _DIFFS, _GOLDEN_BASELINE_PROMPT
+
+_TB = (
+ "Traceback (most recent call last):\n"
+ ' File "/srv/orders/app/db.py", line 24, in fetch_all\n'
+ " conn = pool.getconn()\n"
+ "psycopg2.pool.PoolError: connection pool exhausted"
+)
+_FILED = [{**_DIFFS[0], "files": ["config.json", "app/db.py"]}, {**_DIFFS[1], "files": ["README.md"]}]
+
+
+def _prompt(diffs, alert):
+ captured = []
+
+ def fake_chat(messages, **_):
+ captured.append(messages[0]["content"])
+ return NS(
+ stop_reason="tool_use",
+ content=[NS(type="tool_use", name="rank_commits", input={"ranked_commits": []})],
+ )
+
+ with patch.object(llm_analyzer, "chat", fake_chat):
+ llm_analyzer.rank_suspect_commits(diffs, alert)
+ return captured[0]
+
+
+def test_no_traceback_prompt_is_byte_identical_to_baseline():
+ assert _prompt(_FILED, _ALERT) == _GOLDEN_BASELINE_PROMPT
+ assert _prompt(_FILED, {**_ALERT, "traceback": ""}) == _GOLDEN_BASELINE_PROMPT
+
+
+def test_traceback_sits_after_alert_and_marks_only_overlapping_commits():
+ prompt = _prompt(_FILED, {**_ALERT, "traceback": _TB})
+ assert prompt.index("ALERT:") < prompt.index("") < prompt.index("RECENT COMMITS:")
+ assert prompt.count("Touches files in the stack trace:") == 1
+ assert "Subject: config: raise pool size\nTouches files in the stack trace: app/db.py\nDiff:" in prompt
+
+
+def test_overlap_matches_whole_path_segments():
+ tb = 'File "/srv/orders/app/db.py", line 1\nFile "C:\\svc\\app\\cart.py", line 2'
+ files = ["app/db.py", "db.py", "pp/db.py", "app/cart.py", "app/orders.py"]
+ assert llm_analyzer.files_in_traceback(files, tb) == ["app/db.py", "db.py", "app/cart.py"]
+ assert llm_analyzer.files_in_traceback(files, None) == []
+
+
+def test_long_traceback_keeps_newest_frames():
+ tb = "x" * 10_000 + "\nTypeError: newest"
+ block = llm_analyzer._traceback_block(tb)
+ assert "TypeError: newest" in block and len(block) < llm_analyzer._TRACEBACK_MAX_CHARS + 200
+
+
+def test_sandbox_app_traceback_matches_its_repo_path():
+ from fastapi.testclient import TestClient
+
+ from sandbox.app import main as sandbox_app
+
+ with patch.object(sandbox_app, "_bug", return_value="db_failure"):
+ r = TestClient(sandbox_app.app, raise_server_exceptions=False).get("/api/data")
+ assert r.status_code == 500 and r.json()["error"].startswith("RuntimeError: OperationalError")
+ tb = r.json()["traceback"]
+ assert llm_analyzer.files_in_traceback(["sandbox/app/main.py", "core/db.py"], tb) == [
+ "sandbox/app/main.py"
+ ]
diff --git a/dashboard/src/app/incidents/[id]/page.tsx b/dashboard/src/app/incidents/[id]/page.tsx
index 9513367..c45407d 100644
--- a/dashboard/src/app/incidents/[id]/page.tsx
+++ b/dashboard/src/app/incidents/[id]/page.tsx
@@ -48,6 +48,12 @@ export default function IncidentDetail() {
Trigger
{trigger.alert_name} — {trigger.source} — {new Date(trigger.timestamp).toLocaleString()}
{trigger.error_signature}
+ {trigger.traceback && (
+
+ Stack trace
+ {trigger.traceback}
+
+ )}
{diagnostics && (
diff --git a/dashboard/src/lib/api.ts b/dashboard/src/lib/api.ts
index e0caa53..2004182 100644
--- a/dashboard/src/lib/api.ts
+++ b/dashboard/src/lib/api.ts
@@ -26,6 +26,7 @@ export type Trigger = {
alert_name: string;
timestamp: string;
error_signature: string;
+ traceback?: string;
};
// shape returned by core's db rows (see core/db.py)
diff --git a/eval/cases/auth_issuer_mesh_hosts.json b/eval/cases/auth_issuer_mesh_hosts.json
index 691d55d..0afde5d 100644
--- a/eval/cases/auth_issuer_mesh_hosts.json
+++ b/eval/cases/auth_issuer_mesh_hosts.json
@@ -85,5 +85,43 @@
"culprit_index": 4,
"expected_runbook_id": "auth_failures",
"note": "Tokens still carry iss=https://auth.internal; the mesh rename changed the expected issuer, not just a transport URL. Leeway affects exp/iat checks and the algorithm list only widens what is accepted."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/auth.py",
+ "current_user_id",
+ "return verify_token("
+ ],
+ [
+ "app/auth.py",
+ "verify_token",
+ "return jwt.decode("
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/jwt/api_jwt.py",
+ 210,
+ "decode",
+ "decoded = self.decode_complete("
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/jwt/api_jwt.py",
+ 151,
+ "decode_complete",
+ "self._validate_claims("
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/jwt/api_jwt.py",
+ 244,
+ "_validate_claims",
+ "self._validate_iss(payload, issuer)"
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/jwt/api_jwt.py",
+ 330,
+ "_validate_iss",
+ "raise InvalidIssuerError(\"Invalid issuer\")"
+ ]
+ ]
}
}
diff --git a/eval/cases/cache_contains_recursion.json b/eval/cases/cache_contains_recursion.json
index ab6912c..4dc731e 100644
--- a/eval/cases/cache_contains_recursion.json
+++ b/eval/cases/cache_contains_recursion.json
@@ -85,5 +85,59 @@
"culprit_index": 2,
"expected_runbook_id": null,
"note": "__contains__ now calls get() and get() starts with 'key not in self', so each calls the other. The earlier __contains__ read _data directly and was fine."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/handlers/orders.py",
+ "create_order",
+ "stock = inventory.get_stock_many("
+ ],
+ [
+ "app/clients/inventory.py",
+ "get_stock_many",
+ "get_stock(sku"
+ ],
+ [
+ "app/clients/inventory.py",
+ "get_stock",
+ "cached = _stock_cache.get("
+ ],
+ [
+ "app/cache.py",
+ "get",
+ "if key not in self"
+ ],
+ [
+ "app/cache.py",
+ "__contains__",
+ "return self.get(key) is not None"
+ ],
+ [
+ "app/cache.py",
+ "get",
+ "if key not in self"
+ ],
+ [
+ "app/cache.py",
+ "__contains__",
+ "return self.get(key) is not None"
+ ],
+ [
+ "app/cache.py",
+ "get",
+ "if key not in self"
+ ],
+ [
+ "app/cache.py",
+ "__contains__",
+ "return self.get(key) is not None"
+ ],
+ [
+ "app/cache.py",
+ "get",
+ "if key not in self"
+ ]
+ ]
}
}
diff --git a/eval/cases/cart_price_field_v2.json b/eval/cases/cart_price_field_v2.json
index 2de684d..1e61a5c 100644
--- a/eval/cases/cart_price_field_v2.json
+++ b/eval/cases/cart_price_field_v2.json
@@ -81,5 +81,29 @@
"culprit_index": 2,
"expected_runbook_id": "null_field_handling",
"note": "Carts saved before the v2 payload only carry unit_price_cents, so .get('price_cents') yields None and qty * None raises inside _line_total. The helper extraction is behaviour-preserving."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/cart.py",
+ "checkout",
+ "return create_order("
+ ],
+ [
+ "app/handlers/orders.py",
+ "create_order",
+ "total = sum(_line_total(line)"
+ ],
+ [
+ "app/handlers/orders.py",
+ "",
+ "total = sum(_line_total(line)"
+ ],
+ [
+ "app/handlers/orders.py",
+ "_line_total",
+ "return line[\"qty\"] * line[\"unit_price_cents\"]"
+ ]
+ ]
}
}
diff --git a/eval/cases/cart_sku_lowercased.json b/eval/cases/cart_sku_lowercased.json
index 7117ee9..3a08442 100644
--- a/eval/cases/cart_sku_lowercased.json
+++ b/eval/cases/cart_sku_lowercased.json
@@ -68,5 +68,35 @@
"culprit_index": 2,
"expected_runbook_id": null,
"note": "SKUs are upper-case (SKU-10293); the cart now lower-cases them before the stock lookup. quote() leaves 'SKU-10293' unchanged."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/cart.py",
+ "checkout",
+ "return create_order("
+ ],
+ [
+ "app/handlers/orders.py",
+ "create_order",
+ "stock = inventory.get_stock_many("
+ ],
+ [
+ "app/clients/inventory.py",
+ "get_stock_many",
+ "get_stock(sku"
+ ],
+ [
+ "app/clients/inventory.py",
+ "get_stock",
+ "resp.raise_for_status()"
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/httpx/_models.py",
+ 763,
+ "raise_for_status",
+ "raise HTTPStatusError(message, request=request, response=self)"
+ ]
+ ]
}
}
diff --git a/eval/cases/config_reformat_hidden_string.json b/eval/cases/config_reformat_hidden_string.json
index 2e67eea..ce0a03e 100644
--- a/eval/cases/config_reformat_hidden_string.json
+++ b/eval/cases/config_reformat_hidden_string.json
@@ -72,5 +72,29 @@
"culprit_index": 1,
"expected_runbook_id": "config_parse_errors",
"note": "The one-key-per-line reformat quietly turned cache.max_entries into the string \"5000\"; TTLCache.set compares len(...) >= max_entries. The TTL change keeps an int."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/handlers/orders.py",
+ "create_order",
+ "stock = inventory.get_stock_many("
+ ],
+ [
+ "app/clients/inventory.py",
+ "get_stock_many",
+ "get_stock(sku"
+ ],
+ [
+ "app/clients/inventory.py",
+ "get_stock",
+ "_stock_cache.set("
+ ],
+ [
+ "app/cache.py",
+ "set",
+ "if len(self._data) >= self.max_entries"
+ ]
+ ]
}
}
diff --git a/eval/cases/db_fetch_one_leak.json b/eval/cases/db_fetch_one_leak.json
index 60e2cf9..5b30a41 100644
--- a/eval/cases/db_fetch_one_leak.json
+++ b/eval/cases/db_fetch_one_leak.json
@@ -77,5 +77,31 @@
"culprit_index": 1,
"expected_runbook_id": "db_failures",
"note": "fetch_one returns before putconn on every miss (and on any exception), leaking one connection per lookup that finds nothing; a pool of 6 only lowers the ceiling, it cannot be exhausted without a leak."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/handlers/orders.py",
+ "list_orders",
+ "rows = db.fetch_all(_LIST_SQL"
+ ],
+ [
+ "app/db.py",
+ "fetch_all",
+ "conn = pool.getconn()"
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/psycopg2/pool.py",
+ 169,
+ "getconn",
+ "return self._getconn(key)"
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/psycopg2/pool.py",
+ 93,
+ "_getconn",
+ "raise PoolError(\"connection pool exhausted\")"
+ ]
+ ]
}
}
diff --git a/eval/cases/export_tempfile_never_closed.json b/eval/cases/export_tempfile_never_closed.json
index 6668fbd..77ba39d 100644
--- a/eval/cases/export_tempfile_never_closed.json
+++ b/eval/cases/export_tempfile_never_closed.json
@@ -86,5 +86,20 @@
"culprit_index": 1,
"expected_runbook_id": "file_handle_exhaustion",
"note": "The temp-file rewrite never closes the NamedTemporaryFile, leaking a descriptor per export. The audit log opens a file per order but closes it via the with block."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/handlers/orders.py",
+ "create_order",
+ "audit.record(\"order.created\""
+ ],
+ [
+ "app/audit.py",
+ "record",
+ "with open(AUDIT_LOG"
+ ]
+ ],
+ "error": "OSError: [Errno 24] Too many open files: '/var/log/orders/audit.jsonl'"
}
}
diff --git a/eval/cases/guest_email_normalise.json b/eval/cases/guest_email_normalise.json
index 7c32011..416fff6 100644
--- a/eval/cases/guest_email_normalise.json
+++ b/eval/cases/guest_email_normalise.json
@@ -81,5 +81,19 @@
"culprit_index": 4,
"expected_runbook_id": "null_field_handling",
"note": "Guest orders have customer_email NULL; .strip() on None raises. The email_domain commit guards None."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/handlers/orders.py",
+ "get_order",
+ "serializers.order_to_dict(row)"
+ ],
+ [
+ "app/serializers.py",
+ "order_to_dict",
+ "row[\"customer_email\"].strip()"
+ ]
+ ]
}
}
diff --git a/eval/cases/httpx_028_drops_proxies.json b/eval/cases/httpx_028_drops_proxies.json
index 1f9356d..5c29fce 100644
--- a/eval/cases/httpx_028_drops_proxies.json
+++ b/eval/cases/httpx_028_drops_proxies.json
@@ -82,5 +82,19 @@
"culprit_index": 3,
"expected_runbook_id": "dependency_regressions",
"note": "httpx 0.28 removed the deprecated proxies= argument (replaced by proxy=). The proxy commit was valid against the pinned 0.27.2; the version bump is what broke it."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/handlers/orders.py",
+ "",
+ "from app.clients import inventory, payments"
+ ],
+ [
+ "app/clients/payments.py",
+ "",
+ "_http = httpx.Client("
+ ]
+ ]
}
}
diff --git a/eval/cases/inventory_cache_key_per_request.json b/eval/cases/inventory_cache_key_per_request.json
index 7f689e4..1eaa2f9 100644
--- a/eval/cases/inventory_cache_key_per_request.json
+++ b/eval/cases/inventory_cache_key_per_request.json
@@ -101,5 +101,31 @@
"culprit_index": 4,
"expected_runbook_id": "rate_limiting",
"note": "The cache key now includes a fresh uuid per batch, so the stock cache never hits; a 60s TTL alone can raise lookups at most ~5x over 300s, not 31x."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/handlers/orders.py",
+ "create_order",
+ "stock = inventory.get_stock_many("
+ ],
+ [
+ "app/clients/inventory.py",
+ "get_stock_many",
+ "get_stock(sku"
+ ],
+ [
+ "app/clients/inventory.py",
+ "get_stock",
+ "resp.raise_for_status()"
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/httpx/_models.py",
+ 763,
+ "raise_for_status",
+ "raise HTTPStatusError(message, request=request, response=self)"
+ ]
+ ],
+ "error": "httpx.HTTPStatusError: Client error '429 Too Many Requests' for url 'https://inventory.internal/v2/stock/SKU-30117'"
}
}
diff --git a/eval/cases/inventory_v3_payload.json b/eval/cases/inventory_v3_payload.json
index b893e40..b996b90 100644
--- a/eval/cases/inventory_v3_payload.json
+++ b/eval/cases/inventory_v3_payload.json
@@ -89,5 +89,24 @@
"culprit_index": 2,
"expected_runbook_id": "null_field_handling",
"note": "The response parser still reads ['available'] while the client now calls a different API version; the only change to what the response contains. The other inventory and order commits never touch parsing."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/handlers/orders.py",
+ "create_order",
+ "stock = inventory.get_stock_many("
+ ],
+ [
+ "app/clients/inventory.py",
+ "get_stock_many",
+ "get_stock(sku"
+ ],
+ [
+ "app/clients/inventory.py",
+ "get_stock",
+ "resp.json()[\"available\"]"
+ ]
+ ]
}
}
diff --git a/eval/cases/jwt_leeway_milliseconds.json b/eval/cases/jwt_leeway_milliseconds.json
index cee8aa4..c455a5b 100644
--- a/eval/cases/jwt_leeway_milliseconds.json
+++ b/eval/cases/jwt_leeway_milliseconds.json
@@ -85,5 +85,44 @@
"culprit_index": 3,
"expected_runbook_id": "auth_failures",
"note": "leeway_s is seconds but is now wrapped as milliseconds (45ms), so ~4s of clock skew fails the iat check. Widening leeway_s to 45 would tolerate it; the PyJWT minor bump does not change leeway handling."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/auth.py",
+ "current_user_id",
+ "return verify_token("
+ ],
+ [
+ "app/auth.py",
+ "verify_token",
+ "return jwt.decode("
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/jwt/api_jwt.py",
+ 210,
+ "decode",
+ "decoded = self.decode_complete("
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/jwt/api_jwt.py",
+ 151,
+ "decode_complete",
+ "self._validate_claims("
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/jwt/api_jwt.py",
+ 244,
+ "_validate_claims",
+ "self._validate_iat(payload, now, leeway)"
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/jwt/api_jwt.py",
+ 330,
+ "_validate_iat",
+ "raise ImmatureSignatureError(\"The token is not yet valid (iat)\")"
+ ]
+ ],
+ "error": "jwt.exceptions.ImmatureSignatureError: The token is not yet valid (iat)"
}
}
diff --git a/eval/cases/money_returns_decimal.json b/eval/cases/money_returns_decimal.json
index cb2f9b6..51f5491 100644
--- a/eval/cases/money_returns_decimal.json
+++ b/eval/cases/money_returns_decimal.json
@@ -81,5 +81,39 @@
"culprit_index": 4,
"expected_runbook_id": null,
"note": "money() now returns a Decimal that lands in every order response body. The thousands-separator change still returned a str."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "/usr/local/lib/python3.12/site-packages/starlette/responses.py",
+ 181,
+ "render",
+ "return json.dumps("
+ ],
+ [
+ "/usr/local/lib/python3.12/json/__init__.py",
+ 238,
+ "dumps",
+ "**kw).encode(obj)"
+ ],
+ [
+ "/usr/local/lib/python3.12/json/encoder.py",
+ 200,
+ "encode",
+ "chunks = self.iterencode(o, _one_shot=True)"
+ ],
+ [
+ "/usr/local/lib/python3.12/json/encoder.py",
+ 258,
+ "iterencode",
+ "return _iterencode(o, 0)"
+ ],
+ [
+ "/usr/local/lib/python3.12/json/encoder.py",
+ 180,
+ "default",
+ "raise TypeError(f'Object of type {o.__class__.__name__} '"
+ ]
+ ]
}
}
diff --git a/eval/cases/order_sql_missing_param.json b/eval/cases/order_sql_missing_param.json
index c6ebc06..65b7e25 100644
--- a/eval/cases/order_sql_missing_param.json
+++ b/eval/cases/order_sql_missing_param.json
@@ -81,5 +81,24 @@
"culprit_index": 3,
"expected_runbook_id": null,
"note": "_ORDER_SQL gained a second placeholder but get_order still passes (order_id,). The list query added a placeholder and its parameter together."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/handlers/orders.py",
+ "get_order",
+ "row = db.fetch_one(_ORDER_SQL"
+ ],
+ [
+ "app/db.py",
+ "fetch_one",
+ "rows = fetch_all(sql, params)"
+ ],
+ [
+ "app/db.py",
+ "fetch_all",
+ "cur.execute(sql, params)"
+ ]
+ ]
}
}
diff --git a/eval/cases/page_size_zero_division.json b/eval/cases/page_size_zero_division.json
index 2718c88..1bf7fff 100644
--- a/eval/cases/page_size_zero_division.json
+++ b/eval/cases/page_size_zero_division.json
@@ -77,5 +77,19 @@
"culprit_index": 0,
"expected_runbook_id": "pagination_errors",
"note": "Dropping the max(1, ...) clamp lets page_size=0 through; the later total_pages line is where it divides, but that line is safe for any page_size >= 1."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/handlers/orders.py",
+ "list_orders",
+ "body = paginate("
+ ],
+ [
+ "app/pagination.py",
+ "paginate",
+ "total_pages = -(-len(items) // page_size)"
+ ]
+ ]
}
}
diff --git a/eval/cases/payments_body_without_content_type.json b/eval/cases/payments_body_without_content_type.json
index 7b1d472..c2953cc 100644
--- a/eval/cases/payments_body_without_content_type.json
+++ b/eval/cases/payments_body_without_content_type.json
@@ -89,5 +89,25 @@
"culprit_index": 2,
"expected_runbook_id": null,
"note": "httpx sets Content-Type: application/json only for json=; with content= the body is sent untyped. The idempotency header commit leaves the json= body intact."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/handlers/orders.py",
+ "create_order",
+ "payments.charge(order_id, total, card_token)"
+ ],
+ [
+ "app/clients/payments.py",
+ "charge",
+ "resp.raise_for_status()"
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/httpx/_models.py",
+ 763,
+ "raise_for_status",
+ "raise HTTPStatusError(message, request=request, response=self)"
+ ]
+ ]
}
}
diff --git a/eval/cases/payments_retry_loop.json b/eval/cases/payments_retry_loop.json
index eb66ae6..499f028 100644
--- a/eval/cases/payments_retry_loop.json
+++ b/eval/cases/payments_retry_loop.json
@@ -85,5 +85,26 @@
"culprit_index": 3,
"expected_runbook_id": "rate_limiting",
"note": "payments.retries was not read by any code until the retry loop; the loop retries 429s immediately with no backoff, giving 1+4=5 attempts per order. The config bump was inert on its own."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/handlers/orders.py",
+ "create_order",
+ "payments.charge(order_id, total, card_token)"
+ ],
+ [
+ "app/clients/payments.py",
+ "charge",
+ "resp.raise_for_status()"
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/httpx/_models.py",
+ 763,
+ "raise_for_status",
+ "raise HTTPStatusError(message, request=request, response=self)"
+ ]
+ ],
+ "error": "httpx.HTTPStatusError: Client error '429 Too Many Requests' for url 'https://payments.internal/v1/charges'"
}
}
diff --git a/eval/cases/redis_legacy_lrem_args.json b/eval/cases/redis_legacy_lrem_args.json
index ba5349d..310acb4 100644
--- a/eval/cases/redis_legacy_lrem_args.json
+++ b/eval/cases/redis_legacy_lrem_args.json
@@ -77,5 +77,38 @@
"culprit_index": 0,
"expected_runbook_id": "dependency_regressions",
"note": "redis-py 2.x uses lrem(name, value, num); 3.0+ uses lrem(name, count, value). Under 2.10.6 the job payload is sent as the count. A count of -1 is valid in the current argument order."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/worker.py",
+ "run_forever",
+ "_r.lrem(PROCESSING"
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/redis/client.py",
+ 1418,
+ "lrem",
+ "return self.execute_command('LREM', name, num, value)"
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/redis/client.py",
+ 673,
+ "execute_command",
+ "return self.parse_response(connection, command_name, **options)"
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/redis/client.py",
+ 680,
+ "parse_response",
+ "response = connection.read_response()"
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/redis/connection.py",
+ 629,
+ "read_response",
+ "raise response"
+ ]
+ ]
}
}
diff --git a/eval/cases/settings_env_quotes.json b/eval/cases/settings_env_quotes.json
index 20050b1..20af029 100644
--- a/eval/cases/settings_env_quotes.json
+++ b/eval/cases/settings_env_quotes.json
@@ -77,5 +77,53 @@
"culprit_index": 2,
"expected_runbook_id": "config_parse_errors",
"note": "Deployments set overrides JSON-encoded (README), e.g. ORDERS_DB_HOST='\"10.0.4.12\"'; without json.loads the quotes stay in the host name. Not a database or network problem."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/handlers/orders.py",
+ "get_order",
+ "row = db.fetch_one(_ORDER_SQL"
+ ],
+ [
+ "app/db.py",
+ "fetch_one",
+ "rows = fetch_all("
+ ],
+ [
+ "app/db.py",
+ "fetch_all",
+ "pool = get_pool()"
+ ],
+ [
+ "app/db.py",
+ "get_pool",
+ "_pool = psycopg2.pool.ThreadedConnectionPool("
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/psycopg2/pool.py",
+ 161,
+ "__init__",
+ "AbstractConnectionPool.__init__("
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/psycopg2/pool.py",
+ 59,
+ "__init__",
+ "self._connect()"
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/psycopg2/pool.py",
+ 63,
+ "_connect",
+ "conn = psycopg2.connect(*self._args, **self._kwargs)"
+ ],
+ [
+ "/usr/local/lib/python3.12/site-packages/psycopg2/__init__.py",
+ 122,
+ "connect",
+ "conn = _connect(dsn, connection_factory=connection_factory, **kwasync)"
+ ]
+ ]
}
}
diff --git a/eval/cases/since_filter_naive_cutoff.json b/eval/cases/since_filter_naive_cutoff.json
index 94db78c..fc84aa7 100644
--- a/eval/cases/since_filter_naive_cutoff.json
+++ b/eval/cases/since_filter_naive_cutoff.json
@@ -85,5 +85,14 @@
"culprit_index": 2,
"expected_runbook_id": null,
"note": "The new filter strips tzinfo from the cutoff and compares it with timestamptz rows. The export change only formats a file name and never compares datetimes."
+ },
+ "traceback": {
+ "frames": [
+ [
+ "app/handlers/orders.py",
+ "list_orders",
+ "r[\"created_at\"] >= cutoff"
+ ]
+ ]
}
}
diff --git a/eval/config.py b/eval/config.py
index 9eb896f..5491220 100644
--- a/eval/config.py
+++ b/eval/config.py
@@ -2,6 +2,10 @@
RUNBOOK_CORPUS_VERSION = "v1" # sandbox/runbooks/*.md, 10 docs, frozen at Phase 9B
# v2 replaced v1 after the v1 baseline hit 100% top-1 (ceiling); v1 lives in git history at d6f6bd1.
EVAL_SET_VERSION = "v2" # eval/cases/*.json + eval/bases/, 24 cases, frozen at Phase 9C (rev)
+# Case "traceback" specs, read only by the traceback condition (baseline and rag prompts never see them).
+# Frames follow the real runtime path. Where the raising call is ambiguous (a leak surfaces at any getconn
+# or open), the path NOT through the culprit's lines was chosen, so the culprit file is in 13/20 traces.
+TRACEBACK_SET_VERSION = "v1" # 20 of 24 cases; the 4 metric alerts (timeouts, OOM, idle queue) have none
ALERT_TS = "2026-09-01T12:00:00+00:00" # fixed so fixture commit dates are reproducible
DEFAULT_TRIALS = 3
diff --git a/eval/fixture.py b/eval/fixture.py
index 3a24db5..aa69190 100644
--- a/eval/fixture.py
+++ b/eval/fixture.py
@@ -2,6 +2,7 @@
import json
import os
+import re
import shutil
import stat
import subprocess
@@ -69,6 +70,40 @@ def _chmod_retry(func, p, _exc):
shutil.rmtree(path, onerror=_chmod_retry)
+APP_ROOT = "/srv/orders" # deploy path frames carry; repo paths are relative to it
+
+
+def render_traceback(repo: Path, spec: dict, error: str) -> str:
+ """Python-style traceback whose app frames point at real lines of the post-commit repo.
+
+ Frame forms: [repo_file, func, anchor] (line = first line containing anchor after `def func(`, or the
+ file's only line containing it for /), [abs_path, line, func, code] for library frames,
+ or [text] for a literal line such as "[Previous line repeated 996 more times]".
+ """
+ out = ["Traceback (most recent call last):"]
+ for fr in spec["frames"]:
+ if len(fr) == 1:
+ out.append(f" {fr[0]}")
+ continue
+ if len(fr) == 4:
+ path, line, func, code = fr
+ else:
+ rel, func, anchor = fr
+ lines = (repo / rel).read_text(encoding="utf-8").splitlines()
+ start = next(
+ (i for i, x in enumerate(lines) if re.match(rf"\s*def {re.escape(func)}\(", x)), None
+ )
+ hits = [i for i, x in enumerate(lines) if anchor in x and (start is None or i >= start)]
+ if not hits or (start is None and len(hits) != 1):
+ raise ValueError(
+ f"{rel}:{func}: anchor must match {'once' if start is None else ''}: {anchor!r}"
+ )
+ path, line, code = f"{APP_ROOT}/{rel}", hits[0] + 1, lines[hits[0]].strip()
+ out += [f' File "{path}", line {line}, in {func}', f" {code}"]
+ out.append(spec.get("error", error))
+ return "\n".join(out)
+
+
@contextmanager
def build(case: dict):
"""Yields (repo_path, alert, label). label gains culprit_hash for the scorer."""
@@ -101,6 +136,8 @@ def build(case: dict):
"timestamp": ALERT_TS,
"error_signature": case["alert"]["error_signature"],
}
+ if case.get("traceback"): # only the traceback condition sends it; run_eval strips it otherwise
+ alert["traceback"] = render_traceback(repo, case["traceback"], case["alert"]["error_signature"])
label = {**case["label"], "culprit_hash": hashes[case["label"]["culprit_index"]]}
yield str(repo), alert, label
finally:
diff --git a/eval/report.py b/eval/report.py
index 37021ee..416ed5a 100644
--- a/eval/report.py
+++ b/eval/report.py
@@ -110,6 +110,19 @@ def paired(a: list[dict], b: list[dict]) -> list[tuple]:
return [(cid, *h) for cid, h in sorted(hits.items())]
+def traceback_groups(records: list[dict]) -> dict[str, str]:
+ """case_id -> traceback group, from whichever run recorded it (pre-traceback runs did not)."""
+ groups = {}
+ for r in records:
+ if r.get("has_traceback"):
+ groups[r["case_id"]] = (
+ "culprit file in trace" if r["culprit_in_traceback"] else "trace misses culprit file"
+ )
+ elif "has_traceback" in r and r["case_id"] not in groups:
+ groups[r["case_id"]] = "no trace"
+ return groups
+
+
def _fmt_conf(x):
return f"{x:.2f}" if x is not None else "n/a"
@@ -165,6 +178,14 @@ def print_comparison(a: dict, b: dict):
ties = len(rows) - wins - losses
print(f" {lb} wins {wins}, losses {losses}, ties {ties} (n={len(rows)} cases)")
+ groups = traceback_groups(a["records"] + b["records"])
+ if groups:
+ print("\n== By traceback content (does a trace that names only other files mislead?)")
+ for group in ("culprit file in trace", "trace misses culprit file", "no trace"):
+ for label, res in ((la, a), (lb, b)):
+ s = summarize([r for r in res["records"] if groups.get(r["case_id"]) == group])
+ print(f" {group:<30} {label:<15} top-1 {_pct(s['top1'], s['n'])}, MRR {s['mrr']:.3f}")
+
print("\n== Null-runbook cases only (does irrelevant context hurt?)")
for label, res in ((la, a), (lb, b)):
s = summarize([r for r in res["records"] if not r["expected_runbook_id"]])
diff --git a/eval/results/traceback_v2_sonnet5_2026-10-11.json b/eval/results/traceback_v2_sonnet5_2026-10-11.json
new file mode 100644
index 0000000..3537cd6
--- /dev/null
+++ b/eval/results/traceback_v2_sonnet5_2026-10-11.json
@@ -0,0 +1,3881 @@
+{
+ "condition": "traceback",
+ "dry_run": false,
+ "eval_set_version": "v2",
+ "traceback_set_version": "v1",
+ "runbook_corpus_version": "v1",
+ "runbooks_ingested": 10,
+ "runbook_floor": 0.35,
+ "model": "claude-sonnet-5",
+ "sentinel_git_sha": "6a345e231aa34c9d4899c669f875aad08bf7355c",
+ "started_at": "2026-10-11T19:36:38.073186+00:00",
+ "trials": 3,
+ "n_cases": 24,
+ "records": [
+ {
+ "trial": 0,
+ "case_id": "auth_issuer_mesh_hosts",
+ "category": "auth_rejection",
+ "culprit_hash": "5e68f8a",
+ "expected_runbook_id": "auth_failures",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.317
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.245
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.236
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "5e68f8a",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "45a24d5",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "df0ec4c",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "298be8d",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "d20025a",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 8.58,
+ "usage": {
+ "input_tokens": 3560,
+ "output_tokens": 738
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "cache_contains_recursion",
+ "category": "infinite_recursion",
+ "culprit_hash": "4584056",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.326
+ },
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.31
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.231
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "4584056",
+ "confidence_score": 0.98
+ },
+ {
+ "commit_hash": "6484609",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "7e63ea3",
+ "confidence_score": 0.03
+ },
+ {
+ "commit_hash": "6cc141a",
+ "confidence_score": 0.0
+ },
+ {
+ "commit_hash": "a98de00",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.52,
+ "usage": {
+ "input_tokens": 3446,
+ "output_tokens": 591
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "cart_price_field_v2",
+ "category": "null_or_missing_field",
+ "culprit_hash": "d0ab213",
+ "expected_runbook_id": "null_field_handling",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.26
+ },
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.207
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.17
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "d0ab213",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "73e1933",
+ "confidence_score": 0.3
+ },
+ {
+ "commit_hash": "4fbde4c",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "2bbbf3f",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "f618f59",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.7,
+ "usage": {
+ "input_tokens": 3384,
+ "output_tokens": 727
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "cart_sku_lowercased",
+ "category": "identifier_format",
+ "culprit_hash": "c2ec824",
+ "expected_runbook_id": null,
+ "commits_in_window": 4,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.293
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.247
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.229
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "c2ec824",
+ "confidence_score": 0.9
+ },
+ {
+ "commit_hash": "64dde29",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "d0cb0b3",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "91f8d36",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.97,
+ "usage": {
+ "input_tokens": 2991,
+ "output_tokens": 623
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "checkout_deadline_extra_call",
+ "category": "latency_timeout",
+ "culprit_hash": "efebc07",
+ "expected_runbook_id": "request_timeouts",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": false,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.487
+ },
+ {
+ "id": "db_failures",
+ "similarity_score": 0.373
+ },
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.353
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "efebc07",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "a035477",
+ "confidence_score": 0.55
+ },
+ {
+ "commit_hash": "2b765f4",
+ "confidence_score": 0.3
+ },
+ {
+ "commit_hash": "be85cc9",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "91b8b07",
+ "confidence_score": 0.02
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 7.62,
+ "usage": {
+ "input_tokens": 3255,
+ "output_tokens": 808
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "config_reformat_hidden_string",
+ "category": "config_type_error",
+ "culprit_hash": "c3629ef",
+ "expected_runbook_id": "config_parse_errors",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.246
+ },
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.19
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.172
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "c3629ef",
+ "confidence_score": 0.9
+ },
+ {
+ "commit_hash": "3af442c",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "976c988",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "a6a3c41",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "89436d3",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.56,
+ "usage": {
+ "input_tokens": 3480,
+ "output_tokens": 700
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "db_fetch_one_leak",
+ "category": "db_pool_exhaustion",
+ "culprit_hash": "af9b861",
+ "expected_runbook_id": "db_failures",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "db_failures",
+ "similarity_score": 0.628
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.344
+ },
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.28
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "af9b861",
+ "confidence_score": 0.9
+ },
+ {
+ "commit_hash": "f46cd8c",
+ "confidence_score": 0.5
+ },
+ {
+ "commit_hash": "6d50d20",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "1618815",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "1bf1024",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.22,
+ "usage": {
+ "input_tokens": 3516,
+ "output_tokens": 690
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "export_tempfile_never_closed",
+ "category": "file_handle_leak",
+ "culprit_hash": "6eea0fc",
+ "expected_runbook_id": "file_handle_exhaustion",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.625
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.411
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.291
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "5cedb21",
+ "confidence_score": 0.9
+ },
+ {
+ "commit_hash": "b245524",
+ "confidence_score": 0.25
+ },
+ {
+ "commit_hash": "6eea0fc",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "0be651d",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "7453733",
+ "confidence_score": 0.01
+ }
+ ],
+ "culprit_rank": 3,
+ "error": null,
+ "rank_latency_s": 7.86,
+ "usage": {
+ "input_tokens": 3722,
+ "output_tokens": 813
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "guest_email_normalise",
+ "category": "null_or_missing_field",
+ "culprit_hash": "7dd25fe",
+ "expected_runbook_id": "null_field_handling",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.241
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.208
+ },
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.161
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "7dd25fe",
+ "confidence_score": 0.97
+ },
+ {
+ "commit_hash": "9519064",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "30a7434",
+ "confidence_score": 0.0
+ },
+ {
+ "commit_hash": "1c0135b",
+ "confidence_score": 0.0
+ },
+ {
+ "commit_hash": "cde392a",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.23,
+ "usage": {
+ "input_tokens": 3135,
+ "output_tokens": 563
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "httpx_028_drops_proxies",
+ "category": "dependency_version",
+ "culprit_hash": "b35322f",
+ "expected_runbook_id": "dependency_regressions",
+ "commits_in_window": 4,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.294
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.285
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.244
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "b35322f",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "741f699",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "b45189f",
+ "confidence_score": 0.03
+ },
+ {
+ "commit_hash": "e3601f6",
+ "confidence_score": 0.02
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.28,
+ "usage": {
+ "input_tokens": 2993,
+ "output_tokens": 599
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "inventory_cache_key_per_request",
+ "category": "rate_limit_429",
+ "culprit_hash": "526f8dd",
+ "expected_runbook_id": "rate_limiting",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.382
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.381
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.309
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "526f8dd",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "bf4cc5f",
+ "confidence_score": 0.35
+ },
+ {
+ "commit_hash": "93320f2",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "2e1b06b",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "d9caa65",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.09,
+ "usage": {
+ "input_tokens": 3709,
+ "output_tokens": 678
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "inventory_v3_payload",
+ "category": "null_or_missing_field",
+ "culprit_hash": "1c29d94",
+ "expected_runbook_id": "null_field_handling",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.33
+ },
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.28
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.271
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "1c29d94",
+ "confidence_score": 0.9
+ },
+ {
+ "commit_hash": "16d2191",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "c6f0797",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "5823d1f",
+ "confidence_score": 0.03
+ },
+ {
+ "commit_hash": "fb9b710",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.17,
+ "usage": {
+ "input_tokens": 3556,
+ "output_tokens": 582
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "jwt_leeway_milliseconds",
+ "category": "auth_rejection",
+ "culprit_hash": "2eeb222",
+ "expected_runbook_id": "auth_failures",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.488
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.198
+ },
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.159
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "2eeb222",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "e8548c5",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "2e11931",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "a028c62",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "716e22d",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 7.0,
+ "usage": {
+ "input_tokens": 3526,
+ "output_tokens": 674
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "money_returns_decimal",
+ "category": "json_serialization",
+ "culprit_hash": "b14897b",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.298
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.151
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.13
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "b14897b",
+ "confidence_score": 0.97
+ },
+ {
+ "commit_hash": "dedc41a",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "626c22d",
+ "confidence_score": 0.03
+ },
+ {
+ "commit_hash": "cd4a49f",
+ "confidence_score": 0.01
+ },
+ {
+ "commit_hash": "92cb3da",
+ "confidence_score": 0.01
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.45,
+ "usage": {
+ "input_tokens": 3313,
+ "output_tokens": 719
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "order_read_sync_audit",
+ "category": "latency_timeout",
+ "culprit_hash": "8aed86c",
+ "expected_runbook_id": "request_timeouts",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": false,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.539
+ },
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.369
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.35
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "8aed86c",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "d3d6f87",
+ "confidence_score": 0.45
+ },
+ {
+ "commit_hash": "94578cc",
+ "confidence_score": 0.3
+ },
+ {
+ "commit_hash": "5abe2e4",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "b7ea08d",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 7.84,
+ "usage": {
+ "input_tokens": 3260,
+ "output_tokens": 813
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "order_sql_missing_param",
+ "category": "sql_param_mismatch",
+ "culprit_hash": "9ef79b2",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.322
+ },
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.23
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.188
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "9ef79b2",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "661f871",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "66d5f2f",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "93ebe2f",
+ "confidence_score": 0.01
+ },
+ {
+ "commit_hash": "411ddfa",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.61,
+ "usage": {
+ "input_tokens": 3452,
+ "output_tokens": 612
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "page_size_zero_division",
+ "category": "pagination_boundary",
+ "culprit_hash": "6265f24",
+ "expected_runbook_id": "pagination_errors",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.146
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.13
+ },
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.127
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "6265f24",
+ "confidence_score": 0.92
+ },
+ {
+ "commit_hash": "ed3861b",
+ "confidence_score": 0.65
+ },
+ {
+ "commit_hash": "0d241ff",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "ef48acf",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "debe268",
+ "confidence_score": 0.01
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.5,
+ "usage": {
+ "input_tokens": 3189,
+ "output_tokens": 599
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "payments_body_without_content_type",
+ "category": "http_content_type",
+ "culprit_hash": "33a9e2a",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.274
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.164
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.143
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "33a9e2a",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "e60edf2",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "5430f37",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "91f3040",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "64ee5a8",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.94,
+ "usage": {
+ "input_tokens": 3771,
+ "output_tokens": 559
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "payments_retry_loop",
+ "category": "rate_limit_429",
+ "culprit_hash": "f07e927",
+ "expected_runbook_id": "rate_limiting",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.357
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.352
+ },
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.323
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "f07e927",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "07a85f8",
+ "confidence_score": 0.8
+ },
+ {
+ "commit_hash": "c1fc55f",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "429c4dd",
+ "confidence_score": 0.0
+ },
+ {
+ "commit_hash": "51bc043",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.97,
+ "usage": {
+ "input_tokens": 3682,
+ "output_tokens": 631
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "redis_legacy_lrem_args",
+ "category": "dependency_version",
+ "culprit_hash": "0751831",
+ "expected_runbook_id": "dependency_regressions",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.324
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.317
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.269
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "0751831",
+ "confidence_score": 0.55
+ },
+ {
+ "commit_hash": "fd92e84",
+ "confidence_score": 0.4
+ },
+ {
+ "commit_hash": "3059910",
+ "confidence_score": 0.03
+ },
+ {
+ "commit_hash": "7b9dd2c",
+ "confidence_score": 0.01
+ },
+ {
+ "commit_hash": "957a42e",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.84,
+ "usage": {
+ "input_tokens": 3032,
+ "output_tokens": 723
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "settings_env_quotes",
+ "category": "config_type_error",
+ "culprit_hash": "15aba07",
+ "expected_runbook_id": "config_parse_errors",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "db_failures",
+ "similarity_score": 0.374
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.278
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.233
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "15aba07",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "cae9a6c",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "423c51f",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "3e36aa4",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "55cfa0d",
+ "confidence_score": 0.02
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 7.28,
+ "usage": {
+ "input_tokens": 3391,
+ "output_tokens": 740
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "since_filter_naive_cutoff",
+ "category": "timezone_mismatch",
+ "culprit_hash": "266c09c",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.279
+ },
+ {
+ "id": "dependency_regressions",
+ "similarity_score": 0.224
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.21
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "266c09c",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "f3ad8b7",
+ "confidence_score": 0.4
+ },
+ {
+ "commit_hash": "7f71cc9",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "ef6b4a1",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "5948e6d",
+ "confidence_score": 0.01
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.55,
+ "usage": {
+ "input_tokens": 3219,
+ "output_tokens": 708
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "worker_queue_renamed",
+ "category": "queue_stall",
+ "culprit_hash": "f8d8258",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": false,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.411
+ },
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.409
+ },
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.383
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "f8d8258",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "0f497ec",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "110eec6",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "aacbcd7",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "3434d6b",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.81,
+ "usage": {
+ "input_tokens": 2774,
+ "output_tokens": 639
+ }
+ },
+ {
+ "trial": 0,
+ "case_id": "worker_retains_job_payloads",
+ "category": "memory_growth",
+ "culprit_hash": "3037583",
+ "expected_runbook_id": "memory_leaks",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": false,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.48
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.434
+ },
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.424
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "3037583",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "088e964",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "947ac52",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "15f34e0",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "63c2633",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 7.58,
+ "usage": {
+ "input_tokens": 3095,
+ "output_tokens": 817
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "auth_issuer_mesh_hosts",
+ "category": "auth_rejection",
+ "culprit_hash": "5e68f8a",
+ "expected_runbook_id": "auth_failures",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.317
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.245
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.236
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "5e68f8a",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "45a24d5",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "df0ec4c",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "298be8d",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "d20025a",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.12,
+ "usage": {
+ "input_tokens": 3560,
+ "output_tokens": 587
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "cache_contains_recursion",
+ "category": "infinite_recursion",
+ "culprit_hash": "4584056",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.326
+ },
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.31
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.231
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "4584056",
+ "confidence_score": 0.98
+ },
+ {
+ "commit_hash": "6484609",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "7e63ea3",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "6cc141a",
+ "confidence_score": 0.0
+ },
+ {
+ "commit_hash": "a98de00",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.78,
+ "usage": {
+ "input_tokens": 3446,
+ "output_tokens": 591
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "cart_price_field_v2",
+ "category": "null_or_missing_field",
+ "culprit_hash": "d0ab213",
+ "expected_runbook_id": "null_field_handling",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.26
+ },
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.207
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.17
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "d0ab213",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "73e1933",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "4fbde4c",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "2bbbf3f",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "f618f59",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 7.0,
+ "usage": {
+ "input_tokens": 3384,
+ "output_tokens": 711
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "cart_sku_lowercased",
+ "category": "identifier_format",
+ "culprit_hash": "c2ec824",
+ "expected_runbook_id": null,
+ "commits_in_window": 4,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.293
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.247
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.229
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "c2ec824",
+ "confidence_score": 0.9
+ },
+ {
+ "commit_hash": "64dde29",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "d0cb0b3",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "91f8d36",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.47,
+ "usage": {
+ "input_tokens": 2991,
+ "output_tokens": 618
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "checkout_deadline_extra_call",
+ "category": "latency_timeout",
+ "culprit_hash": "efebc07",
+ "expected_runbook_id": "request_timeouts",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": false,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.487
+ },
+ {
+ "id": "db_failures",
+ "similarity_score": 0.373
+ },
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.353
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "efebc07",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "a035477",
+ "confidence_score": 0.55
+ },
+ {
+ "commit_hash": "2b765f4",
+ "confidence_score": 0.3
+ },
+ {
+ "commit_hash": "be85cc9",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "91b8b07",
+ "confidence_score": 0.02
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.94,
+ "usage": {
+ "input_tokens": 3255,
+ "output_tokens": 755
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "config_reformat_hidden_string",
+ "category": "config_type_error",
+ "culprit_hash": "c3629ef",
+ "expected_runbook_id": "config_parse_errors",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.246
+ },
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.19
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.172
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "c3629ef",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "3af442c",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "976c988",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "a6a3c41",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "89436d3",
+ "confidence_score": 0.01
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.36,
+ "usage": {
+ "input_tokens": 3480,
+ "output_tokens": 677
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "db_fetch_one_leak",
+ "category": "db_pool_exhaustion",
+ "culprit_hash": "af9b861",
+ "expected_runbook_id": "db_failures",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "db_failures",
+ "similarity_score": 0.628
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.344
+ },
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.28
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "af9b861",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "f46cd8c",
+ "confidence_score": 0.55
+ },
+ {
+ "commit_hash": "1618815",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "6d50d20",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "1bf1024",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 8.09,
+ "usage": {
+ "input_tokens": 3516,
+ "output_tokens": 822
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "export_tempfile_never_closed",
+ "category": "file_handle_leak",
+ "culprit_hash": "6eea0fc",
+ "expected_runbook_id": "file_handle_exhaustion",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.625
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.411
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.291
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "5cedb21",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "b245524",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "6eea0fc",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "0be651d",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "7453733",
+ "confidence_score": 0.01
+ }
+ ],
+ "culprit_rank": 3,
+ "error": null,
+ "rank_latency_s": 6.69,
+ "usage": {
+ "input_tokens": 3722,
+ "output_tokens": 720
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "guest_email_normalise",
+ "category": "null_or_missing_field",
+ "culprit_hash": "7dd25fe",
+ "expected_runbook_id": "null_field_handling",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.241
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.208
+ },
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.161
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "7dd25fe",
+ "confidence_score": 0.98
+ },
+ {
+ "commit_hash": "9519064",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "30a7434",
+ "confidence_score": 0.0
+ },
+ {
+ "commit_hash": "1c0135b",
+ "confidence_score": 0.0
+ },
+ {
+ "commit_hash": "cde392a",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.31,
+ "usage": {
+ "input_tokens": 3135,
+ "output_tokens": 629
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "httpx_028_drops_proxies",
+ "category": "dependency_version",
+ "culprit_hash": "b35322f",
+ "expected_runbook_id": "dependency_regressions",
+ "commits_in_window": 4,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.294
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.285
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.244
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "b35322f",
+ "confidence_score": 0.9
+ },
+ {
+ "commit_hash": "741f699",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "b45189f",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "e3601f6",
+ "confidence_score": 0.01
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.0,
+ "usage": {
+ "input_tokens": 2993,
+ "output_tokens": 585
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "inventory_cache_key_per_request",
+ "category": "rate_limit_429",
+ "culprit_hash": "526f8dd",
+ "expected_runbook_id": "rate_limiting",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.382
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.381
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.309
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "526f8dd",
+ "confidence_score": 0.92
+ },
+ {
+ "commit_hash": "bf4cc5f",
+ "confidence_score": 0.55
+ },
+ {
+ "commit_hash": "93320f2",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "2e1b06b",
+ "confidence_score": 0.01
+ },
+ {
+ "commit_hash": "d9caa65",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.34,
+ "usage": {
+ "input_tokens": 3709,
+ "output_tokens": 665
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "inventory_v3_payload",
+ "category": "null_or_missing_field",
+ "culprit_hash": "1c29d94",
+ "expected_runbook_id": "null_field_handling",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.33
+ },
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.28
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.271
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "1c29d94",
+ "confidence_score": 0.92
+ },
+ {
+ "commit_hash": "16d2191",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "c6f0797",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "5823d1f",
+ "confidence_score": 0.01
+ },
+ {
+ "commit_hash": "fb9b710",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.14,
+ "usage": {
+ "input_tokens": 3556,
+ "output_tokens": 586
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "jwt_leeway_milliseconds",
+ "category": "auth_rejection",
+ "culprit_hash": "2eeb222",
+ "expected_runbook_id": "auth_failures",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.488
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.198
+ },
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.159
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "2eeb222",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "2e11931",
+ "confidence_score": 0.3
+ },
+ {
+ "commit_hash": "e8548c5",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "a028c62",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "716e22d",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 7.02,
+ "usage": {
+ "input_tokens": 3526,
+ "output_tokens": 705
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "money_returns_decimal",
+ "category": "json_serialization",
+ "culprit_hash": "b14897b",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.298
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.151
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.13
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "b14897b",
+ "confidence_score": 0.98
+ },
+ {
+ "commit_hash": "dedc41a",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "626c22d",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "cd4a49f",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "92cb3da",
+ "confidence_score": 0.02
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.92,
+ "usage": {
+ "input_tokens": 3313,
+ "output_tokens": 708
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "order_read_sync_audit",
+ "category": "latency_timeout",
+ "culprit_hash": "8aed86c",
+ "expected_runbook_id": "request_timeouts",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": false,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.539
+ },
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.369
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.35
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "8aed86c",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "d3d6f87",
+ "confidence_score": 0.6
+ },
+ {
+ "commit_hash": "94578cc",
+ "confidence_score": 0.55
+ },
+ {
+ "commit_hash": "5abe2e4",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "b7ea08d",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 7.72,
+ "usage": {
+ "input_tokens": 3260,
+ "output_tokens": 802
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "order_sql_missing_param",
+ "category": "sql_param_mismatch",
+ "culprit_hash": "9ef79b2",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.322
+ },
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.23
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.188
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "9ef79b2",
+ "confidence_score": 0.97
+ },
+ {
+ "commit_hash": "661f871",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "66d5f2f",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "93ebe2f",
+ "confidence_score": 0.01
+ },
+ {
+ "commit_hash": "411ddfa",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.59,
+ "usage": {
+ "input_tokens": 3452,
+ "output_tokens": 626
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "page_size_zero_division",
+ "category": "pagination_boundary",
+ "culprit_hash": "6265f24",
+ "expected_runbook_id": "pagination_errors",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.146
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.13
+ },
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.127
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "6265f24",
+ "confidence_score": 0.92
+ },
+ {
+ "commit_hash": "ed3861b",
+ "confidence_score": 0.65
+ },
+ {
+ "commit_hash": "0d241ff",
+ "confidence_score": 0.08
+ },
+ {
+ "commit_hash": "ef48acf",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "debe268",
+ "confidence_score": 0.01
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.03,
+ "usage": {
+ "input_tokens": 3189,
+ "output_tokens": 655
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "payments_body_without_content_type",
+ "category": "http_content_type",
+ "culprit_hash": "33a9e2a",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.274
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.164
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.143
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "33a9e2a",
+ "confidence_score": 0.97
+ },
+ {
+ "commit_hash": "e60edf2",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "91f3040",
+ "confidence_score": 0.03
+ },
+ {
+ "commit_hash": "5430f37",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "64ee5a8",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.28,
+ "usage": {
+ "input_tokens": 3771,
+ "output_tokens": 639
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "payments_retry_loop",
+ "category": "rate_limit_429",
+ "culprit_hash": "f07e927",
+ "expected_runbook_id": "rate_limiting",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.357
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.352
+ },
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.323
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "f07e927",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "07a85f8",
+ "confidence_score": 0.75
+ },
+ {
+ "commit_hash": "c1fc55f",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "429c4dd",
+ "confidence_score": 0.0
+ },
+ {
+ "commit_hash": "51bc043",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.58,
+ "usage": {
+ "input_tokens": 3682,
+ "output_tokens": 572
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "redis_legacy_lrem_args",
+ "category": "dependency_version",
+ "culprit_hash": "0751831",
+ "expected_runbook_id": "dependency_regressions",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.324
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.317
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.269
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "0751831",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "fd92e84",
+ "confidence_score": 0.55
+ },
+ {
+ "commit_hash": "3059910",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "7b9dd2c",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "957a42e",
+ "confidence_score": 0.01
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.73,
+ "usage": {
+ "input_tokens": 3032,
+ "output_tokens": 671
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "settings_env_quotes",
+ "category": "config_type_error",
+ "culprit_hash": "15aba07",
+ "expected_runbook_id": "config_parse_errors",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "db_failures",
+ "similarity_score": 0.374
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.278
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.233
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "15aba07",
+ "confidence_score": 0.9
+ },
+ {
+ "commit_hash": "423c51f",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "cae9a6c",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "3e36aa4",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "55cfa0d",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.5,
+ "usage": {
+ "input_tokens": 3391,
+ "output_tokens": 720
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "since_filter_naive_cutoff",
+ "category": "timezone_mismatch",
+ "culprit_hash": "266c09c",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.279
+ },
+ {
+ "id": "dependency_regressions",
+ "similarity_score": 0.224
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.21
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "266c09c",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "f3ad8b7",
+ "confidence_score": 0.35
+ },
+ {
+ "commit_hash": "7f71cc9",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "ef6b4a1",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "5948e6d",
+ "confidence_score": 0.01
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.22,
+ "usage": {
+ "input_tokens": 3219,
+ "output_tokens": 700
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "worker_queue_renamed",
+ "category": "queue_stall",
+ "culprit_hash": "f8d8258",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": false,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.411
+ },
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.409
+ },
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.383
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "f8d8258",
+ "confidence_score": 0.9
+ },
+ {
+ "commit_hash": "0f497ec",
+ "confidence_score": 0.25
+ },
+ {
+ "commit_hash": "110eec6",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "aacbcd7",
+ "confidence_score": 0.03
+ },
+ {
+ "commit_hash": "3434d6b",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.12,
+ "usage": {
+ "input_tokens": 2774,
+ "output_tokens": 661
+ }
+ },
+ {
+ "trial": 1,
+ "case_id": "worker_retains_job_payloads",
+ "category": "memory_growth",
+ "culprit_hash": "3037583",
+ "expected_runbook_id": "memory_leaks",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": false,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.48
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.434
+ },
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.424
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "3037583",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "088e964",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "947ac52",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "15f34e0",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "63c2633",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.39,
+ "usage": {
+ "input_tokens": 3095,
+ "output_tokens": 568
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "auth_issuer_mesh_hosts",
+ "category": "auth_rejection",
+ "culprit_hash": "5e68f8a",
+ "expected_runbook_id": "auth_failures",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.317
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.245
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.236
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "5e68f8a",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "45a24d5",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "df0ec4c",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "298be8d",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "d20025a",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.33,
+ "usage": {
+ "input_tokens": 3560,
+ "output_tokens": 598
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "cache_contains_recursion",
+ "category": "infinite_recursion",
+ "culprit_hash": "4584056",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.326
+ },
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.31
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.231
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "4584056",
+ "confidence_score": 0.98
+ },
+ {
+ "commit_hash": "6484609",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "7e63ea3",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "6cc141a",
+ "confidence_score": 0.0
+ },
+ {
+ "commit_hash": "a98de00",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.05,
+ "usage": {
+ "input_tokens": 3446,
+ "output_tokens": 572
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "cart_price_field_v2",
+ "category": "null_or_missing_field",
+ "culprit_hash": "d0ab213",
+ "expected_runbook_id": "null_field_handling",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.26
+ },
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.207
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.17
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "d0ab213",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "73e1933",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "4fbde4c",
+ "confidence_score": 0.03
+ },
+ {
+ "commit_hash": "2bbbf3f",
+ "confidence_score": 0.01
+ },
+ {
+ "commit_hash": "f618f59",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.7,
+ "usage": {
+ "input_tokens": 3384,
+ "output_tokens": 708
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "cart_sku_lowercased",
+ "category": "identifier_format",
+ "culprit_hash": "c2ec824",
+ "expected_runbook_id": null,
+ "commits_in_window": 4,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.293
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.247
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.229
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "c2ec824",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "64dde29",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "d0cb0b3",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "91f8d36",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.98,
+ "usage": {
+ "input_tokens": 2991,
+ "output_tokens": 642
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "checkout_deadline_extra_call",
+ "category": "latency_timeout",
+ "culprit_hash": "efebc07",
+ "expected_runbook_id": "request_timeouts",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": false,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.487
+ },
+ {
+ "id": "db_failures",
+ "similarity_score": 0.373
+ },
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.353
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "efebc07",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "a035477",
+ "confidence_score": 0.55
+ },
+ {
+ "commit_hash": "2b765f4",
+ "confidence_score": 0.3
+ },
+ {
+ "commit_hash": "be85cc9",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "91b8b07",
+ "confidence_score": 0.02
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.78,
+ "usage": {
+ "input_tokens": 3255,
+ "output_tokens": 746
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "config_reformat_hidden_string",
+ "category": "config_type_error",
+ "culprit_hash": "c3629ef",
+ "expected_runbook_id": "config_parse_errors",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.246
+ },
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.19
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.172
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "c3629ef",
+ "confidence_score": 0.9
+ },
+ {
+ "commit_hash": "3af442c",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "976c988",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "a6a3c41",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "89436d3",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.12,
+ "usage": {
+ "input_tokens": 3480,
+ "output_tokens": 693
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "db_fetch_one_leak",
+ "category": "db_pool_exhaustion",
+ "culprit_hash": "af9b861",
+ "expected_runbook_id": "db_failures",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "db_failures",
+ "similarity_score": 0.628
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.344
+ },
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.28
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "af9b861",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "f46cd8c",
+ "confidence_score": 0.45
+ },
+ {
+ "commit_hash": "1618815",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "6d50d20",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "1bf1024",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.5,
+ "usage": {
+ "input_tokens": 3516,
+ "output_tokens": 684
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "export_tempfile_never_closed",
+ "category": "file_handle_leak",
+ "culprit_hash": "6eea0fc",
+ "expected_runbook_id": "file_handle_exhaustion",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.625
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.411
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.291
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "5cedb21",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "6eea0fc",
+ "confidence_score": 0.3
+ },
+ {
+ "commit_hash": "b245524",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "0be651d",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "7453733",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 2,
+ "error": null,
+ "rank_latency_s": 6.98,
+ "usage": {
+ "input_tokens": 3722,
+ "output_tokens": 765
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "guest_email_normalise",
+ "category": "null_or_missing_field",
+ "culprit_hash": "7dd25fe",
+ "expected_runbook_id": "null_field_handling",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.241
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.208
+ },
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.161
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "7dd25fe",
+ "confidence_score": 0.98
+ },
+ {
+ "commit_hash": "9519064",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "30a7434",
+ "confidence_score": 0.0
+ },
+ {
+ "commit_hash": "1c0135b",
+ "confidence_score": 0.0
+ },
+ {
+ "commit_hash": "cde392a",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.77,
+ "usage": {
+ "input_tokens": 3135,
+ "output_tokens": 609
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "httpx_028_drops_proxies",
+ "category": "dependency_version",
+ "culprit_hash": "b35322f",
+ "expected_runbook_id": "dependency_regressions",
+ "commits_in_window": 4,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.294
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.285
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.244
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "b35322f",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "741f699",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "b45189f",
+ "confidence_score": 0.03
+ },
+ {
+ "commit_hash": "e3601f6",
+ "confidence_score": 0.02
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.44,
+ "usage": {
+ "input_tokens": 2993,
+ "output_tokens": 600
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "inventory_cache_key_per_request",
+ "category": "rate_limit_429",
+ "culprit_hash": "526f8dd",
+ "expected_runbook_id": "rate_limiting",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.382
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.381
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.309
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "526f8dd",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "bf4cc5f",
+ "confidence_score": 0.3
+ },
+ {
+ "commit_hash": "93320f2",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "2e1b06b",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "d9caa65",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 7.17,
+ "usage": {
+ "input_tokens": 3709,
+ "output_tokens": 806
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "inventory_v3_payload",
+ "category": "null_or_missing_field",
+ "culprit_hash": "1c29d94",
+ "expected_runbook_id": "null_field_handling",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.33
+ },
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.28
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.271
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "1c29d94",
+ "confidence_score": 0.93
+ },
+ {
+ "commit_hash": "16d2191",
+ "confidence_score": 0.07
+ },
+ {
+ "commit_hash": "c6f0797",
+ "confidence_score": 0.03
+ },
+ {
+ "commit_hash": "5823d1f",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "fb9b710",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.84,
+ "usage": {
+ "input_tokens": 3556,
+ "output_tokens": 729
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "jwt_leeway_milliseconds",
+ "category": "auth_rejection",
+ "culprit_hash": "2eeb222",
+ "expected_runbook_id": "auth_failures",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.488
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.198
+ },
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.159
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "2eeb222",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "e8548c5",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "2e11931",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "a028c62",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "716e22d",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.75,
+ "usage": {
+ "input_tokens": 3526,
+ "output_tokens": 630
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "money_returns_decimal",
+ "category": "json_serialization",
+ "culprit_hash": "b14897b",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.298
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.151
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.13
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "b14897b",
+ "confidence_score": 0.98
+ },
+ {
+ "commit_hash": "dedc41a",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "626c22d",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "cd4a49f",
+ "confidence_score": 0.01
+ },
+ {
+ "commit_hash": "92cb3da",
+ "confidence_score": 0.01
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.27,
+ "usage": {
+ "input_tokens": 3313,
+ "output_tokens": 683
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "order_read_sync_audit",
+ "category": "latency_timeout",
+ "culprit_hash": "8aed86c",
+ "expected_runbook_id": "request_timeouts",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": false,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.539
+ },
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.369
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.35
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "8aed86c",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "d3d6f87",
+ "confidence_score": 0.35
+ },
+ {
+ "commit_hash": "94578cc",
+ "confidence_score": 0.3
+ },
+ {
+ "commit_hash": "5abe2e4",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "b7ea08d",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 7.3,
+ "usage": {
+ "input_tokens": 3260,
+ "output_tokens": 780
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "order_sql_missing_param",
+ "category": "sql_param_mismatch",
+ "culprit_hash": "9ef79b2",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.322
+ },
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.23
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.188
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "9ef79b2",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "661f871",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "66d5f2f",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "93ebe2f",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "411ddfa",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.97,
+ "usage": {
+ "input_tokens": 3452,
+ "output_tokens": 632
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "page_size_zero_division",
+ "category": "pagination_boundary",
+ "culprit_hash": "6265f24",
+ "expected_runbook_id": "pagination_errors",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.146
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.13
+ },
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.127
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "6265f24",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "ed3861b",
+ "confidence_score": 0.55
+ },
+ {
+ "commit_hash": "0d241ff",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "ef48acf",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "debe268",
+ "confidence_score": 0.01
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.56,
+ "usage": {
+ "input_tokens": 3189,
+ "output_tokens": 620
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "payments_body_without_content_type",
+ "category": "http_content_type",
+ "culprit_hash": "33a9e2a",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.274
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.164
+ },
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.143
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "33a9e2a",
+ "confidence_score": 0.97
+ },
+ {
+ "commit_hash": "e60edf2",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "5430f37",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "91f3040",
+ "confidence_score": 0.01
+ },
+ {
+ "commit_hash": "64ee5a8",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.19,
+ "usage": {
+ "input_tokens": 3771,
+ "output_tokens": 568
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "payments_retry_loop",
+ "category": "rate_limit_429",
+ "culprit_hash": "f07e927",
+ "expected_runbook_id": "rate_limiting",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.357
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.352
+ },
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.323
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "f07e927",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "07a85f8",
+ "confidence_score": 0.8
+ },
+ {
+ "commit_hash": "c1fc55f",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "429c4dd",
+ "confidence_score": 0.0
+ },
+ {
+ "commit_hash": "51bc043",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 7.3,
+ "usage": {
+ "input_tokens": 3682,
+ "output_tokens": 725
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "redis_legacy_lrem_args",
+ "category": "dependency_version",
+ "culprit_hash": "0751831",
+ "expected_runbook_id": "dependency_regressions",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "null_field_handling",
+ "similarity_score": 0.324
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.317
+ },
+ {
+ "id": "auth_failures",
+ "similarity_score": 0.269
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "0751831",
+ "confidence_score": 0.85
+ },
+ {
+ "commit_hash": "fd92e84",
+ "confidence_score": 0.55
+ },
+ {
+ "commit_hash": "3059910",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "7b9dd2c",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "957a42e",
+ "confidence_score": 0.01
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.42,
+ "usage": {
+ "input_tokens": 3032,
+ "output_tokens": 701
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "settings_env_quotes",
+ "category": "config_type_error",
+ "culprit_hash": "15aba07",
+ "expected_runbook_id": "config_parse_errors",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "db_failures",
+ "similarity_score": 0.374
+ },
+ {
+ "id": "config_parse_errors",
+ "similarity_score": 0.278
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.233
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "15aba07",
+ "confidence_score": 0.9
+ },
+ {
+ "commit_hash": "cae9a6c",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "423c51f",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "3e36aa4",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "55cfa0d",
+ "confidence_score": 0.01
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.2,
+ "usage": {
+ "input_tokens": 3391,
+ "output_tokens": 703
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "since_filter_naive_cutoff",
+ "category": "timezone_mismatch",
+ "culprit_hash": "266c09c",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": true,
+ "culprit_in_traceback": true,
+ "retrieved_runbooks": [
+ {
+ "id": "pagination_errors",
+ "similarity_score": 0.279
+ },
+ {
+ "id": "dependency_regressions",
+ "similarity_score": 0.224
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.21
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "266c09c",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "f3ad8b7",
+ "confidence_score": 0.3
+ },
+ {
+ "commit_hash": "7f71cc9",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "ef6b4a1",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "5948e6d",
+ "confidence_score": 0.02
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.69,
+ "usage": {
+ "input_tokens": 3219,
+ "output_tokens": 679
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "worker_queue_renamed",
+ "category": "queue_stall",
+ "culprit_hash": "f8d8258",
+ "expected_runbook_id": null,
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": false,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "request_timeouts",
+ "similarity_score": 0.411
+ },
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.409
+ },
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.383
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "f8d8258",
+ "confidence_score": 0.9
+ },
+ {
+ "commit_hash": "0f497ec",
+ "confidence_score": 0.2
+ },
+ {
+ "commit_hash": "110eec6",
+ "confidence_score": 0.05
+ },
+ {
+ "commit_hash": "aacbcd7",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "3434d6b",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 6.28,
+ "usage": {
+ "input_tokens": 2774,
+ "output_tokens": 728
+ }
+ },
+ {
+ "trial": 2,
+ "case_id": "worker_retains_job_payloads",
+ "category": "memory_growth",
+ "culprit_hash": "3037583",
+ "expected_runbook_id": "memory_leaks",
+ "commits_in_window": 5,
+ "truncated_diffs": 0,
+ "has_traceback": false,
+ "culprit_in_traceback": false,
+ "retrieved_runbooks": [
+ {
+ "id": "file_handle_exhaustion",
+ "similarity_score": 0.48
+ },
+ {
+ "id": "memory_leaks",
+ "similarity_score": 0.434
+ },
+ {
+ "id": "rate_limiting",
+ "similarity_score": 0.424
+ }
+ ],
+ "ranking": [
+ {
+ "commit_hash": "3037583",
+ "confidence_score": 0.95
+ },
+ {
+ "commit_hash": "088e964",
+ "confidence_score": 0.15
+ },
+ {
+ "commit_hash": "947ac52",
+ "confidence_score": 0.1
+ },
+ {
+ "commit_hash": "15f34e0",
+ "confidence_score": 0.02
+ },
+ {
+ "commit_hash": "63c2633",
+ "confidence_score": 0.0
+ }
+ ],
+ "culprit_rank": 1,
+ "error": null,
+ "rank_latency_s": 5.53,
+ "usage": {
+ "input_tokens": 3095,
+ "output_tokens": 597
+ }
+ }
+ ],
+ "finished_at": "2026-10-11T19:49:07.030635+00:00"
+}
\ No newline at end of file
diff --git a/eval/run_eval.py b/eval/run_eval.py
index 74161af..8a2c813 100644
--- a/eval/run_eval.py
+++ b/eval/run_eval.py
@@ -26,6 +26,8 @@
def run_trial(case: dict, condition: str) -> dict:
with fixture.build(case) as (repo, alert, label):
+ if condition != "traceback":
+ alert.pop("traceback", None) # baseline and rag prompts stay byte-identical to Phase 9
diffs = git_client.get_recent_diffs(repo, alert["timestamp"])
rag = condition == "rag"
runbooks = (
@@ -40,6 +42,13 @@ def run_trial(case: dict, condition: str) -> dict:
"expected_runbook_id": label["expected_runbook_id"],
"commits_in_window": len(diffs),
"truncated_diffs": sum(len(d["diff"]) >= 3000 for d in diffs),
+ "has_traceback": "traceback" in alert,
+ "culprit_in_traceback": bool(
+ llm_analyzer.files_in_traceback(
+ next(d["files"] for d in diffs if d["commit_hash"] == label["culprit_hash"]),
+ alert.get("traceback"),
+ )
+ ),
"retrieved_runbooks": [
{"id": r["id"], "similarity_score": r["similarity_score"]} for r in runbooks
],
@@ -108,7 +117,7 @@ def _git_sha() -> str:
def main(argv=None):
p = argparse.ArgumentParser()
- p.add_argument("--condition", choices=["baseline", "rag"], required=True)
+ p.add_argument("--condition", choices=["baseline", "rag", "traceback"], required=True)
p.add_argument("--trials", type=int, default=config.DEFAULT_TRIALS)
p.add_argument("--cases", default="*.json", help="glob inside eval/cases")
p.add_argument("--out", required=True)
@@ -126,6 +135,7 @@ def main(argv=None):
"condition": args.condition,
"dry_run": args.dry_run,
"eval_set_version": config.EVAL_SET_VERSION,
+ "traceback_set_version": config.TRACEBACK_SET_VERSION if args.condition == "traceback" else None,
"runbook_corpus_version": config.RUNBOOK_CORPUS_VERSION,
"runbooks_ingested": n_runbooks,
"runbook_floor": llm_analyzer.RUNBOOK_FLOOR,
diff --git a/eval/tests/test_eval.py b/eval/tests/test_eval.py
index 3aa6f1b..461b6f3 100644
--- a/eval/tests/test_eval.py
+++ b/eval/tests/test_eval.py
@@ -74,6 +74,11 @@ def test_fixture_commits_land_in_window_in_order(case):
assert [d["subject"] for d in diffs] == [c["message"] for c in reversed(case["commits"])]
assert label["culprit_hash"] in {d["commit_hash"] for d in diffs}
assert all(len(d["diff"]) < 3000 for d in diffs), "diff hit the truncation limit"
+ if case.get("traceback"): # every app frame resolved to a real line; no answer-announcing words
+ assert alert["traceback"].startswith("Traceback (most recent call last):")
+ assert not _BANNED.search(alert["traceback"]), alert["traceback"]
+ else:
+ assert "traceback" not in alert
# ---------- scorer ----------
@@ -123,6 +128,7 @@ def test_paired():
"condition",
[
"baseline",
+ "traceback",
pytest.param("rag", marks=pytest.mark.skipif(not _RAG_READY, reason="runbooks param lands in 9E")),
],
)
@@ -145,6 +151,7 @@ def capture(messages, **kw):
for value in filter(None, leaks):
assert value not in prompt, f"{case['id']}: {value!r} leaked into prompt"
assert "culprit" not in prompt.lower()
+ assert ("" in prompt) == (condition == "traceback" and bool(case.get("traceback")))
assert len(prompts) == len(CASES)
@@ -163,3 +170,18 @@ def dead_key(*_a, **_kw):
run_eval.main(["--condition", "baseline", "--trials", "3", "--out", str(out)])
res = report.load(out)
assert len(res["records"]) == run_eval.MAX_CONSECUTIVE_ERRORS and "aborted" in res
+
+
+def test_traceback_groups_prefer_the_run_that_saw_the_trace():
+ base = [{"case_id": "a", "has_traceback": False}, {"case_id": "b", "has_traceback": False}]
+ tb = [
+ {"case_id": "a", "has_traceback": True, "culprit_in_traceback": False},
+ {"case_id": "b", "has_traceback": False},
+ {"case_id": "c", "has_traceback": True, "culprit_in_traceback": True},
+ ]
+ old = [{"case_id": "d"}] # pre-traceback runs carry no field
+ assert report.traceback_groups(base + tb + old) == {
+ "a": "trace misses culprit file",
+ "b": "no trace",
+ "c": "culprit file in trace",
+ }
diff --git a/sandbox/app/main.py b/sandbox/app/main.py
index 25345c6..a01e67e 100644
--- a/sandbox/app/main.py
+++ b/sandbox/app/main.py
@@ -1,5 +1,6 @@
import json
import os
+import traceback
from datetime import datetime, timezone
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
@@ -44,7 +45,11 @@ async def catch_all(request: Request, exc: Exception):
"level": "ERROR",
"message": str(exc),
"endpoint": request.url.path,
- "traceback": repr(exc),
+ "traceback": "".join(traceback.format_exception(exc)),
}
print(json.dumps(entry))
- return JSONResponse(status_code=500, content={"error": str(exc)})
+ # ponytail: debug-mode error body (like a Sentry event) so the load generator can forward the trace
+ return JSONResponse(
+ status_code=500,
+ content={"error": f"{type(exc).__name__}: {exc}", "traceback": entry["traceback"]},
+ )
diff --git a/sandbox/chaos_cli.py b/sandbox/chaos_cli.py
index de8b2a0..2ae4f09 100644
--- a/sandbox/chaos_cli.py
+++ b/sandbox/chaos_cli.py
@@ -16,6 +16,9 @@
REPO_ROOT = Path(__file__).parent.parent
CONFIG = REPO_ROOT / "sandbox" / "app" / "config.json"
CORE_ALERT = "http://localhost:8000/alert"
+APP = "http://localhost:8001"
+# endpoint that raises for each bug; memory_leak degrades without raising, so it has none
+_PROBES = {"db_failure": ("GET", "/api/data"), "null_pointer": ("POST", "/api/process")}
_ERROR_SIGS = {
"db_failure": "OperationalError: connection to server at '127.0.0.1' failed: Connection refused",
@@ -33,6 +36,18 @@ def _write_config_and_commit(bug: str | None):
subprocess.run(["git", "commit", "-m", msg], cwd=REPO_ROOT, check=True)
+def _probe_traceback(bug_type: str) -> str | None:
+ """Hit the sandbox app once so the alert carries the real stack trace, as Sentry would."""
+ if bug_type not in _PROBES:
+ return None
+ method, path = _PROBES[bug_type]
+ try:
+ r = httpx.request(method, APP + path, json={} if method == "POST" else None, timeout=3.0)
+ return r.json().get("traceback") if r.status_code >= 500 else None
+ except (httpx.HTTPError, ValueError):
+ return None
+
+
@click.group()
def cli():
pass
@@ -53,6 +68,10 @@ def trigger_bug(bug_type: str):
"error_signature": _ERROR_SIGS[bug_type],
"bug_type": bug_type,
}
+ tb = _probe_traceback(bug_type)
+ if tb:
+ alert["traceback"] = tb
+ click.echo(f"[chaos] traceback: {'captured from sandbox app' if tb else 'none (app not running?)'}")
try:
r = httpx.post(CORE_ALERT, json=alert, timeout=5.0)
data = r.json()
diff --git a/sandbox/load_generator.py b/sandbox/load_generator.py
index 2f9dff9..a7e7041 100644
--- a/sandbox/load_generator.py
+++ b/sandbox/load_generator.py
@@ -18,6 +18,7 @@
async def run(tps: int, duration: int):
counts = {"ok": 0, "err": 0}
+ last_error = {} # most recent 500 body: real signature + traceback for the alert
alerted = False
deadline = time.monotonic() + duration
interval = 1.0 / tps
@@ -28,6 +29,11 @@ async def run(tps: int, duration: int):
try:
r = await client.get(f"{APP}/api/data")
counts["err" if r.status_code >= 500 else "ok"] += 1
+ if r.status_code >= 500:
+ try:
+ last_error = r.json()
+ except ValueError:
+ pass
except Exception:
counts["err"] += 1
@@ -41,9 +47,11 @@ async def run(tps: int, duration: int):
"source": "load_generator",
"alert_name": "HTTP_500_Internal_Server_Error",
"timestamp": datetime.now(timezone.utc).isoformat(),
- "error_signature": "Error rate spike detected",
+ "error_signature": last_error.get("error", "Error rate spike detected"),
"error_rate_pct": round(err_pct, 1),
}
+ if last_error.get("traceback"):
+ alert["traceback"] = last_error["traceback"]
try:
await client.post(CORE, json=alert, timeout=5.0)
print("\n[load] alert fired -> sentinel core")