Stack traces in commit ranking (95.8% top-1 on eval set v2)

ADR.md

@@ -145,3 +145,23 @@ depends on two things that could change:
 - **Baseline headroom.** At 90% top-1 there is little left to win. A larger,
   harder or real incident corpus — especially real runbooks written by the
   team that wrote the code — would be the test that could change this decision.
+
+## ADR-8: Stack traces in commit ranking — on whenever the alert has one
+
+**Context.** The alert was one line of error text; Sentinel never used the
+stack trace a real error tracker attaches. A trace says where an error
+surfaced, which is often not where it was introduced, so it could just as
+easily mislead the ranker as help it.
+
+**Decision.** An alert may carry an optional `traceback`. Ranking gets the
+newest 4000 characters of it, and each commit that touches a file named in a
+frame is marked (frames carry deploy paths, so files match on a path-segment
+suffix). The prompt says frames show where the error surfaced, not
+necessarily its cause. Without a trace the prompt is byte-identical to the
+baseline. No flag: having a trace is what switches it on.
+
+**Measured result** (METRICS.md, 2026-10-11): 95.8% top-1 against baselines
+of 90.3% and 91.7%. The gain comes from one case that went from 0/9 to 3/3;
+cases whose trace named only decoy files improved rather than regressed.
+These traces were written to go through decoys wherever the raising call
+was ambiguous, so they are harder than typical traces.

METRICS.md

@@ -227,3 +227,60 @@ false alarms** (in-sample it was 9/13 and 9/131), so it holds. On Haiku 4.5 it
 flags 7/14 wrong but 22/58 right picks, and Haiku's raw confidence is identical
 for right and wrong picks (0.93 / 0.93): the rule is calibrated for Sonnet 5
 and should be re-checked if the model changes.
+
+## Stack traces in commit ranking (2026-10-11)
+
+**Question:** does sending the alert's stack trace to the ranker, and marking
+commits that touch a file named in a frame, improve top-1?
+
+**Answer: yes, by one robust case flip; everything else ties.** Traces are
+used whenever an alert carries one (no flag).
+
+- **Inputs:** eval set v2 unchanged; 20/24 cases gained a trace
+  (`TRACEBACK_SET_VERSION = "v1"`), rendered against the post-commit fixture
+  repo so app frames carry real line numbers. The 4 metric alerts (timeouts,
+  OOM kill, idle queue) carry none, and their prompts are identical to
+  baseline.
+- **Traces were built against the treatment.** Where the raising call is
+  ambiguous (a leak surfaces at whichever `getconn` or `open` runs next), the
+  frames follow a path *not* through the culprit's lines. The culprit's file
+  appears in 13/20 traces; the other 7 name only decoy files or no app files.
+- **Control:** the two existing Sonnet 5 baselines. The baseline prompt is
+  byte-identical to the one they used (golden-prompt test), so no new
+  baseline run was needed. Same model, `claude-sonnet-5`, 3 trials per case.
+
+Reproduce: `python -m eval.report eval/results/baseline_v2_sonnet5_rerun_2026-10-07.json eval/results/traceback_v2_sonnet5_2026-10-11.json`
+
+| Metric | Baseline 10-04 | Baseline 10-07 | Traceback 10-11 |
+|---|---|---|---|
+| Top-1 | 90.3% (65/72) | 91.7% (66/72) | **95.8% (69/72)** |
+| Top-3 | 100% | 100% | 100% |
+| MRR | 0.947 | 0.958 | 0.975 |
+| LLM failures | 0/72 | 0/72 | 0/72 |
+| Ranking latency, median | 5.8s | 6.5s | 6.3s |
+| Input tokens per call, mean | n/a | 3066 | 3352 (+9%) |
+
+**Where the gain comes from.** `httpx_028_drops_proxies` went from **0/3 in
+every earlier Sonnet run (0/9 across baseline, RAG and re-run) to 3/3**. Its
+trace names only the decoy (`payments.py`, where `httpx.Client(proxies=...)`
+raises at import), yet seeing the call fail at construction pointed the model
+at the dependency bump. The other paired differences are single-trial swings
+already seen between the two baselines (`since_filter_naive_cutoff`,
+`page_size_zero_division`, `export_tempfile_never_closed`).
+
+| Group (trials) | Baseline 10-04 | Baseline 10-07 | Traceback |
+|---|---|---|---|
+| Culprit file in trace (39) | 38/39 | 38/39 | 39/39 |
+| Trace misses culprit file (21) | 15/21 | 16/21 | 18/21 |
+| No trace (12) | 12/12 | 12/12 | 12/12 |
+
+A trace that pointed at a decoy did not pull the ranking towards it: that
+group improved too. `export_tempfile_never_closed`, whose trace points at the
+audit-log decoy, stayed at 0/3, where baseline had already put it (0/3 and
+1/3).
+
+**Limitation:** a 4–5 point gain on 24 cases rests on one case. Read it as
+"traces help and did not hurt here", not as a precise effect size.
+
+**Truncation check:** none of the 216 baseline/RAG/re-run trials hit the
+3000-character diff limit, so truncation did not cause any of the misses.

README.md

@@ -85,6 +85,7 @@ Every hop reads and writes one auditable incident row conforming to
 
 - [METRICS.md](METRICS.md) — measured end-to-end timings, methodology, and per-run results
 - Measured diagnostic accuracy: faulty commit ranked #1 in 90.3% of 72 trials across 24 injected-fault cases with decoy commits ([METRICS.md, Phase 9](METRICS.md#phase-9--diagnostic-accuracy-and-runbook-context-in-ranking))
+- With the alert's stack trace in the ranking prompt: 95.8% of 72 trials, against 90.3% and 91.7% without, the gain coming from one case that went from 0/9 to 3/3 ([METRICS.md](METRICS.md#stack-traces-in-commit-ranking-2026-10-11))
 - [ADR.md](ADR.md) — why explicit state loops over LangChain, polling over WebSockets, Chroma over Pinecone, FastAPI over Flask, tool-call JSON over free-form parsing, sandbox over live infra
 
 ## Sandbox reference

contract/pipeline_schema.json

@@ -13,7 +13,8 @@
         "source": { "type": "string" },
         "alert_name": { "type": "string" },
         "timestamp": { "type": "string", "format": "date-time" },
-        "error_signature": { "type": "string" }
+        "error_signature": { "type": "string" },
+        "traceback": { "type": "string", "description": "Optional stack trace; fed to commit ranking when present" }
       }
     },
     "diagnostics": {

core/services/git_client.py

@@ -1,3 +1,4 @@
+import re
 import subprocess
 from datetime import datetime, timedelta
 
@@ -41,6 +42,8 @@ def get_recent_diffs(repo_path: str, alert_ts: str, window_minutes: int = 60) ->
                 "author": author,
                 "timestamp": ts,
                 "subject": subject,
+                # from the full patch, before truncation, so traceback overlap sees every file
+                "files": re.findall(r"^diff --git a/.* b/(.*)$", diff.stdout, flags=re.M),
                 "diff": diff.stdout[:3000],  # ponytail: truncated for LLM context; raise if diffs are large
             }
         )

core/services/llm_analyzer.py

@@ -1,4 +1,5 @@
 import os
+import re
 
 import anthropic
 
@@ -27,6 +28,28 @@ def _runbooks_block(runbooks: list[dict] | None) -> str:
     )
 
 
+# ponytail: keep the newest frames only; recursion traces run to thousands of lines
+_TRACEBACK_MAX_CHARS = 4000
+
+
+def _traceback_block(traceback: str | None) -> str:
+    """'' without a traceback, so an alert that carries none ranks with the baseline prompt."""
+    if not traceback:
+        return ""
+    return (
+        "STACK TRACE (from the alert; frames show where the error surfaced, which is not necessarily "
+        "where it was introduced):\n"
+        f"<traceback>\n{traceback[-_TRACEBACK_MAX_CHARS:]}\n</traceback>\n\n"
+    )
+
+
+def files_in_traceback(commit_files: list[str], traceback: str | None) -> list[str]:
+    """Commit files that appear as a frame in the traceback. Frames carry deploy paths
+    (/srv/app/x.py), commits carry repo paths (x.py), so match on a path-segment suffix."""
+    frames = {p.replace("\\", "/") for p in re.findall(r'File "([^"]+)", line \d+', traceback or "")}
+    return [f for f in commit_files if any(p == f or p.endswith("/" + f) for p in frames)]
+
+
 class LLMUnavailable(Exception):
     """LLM output could not be produced: timeout/network, malformed JSON, or refusal."""
 
@@ -61,9 +84,16 @@ def rank_suspect_commits(diffs: list[dict], alert: dict, runbooks: list[dict] |
     if not diffs:
         return []
 
+    traceback = alert.get("traceback")
+
+    def _overlap_line(c: dict) -> str:
+        hits = files_in_traceback(c.get("files", []), traceback)
+        return f"Touches files in the stack trace: {', '.join(hits)}\n" if hits else ""
+
     commits_block = "\n\n".join(
         f"COMMIT {c['commit_hash']} | {c['author']} | {c['timestamp']}\n"
         f"Subject: {c['subject']}\n"
+        f"{_overlap_line(c)}"
         f"Diff:\n{c['diff']}"
         for c in diffs
     )
@@ -74,6 +104,7 @@ def rank_suspect_commits(diffs: list[dict], alert: dict, runbooks: list[dict] |
         f"  name: {alert['alert_name']}\n"
         f"  error: {alert['error_signature']}\n"
         f"  time: {alert['timestamp']}\n\n"
+        f"{_traceback_block(traceback)}"
         f"{_runbooks_block(runbooks)}"
         f"RECENT COMMITS:\n{commits_block}\n\n"
         f"Use rank_commits to return every commit with a confidence_score (0=unrelated, 1=certain cause) "

core/tests/test_traceback_ranking.py

@@ -0,0 +1,68 @@
+# ponytail: traceback context must not move the baseline prompt; overlap must match path segments only
+from types import SimpleNamespace as NS
+from unittest.mock import patch
+
+from core.services import llm_analyzer
+from core.tests.test_rag_ranking import _ALERT, _DIFFS, _GOLDEN_BASELINE_PROMPT
+
+_TB = (
+    "Traceback (most recent call last):\n"
+    '  File "/srv/orders/app/db.py", line 24, in fetch_all\n'
+    "    conn = pool.getconn()\n"
+    "psycopg2.pool.PoolError: connection pool exhausted"
+)
+_FILED = [{**_DIFFS[0], "files": ["config.json", "app/db.py"]}, {**_DIFFS[1], "files": ["README.md"]}]
+
+
+def _prompt(diffs, alert):
+    captured = []
+
+    def fake_chat(messages, **_):
+        captured.append(messages[0]["content"])
+        return NS(
+            stop_reason="tool_use",
+            content=[NS(type="tool_use", name="rank_commits", input={"ranked_commits": []})],
+        )
+
+    with patch.object(llm_analyzer, "chat", fake_chat):
+        llm_analyzer.rank_suspect_commits(diffs, alert)
+    return captured[0]
+
+
+def test_no_traceback_prompt_is_byte_identical_to_baseline():
+    assert _prompt(_FILED, _ALERT) == _GOLDEN_BASELINE_PROMPT
+    assert _prompt(_FILED, {**_ALERT, "traceback": ""}) == _GOLDEN_BASELINE_PROMPT
+
+
+def test_traceback_sits_after_alert_and_marks_only_overlapping_commits():
+    prompt = _prompt(_FILED, {**_ALERT, "traceback": _TB})
+    assert prompt.index("ALERT:") < prompt.index("<traceback>") < prompt.index("RECENT COMMITS:")
+    assert prompt.count("Touches files in the stack trace:") == 1
+    assert "Subject: config: raise pool size\nTouches files in the stack trace: app/db.py\nDiff:" in prompt
+
+
+def test_overlap_matches_whole_path_segments():
+    tb = 'File "/srv/orders/app/db.py", line 1\nFile "C:\\svc\\app\\cart.py", line 2'
+    files = ["app/db.py", "db.py", "pp/db.py", "app/cart.py", "app/orders.py"]
+    assert llm_analyzer.files_in_traceback(files, tb) == ["app/db.py", "db.py", "app/cart.py"]
+    assert llm_analyzer.files_in_traceback(files, None) == []
+
+
+def test_long_traceback_keeps_newest_frames():
+    tb = "x" * 10_000 + "\nTypeError: newest"
+    block = llm_analyzer._traceback_block(tb)
+    assert "TypeError: newest" in block and len(block) < llm_analyzer._TRACEBACK_MAX_CHARS + 200
+
+
+def test_sandbox_app_traceback_matches_its_repo_path():
+    from fastapi.testclient import TestClient
+
+    from sandbox.app import main as sandbox_app
+
+    with patch.object(sandbox_app, "_bug", return_value="db_failure"):
+        r = TestClient(sandbox_app.app, raise_server_exceptions=False).get("/api/data")
+    assert r.status_code == 500 and r.json()["error"].startswith("RuntimeError: OperationalError")
+    tb = r.json()["traceback"]
+    assert llm_analyzer.files_in_traceback(["sandbox/app/main.py", "core/db.py"], tb) == [
+        "sandbox/app/main.py"
+    ]

dashboard/src/app/incidents/[id]/page.tsx

@@ -48,6 +48,12 @@ export default function IncidentDetail() {
         <h2>Trigger</h2>
         <p>{trigger.alert_name} — {trigger.source} — {new Date(trigger.timestamp).toLocaleString()}</p>
         <pre className="error-sig">{trigger.error_signature}</pre>
+        {trigger.traceback && (
+          <details>
+            <summary>Stack trace</summary>
+            <pre>{trigger.traceback}</pre>
+          </details>
+        )}
       </div>
 
       {diagnostics && (

dashboard/src/lib/api.ts

@@ -26,6 +26,7 @@ export type Trigger = {
   alert_name: string;
   timestamp: string;
   error_signature: string;
+  traceback?: string;
 };
 
 // shape returned by core's db rows (see core/db.py)

eval/cases/auth_issuer_mesh_hosts.json

@@ -85,5 +85,43 @@
     "culprit_index": 4,
     "expected_runbook_id": "auth_failures",
     "note": "Tokens still carry iss=https://auth.internal; the mesh rename changed the expected issuer, not just a transport URL. Leeway affects exp/iat checks and the algorithm list only widens what is accepted."
+  },
+  "traceback": {
+    "frames": [
+      [
+        "app/auth.py",
+        "current_user_id",
+        "return verify_token("
+      ],
+      [
+        "app/auth.py",
+        "verify_token",
+        "return jwt.decode("
+      ],
+      [
+        "/usr/local/lib/python3.12/site-packages/jwt/api_jwt.py",
+        210,
+        "decode",
+        "decoded = self.decode_complete("
+      ],
+      [
+        "/usr/local/lib/python3.12/site-packages/jwt/api_jwt.py",
+        151,
+        "decode_complete",
+        "self._validate_claims("
+      ],
+      [
+        "/usr/local/lib/python3.12/site-packages/jwt/api_jwt.py",
+        244,
+        "_validate_claims",
+        "self._validate_iss(payload, issuer)"
+      ],
+      [
+        "/usr/local/lib/python3.12/site-packages/jwt/api_jwt.py",
+        330,
+        "_validate_iss",
+        "raise InvalidIssuerError(\"Invalid issuer\")"
+      ]
+    ]
   }
 }

eval/cases/cache_contains_recursion.json

@@ -85,5 +85,59 @@
     "culprit_index": 2,
     "expected_runbook_id": null,
     "note": "__contains__ now calls get() and get() starts with 'key not in self', so each calls the other. The earlier __contains__ read _data directly and was fine."
+  },
+  "traceback": {
+    "frames": [
+      [
+        "app/handlers/orders.py",
+        "create_order",
+        "stock = inventory.get_stock_many("
+      ],
+      [
+        "app/clients/inventory.py",
+        "get_stock_many",
+        "get_stock(sku"
+      ],
+      [
+        "app/clients/inventory.py",
+        "get_stock",
+        "cached = _stock_cache.get("
+      ],
+      [
+        "app/cache.py",
+        "get",
+        "if key not in self"
+      ],
+      [
+        "app/cache.py",
+        "__contains__",
+        "return self.get(key) is not None"
+      ],
+      [
+        "app/cache.py",
+        "get",
+        "if key not in self"
+      ],
+      [
+        "app/cache.py",
+        "__contains__",
+        "return self.get(key) is not None"
+      ],
+      [
+        "app/cache.py",
+        "get",
+        "if key not in self"
+      ],
+      [
+        "app/cache.py",
+        "__contains__",
+        "return self.get(key) is not None"
+      ],
+      [
+        "app/cache.py",
+        "get",
+        "if key not in self"
+      ]
+    ]
   }
 }