Repository navigation
Stack traces in commit ranking (95.8% top-1 on eval set v2) - #12
Merged
Merged
Conversation
An alert may now carry an optional `traceback`. When present, ranking gets the trace (newest 4000 chars) and each commit that touches a file named in a frame is marked "Touches files in the stack trace". Without a traceback the prompt is byte-identical to the Phase 9 baseline (golden-prompt test). git_client records each commit's changed files from the full patch, before the 3000-char diff truncation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The toy app logged repr(exc) as its "traceback"; it now logs and returns the real formatted trace in its 500 body. The load generator forwards the last error's real signature and trace instead of a fixed "Error rate spike detected", and chaos_cli probes the app once after injecting a bug so its alert carries the trace too (skipped when the app isn't running). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collapsible under the error signature; absent when the alert carried none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cases gain a "traceback" spec: app frames are anchored to source text and rendered with the line numbers of the post-commit fixture repo; library frames are literal. The 4 metric alerts (timeouts, OOM, idle consumer) have none. Where the raising call is ambiguous, frames take the path that does not run through the culprit's lines, so the culprit's file is in 13/20 traces. run_eval --condition traceback sends the trace; baseline and rag strip it, so their prompts are unchanged. Records note has_traceback and culprit_in_traceback, and the report splits a comparison by those groups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
72 live trials (24 cases x 3, claude-sonnet-5): 95.8% top-1 against the two existing baselines' 90.3% and 91.7%. The gain is one case, httpx_028_drops_proxies, which went from 0/9 to 3/3. Traces naming only decoy files did not pull rankings towards the decoys. Adds ADR-8 and a README line. Also records that no earlier trial hit diff truncation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Alerts can carry an optional
traceback. Commit ranking gets the trace, and each commit that touches a file named in a frame is marked. Without a trace the prompt is byte-identical to the baseline (golden-prompt test), so nothing changes for alerts that have none.llm_analyzerbuilds the trace block and finds the overlap;git_clientrecords each commit's changed files from the full patch, before truncation.tracebackadded to the contract.repr(exc)as its "traceback"; it now returns the real one. The load generator andchaos_clisend it in the alert.--condition tracebackand a report breakdown by whether the trace names the culprit's file.Result (METRICS.md, 2026-10-11)
The gain is one case that went from 0/9 to 3/3 (
httpx_028_drops_proxies). The traces were written to go through decoy code wherever the raising call was ambiguous (the culprit's file is in only 13/20), and cases whose trace named only decoys still improved.Tests
92 passing locally (black, flake8, pytest); dashboard lint and jest pass.
🤖 Generated with Claude Code