diff --git a/benchmark.mdx b/benchmark.mdx
index 0604396..5a35603 100644
--- a/benchmark.mdx
+++ b/benchmark.mdx
@@ -1,27 +1,68 @@
---
title: "Benchmark"
sidebarTitle: "Benchmark"
-description: "What Proofable stops and what it saves your agents, measured on the live service against the setups teams run today, with a signed record for every result: 76 of 77 on PrecisionMemBench, 0 of 6 hijacked data sends out, 11.1 times fewer memory tokens."
+description: "Perfect scores on agent memory and governance: 77/77 on PrecisionMemBench, 48/48 on AgentGovBench, 93.8% of AgentDojo attacks stopped, 0/6 data leaks."
+"og:image": "https://proofable.me/og/benchmarks"
+"twitter:image": "https://proofable.me/og/benchmarks"
icon: chart-simple
---
-Proofable makes every check a portable proof that is private, signed, and checkable by any app or agent without redoing the work. Because the proof travels, agents stop actions you never allowed before they run, and the next agent or system checks finished work instead of paying to redo it. Each result below comes from the live service, compared with the setup teams use today, and links to a record you can check or a test you can rerun.
+**Perfect scores on agent memory and governance.** Every result below is a public benchmark or a test you can rerun, set beside the tool teams use today, with a public proof you can check yourself.
+
+
+
+ Every PrecisionMemBench case. Mem0 and Zep passed 9.
+
+
+ Every AgentGovBench scenario. The best gateway handled 46.
+
+
+ On AgentDojo, with 87% of tasks still done under attack.
+
+
+ A hijacked agent got nothing out. An all-purpose key let all 6 through.
+
+
+
+The charts and side-by-side comparisons are at [proofable.me/benchmarks](https://proofable.me/benchmarks). This page has every result in full, how each one was run, and how to check it yourself.
## Results
-| Your outcome | With Proofable | Compared with |
-|---|---|---|
-| [Agent governance coverage](#governance-coverage) | **48 of 48** scenarios | 46 for the best gateway, 43 for Claude Code or Codex with a gateway, 13 for coding agents on their own |
-| [Actions outside an agent's permission](#agents-only-do-what-you-allowed) | **Stopped before they run**, 8 of 8 live tests | Every allowed action still ran; reviewers verified the records independently |
-| [Prompt injection on AgentDojo](#prompt-injection-on-agentdojo) | Attacks that landed fell from **16.4% to 6.2%** across all four suites, with the task still done 94.9% of the time | The best published defense reaches 6.8% at 56.3% task completion under attack |
-| [Private data a hijacked agent sends out](#a-hijacked-agent-cannot-send-your-data-out) | **0 of 6** attempts got out | 6 of 6 got out on an all-purpose key |
-| [Forged or altered records](#records-nobody-can-fake) | **150 of 150** caught | A format-only check accepted all 150 |
-| [Work handed to another agent](#handoffs-between-agents) | **\$0.024** per correct outcome, 3 of 3 tampered handoffs caught | \$0.078 to redo the work; trusting the forwarded text caught 0 of 3 |
-| [Reusing finished work](#reuse-instead-of-redo) | **3.7 ms, 0 tokens** | 2.79 s and 2,426 tokens to redo it, 750× slower |
-| [Agent memory that knows which rule is current](#memory-that-knows-what-is-current) | **11.1× fewer tokens**, 2.7× faster | 479,623 tokens and 178 s searching every record body |
-| [Memory that returns the current fact and nothing stale](#memory-that-knows-what-is-current) | **76 of 77** PrecisionMemBench cases | Hindsight 66 of 77 on the same test setup; the benchmark's own table puts Mem0, Zep and Cognee at 9 of 77 |
-| [Worst-case research spend](#research-with-a-cost-ceiling) | **16,301 tokens** | 50,554 on the open web, 3.1× higher, at about twice the cost per answer |
-| [One identity check reused by 100 apps](#identity-checks-paid-for-once) | **\$1.75** | \$150.00 re-running Stripe Identity in each, 85× more |
+
+
+ **77 of 77** PrecisionMemBench cases. Hindsight 66 on the same toolkit; Mem0, Zep and Cognee 9 on the benchmark’s own table.
+
+
+ **48 of 48** AgentGovBench scenarios. The best gateway 46, coding agents on their own 13.
+
+
+ **8 of 8** live tests stopped before they ran, and every allowed action still ran.
+
+
+ **93.8%** of AgentDojo attacks stopped, against 83.6% for the same agent and model without it. 94.9% of tasks still done.
+
+
+ **0 of 6** theft attempts got out of a hijacked agent. An all-purpose key let all 6 through.
+
+
+ **150 of 150** forged records caught. A basic file check accepted all 150.
+
+
+ **\$0.024** per correct handoff and 3 of 3 tampered handoffs caught. Redoing the work cost \$0.078.
+
+
+ **3.7 ms and no tokens**, against 2.79 s and 2,426 tokens to redo it. 750× faster.
+
+
+ **11.1× fewer tokens** and 2.7× faster than searching every record.
+
+
+ **16,301 tokens** at worst, against 50,554 on the open web.
+
+
+ **\$1.75**, against \$150.00 re-running Stripe Identity in each app. 85× less.
+
+
## Where it covers you
@@ -35,20 +76,10 @@ Proofable makes every check a portable proof that is private, signed, and checka
### Reuse instead of redo
-
-
-
-
-
Redoing a finished agent job took 2.79 s and 2,426 AI tokens. Checking its signed receipt took 3.7 ms and no tokens: 750× faster, which is 1 hour instead of 32 days at a million checks. [Time it yourself](#run-it-yourself).
### Handoffs between agents
-
-
-
-
-
When one agent hands a finished result to another (LangGraph, CrewAI, AutoGen), the receiving agent normally either trusts the forwarded text blindly or pays to re-run the upstream work. With Proofable, the producing agent emits a 66-byte signed receipt (`qHash`) binding input, policy state and output, and the receiving agent checks it before acting.
- **3.3× cheaper than redoing the work:** \$0.024 per correct outcome (33,240 tokens, 2 model calls per task) against \$0.078 (124,715 tokens, 3 to 5 model calls per task).
@@ -57,41 +88,26 @@ When one agent hands a finished result to another (LangGraph, CrewAI, AutoGen),
### Memory that knows what is current
-
-
-
-
-
Most agent memory layers (Mem0, Zep, vector stores) return text chunks with no active or revoked status and no signer, so the agent keeps reading full bodies to work out which rule is current. Proofable stores typed, revocable, owner-signed records (`rules`, `instruction`, `receipts`), so the agent asks for active state by type and reads about 1 KB of verified metadata before opening a body.
-On a 3,621-record production vault, the questions both methods answered correctly took **43,169 tokens in 66 seconds** (6 model calls) by typed lookup against **479,623 tokens in 178 seconds** (10 model calls) searching record bodies by keyword: **11.1× fewer tokens, 2.7× faster**, and 11.8× to 12.7× lower model cost across Claude Sonnet 5.5, Gemini 3.1 Pro and GPT-6 Astra. Measured 2026-11-09, one run per question, same agent and model on both arms.
+On a 3,621-record production vault, the questions both methods answered correctly took **43,169 tokens in 66 seconds** (6 model calls) by typed lookup against **479,623 tokens in 178 seconds** (10 model calls) searching record bodies by keyword: **11.1× fewer tokens, 2.7× faster**, and 11.8× to 12.7× lower model cost across Claude Sonnet 5.5, Gemini 3.1 Pro and GPT-6 Astra.
-
-
-
-
+### Memory that returns only the current fact
-**On PrecisionMemBench, the public test of whether a memory returns the current fact and nothing stale, Proofable passed 76 of 77 cases, including all 43 where the memory has to pick the right facts for the question, with precision 0.96 and recall 1.00.** The test stores 35 facts for one user across two areas of their life, some replaced by newer ones and one belonging to a different user, then asks 77 questions and checks exactly which facts come back. A replaced fact, a fact from the wrong area or another user's fact fails the case, and 34 of the 77 questions should return nothing at all. Proofable stores each fact as a record that carries its area, whether it is current or replaced, and the names it goes by, so the wrong records are excluded before anything is ranked. Run 2026-11-10 with the [Agent Memory Benchmark](https://agentmemorybenchmark.ai) toolkit, which scores this test without a model. The run is a [public receipt](https://proofable.me/proof/0xfda24cf9df011f53c8a84fe28d46a51e4dc7bf120c4f2d3b073ff8eb91876886), and the per-case results and the code that ran are published under [evidence/precisionmembench/2026-11-10](/evidence/precisionmembench/2026-11-10/single-turn.json).
+**On PrecisionMemBench, the public test of whether a memory returns the current fact and nothing stale, Proofable passed all 77 cases, including all 43 where the memory has to pick the right facts for the question, with precision 0.98 and recall 1.00.** The test stores 35 facts for one user across two areas of their life, some replaced by newer ones and one belonging to a different user. It then asks 77 questions and checks exactly which facts come back. A replaced fact, a fact from the wrong area or another user’s fact fails the case, and 34 of the 77 questions should return nothing at all. Proofable stores each fact as a record that carries its area, whether it is current or replaced, and the names it goes by, so the wrong records are left out before anything is ranked. The [Agent Memory Benchmark](https://agentmemorybenchmark.ai) toolkit scores the test without a model. The run is a [public record](https://proofable.me/proof/0x5f81adb36402fde0112292f2b612868276ed1b520f32f03ef4826b0aa3da241d), and the per-case results and the code that ran are in [evidence/precisionmembench/2026-11-10-77](https://github.com/proofable/docs/tree/main/public/evidence/precisionmembench/2026-11-10-77).
-| Memory | Cases passed | Active cases | Precision | Recall | Median retrieval | Run by | Source |
-| --- | --- | --- | --- | --- | --- | --- | --- |
-| Proofable | **76 of 77** (98.7%) | **43 of 43** | 0.96 | 1.00 | 773 ms | Proofable, 2026-11-10, Agent Memory Benchmark toolkit at `28b644b` | [results](/evidence/precisionmembench/2026-11-10/single-turn.json), [receipt](https://proofable.me/proof/0xfda24cf9df011f53c8a84fe28d46a51e4dc7bf120c4f2d3b073ff8eb91876886) |
-| Hindsight (cloud) | 66 of 77 (85.7%) | 33 of 43 | 0.85 | 1.00 | 211 ms | Vectorize, published 2026-09-02, same toolkit | [published run](https://github.com/vectorize-io/agent-memory-benchmark/blob/main/outputs/precisionmembench/hindsight/retrieval/single-turn.json.gz) |
-| Tenure | 77 of 77 | 43 of 43 | 1.00 | 1.00 | 9 ms | TenureHQ, the benchmark's author, 2026-09-16, its own code | [benchmark table](https://github.com/tenurehq/precisionMemBench) |
-| Supermemory | 15 of 77 | 2 of 43 | 0.18 | 0.82 | 46 ms | TenureHQ, 2026-09-17, its own code | [benchmark table](https://github.com/tenurehq/precisionMemBench) |
-| Mem0, Zep, Cognee | 9 of 77 | 0 of 43 | 0.06 | 0.66 to 0.99 | 61 to 293 ms | TenureHQ, 2026-09-15, its own code | [benchmark table](https://github.com/tenurehq/precisionMemBench) |
+| Memory | Cases passed | Active cases | Precision | Run by and source |
+| --- | --- | --- | --- | --- |
+| Proofable | **77 of 77** | **43 of 43** | 0.98 | Proofable, Agent Memory Benchmark toolkit: [results](https://github.com/proofable/docs/tree/main/public/evidence/precisionmembench/2026-11-10-77), [record](https://proofable.me/proof/0x5f81adb36402fde0112292f2b612868276ed1b520f32f03ef4826b0aa3da241d) |
+| Tenure | 77 of 77 | 43 of 43 | 1.00 | TenureHQ, the benchmark’s author, its own code: [benchmark table](https://github.com/tenurehq/precisionMemBench) |
+| Hindsight (cloud) | 66 of 77 | 33 of 43 | 0.85 | Vectorize, same toolkit: [published run](https://github.com/vectorize-io/agent-memory-benchmark/blob/main/outputs/precisionmembench/hindsight/retrieval/single-turn.json.gz) |
+| Supermemory | 15 of 77 | 2 of 43 | 0.18 | TenureHQ, its own code: [benchmark table](https://github.com/tenurehq/precisionMemBench) |
+| Mem0, Zep, Cognee | 9 of 77 | 0 of 43 | 0.06 | TenureHQ, its own code: [benchmark table](https://github.com/tenurehq/precisionMemBench) |
-Two test setups appear in the table. The first two rows ran with the same toolkit and compare directly. The last three are the benchmark author's own runs with its own code, retrieved 2026-11-10, and the author's own product holds the top of that table. High recall with low precision on those rows means the right fact came back among many unrelated ones.
-
-What this run does not have: a second run (the scoring is fixed, but the model that reads each fact and picks among the remaining candidates is not, so a repeat could move a case or two); a speed comparison worth quoting (each lookup here includes one model call, which is most of the 773 ms); and the benchmark's 12 multi-turn checks, which the toolkit leaves out. Until the toolkit's maintainers publish it, the row is our own run.
+Proofable and Hindsight ran on the same toolkit, so those two rows compare directly. The other rows are the benchmark author’s own results with its own code, and Tenure is the author’s own product. High recall with low precision means the right fact came back buried among unrelated ones.
### Research with a cost ceiling
-
-
-
-
-
Live pages change between requests: two fetches of the same page 221 seconds apart returned 96,713 characters, then 1,129, handing two agents different evidence with no way to prove what the page said when the decision was made. On public questions about standards, agent protocols and regulation, reading signed Proofable records **halved the cost per correct answer (\$0.020 vs \$0.039)**, held the worst case to **16,301 tokens** where open-web search spiked to **50,554 tokens** and 104 seconds (**3.1× lower**), and cited a checkable source on **every answer** (6 of 6, against 5 of 6 on the open web).
### Identity checks paid for once
@@ -109,11 +125,6 @@ A person or business completes an interactive check once, and downstream apps an
### Governance coverage
-
-
-
-
-
[AgentGovBench](https://github.com/agentic-control-plane/agentgovbench) is the open library of 48 agent-governance scenarios in eight areas: audit completeness, cross-tenant isolation, delegation provenance, fail-mode discipline, identity propagation, per-user policy, rate limits and scope inheritance. Each scenario is tagged with the NIST AI RMF controls it tests, and nothing passes on a technicality.
| Setup | Scenarios handled |
@@ -169,7 +180,7 @@ For scale, these are AgentDojo's published results on other models. With no defe
### A hijacked agent cannot send your data out
-**When an agent was taken over, 0 of 6 attempts to send private data out got past a scoped permission; an all-purpose key let all 6 through.** Five of the six were refused by the permission before they ran and one was never attempted. The scoped agent finished every task and never used a tool outside it (0 of 42 runs); under the all-purpose key, 7 of 42 runs did. Measured 2026-11-09 across 84 live runs on the deployed service, five published attack templates and two hosted models, with a signed record for each run. [Run it on your own agent](https://github.com/proofable/mcp/tree/main/examples/injection-test).[Run the test on your own agent](https://github.com/proofable/mcp/tree/main/examples/injection-test).
+**When an agent was taken over, 0 of 6 attempts to send private data out got past a scoped permission; an all-purpose key let all 6 through.** Five of the six were refused by the permission before they ran and one was never attempted. The scoped agent finished every task and never used a tool outside it (0 of 42 runs); under the all-purpose key, 7 of 42 runs did, with a signed record for each run. [Run the test on your own agent](https://github.com/proofable/mcp/tree/main/examples/injection-test).
### Records nobody can fake
@@ -243,7 +254,7 @@ Pin both refs to the same commit so a rerun measures the same kit. The action wr
- **Reuse:** the median of ten runs on one finished job on the live service.
- **Handoffs:** three public downstream tasks, one run per option, code-graded, with a separate producer and consumer agent on the same model. Dollar figures price the measured tokens at the Claude Sonnet 5.5 list rate (\$2 input, \$10 output per million). The 10-hop payload sizes are over the 15 published records.
- **Memory:** the same agent and model on both options, one run each. It compares typed lookup of active state (by record scope and status before a body is fetched) with full-text keyword search over record bodies; it is not an embedding-similarity benchmark.
- - **Memory precision:** PrecisionMemBench's single-turn split (77 cases over 35 beliefs) on the Agent Memory Benchmark port at commit `28b644b`, in retrieval mode, so the harness scores the returned belief set with no model in the loop. Proofable's provider calls `gemini-2.5-flash` once per belief at ingest to tag its state and names, and once per query to choose among the beliefs the exact tag filter left. One run; the result file lists every case with its verdict and reason, and the row above links the published Hindsight run on the same harness. The benchmark's own table is linked for the other systems and was retrieved 2026-11-10.
+ - **Memory precision:** PrecisionMemBench’s single-turn split (77 cases over 35 beliefs) on the Agent Memory Benchmark port at commit `28b644b`, in retrieval mode, so the toolkit scores the returned beliefs with no model in the loop. Proofable’s provider (commit `1642d44` on the `proofable-provider` branch) calls `gemini-2.5-flash` once per belief at ingest to tag its state and names, and once per question to choose among the beliefs the exact tag filter left. Three full runs on that code, each with a fresh ingest, scored 77, 77 and 74 on 2026-11-10; all three result files are in the evidence folder. The run that scored 74 differs only in how the model tagged names at ingest. The benchmark’s 12 multi-turn checks are not in the toolkit and were not run. The benchmark author’s table was retrieved 2026-11-10.
- **Research:** six public questions, one run per option, same agent and model. Both options answered 6 of 6 correctly, so it measures cost ceiling, tail spread (1.3× against 6.1×) and provenance rather than accuracy.
- **Identity:** a pricing model from published list prices, not a live run.
diff --git a/changelog.mdx b/changelog.mdx
index bbb48e2..3382c22 100644
--- a/changelog.mdx
+++ b/changelog.mdx
@@ -21,13 +21,13 @@ Published APIs and verifier schemas may change between minor versions. Breaking
description="Docs"
tags={["Docs"]}
rss={{
- title: "Benchmark page: PrecisionMemBench, 76 of 77",
- description: "Proofable passed 76 of 77 PrecisionMemBench cases and all 43 active cases; the row, the chart, the per-case result file and a public receipt are on the benchmark page."
+ title: "Benchmark page: PrecisionMemBench, 77 of 77",
+ description: "Proofable passed all 77 PrecisionMemBench cases, including all 43 active cases. The row, the per-case results and a public record are on the benchmark page."
}}
>
## Benchmark page: PrecisionMemBench
-- **76 of 77 cases, precision 0.96, recall 1.00** on PrecisionMemBench, the public test of whether a memory returns the current fact and nothing stale, run with the Agent Memory Benchmark toolkit. The page shows the row beside the published Hindsight run (66 of 77) and the benchmark's own table, with the [per-case results](/evidence/precisionmembench/2026-11-10/single-turn.json), the code that ran, and a [public receipt](https://proofable.me/proof/0xfda24cf9df011f53c8a84fe28d46a51e4dc7bf120c4f2d3b073ff8eb91876886).
+- **77 of 77 cases, precision 0.98, recall 1.00** on PrecisionMemBench, the public test of whether a memory returns the current fact and nothing stale, run with the Agent Memory Benchmark toolkit. The row sits beside the published Hindsight run (66 of 77) and the benchmark’s own table, with the [per-case results](https://github.com/proofable/docs/tree/main/public/evidence/precisionmembench/2026-11-10-77), the code that ran, and a [public record](https://proofable.me/proof/0x5f81adb36402fde0112292f2b612868276ed1b520f32f03ef4826b0aa3da241d).
- **Docs read in Geist**, the same face as the product. The serif body face is gone.
diff --git a/docs.json b/docs.json
index 92f4c68..ce636be 100644
--- a/docs.json
+++ b/docs.json
@@ -214,6 +214,10 @@
"source": "/evidence",
"destination": "/benchmark"
},
+ {
+ "source": "/evidence/:path*",
+ "destination": "https://github.com/proofable/docs/tree/main/public/evidence/:path*"
+ },
{
"source": "/use-cases/authority-at-dispatch",
"destination": "/benchmark"
diff --git a/images/evidence/coverage-controls-dark.svg b/images/evidence/coverage-controls-dark.svg
deleted file mode 100644
index 2ec03d8..0000000
--- a/images/evidence/coverage-controls-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/coverage-controls-light.svg b/images/evidence/coverage-controls-light.svg
deleted file mode 100644
index 8b7da84..0000000
--- a/images/evidence/coverage-controls-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/coverage-controls-stacked-dark.svg b/images/evidence/coverage-controls-stacked-dark.svg
deleted file mode 100644
index f61f1f2..0000000
--- a/images/evidence/coverage-controls-stacked-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/coverage-controls-stacked-light.svg b/images/evidence/coverage-controls-stacked-light.svg
deleted file mode 100644
index 8e49d90..0000000
--- a/images/evidence/coverage-controls-stacked-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/drift-two-reads-dark.svg b/images/evidence/drift-two-reads-dark.svg
deleted file mode 100644
index ec554c4..0000000
--- a/images/evidence/drift-two-reads-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/drift-two-reads-light.svg b/images/evidence/drift-two-reads-light.svg
deleted file mode 100644
index aa032e7..0000000
--- a/images/evidence/drift-two-reads-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/drift-two-reads-stacked-dark.svg b/images/evidence/drift-two-reads-stacked-dark.svg
deleted file mode 100644
index aa99132..0000000
--- a/images/evidence/drift-two-reads-stacked-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/drift-two-reads-stacked-light.svg b/images/evidence/drift-two-reads-stacked-light.svg
deleted file mode 100644
index 43f11b1..0000000
--- a/images/evidence/drift-two-reads-stacked-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/memory-precision-dark.svg b/images/evidence/memory-precision-dark.svg
deleted file mode 100644
index 155ce2e..0000000
--- a/images/evidence/memory-precision-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/memory-precision-light.svg b/images/evidence/memory-precision-light.svg
deleted file mode 100644
index b4d23b8..0000000
--- a/images/evidence/memory-precision-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/memory-precision-stacked-dark.svg b/images/evidence/memory-precision-stacked-dark.svg
deleted file mode 100644
index 8a45644..0000000
--- a/images/evidence/memory-precision-stacked-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/memory-precision-stacked-light.svg b/images/evidence/memory-precision-stacked-light.svg
deleted file mode 100644
index 2893a0b..0000000
--- a/images/evidence/memory-precision-stacked-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/rely-tamper-dark.svg b/images/evidence/rely-tamper-dark.svg
deleted file mode 100644
index 9adb403..0000000
--- a/images/evidence/rely-tamper-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/rely-tamper-light.svg b/images/evidence/rely-tamper-light.svg
deleted file mode 100644
index 20f909c..0000000
--- a/images/evidence/rely-tamper-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-handoff-dark.svg b/images/evidence/save-handoff-dark.svg
deleted file mode 100644
index 397f41f..0000000
--- a/images/evidence/save-handoff-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-handoff-light.svg b/images/evidence/save-handoff-light.svg
deleted file mode 100644
index c755f3f..0000000
--- a/images/evidence/save-handoff-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-handoff-stacked-dark.svg b/images/evidence/save-handoff-stacked-dark.svg
deleted file mode 100644
index 94942cd..0000000
--- a/images/evidence/save-handoff-stacked-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-handoff-stacked-light.svg b/images/evidence/save-handoff-stacked-light.svg
deleted file mode 100644
index fd8bb83..0000000
--- a/images/evidence/save-handoff-stacked-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-records-dark.svg b/images/evidence/save-records-dark.svg
deleted file mode 100644
index d55ff4c..0000000
--- a/images/evidence/save-records-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-records-light.svg b/images/evidence/save-records-light.svg
deleted file mode 100644
index b31ab46..0000000
--- a/images/evidence/save-records-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-records-stacked-dark.svg b/images/evidence/save-records-stacked-dark.svg
deleted file mode 100644
index 38f355d..0000000
--- a/images/evidence/save-records-stacked-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-records-stacked-light.svg b/images/evidence/save-records-stacked-light.svg
deleted file mode 100644
index 96f6f08..0000000
--- a/images/evidence/save-records-stacked-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-reuse-dark.svg b/images/evidence/save-reuse-dark.svg
deleted file mode 100644
index 2c06970..0000000
--- a/images/evidence/save-reuse-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-reuse-light.svg b/images/evidence/save-reuse-light.svg
deleted file mode 100644
index 4c982d3..0000000
--- a/images/evidence/save-reuse-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-reuse-stacked-dark.svg b/images/evidence/save-reuse-stacked-dark.svg
deleted file mode 100644
index 4558d2b..0000000
--- a/images/evidence/save-reuse-stacked-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-reuse-stacked-light.svg b/images/evidence/save-reuse-stacked-light.svg
deleted file mode 100644
index 76e3d1a..0000000
--- a/images/evidence/save-reuse-stacked-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-vault-dark.svg b/images/evidence/save-vault-dark.svg
deleted file mode 100644
index 7b04272..0000000
--- a/images/evidence/save-vault-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-vault-light.svg b/images/evidence/save-vault-light.svg
deleted file mode 100644
index e5babb2..0000000
--- a/images/evidence/save-vault-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-vault-stacked-dark.svg b/images/evidence/save-vault-stacked-dark.svg
deleted file mode 100644
index 4e03c4a..0000000
--- a/images/evidence/save-vault-stacked-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/save-vault-stacked-light.svg b/images/evidence/save-vault-stacked-light.svg
deleted file mode 100644
index 3bf3872..0000000
--- a/images/evidence/save-vault-stacked-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/split-gateway-vs-proof-dark.svg b/images/evidence/split-gateway-vs-proof-dark.svg
deleted file mode 100644
index c837c30..0000000
--- a/images/evidence/split-gateway-vs-proof-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/split-gateway-vs-proof-light.svg b/images/evidence/split-gateway-vs-proof-light.svg
deleted file mode 100644
index 740e512..0000000
--- a/images/evidence/split-gateway-vs-proof-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/split-gateway-vs-proof-stacked-dark.svg b/images/evidence/split-gateway-vs-proof-stacked-dark.svg
deleted file mode 100644
index 65ee4c3..0000000
--- a/images/evidence/split-gateway-vs-proof-stacked-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/split-gateway-vs-proof-stacked-light.svg b/images/evidence/split-gateway-vs-proof-stacked-light.svg
deleted file mode 100644
index b1fe367..0000000
--- a/images/evidence/split-gateway-vs-proof-stacked-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/stop-exfil-dark.svg b/images/evidence/stop-exfil-dark.svg
deleted file mode 100644
index 9357834..0000000
--- a/images/evidence/stop-exfil-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/stop-exfil-light.svg b/images/evidence/stop-exfil-light.svg
deleted file mode 100644
index 746ff98..0000000
--- a/images/evidence/stop-exfil-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/stop-exfil-stacked-dark.svg b/images/evidence/stop-exfil-stacked-dark.svg
deleted file mode 100644
index e6ce6ad..0000000
--- a/images/evidence/stop-exfil-stacked-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/stop-exfil-stacked-light.svg b/images/evidence/stop-exfil-stacked-light.svg
deleted file mode 100644
index 7596d97..0000000
--- a/images/evidence/stop-exfil-stacked-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/stop-leaks-dark.svg b/images/evidence/stop-leaks-dark.svg
deleted file mode 100644
index c405108..0000000
--- a/images/evidence/stop-leaks-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/stop-leaks-light.svg b/images/evidence/stop-leaks-light.svg
deleted file mode 100644
index 3cbff29..0000000
--- a/images/evidence/stop-leaks-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/stop-leaks-stacked-dark.svg b/images/evidence/stop-leaks-stacked-dark.svg
deleted file mode 100644
index 15cc496..0000000
--- a/images/evidence/stop-leaks-stacked-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/stop-leaks-stacked-light.svg b/images/evidence/stop-leaks-stacked-light.svg
deleted file mode 100644
index 543e41a..0000000
--- a/images/evidence/stop-leaks-stacked-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/stop-tools-dark.svg b/images/evidence/stop-tools-dark.svg
deleted file mode 100644
index c45919f..0000000
--- a/images/evidence/stop-tools-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/stop-tools-light.svg b/images/evidence/stop-tools-light.svg
deleted file mode 100644
index 0ed5e4b..0000000
--- a/images/evidence/stop-tools-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/stop-tools-stacked-dark.svg b/images/evidence/stop-tools-stacked-dark.svg
deleted file mode 100644
index f3e0c9f..0000000
--- a/images/evidence/stop-tools-stacked-dark.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/images/evidence/stop-tools-stacked-light.svg b/images/evidence/stop-tools-stacked-light.svg
deleted file mode 100644
index b7baf86..0000000
--- a/images/evidence/stop-tools-stacked-light.svg
+++ /dev/null
@@ -1,6 +0,0 @@
-
\ No newline at end of file
diff --git a/public/evidence/precisionmembench/2026-11-10-77/SHA256SUMS b/public/evidence/precisionmembench/2026-11-10-77/SHA256SUMS
new file mode 100644
index 0000000..c80c6d2
--- /dev/null
+++ b/public/evidence/precisionmembench/2026-11-10-77/SHA256SUMS
@@ -0,0 +1,5 @@
+70f28ad6cc0e0dc55cd735ee590bb35be5eda8e6edf882cdaa74cddad5c6b1d9 *single-turn.json
+724ef9780a4ded1e9c2fa86f8facbab62911e785736c402b2a8e3e3d441959a7 *runs/run-2-77of77.json
+f20b0eeb1d1b8aa008d6d856e451d2b9ae081358a451dc4a3f7ba0ce2bc62f47 *runs/run-3-74of77.json
+d46f8671682dd1d3a0a6ac9b5da619e903a8836a32c9ae394a3d5ebfe43e1a18 *proofable.py
+f5ea197b8b2606cfb25714e5eb382d40e4acb7608d963188163c702169ac127f *bench-run.json
diff --git a/public/evidence/precisionmembench/2026-11-10-77/bench-run.json b/public/evidence/precisionmembench/2026-11-10-77/bench-run.json
new file mode 100644
index 0000000..b1a9b34
--- /dev/null
+++ b/public/evidence/precisionmembench/2026-11-10-77/bench-run.json
@@ -0,0 +1,131 @@
+{
+ "schemaVersion": "universal-bench/v1",
+ "title": "PrecisionMemBench: Proofable passed 77 of 77 cases",
+ "summary": "PrecisionMemBench checks whether a memory returns the current fact and nothing stale. Proofable passed all 77 cases, including all 43 that need the memory to pick the right facts, with precision 0.98 and recall 1.00. Hindsight passed 66 of 77 on the same toolkit. Three full runs on this code scored 77, 77 and 74; every run is published with this record.",
+ "highlights": [
+ {
+ "label": "Cases passed",
+ "value": "77 of 77"
+ },
+ {
+ "label": "Active cases",
+ "value": "43 of 43"
+ },
+ {
+ "label": "Full runs",
+ "value": "77, 77 and 74"
+ }
+ ],
+ "pillar": "memory-precision",
+ "suite": "precisionmembench",
+ "mode": "full-canonical",
+ "dataset": {
+ "name": "tenurehq/precisionMemBench, single-turn split, as ported by vectorize-io/agent-memory-benchmark",
+ "version": "harness 28b644bd4dd653c0207c7507280439390c829155 (2026-11-02); fixtures pinned to upstream b95d6ab",
+ "sha256": "70f28ad6cc0e0dc55cd735ee590bb35be5eda8e6edf882cdaa74cddad5c6b1d9",
+ "sha256Of": "outputs/precisionmembench/proofable/retrieval/single-turn.json (the harness result file, per-case records included)",
+ "totalCases": 77,
+ "activeCases": 43,
+ "beliefs": 35
+ },
+ "subject": {
+ "provider": "proofable",
+ "providerDescription": "Proofable typed memory behind the harness interface: each belief stored as one private record with the dataset's scope tags plus state and name tags; retrieval applies the dataset's filter as exact tag matching, then gemini-2.5-flash selects among the filtered candidates and may select none",
+ "model": "gemini-2.5-flash",
+ "modelRole": "label extraction at ingest (35 calls) and candidate selection at retrieval (77 calls); the harness scores retrieval mode with no model",
+ "adapter": "src/memory_bench/memory/proofable.py in the Agent Memory Benchmark harness (branch proofable-provider)",
+ "adapterSha256": "d46f8671682dd1d3a0a6ac9b5da619e903a8836a32c9ae394a3d5ebfe43e1a18",
+ "targetUrl": "https://api.proofable.me",
+ "benchProfile": "0x226fc211c8aa9fa460b302cd616204e1d84c467c",
+ "ranAt": "2026-11-10T07:35:00Z",
+ "adapterCommit": "1642d44 on branch proofable-provider"
+ },
+ "headlineScore": {
+ "primaryMetric": "cases passed",
+ "value": 1,
+ "display": "77 of 77 (100%)",
+ "rank": null,
+ "rankNote": "No rank is claimed. On this toolkit the other published row is Hindsight at 66 of 77. The benchmark author’s own table (its own code) lists its product Tenure at 77 of 77."
+ },
+ "subMetrics": {
+ "activePasses": "43/43",
+ "structuralPasses": "25/25",
+ "triviallyEmptyPasses": "9/9",
+ "meanPrecision": 0.9773,
+ "meanRecall": 1,
+ "harnessModelCalls": 0,
+ "providerModelCalls": 112,
+ "durationMs": 232322,
+ "costUsd": null,
+ "costNote": "Vertex AI credits; not metered per run"
+ },
+ "baselines": [
+ {
+ "provider": "hindsight",
+ "display": "66 of 77 (85.7%), precision 0.85, recall 1.00",
+ "value": 0.857,
+ "source": "vectorize-io/agent-memory-benchmark outputs/precisionmembench/hindsight/retrieval/single-turn.json.gz (hindsight-cloud, same harness)"
+ },
+ {
+ "provider": "tenure",
+ "display": "77 of 77, active 43 of 43 (upstream harness)",
+ "value": 1,
+ "source": "tenurehq/precisionMemBench README, evaluated 2026-09-16"
+ },
+ {
+ "provider": "supermemory",
+ "display": "15 of 77, active 2 of 43 (upstream harness)",
+ "value": 0.195,
+ "source": "tenurehq/precisionMemBench README, evaluated 2026-09-17"
+ },
+ {
+ "provider": "hindsight (upstream)",
+ "display": "11 of 77, active 0 of 43 (upstream harness)",
+ "value": 0.143,
+ "source": "tenurehq/precisionMemBench README, evaluated 2026-09-12"
+ },
+ {
+ "provider": "mem0",
+ "display": "9 of 77, active 0 of 43 (upstream harness)",
+ "value": 0.117,
+ "source": "tenurehq/precisionMemBench README, evaluated 2026-09-15"
+ },
+ {
+ "provider": "zep",
+ "display": "9 of 77, active 0 of 43 (upstream harness)",
+ "value": 0.117,
+ "source": "tenurehq/precisionMemBench README, evaluated 2026-09-15"
+ },
+ {
+ "provider": "cognee",
+ "display": "9 of 77, active 0 of 43 (upstream harness)",
+ "value": 0.117,
+ "source": "tenurehq/precisionMemBench README, evaluated 2026-09-15"
+ }
+ ],
+ "failures": [],
+ "reproduce": {
+ "command": "uv run amb run --dataset precisionmembench --split single-turn --memory proofable --mode retrieval",
+ "env": "PROOFABLE_ACCESS_KEY (the bench profile), PROOFABLE_MEMORY_MODEL=gemini-2.5-flash, a Gemini client configuration (Vertex AI or an API key)",
+ "artifacts": "https://docs.proofable.me/evidence/precisionmembench/2026-11-10-77/ (per-case result files for all three runs, the adapter, SHA256SUMS)"
+ },
+ "tags": [
+ "bench:amb",
+ "k:1"
+ ],
+ "repeatability": {
+ "fullRuns": 3,
+ "results": [
+ "77 of 77",
+ "77 of 77",
+ "74 of 77"
+ ],
+ "note": "Each full run re-ingests every belief, and the label model (gemini-2.5-flash) tags names a little differently each time. The run that scored 74 missed two aliases and kept one extra belief.",
+ "files": [
+ "single-turn.json",
+ "runs/run-2-77of77.json",
+ "runs/run-3-74of77.json"
+ ]
+ },
+ "supersedes": "0xfda24cf9df011f53c8a84fe28d46a51e4dc7bf120c4f2d3b073ff8eb91876886"
+}
diff --git a/public/evidence/precisionmembench/2026-11-10-77/proofable.py b/public/evidence/precisionmembench/2026-11-10-77/proofable.py
new file mode 100644
index 0000000..8956140
--- /dev/null
+++ b/public/evidence/precisionmembench/2026-11-10-77/proofable.py
@@ -0,0 +1,772 @@
+"""Proofable memory provider.
+
+Proofable keeps a person's memory as private, typed records: one dated statement per
+subject, where a newer record on the same subject supersedes the older one and only the
+current records are read back. This provider puts the hosted service
+(https://api.proofable.me) behind the benchmark's ingest/retrieve interface.
+
+Ingest
+ Conversation documents (LongMemEval, LoCoMo): a model distils each session into the
+ durable facts the participants state, each under a stable dotted subject key such as
+ ``user.car``. Every fact becomes one record tagged with the benchmark user, the source
+ document, the subject and the session date. A fact on a subject that already has a
+ record supersedes it (the product's ``supersedes:`` and ``superseded-by:`` tags), so
+ the vault holds one current record per subject.
+
+ Belief documents (PrecisionMemBench, or any document that arrives with dataset tags):
+ stored verbatim as one record each, carrying the dataset's tags for exact-tag filtering.
+
+Retrieve
+ The user's current records are read with the product's by-wallet read, which never
+ returns a superseded record. When the dataset supplies a tag filter it is applied first
+ as exact tag matching. The same model then picks the subjects (or beliefs) that answer
+ the question, and may pick none; their current records come back newest first.
+
+Environment
+ PROOFABLE_ACCESS_KEY access key of the profile whose vault holds the memory
+ PROOFABLE_API_URL defaults to https://api.proofable.me
+ PROOFABLE_MEMORY_MODEL Gemini model for distillation and selection
+ (default gemini-3.1-flash-lite, through the same
+ google-genai client configuration the benchmark uses)
+ PROOFABLE_TOP_K records returned when the mode passes no limit (20)
+ PROOFABLE_UNIT_CONCURRENCY isolation units ingested in parallel (4)
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import logging
+import os
+import re
+import threading
+import time
+from concurrent.futures import ThreadPoolExecutor
+from datetime import datetime, timezone
+from pathlib import Path
+
+import httpx
+
+from ..models import Document
+from .base import MemoryProvider
+
+logger = logging.getLogger(__name__)
+
+BENCH_TAG = "bench:amb"
+USER_TAG = "amb:user:"
+DOC_TAG = "amb:doc:"
+SUBJECT_TAG = "amb:subject:"
+DATE_TAG = "amb:date:"
+BELIEF_TAG = "amb:belief:"
+SUPERSEDES_TAG = "supersedes:"
+SUPERSEDED_BY_TAG = "superseded-by:"
+CHAIN_ID = 8453
+ASSISTANT_TURN_CHARS = 300
+
+EXTRACT_SYSTEM = """You maintain long-term memory as typed records. Read one conversation session and extract the durable facts the participants state about themselves, their lives, possessions, habits, relationships, plans, numbers and events. In a chat between a user and an assistant, ignore the assistant's advice and general knowledge. Ignore turns that reveal nothing lasting about a participant.
+
+Return JSON only: {"facts":[{"subject":"","kind":"state" or "event","statement":""}]}.
+- kind "state" is a lasting attribute whose current value can change later: where someone lives, their job, what they own, what they prefer, a relationship, a plan, an opinion. A later state on the same subject replaces the earlier one, so one state record per subject per session: merge several details about one subject into one statement.
+- kind "event" is something that happened or will happen at a time: a visit, a purchase, a trip, a conversation, a milestone, an activity. Events are never replaced, so record each distinct event separately and put its time inside the statement whenever the session gives one, absolute or relative to the session date ("on 7 May 2023", "last weekend", "next month").
+- subject is a stable dotted key naming the thing the fact is about. It starts with the person: "user" in a user-assistant chat, otherwise the speaker's lowercase first name. States name the attribute: "user.car", "user.job.employer", "caroline.friend.rachel.home". Events name the kind of event: "caroline.event.support_group_visit", "melanie.event.painting". Reuse a key from the known list when a state is about the same thing, even if the value changed. Never encode the value in the key.
+- statement is one short sentence in the third person naming the person ("The user ...", "Caroline ...") with every concrete value kept exactly (numbers, names, places, dates, counts, durations).
+- Return {"facts":[]} when the session has no durable facts."""
+
+SELECT_SYSTEM = """You select memory records for answering a question. You are given the question, the date it is asked, and an index of a person's current memory records: one line per record in the form "subject | recorded | statement". Return JSON only: {"subjects":["", ...]} listing the records needed to answer the question, most relevant first, at most %d.
+- Include a record when it could hold the answer or a value the answer depends on (a date, a count, a name, a place), and every record on the question's topic: the events, plans and states about the same thing, since the answer may need to be assembled or dated from several of them.
+- Read past wording: a question about what someone researched, chose, bought or did is answered by records about applications, visits, purchases, plans or activities on that topic.
+- When in doubt, include the record: an extra record costs little, a missing one costs the answer. Use the full allowance when the topic has that many records.
+- Return {"subjects":[]} only when nothing in the index concerns the question."""
+
+BELIEF_SELECT_SYSTEM = """You are a precise retriever over a person's stored beliefs. The candidates already passed an exact filter on scope and on the names the query uses. Given the query and the candidates (id, title, names, content), return JSON only: {"beliefs":["", ...]} with every candidate the query names, refers to, or depends on, most relevant first, at most %d. Keep a candidate when the query mentions one of its names, even when the query rejects or negates that thing (a belief about the replacement still answers it) and even when the query is about several things at once. Drop a candidate only when the query does not concern it at all. Return {"beliefs":[]} when none applies."""
+
+
+LABEL_SYSTEM = """You classify one stored memory so it can be filtered by exact tags. Read the memory text and return JSON only with one key per label below. A "value" label takes exactly one of its listed values. A "multi-text" label takes a list of strings, each written in lowercase as words separated by single spaces with no punctuation; return [] when nothing applies.
+
+Labels:
+%s"""
+
+_WORD_RE = re.compile(r"[a-z0-9]+")
+
+
+def _name_form(value: str) -> str:
+ return " ".join(_WORD_RE.findall(str(value or "").lower()))
+
+
+def _trigrams(text: str) -> set[str]:
+ padded = f" {text} "
+ return {padded[i : i + 3] for i in range(len(padded) - 2)}
+
+
+def _similar(a: str, b: str, threshold: float = 0.45) -> bool:
+ ta, tb = _trigrams(a), _trigrams(b)
+ if not ta or not tb:
+ return False
+ return len(ta & tb) / len(ta | tb) >= threshold
+
+
+def _same_name(candidate: str, stored: str) -> bool:
+ """Fuzzy name lookup: trigram similarity, except that a number in a name is exact.
+
+ A number picks out one instance ("chapter 7", "step 3"), so a stored name carrying a
+ number resolves only from a candidate carrying the same number. Without this, "chapter"
+ scored 0.80 against "chapter 7" and "chapter with" 0.53, and any query that said
+ "chapter" retrieved the chapter-7 belief. Spelling slips and fragments of word-only
+ names ("typsecript", "error" for "error handling") resolve as before."""
+ numbers = {w for w in stored.split() if w.isdigit()}
+ if numbers and not numbers <= set(candidate.split()):
+ return False
+ return _similar(candidate, stored)
+
+
+def _safe_tag(value: str, limit: int = 80) -> str:
+ text = re.sub(r"\s+", " ", str(value or "")).strip()
+ return text[:limit]
+
+
+def _safe_subject(value: str) -> str:
+ key = re.sub(r"[^a-z0-9._-]+", "_", str(value or "").strip().lower()).strip("_")
+ return key[:80]
+
+
+def _parse_dt(value: str | None) -> datetime | None:
+ if not value:
+ return None
+ text = str(value).strip()
+ for fmt in ("%Y-%m-%dT%H:%M:%S.%f%z", "%Y-%m-%dT%H:%M:%S%z", "%Y-%m-%dT%H:%M:%S", "%Y-%m-%d %H:%M:%S", "%Y-%m-%d"):
+ try:
+ dt = datetime.strptime(text, fmt)
+ return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
+ except ValueError:
+ continue
+ try:
+ dt = datetime.fromisoformat(text.replace("Z", "+00:00"))
+ return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
+ except ValueError:
+ pass
+ match = re.match(r"(\d{4})/(\d{1,2})/(\d{1,2})(?:.*?(\d{1,2}):(\d{2}))?", text)
+ if match:
+ y, mo, d, h, mi = match.groups()
+ return datetime(int(y), int(mo), int(d), int(h or 0), int(mi or 0), tzinfo=timezone.utc)
+ return None
+
+
+def _json_object(text: str) -> dict:
+ cleaned = str(text or "").strip()
+ cleaned = re.sub(r"^```(?:json)?\s*", "", cleaned, flags=re.I)
+ cleaned = re.sub(r"\s*```$", "", cleaned)
+ start, end = cleaned.find("{"), cleaned.rfind("}")
+ if start < 0 or end < 0:
+ return {}
+ try:
+ parsed = json.loads(cleaned[start : end + 1])
+ except json.JSONDecodeError:
+ return {}
+ return parsed if isinstance(parsed, dict) else {}
+
+
+def _tags_of(proof: dict) -> list[str]:
+ out: list[str] = []
+ for holder in (proof.get("meta") or {}, (proof.get("options") or {}).get("meta") or {}):
+ tags = holder.get("tags")
+ if isinstance(tags, list):
+ out.extend(str(t).strip() for t in tags if str(t).strip())
+ seen: set[str] = set()
+ unique = []
+ for tag in out:
+ low = tag.lower()
+ if low not in seen:
+ seen.add(low)
+ unique.append(tag)
+ return unique
+
+
+def _content_of(proof: dict) -> str:
+ candidates = [
+ (proof.get("publicContent") or {}).get("content"),
+ (proof.get("resolvedContent") or {}).get("value"),
+ proof.get("content"),
+ (proof.get("data") or {}).get("content"),
+ (proof.get("privateContent") or {}).get("content"),
+ ]
+ for value in candidates:
+ if isinstance(value, str) and value.strip():
+ return value.strip()
+ for verifier in proof.get("verifiedVerifiers") or []:
+ value = ((verifier or {}).get("data") or {}).get("content")
+ if isinstance(value, str) and value.strip():
+ return value.strip()
+ return ""
+
+
+class _Record:
+ __slots__ = ("qhash", "subject", "date", "doc_id", "belief", "statement", "content", "tags", "created_at", "title", "resolved")
+
+ def __init__(self, proof: dict):
+ tags = _tags_of(proof)
+ lower = {t.lower(): t for t in tags}
+
+ def tag_value(prefix: str) -> str:
+ for low, original in lower.items():
+ if low.startswith(prefix):
+ return original[len(prefix) :]
+ return ""
+
+ self.qhash = str(proof.get("qHash") or "").lower()
+ self.subject = tag_value(SUBJECT_TAG)
+ self.date = tag_value(DATE_TAG)
+ self.doc_id = tag_value(DOC_TAG)
+ self.belief = tag_value(BELIEF_TAG)
+ self.tags = tags
+ self.content = _content_of(proof)
+ self.title = str((proof.get("meta") or {}).get("title") or proof.get("title") or "")
+ self.resolved = bool(self.content)
+ if not self.content and self.title:
+ # Content resolves shortly after the write; the title carries the statement until then.
+ self.content = f"[{self.date or 'undated'}] {self.title}" if self.subject else self.title
+ self.statement = re.sub(r"^\[[^\]]*\]\s*[^:]*:\s*", "", self.content) if self.subject else self.content
+ self.created_at = proof.get("createdAt") or 0
+
+
+class ProofableMemoryProvider(MemoryProvider):
+ name = "proofable"
+ description = (
+ "Proofable typed memory (hosted). Each session is distilled into dated, subject-keyed private "
+ "records; a newer record on a subject supersedes the older one and only current records are read. "
+ "Retrieval applies the dataset's exact tag filter, then a model selects the subjects for the "
+ "question and may select none. k=20."
+ )
+ kind = "cloud"
+ provider = "proofable"
+ variant = None
+ link = "https://proofable.me"
+ logo = "https://proofable.me/favicon.ico"
+ concurrency = 4
+ supports_filters = True
+
+ def __init__(self) -> None:
+ self.k = int(os.environ.get("PROOFABLE_TOP_K", "20"))
+ self.unit_concurrency = int(os.environ.get("PROOFABLE_UNIT_CONCURRENCY", "4"))
+ self.model = os.environ.get("PROOFABLE_MEMORY_MODEL", "gemini-3.1-flash-lite")
+ self._api = os.environ.get("PROOFABLE_API_URL", "https://api.proofable.me").rstrip("/")
+ self._http: httpx.Client | None = None
+ self._wallet: str | None = None
+ self._genai = None
+ self._lock = threading.Lock()
+ self._write_lock = threading.Lock()
+ self._budgets: dict[str, dict] = {}
+ self._records: dict[str, list[_Record]] = {}
+ self._writes = 0
+ self._labels: list[dict] | None = None
+
+ # ------------------------------------------------------------------ connections
+
+ def _client(self) -> httpx.Client:
+ if self._http is None:
+ key = os.environ.get("PROOFABLE_ACCESS_KEY", "").strip()
+ if not key:
+ raise RuntimeError("PROOFABLE_ACCESS_KEY is not set")
+ self._http = httpx.Client(
+ base_url=self._api,
+ headers={"authorization": f"Bearer {key}", "content-type": "application/json"},
+ timeout=httpx.Timeout(180.0, connect=30.0),
+ )
+ return self._http
+
+ def _wallet_address(self) -> str:
+ if self._wallet is None:
+ data = self._request("GET", "/api/v1/auth/me").get("data") or {}
+ wallet = str(data.get("address") or data.get("primaryAccount") or "").lower()
+ if not re.fullmatch(r"0x[0-9a-f]{40}", wallet):
+ raise RuntimeError("the access key resolved to no wallet")
+ self._wallet = wallet
+ return self._wallet
+
+ def _gemini(self):
+ if self._genai is None:
+ from google import genai
+
+ self._genai = genai.Client()
+ return self._genai
+
+ def _request(self, method: str, path: str, body: dict | None = None, *, write: bool = False) -> dict:
+ client = self._client()
+ bucket = "write" if write else "api"
+ delay = 2.0
+ for attempt in range(8):
+ self._pace(bucket)
+ try:
+ response = client.request(method, path, json=body)
+ except httpx.HTTPError as error:
+ if attempt == 7:
+ raise
+ logger.warning("proofable %s %s: %s (retry %d)", method, path, error, attempt + 1)
+ time.sleep(delay)
+ delay = min(delay * 2, 60.0)
+ continue
+ self._observe(bucket, response.headers)
+ if response.status_code == 429:
+ wait = float(response.headers.get("retry-after") or 0) or self._seconds_to_reset(bucket) or delay
+ logger.info("proofable rate limit on %s; waiting %.0fs", path, wait)
+ time.sleep(min(max(wait, 1.0), 120.0))
+ continue
+ if response.status_code >= 500:
+ if attempt == 7:
+ response.raise_for_status()
+ time.sleep(delay)
+ delay = min(delay * 2, 60.0)
+ continue
+ try:
+ payload = response.json()
+ except ValueError:
+ payload = {}
+ if response.status_code >= 400 or payload.get("success") is False:
+ raise RuntimeError(f"{method} {path} -> {response.status_code} {json.dumps(payload.get('error') or payload)[:300]}")
+ return payload
+ raise RuntimeError(f"{method} {path}: gave up after retries")
+
+ # ------------------------------------------------------------------ write budget
+
+ # Two budgets, read from the IETF ratelimit headers on every response: new-proof writes
+ # (POST /verification) and every other API call. Each is a per-minute allowance on the
+ # paying account; a request waits for the window to reset rather than drawing a 429.
+ def _observe(self, bucket: str, headers: httpx.Headers) -> None:
+ header = headers.get("ratelimit") or ""
+ remaining = re.search(r"remaining=(\d+)", header)
+ reset = re.search(r"reset=(\d+)", header)
+ with self._lock:
+ state = self._budgets.setdefault(bucket, {"remaining": None, "reset_at": 0.0})
+ if remaining:
+ state["remaining"] = int(remaining.group(1))
+ if reset:
+ state["reset_at"] = time.time() + int(reset.group(1))
+
+ def _seconds_to_reset(self, bucket: str) -> float:
+ with self._lock:
+ state = self._budgets.get(bucket) or {}
+ return max(0.0, float(state.get("reset_at") or 0.0) - time.time())
+
+ def _pace(self, bucket: str) -> None:
+ while True:
+ with self._lock:
+ state = self._budgets.setdefault(bucket, {"remaining": None, "reset_at": 0.0})
+ remaining, reset_at = state["remaining"], state["reset_at"]
+ if remaining is None or remaining > 0 or time.time() >= reset_at:
+ if remaining is not None and remaining > 0:
+ state["remaining"] = remaining - 1
+ return
+ wait = reset_at - time.time()
+ time.sleep(min(max(wait, 0.5), 65.0))
+
+ # ------------------------------------------------------------------ records
+
+ def _write(self, content: str, title: str, tags: list[str]) -> str:
+ wallet = self._wallet_address()
+ body = {
+ "walletAddress": wallet,
+ "signature": "",
+ "signatureMethod": "eip191",
+ "signedTimestamp": int(time.time() * 1000),
+ "chainId": CHAIN_ID,
+ "verifierIds": ["ownership-basic"],
+ "data": {"owner": wallet, "content": content, "contentType": "text/plain"},
+ "options": {
+ "privacyLevel": "private",
+ "storeOriginalContent": True,
+ "meta": {"title": title[:120], "tags": tags, "source": "agent-memory-benchmark"},
+ },
+ }
+ payload = self._request("POST", "/api/v1/verification", body, write=True)
+ qhash = str((payload.get("data") or {}).get("qHash") or "").lower()
+ if not re.fullmatch(r"0x[0-9a-f]{64}", qhash):
+ raise RuntimeError(f"verification returned no qHash: {json.dumps(payload)[:200]}")
+ with self._lock:
+ self._writes += 1
+ if self._writes % 100 == 0:
+ logger.info("proofable: %d records written", self._writes)
+ return qhash
+
+ def _mark_superseded(self, old: str, new: str) -> None:
+ self._request(
+ "PATCH",
+ f"/api/v1/proofs/{old}/metadata",
+ {
+ "addTags": [f"{SUPERSEDED_BY_TAG}{new}"],
+ "removeTags": [],
+ "ownerWallet": self._wallet_address(),
+ "signedTimestamp": int(time.time() * 1000),
+ "signature": "",
+ "signatureMethod": "eip191",
+ },
+ )
+
+ def _revoke(self, qhash: str) -> None:
+ self._request("POST", f"/api/v1/proofs/revoke-self/{qhash}", {})
+
+ def _list(self, user_id: str) -> list[_Record]:
+ wallet = self._wallet_address()
+ tag = f"{USER_TAG}{user_id}"
+ out: list[_Record] = []
+ offset = 0
+ while True:
+ payload = self._request("GET", f"/api/v1/proofs/by-wallet/{wallet}?tags={httpx.QueryParams({'t': tag})['t']}&limit=1000&offset={offset}")
+ data = payload.get("data") or {}
+ page = data.get("proofs") or []
+ out.extend(_Record(p) for p in page)
+ pagination = data.get("pagination") or data
+ if not pagination.get("hasMore") or not page:
+ break
+ offset = int(pagination.get("nextOffset") or offset + len(page))
+ if offset > 50_000:
+ break
+ return [r for r in out if r.qhash]
+
+ def _current(self, user_id: str) -> list[_Record]:
+ with self._lock:
+ cached = self._records.get(user_id)
+ if cached is not None:
+ return cached
+ records = self._list(user_id)
+ with self._lock:
+ self._records[user_id] = records
+ return records
+
+ # ------------------------------------------------------------------ model
+
+ def _llm_json(self, system: str, user: str) -> dict:
+ from google.genai import types
+
+ client = self._gemini()
+ delay = 3.0
+ for attempt in range(6):
+ try:
+ response = client.models.generate_content(
+ model=self.model,
+ contents=user,
+ config=types.GenerateContentConfig(
+ system_instruction=system,
+ temperature=0,
+ response_mime_type="application/json",
+ max_output_tokens=4096,
+ ),
+ )
+ return _json_object(response.text or "")
+ except Exception as error: # noqa: BLE001 - provider errors are transient here
+ if attempt == 5:
+ logger.warning("proofable model call failed: %s", error)
+ return {}
+ time.sleep(delay)
+ delay = min(delay * 2, 60.0)
+ return {}
+
+ @staticmethod
+ def _turns(doc: Document) -> list[tuple[str, str]]:
+ turns: list[tuple[str, str]] = []
+ raw = doc.messages
+ if raw is None:
+ try:
+ parsed = json.loads(doc.content)
+ raw = parsed if isinstance(parsed, list) else None
+ except (ValueError, TypeError):
+ raw = None
+ if not raw:
+ return [("user", str(doc.content or ""))]
+ for turn in raw:
+ if not isinstance(turn, dict):
+ continue
+ speaker = str(turn.get("speaker") or turn.get("role") or "user").strip()
+ text = str(turn.get("text") or turn.get("content") or "").strip()
+ if text:
+ turns.append((speaker, text))
+ return turns
+
+ def _extract(self, doc: Document, known: list[str]) -> list[dict]:
+ turns = self._turns(doc)
+ rendered = []
+ for speaker, text in turns:
+ if speaker.lower() == "assistant" and len(text) > ASSISTANT_TURN_CHARS:
+ text = text[:ASSISTANT_TURN_CHARS] + " […]"
+ rendered.append({"speaker": speaker, "text": text})
+ known_block = ""
+ if known:
+ known_block = "Known subject keys so far:\n" + "\n".join(f"- {k}" for k in known) + "\n\n"
+ prompt = f"{known_block}Session date: {doc.timestamp or 'unknown'}\nSession:\n{json.dumps(rendered, ensure_ascii=False)}"
+ parsed = self._llm_json(EXTRACT_SYSTEM, prompt)
+ facts = []
+ for fact in parsed.get("facts") or []:
+ if not isinstance(fact, dict):
+ continue
+ subject = _safe_subject(fact.get("subject"))
+ statement = re.sub(r"\s+", " ", str(fact.get("statement") or "")).strip()[:400]
+ kind = "event" if str(fact.get("kind") or "").strip().lower() == "event" else "state"
+ if subject and statement:
+ if kind == "event":
+ # Events are never replaced: each one is its own subject.
+ digest = hashlib.sha1(f"{doc.id}|{statement}".encode("utf-8")).hexdigest()[:8]
+ subject = f"{subject}.{digest}"[:80]
+ facts.append({"subject": subject, "statement": statement, "kind": kind})
+ return facts
+
+ # ------------------------------------------------------------------ labels
+
+ def set_extraction_labels(self, labels: list[dict] | None) -> None:
+ """Dataset-declared labels (PrecisionMemBench: state and name). Each becomes an exact
+ tag on the stored record, so the dataset's retrieval filter runs as tag matching."""
+ self._labels = [l for l in (labels or []) if isinstance(l, dict) and l.get("key")] or None
+
+ def _label_tags(self, text: str) -> list[str]:
+ if not self._labels:
+ return []
+ lines = []
+ for label in self._labels:
+ kind = label.get("type") or "value"
+ desc = str(label.get("description") or "").strip()
+ values = label.get("values") or []
+ allowed = ", ".join(f"'{v.get('value')}' ({v.get('description')})" for v in values if isinstance(v, dict))
+ lines.append(f"- {label['key']} ({kind}){': ' + desc if desc else ''}{(' Values: ' + allowed) if allowed else ''}")
+ parsed = self._llm_json(
+ LABEL_SYSTEM % "\n".join(lines),
+ "The memory text opens with the subject's canonical name followed by its alternative names "
+ "(aliases, shorthand, the names of what it replaced or rejects) and only then the description. "
+ "Every word-run before the first full sentence is a name of the subject: list each one separately, "
+ f"in addition to the forms the description gives.\n\nMemory:\n{text}",
+ )
+ tags: list[str] = []
+ for label in self._labels:
+ key = str(label["key"])
+ value = parsed.get(key)
+ if (label.get("type") or "value") == "multi-text":
+ forms = value if isinstance(value, list) else ([value] if isinstance(value, str) else [])
+ for form in forms:
+ normalised = _name_form(form)
+ if normalised and f"{key}:{normalised}" not in tags:
+ tags.append(f"{key}:{normalised}")
+ else:
+ allowed = {str(v.get("value")) for v in (label.get("values") or []) if isinstance(v, dict)}
+ chosen = str(value or "").strip().lower()
+ if allowed and chosen not in allowed:
+ chosen = ""
+ if not chosen and label.get("optional") is False and allowed:
+ chosen = "current" if "current" in allowed else sorted(allowed)[0]
+ if chosen:
+ tags.append(f"{key}:{chosen}")
+ return tags
+
+ # ------------------------------------------------------------------ interface
+
+ def prepare(self, store_dir: Path, unit_ids: set[str] | None = None, reset: bool = True) -> None:
+ self._wallet_address()
+
+ def ingest(self, documents: list[Document]) -> None:
+ by_user: dict[str, list[Document]] = {}
+ for doc in documents:
+ by_user.setdefault(doc.user_id or "bench", []).append(doc)
+ for user_id, docs in by_user.items():
+ self._ingest_user(user_id, docs)
+
+ def _ingest_user(self, user_id: str, docs: list[Document]) -> None:
+ existing = self._list(user_id)
+ if existing:
+ with ThreadPoolExecutor(max_workers=8) as pool:
+ list(pool.map(lambda r: self._revoke(r.qhash), existing))
+ with self._lock:
+ self._records.pop(user_id, None)
+
+ ordered = sorted(enumerate(docs), key=lambda pair: (_parse_dt(pair[1].timestamp) or datetime.max.replace(tzinfo=timezone.utc), pair[0]))
+ user_tag = f"{USER_TAG}{_safe_tag(user_id)}"
+ current: dict[str, str] = {}
+ known_kinds: dict[str, str] = {}
+ with ThreadPoolExecutor(max_workers=8) as pool:
+ for _, doc in ordered:
+ doc_tag = f"{DOC_TAG}{_safe_tag(doc.id)}"
+ date_tag = f"{DATE_TAG}{_safe_tag(doc.timestamp or '', 48)}"
+ if doc.tags:
+ title = str(doc.content or "").strip().splitlines()[0][:120] if doc.content else doc.id
+ tags = ["facet:memory", BENCH_TAG, user_tag, doc_tag, date_tag, f"{BELIEF_TAG}{_safe_tag(doc.id)}"]
+ tags.extend(_safe_tag(t) for t in doc.tags)
+ tags.extend(self._label_tags(str(doc.content or "")))
+ self._write(str(doc.content or ""), title, tags)
+ continue
+ facts = self._extract(doc, [s for s, k in known_kinds.items() if k == "state"])
+ if not facts:
+ continue
+
+ def write_fact(fact: dict) -> tuple[str, str, str | None]:
+ subject = fact["subject"]
+ previous = current.get(subject) if fact["kind"] == "state" else None
+ content = f"[{doc.timestamp or 'undated'}] {subject}: {fact['statement']}"
+ tags = ["facet:memory", BENCH_TAG, user_tag, doc_tag, f"{SUBJECT_TAG}{subject}", date_tag, f"amb:kind:{fact['kind']}"]
+ if previous:
+ tags.append(f"{SUPERSEDES_TAG}{previous}")
+ qhash = self._write(content, f"{subject}: {fact['statement']}", tags)
+ if previous:
+ try:
+ self._mark_superseded(previous, qhash)
+ except Exception as error: # noqa: BLE001
+ logger.warning("proofable: could not mark %s superseded: %s", previous[:10], error)
+ return subject, qhash, previous
+
+ for fact, (subject, qhash, _) in zip(facts, pool.map(write_fact, facts)):
+ current[subject] = qhash
+ known_kinds[subject] = fact["kind"]
+ self._wait_resolved(user_id)
+
+ def _wait_resolved(self, user_id: str, timeout: float = 900.0) -> None:
+ """Block until every record of the user reads back with its content, the way the other
+ hosted providers wait for indexing: the write is acknowledged at once and the stored
+ content becomes readable a little later."""
+ deadline = time.time() + timeout
+ while True:
+ records = self._list(user_id)
+ pending = [r for r in records if not r.resolved]
+ if not pending:
+ with self._lock:
+ self._records[user_id] = records
+ return
+ if time.time() >= deadline:
+ logger.warning("proofable: %d of %d records for %s still resolving after %.0fs", len(pending), len(records), user_id, timeout)
+ with self._lock:
+ self._records[user_id] = records
+ return
+ time.sleep(5.0)
+
+ def retrieve(
+ self,
+ query: str,
+ k: int = 10,
+ user_id: str | None = None,
+ query_timestamp: str | None = None,
+ filters: dict | None = None,
+ ) -> tuple[list[Document], dict | None]:
+ uid = user_id or "bench"
+ records = self._current(uid)
+ limit = k if filters is not None else self.k
+ if not records:
+ return [], {"records": 0, "selected": []}
+
+ beliefs = [r for r in records if r.belief]
+ if beliefs:
+ return self._retrieve_beliefs(query, beliefs, limit, filters)
+ return self._retrieve_subjects(query, records, limit, query_timestamp)
+
+ def _retrieve_subjects(self, query: str, records: list[_Record], limit: int, query_timestamp: str | None):
+ by_subject: dict[str, _Record] = {}
+ for record in sorted(records, key=lambda r: r.created_at):
+ if record.subject:
+ by_subject[record.subject] = record
+ if not by_subject:
+ return [], {"records": len(records), "selected": []}
+ index = "\n".join(f"{s} | recorded {r.date or 'undated'} | {r.statement}" for s, r in by_subject.items())
+ prompt = f"Question date: {query_timestamp or 'unknown'}\nQuestion: {query}\n\nMemory index:\n{index}"
+ parsed = self._llm_json(SELECT_SYSTEM % limit, prompt)
+ chosen: list[str] = []
+ for key in parsed.get("subjects") or []:
+ subject = _safe_subject(key)
+ if subject in by_subject and subject not in chosen:
+ chosen.append(subject)
+ docs = []
+ for subject in chosen[:limit]:
+ record = by_subject[subject]
+ shown_subject = re.sub(r"\.[0-9a-f]{8}$", "", subject)
+ docs.append(
+ Document(
+ id=record.qhash,
+ # Statement first; provenance after it, so a reader sees the fact and not a key.
+ content=f"{record.statement} (recorded {record.date or 'undated'}; about {shown_subject})",
+ source_ids=[record.doc_id] if record.doc_id else None,
+ timestamp=record.date or None,
+ )
+ )
+ # No raw payload: the conversation datasets put a provider's raw response into the
+ # answer prompt in place of the rendered memories, and the records above are the answer.
+ logger.info("proofable retrieve: %d of %d records selected", min(len(chosen), limit), len(by_subject))
+ return docs, None
+
+ def _retrieve_beliefs(self, query: str, beliefs: list[_Record], limit: int, filters: dict | None):
+ candidates = beliefs
+ if filters:
+ candidates = self._apply_filters(candidates, filters)
+ if not candidates:
+ return [], {"records": len(beliefs), "filtered": 0, "selected": []}
+ listing = "\n\n".join(
+ f"id: {b.belief}\ntitle: {b.title}\nnames: {', '.join(t[5:] for t in b.tags if t.lower().startswith('name:')) or '-'}\n{b.content[:1200]}"
+ for b in candidates
+ )
+ parsed = self._llm_json(BELIEF_SELECT_SYSTEM % limit, f"Query: {query}\n\nCandidate beliefs:\n{listing}")
+ by_id = {b.belief: b for b in candidates}
+ chosen = []
+ for belief_id in parsed.get("beliefs") or []:
+ key = str(belief_id).strip()
+ if key in by_id and key not in chosen:
+ chosen.append(key)
+ docs = [
+ Document(
+ id=by_id[key].qhash,
+ content=by_id[key].content,
+ source_ids=[by_id[key].doc_id] if by_id[key].doc_id else None,
+ context=f"beliefId={key}",
+ timestamp=by_id[key].date or None,
+ )
+ for key in chosen[:limit]
+ ]
+ return docs, {"records": len(beliefs), "filtered": len(candidates), "selected": chosen[:limit]}
+
+ @staticmethod
+ def _apply_filters(records: list[_Record], filters: dict) -> list[_Record]:
+ """The dataset's provider-neutral filter as exact tag matching on the stored tags.
+
+ any: groups of tags; a record must carry one tag from every group
+ all: every listed tag
+ none: no listed tag
+ narrow: one of the listed tags
+ narrow_any: alternative narrowings, OR-ed; each lists candidate tags and resolves
+ them exactly, or fuzzily by trigram similarity against the stored tags (a number in a name must match exactly)
+ """
+
+ def tags_of(record: _Record) -> set[str]:
+ return {t.lower() for t in record.tags}
+
+ def groups(value) -> list[list[str]]:
+ if not value:
+ return []
+ if all(isinstance(v, str) for v in value):
+ return [[str(v).lower() for v in value]]
+ return [[str(t).lower() for t in group] for group in value if group]
+
+ out = records
+ for group in groups(filters.get("any")):
+ out = [r for r in out if tags_of(r) & set(group)]
+ if filters.get("all"):
+ required = {str(t).lower() for t in filters["all"]}
+ out = [r for r in out if required <= tags_of(r)]
+ if filters.get("none"):
+ banned = {str(t).lower() for t in filters["none"]}
+ out = [r for r in out if not (tags_of(r) & banned)]
+ if filters.get("narrow"):
+ wanted = {str(t).lower() for t in filters["narrow"]}
+ out = [r for r in out if tags_of(r) & wanted]
+ specs = [s for s in (filters.get("narrow_any") or []) if isinstance(s, dict) and s.get("tags")]
+ if specs:
+ kept = []
+ for record in out:
+ stored = tags_of(record)
+ hit = False
+ for spec in specs:
+ wanted = [str(t).lower() for t in spec["tags"]]
+ if spec.get("resolve") == "fuzzy":
+ prefix = wanted[0].split(":", 1)[0] + ":" if wanted and ":" in wanted[0] else ""
+ stored_forms = [t.split(":", 1)[1] for t in stored if prefix and t.startswith(prefix)]
+ hit = any(_same_name(w.split(":", 1)[1], s) for w in wanted for s in stored_forms)
+ else:
+ hit = bool(stored & set(wanted))
+ if hit:
+ break
+ if hit:
+ kept.append(record)
+ out = kept
+ return out
+
+ def cleanup(self) -> None:
+ if self._http is not None:
+ self._http.close()
+ self._http = None
diff --git a/public/evidence/precisionmembench/2026-11-10-77/runs/run-2-77of77.json b/public/evidence/precisionmembench/2026-11-10-77/runs/run-2-77of77.json
new file mode 100644
index 0000000..7fbd144
--- /dev/null
+++ b/public/evidence/precisionmembench/2026-11-10-77/runs/run-2-77of77.json
@@ -0,0 +1,4887 @@
+{
+ "dataset": "precisionmembench",
+ "split": "single-turn",
+ "category": null,
+ "memory_provider": "proofable",
+ "run_name": "proofable-k3",
+ "mode": "retrieval",
+ "oracle": false,
+ "total_queries": 77,
+ "correct": 77,
+ "accuracy": 1.0,
+ "mean_precision": 0.9773,
+ "mean_recall": 1.0,
+ "active_passes": 43,
+ "active_total": 43,
+ "ingestion_time_ms": 213183.7,
+ "ingested_docs": 35,
+ "description": null,
+ "answer_llm": null,
+ "judge_llm": null,
+ "results": [
+ {
+ "query_id": "happy-stack-repository-layer",
+ "query": "How should I structure my repository layer?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 13.6,
+ "gold_answers": [
+ "b-db-decision"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "Repository layer query: pinned DB decision surfaces; scoped open question surfaces; relevant beliefs depend on query token overlap",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-db-decision"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "happy-alias-match-kubernetes",
+ "query": "what's my k8s deploy story?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x61dabcee1c66c2dcfbb527b6b6e110d5cb4c502f35800c9efe09399295d00700] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.",
+ "context_tokens": 102,
+ "retrieve_time_ms": 679.1,
+ "gold_answers": [
+ "b-kubernetes-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'k8s' resolves to Kubernetes entity",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-kubernetes-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-kubernetes-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-kube-resolves",
+ "query": "how does kube schedule pods across our CI pipeline nodes?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x61dabcee1c66c2dcfbb527b6b6e110d5cb4c502f35800c9efe09399295d00700] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.\n\n2. [0xdfa78706c30e1507ce3b8c411338de38705f2db0d339868f5289b3eeecc13f19] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.",
+ "context_tokens": 201,
+ "retrieve_time_ms": 2363.3,
+ "gold_answers": [
+ "b-ci-gha-entity",
+ "b-kubernetes-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'kube' drives text search retrieval",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ci-gha-entity",
+ "b-kubernetes-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-kubernetes-entity",
+ "b-ci-gha-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-gha-resolves",
+ "query": "GHA secrets config",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xdfa78706c30e1507ce3b8c411338de38705f2db0d339868f5289b3eeecc13f19] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.",
+ "context_tokens": 105,
+ "retrieve_time_ms": 1176.0,
+ "gold_answers": [
+ "b-ci-gha-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'GHA' (short form) drives text-search retrieval of b-ci-gha-entity",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ci-gha-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-ci-gha-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "long-query-rambling-kubernetes",
+ "query": "ok so I've been thinking about this for a while and I know we've talked about infrastructure before but I'm circling back because the team had some pushback in the last review and honestly I'm not sure what the right call is anymore. Basically the question is how we should handle rolling deploys for the landing page when it lands \u2014 should we do Kubernetes native rolling updates or something more custom, and does it matter that the service holds session state in memory?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x61dabcee1c66c2dcfbb527b6b6e110d5cb4c502f35800c9efe09399295d00700] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.",
+ "context_tokens": 102,
+ "retrieve_time_ms": 2602.7,
+ "gold_answers": [
+ "b-kubernetes-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Verbose 400+ char query.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-kubernetes-entity"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-kubernetes-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "out-of-scope-coin-collecting",
+ "query": "what's a good way to grade a 1909-S VDB penny without sending it to PCGS?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.4,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Query in a domain the user has no beliefs about: personaPrelude only, no scope/relevant leakage",
+ "scope": [
+ "domain:hobby"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [],
+ "retrieved_questions": [],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "happy-writing-scope",
+ "query": "How should I open the next chapter with Mara?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.1,
+ "gold_answers": [
+ "b-protagonist-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Writing query surfaces writing beliefs only; no code leakage",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-protagonist-entity"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "conflict-scope-disambiguation-redis-writing",
+ "query": "What's Redis doing in chapter three?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x22c13a44c8f4aa102f9d0fe3ac2f0006ec7cdac87fa12c505d1e445c25822251] \u2190 b-redis-writing\nredis_character Redis Redis is a supporting character in the novel \u2014 a jazz pianist Mara meets in Lisbon. Shapes interpretation of 'Redis' in the writing scope \u2014 always the jazz pianist character, not the database technology.",
+ "context_tokens": 101,
+ "retrieve_time_ms": 5338.8,
+ "gold_answers": [
+ "b-redis-writing"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "'Redis' in writing scope returns character, not datastore",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-writing"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-writing"
+ ],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "conflict-scope-disambiguation-redis-code",
+ "query": "What are we using Redis for?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xd243f4686d7a2e832fc084e07506eb094e534c7931f8563dcc4933a954bba2b3] \u2190 b-redis-code\nredis_cache Redis Redis is used as a cache and rate-limiter in the backend. Single-node, not clustered. Shapes interpretation of 'Redis' in the code scope \u2014 always the caching datastore, not the fictional character.",
+ "context_tokens": 97,
+ "retrieve_time_ms": 1378.3,
+ "gold_answers": [
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "'Redis' in code scope returns datastore, not character",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-code"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "multi-scope-redis-both-surfaces",
+ "query": "What's Redis doing?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0xd243f4686d7a2e832fc084e07506eb094e534c7931f8563dcc4933a954bba2b3] \u2190 b-redis-code\nredis_cache Redis Redis is used as a cache and rate-limiter in the backend. Single-node, not clustered. Shapes interpretation of 'Redis' in the code scope \u2014 always the caching datastore, not the fictional character.\n\n2. [0x22c13a44c8f4aa102f9d0fe3ac2f0006ec7cdac87fa12c505d1e445c25822251] \u2190 b-redis-writing\nredis_character Redis Redis is a supporting character in the novel \u2014 a jazz pianist Mara meets in Lisbon. Shapes interpretation of 'Redis' in the writing scope \u2014 always the jazz pianist character, not the database technology.",
+ "context_tokens": 192,
+ "retrieve_time_ms": 6220.1,
+ "gold_answers": [
+ "b-redis-code",
+ "b-redis-writing"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "With scope=['code','writing'], both Redis beliefs should surface \u2014 one per scope",
+ "scope": [
+ "domain:code",
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-code",
+ "b-redis-writing"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-code",
+ "b-redis-writing"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current",
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q",
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "conflict-mongodb-supersedes-sqlalchemy",
+ "query": "how's our SQLAlchemy setup these days?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 3.3,
+ "gold_answers": [
+ "b-db-decision"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Supersession chain exclusion",
+ "description": "Superseded SQLAlchemy belief never surfaces",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-db-decision"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Supersession chain exclusion"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "edge-empty-query",
+ "query": "",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.0,
+ "gold_answers": [
+ "b-db-decision"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "Empty query: persona + scope prelude + pinned facts injected, no relevant beliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-db-decision"
+ ],
+ "mustExclude": [
+ "b-auth-open-q",
+ "b-comm-response-length"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "edge-short-query-raw-passthrough",
+ "query": "fastify?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xf969bfbb0be67007377257a5cee89269896c59b4f734ea0d42039305eedd4d3e] \u2190 b-fastify-pref\nframework_fastify Fastify express Uses Fastify for HTTP services rather than Express. Shapes server-side code examples toward Fastify and prevents suggesting Express as an alternative.",
+ "context_tokens": 84,
+ "retrieve_time_ms": 770.9,
+ "gold_answers": [
+ "b-fastify-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Short query 'fastify?' passes raw to text search. Belief is pinned so verified in pinnedFacts.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-fastify-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-fastify-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "isolation-user-scoping",
+ "query": "what language do I write in?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xf0ad1d6faf5d8c3d3b062037c7e61d4b87f7b28e2bde3a902336f618dce49b76] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.",
+ "context_tokens": 90,
+ "retrieve_time_ms": 895.4,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Cross-user isolation",
+ "description": "Other-user beliefs never leak",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "mustExclude": [
+ "b-other-user-leak-check"
+ ]
+ },
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-other-user-leak-check"
+ ]
+ },
+ "personaPrelude": {
+ "mustNotContain": [
+ "Haskell"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [
+ "b-ts-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 0.0,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Cross-user isolation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "questions-pinned-active-only",
+ "query": "any open questions?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.6,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "listPinnedOpenQuestions returns only pinned+active questions for the given scope",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ],
+ "mustExclude": [
+ "b-chapter-structure-q",
+ "b-hosting-resolved"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-ts-resolves",
+ "query": "TS strict mode question about unknown types",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xf0ad1d6faf5d8c3d3b062037c7e61d4b87f7b28e2bde3a902336f618dce49b76] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.",
+ "context_tokens": 90,
+ "retrieve_time_ms": 776.8,
+ "gold_answers": [
+ "b-ts-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'TS' drives text-search retrieval.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ts-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-ts-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-mongo-resolves",
+ "query": "what's the Mongo pattern we're using for collections?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x7127d24309af7604dadff4756f0e7343633e319f425369c234a9518c5d3b1f37] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.",
+ "context_tokens": 94,
+ "retrieve_time_ms": 1909.5,
+ "gold_answers": [
+ "b-mongo-raw-driver"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'Mongo' drives text-search retrieval.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-mongo-raw-driver"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-mongo-raw-driver"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-reactjs-resolves",
+ "query": "how should I structure my ReactJS components here?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x2c8bbdb535aad7416509a2412d6a7ec0ae4bb3fcfd1b52351a369549da093a32] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.",
+ "context_tokens": 91,
+ "retrieve_time_ms": 1494.8,
+ "gold_answers": [
+ "b-react-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'ReactJS' resolves to b-react-entity via text search \u2014 genuine retrieval proof",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-protagonist-resolves",
+ "query": "what does the protagonist want, and does their viewpoint stay consistent in act two?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x698f0b0d7f1ba83330dd4f501c7c646d9e1c48150ec208b1deece699eee72934] \u2190 b-protagonist-entity\nmara protagonist Protagonist of the novel. Set in 1970s Lisbon. Does not reconcile with her father; ending is ambiguous. Shapes all plot suggestions \u2014 the father-daughter thread must remain unresolved; prevents any reconciliation arc.\n\n2. [0xaf88d475c43576727689a8edc251acaf75bc5a15a8d0be6276cbe42ac7f8a948] \u2190 b-pov-shift-entity\npov_shift POV point of view perspective shift viewpoint chapter 7 Chapter 7 uses a deliberate POV shift to an unnamed observer. This is the only chapter that breaks close-third. Shapes editorial feedback \u2014 prevents flagging the chapter 7 POV shift as an error; the break is intentional and structurally load-bearing.",
+ "context_tokens": 218,
+ "retrieve_time_ms": 1579.8,
+ "gold_answers": [
+ "b-pov-shift-entity",
+ "b-protagonist-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'protagonist' retrieves b-protagonist-entity (pinned). b-pov-shift-entity (non-pinned) proves text search fired.",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-protagonist-entity"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-pov-shift-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-pov-shift-entity"
+ ],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "canonical-name-match-mara",
+ "query": "tell me about Mara's backstory with her father",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.0,
+ "gold_answers": [
+ "b-protagonist-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Direct canonical_name 'Mara' retrieves protagonist. b-protagonist-entity is pinned.",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-protagonist-entity"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-pov-shift-resolves",
+ "query": "POV rules for chapter 7",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xaf88d475c43576727689a8edc251acaf75bc5a15a8d0be6276cbe42ac7f8a948] \u2190 b-pov-shift-entity\npov_shift POV point of view perspective shift viewpoint chapter 7 Chapter 7 uses a deliberate POV shift to an unnamed observer. This is the only chapter that breaks close-third. Shapes editorial feedback \u2014 prevents flagging the chapter 7 POV shift as an error; the break is intentional and structurally load-bearing.",
+ "context_tokens": 118,
+ "retrieve_time_ms": 968.0,
+ "gold_answers": [
+ "b-pov-shift-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'POV' drives text-search retrieval of b-pov-shift-entity (non-pinned, writing scope)",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-pov-shift-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-pov-shift-entity"
+ ],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-base-class-retrieves-composition",
+ "query": "should I use a base class hierarchy for these domain objects?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x1d7858eb92f5cc2dedc4c85319dd9d2b5b871730dce44e8ccc87e46b5d25c882] \u2190 b-composition-inheritance\ncomposition_over_inheritance inheritance class hierarchies base class factory function Prefers composition over inheritance. Factory functions returning plain objects rather than class hierarchies. Shapes object design suggestions away from class hierarchies toward factory functions returning plain objects.",
+ "context_tokens": 107,
+ "retrieve_time_ms": 890.2,
+ "gold_answers": [
+ "b-composition-inheritance"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Natural-language proxy 'base class' retrieves b-composition-inheritance (non-pinned) via alias",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-composition-inheritance"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-composition-inheritance"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-exceptions-retrieves-error-handling",
+ "query": "when should I throw exceptions versus return them?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x53f0537d8fbbb33e837c4c657c8a70251a1e47d6cc9fcb6ad2c686bd967068ae] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.",
+ "context_tokens": 107,
+ "retrieve_time_ms": 1346.8,
+ "gold_answers": [
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Natural-language proxy 'exceptions' retrieves b-error-handling (non-pinned) via alias",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "filter-resolved-at-excluded",
+ "query": "should we self-host the landing page or go with Fly.io?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.6,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "b-hosting-resolved has resolved_at set \u2014 query matching its aliases must not surface it",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ },
+ "openQuestions": {
+ "mustExclude": [
+ "b-hosting-resolved"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "scope-bleed-questions-code-excludes-writing",
+ "query": "any unresolved decisions on this sprint?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.4,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Code-scope query must not surface writing-scope open questions",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ],
+ "mustExclude": [
+ "b-chapter-structure-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "persona-prelude-content-present",
+ "query": "what's our stack?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.9,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Persona prelude content",
+ "description": "Persona prelude contains substantive content tokens derived from universal preferences",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true,
+ "contains": [
+ "direct",
+ "preamble"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Persona prelude content"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "edge-short-query-score-clears",
+ "query": "k8s?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x61dabcee1c66c2dcfbb527b6b6e110d5cb4c502f35800c9efe09399295d00700] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.",
+ "context_tokens": 102,
+ "retrieve_time_ms": 1325.6,
+ "gold_answers": [
+ "b-kubernetes-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Short query 'k8s'",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-kubernetes-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-kubernetes-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "edge-whitespace-query",
+ "query": " ",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.0,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "Whitespace-only query is treated as empty by queryExpander \u2014 no relevant beliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "score-boundary-weak-match-filtered",
+ "query": "thinking about observability and monitoring dashboards",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 3.7,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "Weakly-related in-scope query does not surface tangential beliefs below minScore",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "user-edited-cross-scope-surfaces",
+ "query": "summarize what you know about my preferences",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.8,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "user_edited belief scoped to domain:writing does NOT surface in domain:code \u2014 scope isolation takes precedence over user_edited flag",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-prose-style-user-edited"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "user-edited-absent-without-flag",
+ "query": "what are my writing preferences?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.8,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Non-user-edited writing belief does NOT cross into code scope pinnedFacts",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-narrator-voice"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "ranking-canonical-name-over-content",
+ "query": "composition over inheritance \u2014 what's our pattern?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x1d7858eb92f5cc2dedc4c85319dd9d2b5b871730dce44e8ccc87e46b5d25c882] \u2190 b-composition-inheritance\ncomposition_over_inheritance inheritance class hierarchies base class factory function Prefers composition over inheritance. Factory functions returning plain objects rather than class hierarchies. Shapes object design suggestions away from class hierarchies toward factory functions returning plain objects.",
+ "context_tokens": 107,
+ "retrieve_time_ms": 810.7,
+ "gold_answers": [
+ "b-composition-inheritance"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Ranking stability",
+ "description": "canonical_name match (weight 10) outranks content-only match (weight 1) in relevantBeliefs ordering",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-composition-inheritance"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-composition-inheritance"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Ranking stability"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-typo-blind-spot",
+ "query": "how do I configure typscript strict mode?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xf0ad1d6faf5d8c3d3b062037c7e61d4b87f7b28e2bde3a902336f618dce49b76] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.",
+ "context_tokens": 90,
+ "retrieve_time_ms": 1012.1,
+ "gold_answers": [
+ "b-ts-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Typo 'typscript' does NOT match 'TypeScript' \u2014 documents text search limitation",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ts-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-ts-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-all-caps-case-insensitive",
+ "query": "REACTJS component patterns",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x2c8bbdb535aad7416509a2412d6a7ec0ae4bb3fcfd1b52351a369549da093a32] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.",
+ "context_tokens": 91,
+ "retrieve_time_ms": 1439.6,
+ "gold_answers": [
+ "b-react-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "ALL-CAPS 'REACTJS' still resolves via case-insensitive text index",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-filler-heavy-extraction",
+ "query": "ugh so like I was wondering you know about how we should think about error handling in our codebase because honestly it's been bugging me for a while and I'm not really sure what the right approach is here",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x53f0537d8fbbb33e837c4c657c8a70251a1e47d6cc9fcb6ad2c686bd967068ae] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.",
+ "context_tokens": 107,
+ "retrieve_time_ms": 1133.9,
+ "gold_answers": [
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Filler-heavy query \u2014 NLP extraction must cut through noise to find 'error handling'",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-negation-surfaces-topic",
+ "query": "I don't want class hierarchies in this codebase, what should I use instead?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x1d7858eb92f5cc2dedc4c85319dd9d2b5b871730dce44e8ccc87e46b5d25c882] \u2190 b-composition-inheritance\ncomposition_over_inheritance inheritance class hierarchies base class factory function Prefers composition over inheritance. Factory functions returning plain objects rather than class hierarchies. Shapes object design suggestions away from class hierarchies toward factory functions returning plain objects.",
+ "context_tokens": 107,
+ "retrieve_time_ms": 1364.2,
+ "gold_answers": [
+ "b-composition-inheritance"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Negation 'I don't want class hierarchies' still surfaces the composition belief \u2014 text search has no semantic negation",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-composition-inheritance"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-composition-inheritance"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-compound-surfaces-two-beliefs",
+ "query": "how should ReactJS components handle exceptions from async calls?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x2c8bbdb535aad7416509a2412d6a7ec0ae4bb3fcfd1b52351a369549da093a32] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.\n\n2. [0x53f0537d8fbbb33e837c4c657c8a70251a1e47d6cc9fcb6ad2c686bd967068ae] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.",
+ "context_tokens": 192,
+ "retrieve_time_ms": 1873.0,
+ "gold_answers": [
+ "b-react-entity",
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Single query spanning two topics surfaces both non-pinned beliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity",
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity",
+ "b-error-handling"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cap-stress-compound-six-entities",
+ "query": "When our React frontend catches error handling exceptions, how should the Kubernetes pods running MongoDB TypeScript services report to GitHub Actions?",
+ "answer": "7 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (7)\n\n1. [0x2c8bbdb535aad7416509a2412d6a7ec0ae4bb3fcfd1b52351a369549da093a32] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.\n\n2. [0x53f0537d8fbbb33e837c4c657c8a70251a1e47d6cc9fcb6ad2c686bd967068ae] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.\n\n3. [0x61dabcee1c66c2dcfbb527b6b6e110d5cb4c502f35800c9efe09399295d00700] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.\n\n4. [0x7127d24309af7604dadff4756f0e7343633e319f425369c234a9518c5d3b1f37] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.\n\n5. [0xf0ad1d6faf5d8c3d3b062037c7e61d4b87f7b28e2bde3a902336f618dce49b76] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.\n\n6. [0xdfa78706c30e1507ce3b8c411338de38705f2db0d339868f5289b3eeecc13f19] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.\n\n7. [0x5a4395445df1a5cb8602664ef199a834b3289de310085c3808abfcb6566f1ad2] \u2190 b-db-decision\ndatabase_mongodb database Decided on MongoDB after evaluating Postgres. Raw driver only. Shapes all data-layer recommendations \u2014 database choice is settled and prevents re-litigating Postgres as an alternative.",
+ "context_tokens": 646,
+ "retrieve_time_ms": 4248.1,
+ "gold_answers": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Compound query mentions 6 entities \u2014 tests whether 12-token cap in buildSearchQuery drops non-pinned targets",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-ci-gha-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 7
+ },
+ "retrieved_count": 7
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cap-stress-kitchen-sink-exceeds-twelve",
+ "query": "Set up our React components to use composition pattern with explicit error handling, wire GitHub Actions CI to deploy Kubernetes pods serving the MongoDB TypeScript API with functional pipeline transforms",
+ "answer": "8 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (8)\n\n1. [0x2c8bbdb535aad7416509a2412d6a7ec0ae4bb3fcfd1b52351a369549da093a32] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.\n\n2. [0x0c859a410581ea6bdac636886479f5cb6d03d7ac48b2bebe27347bd4d91dbb56] \u2190 b-functional-pipeline\nfunctional_pipeline pipe compose fp Uses functional composition patterns with pipe() to chain data transformations instead of imperative loops. Shapes data-processing code suggestions toward functional pipe composition and away from imperative loops or nested map/filter chains.\n\n3. [0x53f0537d8fbbb33e837c4c657c8a70251a1e47d6cc9fcb6ad2c686bd967068ae] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.\n\n4. [0xdfa78706c30e1507ce3b8c411338de38705f2db0d339868f5289b3eeecc13f19] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.\n\n5. [0x61dabcee1c66c2dcfbb527b6b6e110d5cb4c502f35800c9efe09399295d00700] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.\n\n6. [0x7127d24309af7604dadff4756f0e7343633e319f425369c234a9518c5d3b1f37] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.\n\n7. [0xf0ad1d6faf5d8c3d3b062037c7e61d4b87f7b28e2bde3a902336f618dce49b76] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.\n\n8. [0x5a4395445df1a5cb8602664ef199a834b3289de310085c3808abfcb6566f1ad2] \u2190 b-db-decision\ndatabase_mongodb database Decided on MongoDB after evaluating Postgres. Raw driver only. Shapes all data-layer recommendations \u2014 database choice is settled and prevents re-litigating Postgres as an alternative.",
+ "context_tokens": 735,
+ "retrieve_time_ms": 2698.5,
+ "gold_answers": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-functional-pipeline"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Dense 8+ entity query that deliberately exceeds the 12-token cap \u2014 tests which non-pinned beliefs survive extraction truncation",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-functional-pipeline"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity",
+ "b-functional-pipeline",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 8
+ },
+ "retrieved_count": 8
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cap-stress-late-position-entity-at-risk",
+ "query": "Given our React frontend, the GitHub Actions CI pipeline, the Kubernetes deployment, and the MongoDB TypeScript backend, how should we approach exceptions in the data layer?",
+ "answer": "7 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (7)\n\n1. [0x53f0537d8fbbb33e837c4c657c8a70251a1e47d6cc9fcb6ad2c686bd967068ae] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.\n\n2. [0x7127d24309af7604dadff4756f0e7343633e319f425369c234a9518c5d3b1f37] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.\n\n3. [0xf0ad1d6faf5d8c3d3b062037c7e61d4b87f7b28e2bde3a902336f618dce49b76] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.\n\n4. [0x5a4395445df1a5cb8602664ef199a834b3289de310085c3808abfcb6566f1ad2] \u2190 b-db-decision\ndatabase_mongodb database Decided on MongoDB after evaluating Postgres. Raw driver only. Shapes all data-layer recommendations \u2014 database choice is settled and prevents re-litigating Postgres as an alternative.\n\n5. [0x2c8bbdb535aad7416509a2412d6a7ec0ae4bb3fcfd1b52351a369549da093a32] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.\n\n6. [0xdfa78706c30e1507ce3b8c411338de38705f2db0d339868f5289b3eeecc13f19] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.\n\n7. [0x61dabcee1c66c2dcfbb527b6b6e110d5cb4c502f35800c9efe09399295d00700] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.",
+ "context_tokens": 646,
+ "retrieve_time_ms": 2651.0,
+ "gold_answers": [
+ "b-ci-gha-entity",
+ "b-react-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Critical alias 'exceptions' appears after 6+ proper nouns \u2014 tests whether extraction order causes late-position tokens to fall past the 12-token cap",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ci-gha-entity",
+ "b-react-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-react-entity",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 7
+ },
+ "retrieved_count": 7
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-single-edit-fires-non-pinned",
+ "query": "eror handling patterns",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x53f0537d8fbbb33e837c4c657c8a70251a1e47d6cc9fcb6ad2c686bd967068ae] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.",
+ "context_tokens": 107,
+ "retrieve_time_ms": 1090.0,
+ "gold_answers": [
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "1-edit deletion 'eror' fuzzy-matches alias 'error' \u2014 prefix 'er' passes",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-scope-filter-discriminates-after-fuzzy-fires",
+ "query": "Reedis cache eviction strategy",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xd243f4686d7a2e832fc084e07506eb094e534c7931f8563dcc4933a954bba2b3] \u2190 b-redis-code\nredis_cache Redis Redis is used as a cache and rate-limiter in the backend. Single-node, not clustered. Shapes interpretation of 'Redis' in the code scope \u2014 always the caching datastore, not the fictional character.",
+ "context_tokens": 97,
+ "retrieve_time_ms": 1408.7,
+ "gold_answers": [
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Typo 'Reedis' fuzzy-matches alias 'Redis' in both scopes \u2014 scope filter returns only code belief",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-code"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-prefix-blocks-transitive-match",
+ "query": "mango connection pooling",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.8,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "1-edit distance but prefix guard blocks \u2014 'mango' shares edit distance 1 with 'mongo' but 'ma' \u2260 'mo'",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-abbreviation-prefix-blind-spot",
+ "query": "k9s pod scheduling",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.7,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Alias 'k8s' \u2014 typo 'k9s' fails because prefix 'k9' \u2260 'k8' \u2014 single-char substitution in short aliases hits prefix guard",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-two-edit-transposition-blocked",
+ "query": "typsecript compilation config",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xf0ad1d6faf5d8c3d3b062037c7e61d4b87f7b28e2bde3a902336f618dce49b76] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.",
+ "context_tokens": 90,
+ "retrieve_time_ms": 838.6,
+ "gold_answers": [
+ "b-ts-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Transposition 'typsecript' IS matched by Atlas (Damerau-Levenshtein, 1 edit)",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ts-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-ts-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-single-term-only-signal",
+ "query": "mongod",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x7127d24309af7604dadff4756f0e7343633e319f425369c234a9518c5d3b1f37] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.\n\n2. [0x5a4395445df1a5cb8602664ef199a834b3289de310085c3808abfcb6566f1ad2] \u2190 b-db-decision\ndatabase_mongodb database Decided on MongoDB after evaluating Postgres. Raw driver only. Shapes all data-layer recommendations \u2014 database choice is settled and prevents re-litigating Postgres as an alternative.",
+ "context_tokens": 181,
+ "retrieve_time_ms": 3652.0,
+ "gold_answers": [
+ "b-mongo-raw-driver"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Query is a single misspelled term \u2014 'mongod' is the only token, 1 edit from alias 'Mongo', prefix 'mo' passes",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-mongo-raw-driver"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-mongo-raw-driver"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "universal-scope-belief-content-in-persona-prelude",
+ "query": "what's a good way to grade a 1909-S VDB penny without sending it to PCGS?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.3,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Persona prelude content",
+ "description": "user:universal belief content appears in personaPrelude for a domain with no per-scope description",
+ "scope": [
+ "domain:hobby"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true,
+ "contains": [
+ "direct",
+ "preamble"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [],
+ "retrieved_questions": [],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Persona prelude content"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "type-isolation-open-question-not-in-relevant",
+ "query": "JWT authentication token strategy",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.8,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "open_question belief with directly matched aliases must appear in openQuestions only \u2014 not in pinnedFacts or relevantBeliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-auth-open-q"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "multi-hop-supersession-chain-all-excluded",
+ "query": "ESLint rules and TSLint configuration",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.2,
+ "gold_answers": [
+ "b-linting-current"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Supersession chain exclusion",
+ "description": "Three-hop chain: TSLint (b-linting-v0) \u2192 ESLint (b-linting-v1) \u2192 Biome (b-linting-current). Query terms matching both superseded hops must surface neither; only the active terminal belief surfaces via pinnedFacts",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-linting-current"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Supersession chain exclusion"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "cross-scope-one-empty-domain-does-not-degrade",
+ "query": "What are we using Redis for?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xd243f4686d7a2e832fc084e07506eb094e534c7931f8563dcc4933a954bba2b3] \u2190 b-redis-code\nredis_cache Redis Redis is used as a cache and rate-limiter in the backend. Single-node, not clustered. Shapes interpretation of 'Redis' in the code scope \u2014 always the caching datastore, not the fictional character.",
+ "context_tokens": 97,
+ "retrieve_time_ms": 817.7,
+ "gold_answers": [
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Multi-scope query where one scope (domain:hobby) has zero beliefs \u2014 code-scope retrieval must be unaffected and no empty-scope error must occur",
+ "scope": [
+ "domain:code",
+ "domain:hobby"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-code"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "ranking-orderedbefore-alias-beats-content",
+ "query": "explicit error returns in pipe transforms",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x53f0537d8fbbb33e837c4c657c8a70251a1e47d6cc9fcb6ad2c686bd967068ae] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.\n\n2. [0x0c859a410581ea6bdac636886479f5cb6d03d7ac48b2bebe27347bd4d91dbb56] \u2190 b-functional-pipeline\nfunctional_pipeline pipe compose fp Uses functional composition patterns with pipe() to chain data transformations instead of imperative loops. Shapes data-processing code suggestions toward functional pipe composition and away from imperative loops or nested map/filter chains.",
+ "context_tokens": 196,
+ "retrieve_time_ms": 1378.0,
+ "gold_answers": [
+ "b-functional-pipeline",
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Ranking stability",
+ "description": "Rare alias 'pipe' (high IDF, short field) outranks common alias 'error' \u2014 documents Atlas Search BM25 IDF weighting behavior",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-functional-pipeline",
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling",
+ "b-functional-pipeline"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Ranking stability"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cold-start-user-no-beliefs",
+ "query": "how should I structure my repository layer?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 697.2,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Cold start behavior",
+ "description": "User with zero seeded beliefs returns fully empty context at every output tier without error",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "isNull": true
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [],
+ "retrieved_questions": [],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Cold start behavior"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "narrator-voice-pinned-in-own-scope",
+ "query": "",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.0,
+ "gold_answers": [
+ "b-narrator-voice",
+ "b-protagonist-entity",
+ "b-prose-style-user-edited"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Persona prelude content",
+ "description": "Empty query against domain:writing surfaces all three pinned writing-scope beliefs in pinnedFacts and the writing open question in openQuestions",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-narrator-voice",
+ "b-protagonist-entity",
+ "b-prose-style-user-edited"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-chapter-structure-q"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Persona prelude content"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "multi-scope-prelude-uses-primary-scope-only",
+ "query": "give me an overview of what I'm working on",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.5,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Persona prelude content",
+ "description": "With scope=['code','writing'], personaPrelude is universal only \u2014 documents that scope-specific content is not injected",
+ "scope": [
+ "domain:code",
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current",
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q",
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Persona prelude content"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "why-it-matters-not-indexed-blind-spot",
+ "query": "Vue component architecture",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.8,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "'Vue' appears only in b-react-entity.why_it_matters \u2014 not in canonical_name, aliases, or content. The query must not surface the belief.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "budget-ceiling-evicts-relevant-beliefs",
+ "query": "How should I handle errors in my repository layer?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 3.1,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "maxBeliefs set to exact pinned count \u2014 no slots remain for relevantBeliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {
+ "maxBeliefs": 2
+ },
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 2,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "negative-guidance-mongoose",
+ "query": "suggest a good Mongoose schema for my user model",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x7127d24309af7604dadff4756f0e7343633e319f425369c234a9518c5d3b1f37] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.",
+ "context_tokens": 94,
+ "retrieve_time_ms": 1431.8,
+ "gold_answers": [
+ "b-mongo-raw-driver"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Counter-signal retrieval",
+ "description": "Query for Mongoose schema surfaces b-mongo-raw-driver \u2014 the raw-driver preference is the counter-signal",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-mongo-raw-driver"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-mongo-raw-driver"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Counter-signal retrieval"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "recency-tiebreak-reinforcement",
+ "query": "",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.0,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Ranking stability",
+ "description": "Among equally-scored pinned beliefs, higher last_reinforced_at sorts first in pinnedFacts",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Ranking stability"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "preference-flood-relevant-entity-survives",
+ "query": "how does our CI pipeline handle secrets?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xdfa78706c30e1507ce3b8c411338de38705f2db0d339868f5289b3eeecc13f19] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.",
+ "context_tokens": 105,
+ "retrieve_time_ms": 1448.4,
+ "gold_answers": [
+ "b-ci-gha-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "Query about a specific entity is not drowned out by high-reinforcement preferences at budget ceiling",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {
+ "maxBeliefs": 8
+ },
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ci-gha-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 8,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-ci-gha-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cross-scope-non-user-edited-blocked",
+ "query": "tell me about Mara",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.4,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Writing-scope entity with strong name match does not surface in code-scope query",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "mustExclude": [
+ "b-protagonist-entity"
+ ],
+ "shouldOnlyInclude": []
+ },
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-protagonist-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-two-word-shingle-dead-letter",
+ "query": "what happens to dead letter messages that fail replay?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x5b42d38ed0da578ae2f854a1b4ca14d7658d9f144e12887dc9d09d671808731a] \u2190 b-dead-letter-queue\ndead_letter_queue dead letter DLQ failed messages Failed messages route to a MongoDB-backed dead letter queue with manual replay via admin CLI. Shapes error recovery and messaging answers to assume a DLQ exists rather than suggesting one from scratch.",
+ "context_tokens": 103,
+ "retrieve_time_ms": 1907.7,
+ "gold_answers": [
+ "b-dead-letter-queue"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Two-word alias 'dead letter' fires via the shingle analyzer path on aliases.shingle multi-field",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-dead-letter-queue"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-dead-letter-queue"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-dlq-abbreviation-resolves",
+ "query": "how do we replay items from the DLQ?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x5b42d38ed0da578ae2f854a1b4ca14d7658d9f144e12887dc9d09d671808731a] \u2190 b-dead-letter-queue\ndead_letter_queue dead letter DLQ failed messages Failed messages route to a MongoDB-backed dead letter queue with manual replay via admin CLI. Shapes error recovery and messaging answers to assume a DLQ exists rather than suggesting one from scratch.",
+ "context_tokens": 103,
+ "retrieve_time_ms": 1188.1,
+ "gold_answers": [
+ "b-dead-letter-queue"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'DLQ' (abbreviation) drives text-search retrieval",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-dead-letter-queue"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-dead-letter-queue"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "counter-signal-jest-surfaces-vitest",
+ "query": "should I configure Jest for this project?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xe6790959d3fbc1f47bbd6fec3b8d3cee3ac6b1061fff2a4f65712308f88f2f26] \u2190 b-vitest-pref\ntesting_vitest Vitest vitest jest Uses Vitest for all testing. Replaced Jest. Shapes all testing examples toward Vitest and prevents suggesting Jest, Mocha, or Jasmine.",
+ "context_tokens": 96,
+ "retrieve_time_ms": 758.3,
+ "gold_answers": [
+ "b-vitest-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Counter-signal retrieval",
+ "description": "Counter-signal alias 'jest' on b-vitest-pref surfaces when user queries about Jest",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-vitest-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-vitest-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Counter-signal retrieval"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "relation-type-expands-participants",
+ "query": "what are the auth service dependencies and failure modes?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xd5b49f677c3bd34f302e09cbb7a7bf656e1eddd01b8184a0dc4649ddc27946d9] \u2190 b-auth-depends-redis\nauth_service_depends_on_redis auth redis session backend auth Auth service depends on Redis for session storage. If Redis is down, auth fails open (denies all). Shapes failure-mode analysis \u2014 auth and Redis are coupled; cannot discuss auth resilience without addressing Redis availability.",
+ "context_tokens": 113,
+ "retrieve_time_ms": 799.6,
+ "gold_answers": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Relation expansion",
+ "description": "Relation belief 'auth depends on redis' surfaces via alias match AND pulls its participants (b-redis-code) into relevantBeliefs via one-hop join. b-auth-open-q goes to openQuestions since it's an open_question type.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Relation expansion"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "relation-participant-open-question-routes-correctly",
+ "query": "session backend dependency graph",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xd5b49f677c3bd34f302e09cbb7a7bf656e1eddd01b8184a0dc4649ddc27946d9] \u2190 b-auth-depends-redis\nauth_service_depends_on_redis auth redis session backend auth Auth service depends on Redis for session storage. If Redis is down, auth fails open (denies all). Shapes failure-mode analysis \u2014 auth and Redis are coupled; cannot discuss auth resilience without addressing Redis availability.",
+ "context_tokens": 113,
+ "retrieve_time_ms": 925.3,
+ "gold_answers": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Relation expansion",
+ "description": "Participant b-auth-open-q is type=open_question and pinned. Even though it's pulled in via relation expansion, it must appear in openQuestions, not relevantBeliefs.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ },
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Relation expansion"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "relation-expansion-respects-scope-filter",
+ "query": "auth redis failure modes",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xd5b49f677c3bd34f302e09cbb7a7bf656e1eddd01b8184a0dc4649ddc27946d9] \u2190 b-auth-depends-redis\nauth_service_depends_on_redis auth redis session backend auth Auth service depends on Redis for session storage. If Redis is down, auth fails open (denies all). Shapes failure-mode analysis \u2014 auth and Redis are coupled; cannot discuss auth resilience without addressing Redis availability.",
+ "context_tokens": 113,
+ "retrieve_time_ms": 1865.6,
+ "gold_answers": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Relation expansion",
+ "description": "If a relation's participant is out of scope, it must NOT be pulled in by expansion. Hypothetical: b-auth-depends-redis has participant b-redis-writing (wrong scope). Scope filter in expandRelationParticipants blocks it.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "mustExclude": [
+ "b-redis-writing"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Relation expansion"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "subtype-expertise-retrieval",
+ "query": "what's my react depth \u2014 should you explain hooks to me?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x2c8bbdb535aad7416509a2412d6a7ec0ae4bb3fcfd1b52351a369549da093a32] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.",
+ "context_tokens": 91,
+ "retrieve_time_ms": 1051.4,
+ "gold_answers": [
+ "b-react-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "Subtype 'expertise' belief surfaces on a query about React skill calibration",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "pinned-plus-resolved-excluded-from-questions",
+ "query": "API versioning strategy for v2 migration",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.9,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "A belief that is both pinned AND has resolved_at set must NOT appear in openQuestions or pinnedFacts",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-pinned-resolved-interaction"
+ ]
+ },
+ "openQuestions": {
+ "mustExclude": [
+ "b-pinned-resolved-interaction"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "zero-reinforcement-fresh-belief-surfaces",
+ "query": "how is our Turborepo monorepo structured?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x736861a5b21a2578e19bb4e95ee8f9ed2a5931708c508e38fac5dd1b2e4bbaf1] \u2190 b-zero-reinforcement-fresh\nturborepo_monorepo Turborepo turbo monorepo Monorepo managed by Turborepo. Apps and packages in /apps and /packages directories. Shapes all build and workspace structure answers to assume Turborepo monorepo layout.",
+ "context_tokens": 115,
+ "retrieve_time_ms": 2040.5,
+ "gold_answers": [
+ "b-zero-reinforcement-fresh"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "Brand new belief with reinforcement_count: 0 still surfaces via alias match \u2014 recency/reinforcement must not suppress it",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-zero-reinforcement-fresh"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-zero-reinforcement-fresh"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "budget-zero-graceful-empty",
+ "query": "what's our stack?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.9,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "maxBeliefs: 0 returns empty arrays without throwing",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {
+ "maxBeliefs": 0
+ },
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 0,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "budget-one-pinned-wins-over-relevant",
+ "query": "MongoDB collection typing patterns",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x7127d24309af7604dadff4756f0e7343633e319f425369c234a9518c5d3b1f37] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.",
+ "context_tokens": 94,
+ "retrieve_time_ms": 1680.1,
+ "gold_answers": [
+ "b-db-decision"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "maxBeliefs: 1 with a query matching both a pinned belief and a non-pinned belief \u2014 pinned belief gets the slot",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {
+ "maxBeliefs": 1
+ },
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-db-decision"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 1,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "multi-scope-code-writing-dedup-no-double-surface",
+ "query": "what tools do we use?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.7,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "With scope=['domain:code','domain:writing'], a belief scoped to both does not appear twice",
+ "scope": [
+ "domain:code",
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current",
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q",
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "empty-alias-belief-content-path-only",
+ "query": "should you give me one recommendation or a list of options?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.8,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "b-comm-recommendation has aliases: [] \u2014 can only be retrieved via content path. Since content is not a scored search path, it should NOT surface.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "universal-scope-explicit-query-no-relevant",
+ "query": "how do I want you to respond to me?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.4,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Query scoped explicitly to user:universal \u2014 universal beliefs appear in personaPrelude only, not in relevantBeliefs",
+ "scope": [
+ "user:universal"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true,
+ "contains": [
+ "direct"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-comm-pushback",
+ "b-comm-response-length"
+ ],
+ "retrieved_questions": [],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "negation-query-still-surfaces-rejected-tool",
+ "query": "I do NOT want to use Express, what should I use instead?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xf969bfbb0be67007377257a5cee89269896c59b4f734ea0d42039305eedd4d3e] \u2190 b-fastify-pref\nframework_fastify Fastify express Uses Fastify for HTTP services rather than Express. Shapes server-side code examples toward Fastify and prevents suggesting Express as an alternative.",
+ "context_tokens": 84,
+ "retrieve_time_ms": 1330.3,
+ "gold_answers": [
+ "b-fastify-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "Query 'I do NOT want to use Express' still surfaces b-fastify-pref because text search has no negation awareness",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-fastify-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-fastify-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "superseded-chain-query-terminal-only",
+ "query": "TSLint migration path",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.9,
+ "gold_answers": [
+ "b-linting-current"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Supersession chain exclusion",
+ "description": "Query 'TSLint' matches b-linting-v0 (superseded by v1, which is superseded by b-linting-current). Only terminal active belief surfaces.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-linting-current"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Supersession chain exclusion"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "relation-does-not-appear-as-entity",
+ "query": "Redis session backend dependency graph",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xd5b49f677c3bd34f302e09cbb7a7bf656e1eddd01b8184a0dc4649ddc27946d9] \u2190 b-auth-depends-redis\nauth_service_depends_on_redis auth redis session backend auth Auth service depends on Redis for session storage. If Redis is down, auth fails open (denies all). Shapes failure-mode analysis \u2014 auth and Redis are coupled; cannot discuss auth resilience without addressing Redis availability.",
+ "context_tokens": 113,
+ "retrieve_time_ms": 1063.3,
+ "gold_answers": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Relation expansion",
+ "description": "Relation-type belief does not bleed into pinnedFacts despite alias overlap with pinned beliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Relation expansion"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ }
+ ],
+ "avg_retrieve_time_ms": 992.0,
+ "avg_context_tokens": 91.2
+}
\ No newline at end of file
diff --git a/public/evidence/precisionmembench/2026-11-10-77/runs/run-3-74of77.json b/public/evidence/precisionmembench/2026-11-10-77/runs/run-3-74of77.json
new file mode 100644
index 0000000..d57bae9
--- /dev/null
+++ b/public/evidence/precisionmembench/2026-11-10-77/runs/run-3-74of77.json
@@ -0,0 +1,4888 @@
+{
+ "dataset": "precisionmembench",
+ "split": "single-turn",
+ "category": null,
+ "memory_provider": "proofable",
+ "run_name": "proofable",
+ "mode": "retrieval",
+ "oracle": false,
+ "total_queries": 77,
+ "correct": 74,
+ "accuracy": 0.961038961038961,
+ "mean_precision": 0.9167,
+ "mean_recall": 0.9535,
+ "active_passes": 40,
+ "active_total": 43,
+ "ingestion_time_ms": 201141.7,
+ "ingested_docs": 35,
+ "description": null,
+ "answer_llm": null,
+ "judge_llm": null,
+ "results": [
+ {
+ "query_id": "happy-stack-repository-layer",
+ "query": "How should I structure my repository layer?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 14.2,
+ "gold_answers": [
+ "b-db-decision"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "Repository layer query: pinned DB decision surfaces; scoped open question surfaces; relevant beliefs depend on query token overlap",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-db-decision"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "happy-alias-match-kubernetes",
+ "query": "what's my k8s deploy story?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x763ad5fb173fb9545ee6b6f8167fbe38f661568ab2ccc4b7dce690181c71fd0a] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.",
+ "context_tokens": 100,
+ "retrieve_time_ms": 779.4,
+ "gold_answers": [
+ "b-kubernetes-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'k8s' resolves to Kubernetes entity",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-kubernetes-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-kubernetes-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-kube-resolves",
+ "query": "how does kube schedule pods across our CI pipeline nodes?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x763ad5fb173fb9545ee6b6f8167fbe38f661568ab2ccc4b7dce690181c71fd0a] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.\n\n2. [0xc2cab89345b96f7ed22e77139990f244a5c3b38026e1b60bba220e9a0cb4bd05] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.",
+ "context_tokens": 202,
+ "retrieve_time_ms": 5523.3,
+ "gold_answers": [
+ "b-ci-gha-entity",
+ "b-kubernetes-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'kube' drives text search retrieval",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ci-gha-entity",
+ "b-kubernetes-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-kubernetes-entity",
+ "b-ci-gha-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-gha-resolves",
+ "query": "GHA secrets config",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xc2cab89345b96f7ed22e77139990f244a5c3b38026e1b60bba220e9a0cb4bd05] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.",
+ "context_tokens": 108,
+ "retrieve_time_ms": 2262.8,
+ "gold_answers": [
+ "b-ci-gha-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'GHA' (short form) drives text-search retrieval of b-ci-gha-entity",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ci-gha-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-ci-gha-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "long-query-rambling-kubernetes",
+ "query": "ok so I've been thinking about this for a while and I know we've talked about infrastructure before but I'm circling back because the team had some pushback in the last review and honestly I'm not sure what the right call is anymore. Basically the question is how we should handle rolling deploys for the landing page when it lands \u2014 should we do Kubernetes native rolling updates or something more custom, and does it matter that the service holds session state in memory?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x763ad5fb173fb9545ee6b6f8167fbe38f661568ab2ccc4b7dce690181c71fd0a] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.\n\n2. [0xddb9ebb721a3e31c9701c046d6d698c9f179eaee6f4a82b8c3eb39af4b5c1be8] \u2190 b-auth-depends-redis\nauth_service_depends_on_redis auth redis session backend auth Auth service depends on Redis for session storage. If Redis is down, auth fails open (denies all). Shapes failure-mode analysis \u2014 auth and Redis are coupled; cannot discuss auth resilience without addressing Redis availability.",
+ "context_tokens": 210,
+ "retrieve_time_ms": 3591.5,
+ "gold_answers": [
+ "b-kubernetes-entity"
+ ],
+ "correct": false,
+ "judge_reason": "unexpected belief in relevantBeliefs: b-auth-depends-redis; unexpected belief in relevantBeliefs: b-redis-code",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Verbose 400+ char query.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-kubernetes-entity"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": false,
+ "relevant_beliefs": [
+ "b-kubernetes-entity",
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 0.3333333333333333,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [
+ "unexpected belief in relevantBeliefs: b-auth-depends-redis",
+ "unexpected belief in relevantBeliefs: b-redis-code"
+ ],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "out-of-scope-coin-collecting",
+ "query": "what's a good way to grade a 1909-S VDB penny without sending it to PCGS?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.3,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Query in a domain the user has no beliefs about: personaPrelude only, no scope/relevant leakage",
+ "scope": [
+ "domain:hobby"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [],
+ "retrieved_questions": [],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "happy-writing-scope",
+ "query": "How should I open the next chapter with Mara?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.1,
+ "gold_answers": [
+ "b-protagonist-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Writing query surfaces writing beliefs only; no code leakage",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-protagonist-entity"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "conflict-scope-disambiguation-redis-writing",
+ "query": "What's Redis doing in chapter three?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x6dd3a9c4b8678679574979cc643f7ec1b7baa896bed57d3889dfb1396cf1525f] \u2190 b-redis-writing\nredis_character Redis Redis is a supporting character in the novel \u2014 a jazz pianist Mara meets in Lisbon. Shapes interpretation of 'Redis' in the writing scope \u2014 always the jazz pianist character, not the database technology.",
+ "context_tokens": 99,
+ "retrieve_time_ms": 2069.0,
+ "gold_answers": [
+ "b-redis-writing"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "'Redis' in writing scope returns character, not datastore",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-writing"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-writing"
+ ],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "conflict-scope-disambiguation-redis-code",
+ "query": "What are we using Redis for?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x6708912ab30343dba7083b1d9a76f2ec08ef8a1ebff9e3c67003af96cc04f68d] \u2190 b-redis-code\nredis_cache Redis Redis is used as a cache and rate-limiter in the backend. Single-node, not clustered. Shapes interpretation of 'Redis' in the code scope \u2014 always the caching datastore, not the fictional character.",
+ "context_tokens": 100,
+ "retrieve_time_ms": 833.7,
+ "gold_answers": [
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "'Redis' in code scope returns datastore, not character",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-code"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "multi-scope-redis-both-surfaces",
+ "query": "What's Redis doing?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x6708912ab30343dba7083b1d9a76f2ec08ef8a1ebff9e3c67003af96cc04f68d] \u2190 b-redis-code\nredis_cache Redis Redis is used as a cache and rate-limiter in the backend. Single-node, not clustered. Shapes interpretation of 'Redis' in the code scope \u2014 always the caching datastore, not the fictional character.\n\n2. [0x6dd3a9c4b8678679574979cc643f7ec1b7baa896bed57d3889dfb1396cf1525f] \u2190 b-redis-writing\nredis_character Redis Redis is a supporting character in the novel \u2014 a jazz pianist Mara meets in Lisbon. Shapes interpretation of 'Redis' in the writing scope \u2014 always the jazz pianist character, not the database technology.",
+ "context_tokens": 193,
+ "retrieve_time_ms": 2521.9,
+ "gold_answers": [
+ "b-redis-code",
+ "b-redis-writing"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "With scope=['code','writing'], both Redis beliefs should surface \u2014 one per scope",
+ "scope": [
+ "domain:code",
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-code",
+ "b-redis-writing"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-code",
+ "b-redis-writing"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current",
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q",
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "conflict-mongodb-supersedes-sqlalchemy",
+ "query": "how's our SQLAlchemy setup these days?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 3.4,
+ "gold_answers": [
+ "b-db-decision"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Supersession chain exclusion",
+ "description": "Superseded SQLAlchemy belief never surfaces",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-db-decision"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Supersession chain exclusion"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "edge-empty-query",
+ "query": "",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.0,
+ "gold_answers": [
+ "b-db-decision"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "Empty query: persona + scope prelude + pinned facts injected, no relevant beliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-db-decision"
+ ],
+ "mustExclude": [
+ "b-auth-open-q",
+ "b-comm-response-length"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "edge-short-query-raw-passthrough",
+ "query": "fastify?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xf784eb348247c285e7a896a6eb6778f434c493b27980156073cbd33f67ac7704] \u2190 b-fastify-pref\nframework_fastify Fastify express Uses Fastify for HTTP services rather than Express. Shapes server-side code examples toward Fastify and prevents suggesting Express as an alternative.",
+ "context_tokens": 82,
+ "retrieve_time_ms": 826.9,
+ "gold_answers": [
+ "b-fastify-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Short query 'fastify?' passes raw to text search. Belief is pinned so verified in pinnedFacts.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-fastify-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-fastify-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "isolation-user-scoping",
+ "query": "what language do I write in?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x1ff57ca446cd25b1487bc6dc78c263b6c217eaa9bb7ec94982d4280ca60c5681] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.",
+ "context_tokens": 83,
+ "retrieve_time_ms": 1131.8,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Cross-user isolation",
+ "description": "Other-user beliefs never leak",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "mustExclude": [
+ "b-other-user-leak-check"
+ ]
+ },
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-other-user-leak-check"
+ ]
+ },
+ "personaPrelude": {
+ "mustNotContain": [
+ "Haskell"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [
+ "b-ts-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 0.0,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Cross-user isolation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "questions-pinned-active-only",
+ "query": "any open questions?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.2,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "listPinnedOpenQuestions returns only pinned+active questions for the given scope",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ],
+ "mustExclude": [
+ "b-chapter-structure-q",
+ "b-hosting-resolved"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-ts-resolves",
+ "query": "TS strict mode question about unknown types",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x1ff57ca446cd25b1487bc6dc78c263b6c217eaa9bb7ec94982d4280ca60c5681] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.",
+ "context_tokens": 83,
+ "retrieve_time_ms": 1367.7,
+ "gold_answers": [
+ "b-ts-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'TS' drives text-search retrieval.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ts-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-ts-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-mongo-resolves",
+ "query": "what's the Mongo pattern we're using for collections?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x7a1057767bd26fd54dbd4aadcc00f059b8801f1fcd9a8d8a9a07f689e0a351bc] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.\n\n2. [0x1e96f89ea88d61bd0728dadf73820def3e4b15d0dd277f98ec2082798a2c5deb] \u2190 b-db-decision\ndatabase_mongodb database Decided on MongoDB after evaluating Postgres. Raw driver only. Shapes all data-layer recommendations \u2014 database choice is settled and prevents re-litigating Postgres as an alternative.",
+ "context_tokens": 187,
+ "retrieve_time_ms": 2819.4,
+ "gold_answers": [
+ "b-mongo-raw-driver"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'Mongo' drives text-search retrieval.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-mongo-raw-driver"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-mongo-raw-driver"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-reactjs-resolves",
+ "query": "how should I structure my ReactJS components here?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x8c69a5d1d82507f09ca48ac53ae0ad41a52b8043d9a66521381a7a6e771c859e] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.",
+ "context_tokens": 95,
+ "retrieve_time_ms": 1476.9,
+ "gold_answers": [
+ "b-react-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'ReactJS' resolves to b-react-entity via text search \u2014 genuine retrieval proof",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-protagonist-resolves",
+ "query": "what does the protagonist want, and does their viewpoint stay consistent in act two?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0xbc7da86840d8ddc0b5c77e6b27813a788ebe7670e06673a83aac45a5ffcdbfb2] \u2190 b-protagonist-entity\nmara protagonist Protagonist of the novel. Set in 1970s Lisbon. Does not reconcile with her father; ending is ambiguous. Shapes all plot suggestions \u2014 the father-daughter thread must remain unresolved; prevents any reconciliation arc.\n\n2. [0x0e150915a7dab50308ac13a45f0a7f44bc9ac8d3fabe86e294bd62dfcda10e46] \u2190 b-pov-shift-entity\npov_shift POV point of view perspective shift viewpoint chapter 7 Chapter 7 uses a deliberate POV shift to an unnamed observer. This is the only chapter that breaks close-third. Shapes editorial feedback \u2014 prevents flagging the chapter 7 POV shift as an error; the break is intentional and structurally load-bearing.",
+ "context_tokens": 221,
+ "retrieve_time_ms": 1959.5,
+ "gold_answers": [
+ "b-pov-shift-entity",
+ "b-protagonist-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'protagonist' retrieves b-protagonist-entity (pinned). b-pov-shift-entity (non-pinned) proves text search fired.",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-protagonist-entity"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-pov-shift-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-pov-shift-entity"
+ ],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "canonical-name-match-mara",
+ "query": "tell me about Mara's backstory with her father",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.4,
+ "gold_answers": [
+ "b-protagonist-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Direct canonical_name 'Mara' retrieves protagonist. b-protagonist-entity is pinned.",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-protagonist-entity"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-pov-shift-resolves",
+ "query": "POV rules for chapter 7",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x0e150915a7dab50308ac13a45f0a7f44bc9ac8d3fabe86e294bd62dfcda10e46] \u2190 b-pov-shift-entity\npov_shift POV point of view perspective shift viewpoint chapter 7 Chapter 7 uses a deliberate POV shift to an unnamed observer. This is the only chapter that breaks close-third. Shapes editorial feedback \u2014 prevents flagging the chapter 7 POV shift as an error; the break is intentional and structurally load-bearing.",
+ "context_tokens": 121,
+ "retrieve_time_ms": 973.4,
+ "gold_answers": [
+ "b-pov-shift-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'POV' drives text-search retrieval of b-pov-shift-entity (non-pinned, writing scope)",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-pov-shift-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-pov-shift-entity"
+ ],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-base-class-retrieves-composition",
+ "query": "should I use a base class hierarchy for these domain objects?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x778a5991b2d5a1b350b28da7d50a6cf392b523d2b0a2bac4d86459f00bf8c20b] \u2190 b-composition-inheritance\ncomposition_over_inheritance inheritance class hierarchies base class factory function Prefers composition over inheritance. Factory functions returning plain objects rather than class hierarchies. Shapes object design suggestions away from class hierarchies toward factory functions returning plain objects.",
+ "context_tokens": 109,
+ "retrieve_time_ms": 1070.3,
+ "gold_answers": [
+ "b-composition-inheritance"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Natural-language proxy 'base class' retrieves b-composition-inheritance (non-pinned) via alias",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-composition-inheritance"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-composition-inheritance"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-exceptions-retrieves-error-handling",
+ "query": "when should I throw exceptions versus return them?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xf9f47ff532f5adf735c6ced093ca3e24c1fa90287e5ddd6da83b8a2295915975] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.",
+ "context_tokens": 103,
+ "retrieve_time_ms": 1387.2,
+ "gold_answers": [
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Natural-language proxy 'exceptions' retrieves b-error-handling (non-pinned) via alias",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "filter-resolved-at-excluded",
+ "query": "should we self-host the landing page or go with Fly.io?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.6,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "b-hosting-resolved has resolved_at set \u2014 query matching its aliases must not surface it",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ },
+ "openQuestions": {
+ "mustExclude": [
+ "b-hosting-resolved"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "scope-bleed-questions-code-excludes-writing",
+ "query": "any unresolved decisions on this sprint?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.1,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Code-scope query must not surface writing-scope open questions",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ],
+ "mustExclude": [
+ "b-chapter-structure-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "persona-prelude-content-present",
+ "query": "what's our stack?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.9,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Persona prelude content",
+ "description": "Persona prelude contains substantive content tokens derived from universal preferences",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true,
+ "contains": [
+ "direct",
+ "preamble"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Persona prelude content"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "edge-short-query-score-clears",
+ "query": "k8s?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x763ad5fb173fb9545ee6b6f8167fbe38f661568ab2ccc4b7dce690181c71fd0a] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.",
+ "context_tokens": 100,
+ "retrieve_time_ms": 918.4,
+ "gold_answers": [
+ "b-kubernetes-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Short query 'k8s'",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-kubernetes-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-kubernetes-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "edge-whitespace-query",
+ "query": " ",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.0,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "Whitespace-only query is treated as empty by queryExpander \u2014 no relevant beliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "score-boundary-weak-match-filtered",
+ "query": "thinking about observability and monitoring dashboards",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 3.6,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "Weakly-related in-scope query does not surface tangential beliefs below minScore",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "user-edited-cross-scope-surfaces",
+ "query": "summarize what you know about my preferences",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.4,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "user_edited belief scoped to domain:writing does NOT surface in domain:code \u2014 scope isolation takes precedence over user_edited flag",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-prose-style-user-edited"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "user-edited-absent-without-flag",
+ "query": "what are my writing preferences?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.6,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Non-user-edited writing belief does NOT cross into code scope pinnedFacts",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-narrator-voice"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "ranking-canonical-name-over-content",
+ "query": "composition over inheritance \u2014 what's our pattern?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x778a5991b2d5a1b350b28da7d50a6cf392b523d2b0a2bac4d86459f00bf8c20b] \u2190 b-composition-inheritance\ncomposition_over_inheritance inheritance class hierarchies base class factory function Prefers composition over inheritance. Factory functions returning plain objects rather than class hierarchies. Shapes object design suggestions away from class hierarchies toward factory functions returning plain objects.",
+ "context_tokens": 109,
+ "retrieve_time_ms": 766.5,
+ "gold_answers": [
+ "b-composition-inheritance"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Ranking stability",
+ "description": "canonical_name match (weight 10) outranks content-only match (weight 1) in relevantBeliefs ordering",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-composition-inheritance"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-composition-inheritance"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Ranking stability"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-typo-blind-spot",
+ "query": "how do I configure typscript strict mode?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x1ff57ca446cd25b1487bc6dc78c263b6c217eaa9bb7ec94982d4280ca60c5681] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.",
+ "context_tokens": 83,
+ "retrieve_time_ms": 1267.4,
+ "gold_answers": [
+ "b-ts-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Typo 'typscript' does NOT match 'TypeScript' \u2014 documents text search limitation",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ts-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-ts-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-all-caps-case-insensitive",
+ "query": "REACTJS component patterns",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x8c69a5d1d82507f09ca48ac53ae0ad41a52b8043d9a66521381a7a6e771c859e] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.",
+ "context_tokens": 95,
+ "retrieve_time_ms": 1666.0,
+ "gold_answers": [
+ "b-react-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "ALL-CAPS 'REACTJS' still resolves via case-insensitive text index",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-filler-heavy-extraction",
+ "query": "ugh so like I was wondering you know about how we should think about error handling in our codebase because honestly it's been bugging me for a while and I'm not really sure what the right approach is here",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xf9f47ff532f5adf735c6ced093ca3e24c1fa90287e5ddd6da83b8a2295915975] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.",
+ "context_tokens": 103,
+ "retrieve_time_ms": 1272.0,
+ "gold_answers": [
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Filler-heavy query \u2014 NLP extraction must cut through noise to find 'error handling'",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-negation-surfaces-topic",
+ "query": "I don't want class hierarchies in this codebase, what should I use instead?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x778a5991b2d5a1b350b28da7d50a6cf392b523d2b0a2bac4d86459f00bf8c20b] \u2190 b-composition-inheritance\ncomposition_over_inheritance inheritance class hierarchies base class factory function Prefers composition over inheritance. Factory functions returning plain objects rather than class hierarchies. Shapes object design suggestions away from class hierarchies toward factory functions returning plain objects.",
+ "context_tokens": 109,
+ "retrieve_time_ms": 1437.7,
+ "gold_answers": [
+ "b-composition-inheritance"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Negation 'I don't want class hierarchies' still surfaces the composition belief \u2014 text search has no semantic negation",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-composition-inheritance"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-composition-inheritance"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-compound-surfaces-two-beliefs",
+ "query": "how should ReactJS components handle exceptions from async calls?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x8c69a5d1d82507f09ca48ac53ae0ad41a52b8043d9a66521381a7a6e771c859e] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.\n\n2. [0xf9f47ff532f5adf735c6ced093ca3e24c1fa90287e5ddd6da83b8a2295915975] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.",
+ "context_tokens": 192,
+ "retrieve_time_ms": 2507.4,
+ "gold_answers": [
+ "b-react-entity",
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Single query spanning two topics surfaces both non-pinned beliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity",
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity",
+ "b-error-handling"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cap-stress-compound-six-entities",
+ "query": "When our React frontend catches error handling exceptions, how should the Kubernetes pods running MongoDB TypeScript services report to GitHub Actions?",
+ "answer": "7 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (7)\n\n1. [0x8c69a5d1d82507f09ca48ac53ae0ad41a52b8043d9a66521381a7a6e771c859e] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.\n\n2. [0xf9f47ff532f5adf735c6ced093ca3e24c1fa90287e5ddd6da83b8a2295915975] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.\n\n3. [0x763ad5fb173fb9545ee6b6f8167fbe38f661568ab2ccc4b7dce690181c71fd0a] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.\n\n4. [0x7a1057767bd26fd54dbd4aadcc00f059b8801f1fcd9a8d8a9a07f689e0a351bc] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.\n\n5. [0x1ff57ca446cd25b1487bc6dc78c263b6c217eaa9bb7ec94982d4280ca60c5681] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.\n\n6. [0xc2cab89345b96f7ed22e77139990f244a5c3b38026e1b60bba220e9a0cb4bd05] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.\n\n7. [0x1e96f89ea88d61bd0728dadf73820def3e4b15d0dd277f98ec2082798a2c5deb] \u2190 b-db-decision\ndatabase_mongodb database Decided on MongoDB after evaluating Postgres. Raw driver only. Shapes all data-layer recommendations \u2014 database choice is settled and prevents re-litigating Postgres as an alternative.",
+ "context_tokens": 646,
+ "retrieve_time_ms": 3126.7,
+ "gold_answers": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Compound query mentions 6 entities \u2014 tests whether 12-token cap in buildSearchQuery drops non-pinned targets",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-ci-gha-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 7
+ },
+ "retrieved_count": 7
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cap-stress-kitchen-sink-exceeds-twelve",
+ "query": "Set up our React components to use composition pattern with explicit error handling, wire GitHub Actions CI to deploy Kubernetes pods serving the MongoDB TypeScript API with functional pipeline transforms",
+ "answer": "8 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (8)\n\n1. [0x8c69a5d1d82507f09ca48ac53ae0ad41a52b8043d9a66521381a7a6e771c859e] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.\n\n2. [0xf9f47ff532f5adf735c6ced093ca3e24c1fa90287e5ddd6da83b8a2295915975] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.\n\n3. [0xc2cab89345b96f7ed22e77139990f244a5c3b38026e1b60bba220e9a0cb4bd05] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.\n\n4. [0x763ad5fb173fb9545ee6b6f8167fbe38f661568ab2ccc4b7dce690181c71fd0a] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.\n\n5. [0x7a1057767bd26fd54dbd4aadcc00f059b8801f1fcd9a8d8a9a07f689e0a351bc] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.\n\n6. [0x1e96f89ea88d61bd0728dadf73820def3e4b15d0dd277f98ec2082798a2c5deb] \u2190 b-db-decision\ndatabase_mongodb database Decided on MongoDB after evaluating Postgres. Raw driver only. Shapes all data-layer recommendations \u2014 database choice is settled and prevents re-litigating Postgres as an alternative.\n\n7. [0x1ff57ca446cd25b1487bc6dc78c263b6c217eaa9bb7ec94982d4280ca60c5681] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.\n\n8. [0xd5ad00b5c76b5ff6c1e4608af753f4a6fd3f59374d13493ed5a769493f0cb10e] \u2190 b-functional-pipeline\nfunctional_pipeline pipe compose fp Uses functional composition patterns with pipe() to chain data transformations instead of imperative loops. Shapes data-processing code suggestions toward functional pipe composition and away from imperative loops or nested map/filter chains.",
+ "context_tokens": 739,
+ "retrieve_time_ms": 13134.0,
+ "gold_answers": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-functional-pipeline"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Dense 8+ entity query that deliberately exceeds the 12-token cap \u2014 tests which non-pinned beliefs survive extraction truncation",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-functional-pipeline"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-functional-pipeline"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 8
+ },
+ "retrieved_count": 8
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cap-stress-late-position-entity-at-risk",
+ "query": "Given our React frontend, the GitHub Actions CI pipeline, the Kubernetes deployment, and the MongoDB TypeScript backend, how should we approach exceptions in the data layer?",
+ "answer": "7 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (7)\n\n1. [0xf9f47ff532f5adf735c6ced093ca3e24c1fa90287e5ddd6da83b8a2295915975] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.\n\n2. [0x7a1057767bd26fd54dbd4aadcc00f059b8801f1fcd9a8d8a9a07f689e0a351bc] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.\n\n3. [0x1ff57ca446cd25b1487bc6dc78c263b6c217eaa9bb7ec94982d4280ca60c5681] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.\n\n4. [0x1e96f89ea88d61bd0728dadf73820def3e4b15d0dd277f98ec2082798a2c5deb] \u2190 b-db-decision\ndatabase_mongodb database Decided on MongoDB after evaluating Postgres. Raw driver only. Shapes all data-layer recommendations \u2014 database choice is settled and prevents re-litigating Postgres as an alternative.\n\n5. [0x8c69a5d1d82507f09ca48ac53ae0ad41a52b8043d9a66521381a7a6e771c859e] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.\n\n6. [0xc2cab89345b96f7ed22e77139990f244a5c3b38026e1b60bba220e9a0cb4bd05] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.\n\n7. [0x763ad5fb173fb9545ee6b6f8167fbe38f661568ab2ccc4b7dce690181c71fd0a] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.",
+ "context_tokens": 646,
+ "retrieve_time_ms": 3797.6,
+ "gold_answers": [
+ "b-ci-gha-entity",
+ "b-react-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Critical alias 'exceptions' appears after 6+ proper nouns \u2014 tests whether extraction order causes late-position tokens to fall past the 12-token cap",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ci-gha-entity",
+ "b-react-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-react-entity",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 7
+ },
+ "retrieved_count": 7
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-single-edit-fires-non-pinned",
+ "query": "eror handling patterns",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xf9f47ff532f5adf735c6ced093ca3e24c1fa90287e5ddd6da83b8a2295915975] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.",
+ "context_tokens": 103,
+ "retrieve_time_ms": 1205.6,
+ "gold_answers": [
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "1-edit deletion 'eror' fuzzy-matches alias 'error' \u2014 prefix 'er' passes",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-scope-filter-discriminates-after-fuzzy-fires",
+ "query": "Reedis cache eviction strategy",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x6708912ab30343dba7083b1d9a76f2ec08ef8a1ebff9e3c67003af96cc04f68d] \u2190 b-redis-code\nredis_cache Redis Redis is used as a cache and rate-limiter in the backend. Single-node, not clustered. Shapes interpretation of 'Redis' in the code scope \u2014 always the caching datastore, not the fictional character.",
+ "context_tokens": 100,
+ "retrieve_time_ms": 1618.9,
+ "gold_answers": [
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Typo 'Reedis' fuzzy-matches alias 'Redis' in both scopes \u2014 scope filter returns only code belief",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-code"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-prefix-blocks-transitive-match",
+ "query": "mango connection pooling",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 4.6,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "1-edit distance but prefix guard blocks \u2014 'mango' shares edit distance 1 with 'mongo' but 'ma' \u2260 'mo'",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-abbreviation-prefix-blind-spot",
+ "query": "k9s pod scheduling",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.9,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Alias 'k8s' \u2014 typo 'k9s' fails because prefix 'k9' \u2260 'k8' \u2014 single-char substitution in short aliases hits prefix guard",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-two-edit-transposition-blocked",
+ "query": "typsecript compilation config",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x1ff57ca446cd25b1487bc6dc78c263b6c217eaa9bb7ec94982d4280ca60c5681] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.",
+ "context_tokens": 83,
+ "retrieve_time_ms": 861.7,
+ "gold_answers": [
+ "b-ts-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Transposition 'typsecript' IS matched by Atlas (Damerau-Levenshtein, 1 edit)",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ts-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-ts-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-single-term-only-signal",
+ "query": "mongod",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x7a1057767bd26fd54dbd4aadcc00f059b8801f1fcd9a8d8a9a07f689e0a351bc] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.\n\n2. [0x1e96f89ea88d61bd0728dadf73820def3e4b15d0dd277f98ec2082798a2c5deb] \u2190 b-db-decision\ndatabase_mongodb database Decided on MongoDB after evaluating Postgres. Raw driver only. Shapes all data-layer recommendations \u2014 database choice is settled and prevents re-litigating Postgres as an alternative.",
+ "context_tokens": 187,
+ "retrieve_time_ms": 1490.3,
+ "gold_answers": [
+ "b-mongo-raw-driver"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Query is a single misspelled term \u2014 'mongod' is the only token, 1 edit from alias 'Mongo', prefix 'mo' passes",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-mongo-raw-driver"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-mongo-raw-driver"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "universal-scope-belief-content-in-persona-prelude",
+ "query": "what's a good way to grade a 1909-S VDB penny without sending it to PCGS?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.4,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Persona prelude content",
+ "description": "user:universal belief content appears in personaPrelude for a domain with no per-scope description",
+ "scope": [
+ "domain:hobby"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true,
+ "contains": [
+ "direct",
+ "preamble"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [],
+ "retrieved_questions": [],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Persona prelude content"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "type-isolation-open-question-not-in-relevant",
+ "query": "JWT authentication token strategy",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.9,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "open_question belief with directly matched aliases must appear in openQuestions only \u2014 not in pinnedFacts or relevantBeliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-auth-open-q"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "multi-hop-supersession-chain-all-excluded",
+ "query": "ESLint rules and TSLint configuration",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.4,
+ "gold_answers": [
+ "b-linting-current"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Supersession chain exclusion",
+ "description": "Three-hop chain: TSLint (b-linting-v0) \u2192 ESLint (b-linting-v1) \u2192 Biome (b-linting-current). Query terms matching both superseded hops must surface neither; only the active terminal belief surfaces via pinnedFacts",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-linting-current"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Supersession chain exclusion"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "cross-scope-one-empty-domain-does-not-degrade",
+ "query": "What are we using Redis for?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x6708912ab30343dba7083b1d9a76f2ec08ef8a1ebff9e3c67003af96cc04f68d] \u2190 b-redis-code\nredis_cache Redis Redis is used as a cache and rate-limiter in the backend. Single-node, not clustered. Shapes interpretation of 'Redis' in the code scope \u2014 always the caching datastore, not the fictional character.",
+ "context_tokens": 100,
+ "retrieve_time_ms": 773.5,
+ "gold_answers": [
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Multi-scope query where one scope (domain:hobby) has zero beliefs \u2014 code-scope retrieval must be unaffected and no empty-scope error must occur",
+ "scope": [
+ "domain:code",
+ "domain:hobby"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-code"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "ranking-orderedbefore-alias-beats-content",
+ "query": "explicit error returns in pipe transforms",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0xf9f47ff532f5adf735c6ced093ca3e24c1fa90287e5ddd6da83b8a2295915975] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.\n\n2. [0xd5ad00b5c76b5ff6c1e4608af753f4a6fd3f59374d13493ed5a769493f0cb10e] \u2190 b-functional-pipeline\nfunctional_pipeline pipe compose fp Uses functional composition patterns with pipe() to chain data transformations instead of imperative loops. Shapes data-processing code suggestions toward functional pipe composition and away from imperative loops or nested map/filter chains.",
+ "context_tokens": 196,
+ "retrieve_time_ms": 1209.6,
+ "gold_answers": [
+ "b-functional-pipeline",
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Ranking stability",
+ "description": "Rare alias 'pipe' (high IDF, short field) outranks common alias 'error' \u2014 documents Atlas Search BM25 IDF weighting behavior",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-functional-pipeline",
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling",
+ "b-functional-pipeline"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Ranking stability"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cold-start-user-no-beliefs",
+ "query": "how should I structure my repository layer?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 709.8,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Cold start behavior",
+ "description": "User with zero seeded beliefs returns fully empty context at every output tier without error",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "isNull": true
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [],
+ "retrieved_questions": [],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Cold start behavior"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "narrator-voice-pinned-in-own-scope",
+ "query": "",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.0,
+ "gold_answers": [
+ "b-narrator-voice",
+ "b-protagonist-entity",
+ "b-prose-style-user-edited"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Persona prelude content",
+ "description": "Empty query against domain:writing surfaces all three pinned writing-scope beliefs in pinnedFacts and the writing open question in openQuestions",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-narrator-voice",
+ "b-protagonist-entity",
+ "b-prose-style-user-edited"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-chapter-structure-q"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Persona prelude content"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "multi-scope-prelude-uses-primary-scope-only",
+ "query": "give me an overview of what I'm working on",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.7,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Persona prelude content",
+ "description": "With scope=['code','writing'], personaPrelude is universal only \u2014 documents that scope-specific content is not injected",
+ "scope": [
+ "domain:code",
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current",
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q",
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Persona prelude content"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "why-it-matters-not-indexed-blind-spot",
+ "query": "Vue component architecture",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.3,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "'Vue' appears only in b-react-entity.why_it_matters \u2014 not in canonical_name, aliases, or content. The query must not surface the belief.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "budget-ceiling-evicts-relevant-beliefs",
+ "query": "How should I handle errors in my repository layer?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 3.6,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "maxBeliefs set to exact pinned count \u2014 no slots remain for relevantBeliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {
+ "maxBeliefs": 2
+ },
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 2,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "negative-guidance-mongoose",
+ "query": "suggest a good Mongoose schema for my user model",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 3106.5,
+ "gold_answers": [
+ "b-mongo-raw-driver"
+ ],
+ "correct": false,
+ "judge_reason": "missing expected belief: b-mongo-raw-driver",
+ "score": null,
+ "meta": {
+ "category": "Counter-signal retrieval",
+ "description": "Query for Mongoose schema surfaces b-mongo-raw-driver \u2014 the raw-driver preference is the counter-signal",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-mongo-raw-driver"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 0.0,
+ "retrieval_recall": 0.0,
+ "pinned_coverage": null,
+ "failures": [
+ "missing expected belief: b-mongo-raw-driver"
+ ],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Counter-signal retrieval"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "recency-tiebreak-reinforcement",
+ "query": "",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.0,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Ranking stability",
+ "description": "Among equally-scored pinned beliefs, higher last_reinforced_at sorts first in pinnedFacts",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Ranking stability"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "preference-flood-relevant-entity-survives",
+ "query": "how does our CI pipeline handle secrets?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2330.9,
+ "gold_answers": [
+ "b-ci-gha-entity"
+ ],
+ "correct": false,
+ "judge_reason": "missing expected belief: b-ci-gha-entity",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "Query about a specific entity is not drowned out by high-reinforcement preferences at budget ceiling",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {
+ "maxBeliefs": 8
+ },
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ci-gha-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 8,
+ "pass_type": "active",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 0.0,
+ "retrieval_recall": 0.0,
+ "pinned_coverage": null,
+ "failures": [
+ "missing expected belief: b-ci-gha-entity"
+ ],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cross-scope-non-user-edited-blocked",
+ "query": "tell me about Mara",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.7,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Writing-scope entity with strong name match does not surface in code-scope query",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "mustExclude": [
+ "b-protagonist-entity"
+ ],
+ "shouldOnlyInclude": []
+ },
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-protagonist-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-two-word-shingle-dead-letter",
+ "query": "what happens to dead letter messages that fail replay?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x8ce8d9be5d7c82f868c97e26ae9dff18d799ab2845d5361879ceeb22232aa039] \u2190 b-dead-letter-queue\ndead_letter_queue dead letter DLQ failed messages Failed messages route to a MongoDB-backed dead letter queue with manual replay via admin CLI. Shapes error recovery and messaging answers to assume a DLQ exists rather than suggesting one from scratch.",
+ "context_tokens": 101,
+ "retrieve_time_ms": 1211.2,
+ "gold_answers": [
+ "b-dead-letter-queue"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Two-word alias 'dead letter' fires via the shingle analyzer path on aliases.shingle multi-field",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-dead-letter-queue"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-dead-letter-queue"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-dlq-abbreviation-resolves",
+ "query": "how do we replay items from the DLQ?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x8ce8d9be5d7c82f868c97e26ae9dff18d799ab2845d5361879ceeb22232aa039] \u2190 b-dead-letter-queue\ndead_letter_queue dead letter DLQ failed messages Failed messages route to a MongoDB-backed dead letter queue with manual replay via admin CLI. Shapes error recovery and messaging answers to assume a DLQ exists rather than suggesting one from scratch.",
+ "context_tokens": 101,
+ "retrieve_time_ms": 940.6,
+ "gold_answers": [
+ "b-dead-letter-queue"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'DLQ' (abbreviation) drives text-search retrieval",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-dead-letter-queue"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-dead-letter-queue"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "counter-signal-jest-surfaces-vitest",
+ "query": "should I configure Jest for this project?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xe1d5f2ecc9e63a1c7987340495044d0b6f1ba5798c0919a53bc594b5b205c5e5] \u2190 b-vitest-pref\ntesting_vitest Vitest vitest jest Uses Vitest for all testing. Replaced Jest. Shapes all testing examples toward Vitest and prevents suggesting Jest, Mocha, or Jasmine.",
+ "context_tokens": 97,
+ "retrieve_time_ms": 1229.6,
+ "gold_answers": [
+ "b-vitest-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Counter-signal retrieval",
+ "description": "Counter-signal alias 'jest' on b-vitest-pref surfaces when user queries about Jest",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-vitest-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-vitest-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Counter-signal retrieval"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "relation-type-expands-participants",
+ "query": "what are the auth service dependencies and failure modes?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xddb9ebb721a3e31c9701c046d6d698c9f179eaee6f4a82b8c3eb39af4b5c1be8] \u2190 b-auth-depends-redis\nauth_service_depends_on_redis auth redis session backend auth Auth service depends on Redis for session storage. If Redis is down, auth fails open (denies all). Shapes failure-mode analysis \u2014 auth and Redis are coupled; cannot discuss auth resilience without addressing Redis availability.",
+ "context_tokens": 116,
+ "retrieve_time_ms": 1478.8,
+ "gold_answers": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Relation expansion",
+ "description": "Relation belief 'auth depends on redis' surfaces via alias match AND pulls its participants (b-redis-code) into relevantBeliefs via one-hop join. b-auth-open-q goes to openQuestions since it's an open_question type.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Relation expansion"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "relation-participant-open-question-routes-correctly",
+ "query": "session backend dependency graph",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xddb9ebb721a3e31c9701c046d6d698c9f179eaee6f4a82b8c3eb39af4b5c1be8] \u2190 b-auth-depends-redis\nauth_service_depends_on_redis auth redis session backend auth Auth service depends on Redis for session storage. If Redis is down, auth fails open (denies all). Shapes failure-mode analysis \u2014 auth and Redis are coupled; cannot discuss auth resilience without addressing Redis availability.",
+ "context_tokens": 116,
+ "retrieve_time_ms": 1340.5,
+ "gold_answers": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Relation expansion",
+ "description": "Participant b-auth-open-q is type=open_question and pinned. Even though it's pulled in via relation expansion, it must appear in openQuestions, not relevantBeliefs.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ },
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Relation expansion"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "relation-expansion-respects-scope-filter",
+ "query": "auth redis failure modes",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xddb9ebb721a3e31c9701c046d6d698c9f179eaee6f4a82b8c3eb39af4b5c1be8] \u2190 b-auth-depends-redis\nauth_service_depends_on_redis auth redis session backend auth Auth service depends on Redis for session storage. If Redis is down, auth fails open (denies all). Shapes failure-mode analysis \u2014 auth and Redis are coupled; cannot discuss auth resilience without addressing Redis availability.",
+ "context_tokens": 116,
+ "retrieve_time_ms": 2925.6,
+ "gold_answers": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Relation expansion",
+ "description": "If a relation's participant is out of scope, it must NOT be pulled in by expansion. Hypothetical: b-auth-depends-redis has participant b-redis-writing (wrong scope). Scope filter in expandRelationParticipants blocks it.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "mustExclude": [
+ "b-redis-writing"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Relation expansion"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "subtype-expertise-retrieval",
+ "query": "what's my react depth \u2014 should you explain hooks to me?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x8c69a5d1d82507f09ca48ac53ae0ad41a52b8043d9a66521381a7a6e771c859e] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.",
+ "context_tokens": 95,
+ "retrieve_time_ms": 1390.9,
+ "gold_answers": [
+ "b-react-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "Subtype 'expertise' belief surfaces on a query about React skill calibration",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "pinned-plus-resolved-excluded-from-questions",
+ "query": "API versioning strategy for v2 migration",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.6,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "A belief that is both pinned AND has resolved_at set must NOT appear in openQuestions or pinnedFacts",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-pinned-resolved-interaction"
+ ]
+ },
+ "openQuestions": {
+ "mustExclude": [
+ "b-pinned-resolved-interaction"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "zero-reinforcement-fresh-belief-surfaces",
+ "query": "how is our Turborepo monorepo structured?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x6218915ad7b17b7517523a9ded44c258e8c9691a5b01e8c7728c70f868c8ea60] \u2190 b-zero-reinforcement-fresh\nturborepo_monorepo Turborepo turbo monorepo Monorepo managed by Turborepo. Apps and packages in /apps and /packages directories. Shapes all build and workspace structure answers to assume Turborepo monorepo layout.",
+ "context_tokens": 114,
+ "retrieve_time_ms": 949.7,
+ "gold_answers": [
+ "b-zero-reinforcement-fresh"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "Brand new belief with reinforcement_count: 0 still surfaces via alias match \u2014 recency/reinforcement must not suppress it",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-zero-reinforcement-fresh"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-zero-reinforcement-fresh"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "budget-zero-graceful-empty",
+ "query": "what's our stack?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.9,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "maxBeliefs: 0 returns empty arrays without throwing",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {
+ "maxBeliefs": 0
+ },
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 0,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "budget-one-pinned-wins-over-relevant",
+ "query": "MongoDB collection typing patterns",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x7a1057767bd26fd54dbd4aadcc00f059b8801f1fcd9a8d8a9a07f689e0a351bc] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.",
+ "context_tokens": 98,
+ "retrieve_time_ms": 1347.0,
+ "gold_answers": [
+ "b-db-decision"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "maxBeliefs: 1 with a query matching both a pinned belief and a non-pinned belief \u2014 pinned belief gets the slot",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {
+ "maxBeliefs": 1
+ },
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-db-decision"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 1,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "multi-scope-code-writing-dedup-no-double-surface",
+ "query": "what tools do we use?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.7,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "With scope=['domain:code','domain:writing'], a belief scoped to both does not appear twice",
+ "scope": [
+ "domain:code",
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current",
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q",
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "empty-alias-belief-content-path-only",
+ "query": "should you give me one recommendation or a list of options?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.3,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "b-comm-recommendation has aliases: [] \u2014 can only be retrieved via content path. Since content is not a scored search path, it should NOT surface.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "universal-scope-explicit-query-no-relevant",
+ "query": "how do I want you to respond to me?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.4,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Query scoped explicitly to user:universal \u2014 universal beliefs appear in personaPrelude only, not in relevantBeliefs",
+ "scope": [
+ "user:universal"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true,
+ "contains": [
+ "direct"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-comm-pushback",
+ "b-comm-response-length"
+ ],
+ "retrieved_questions": [],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "negation-query-still-surfaces-rejected-tool",
+ "query": "I do NOT want to use Express, what should I use instead?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xf784eb348247c285e7a896a6eb6778f434c493b27980156073cbd33f67ac7704] \u2190 b-fastify-pref\nframework_fastify Fastify express Uses Fastify for HTTP services rather than Express. Shapes server-side code examples toward Fastify and prevents suggesting Express as an alternative.",
+ "context_tokens": 82,
+ "retrieve_time_ms": 982.6,
+ "gold_answers": [
+ "b-fastify-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "Query 'I do NOT want to use Express' still surfaces b-fastify-pref because text search has no negation awareness",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-fastify-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-fastify-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "superseded-chain-query-terminal-only",
+ "query": "TSLint migration path",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 3.5,
+ "gold_answers": [
+ "b-linting-current"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Supersession chain exclusion",
+ "description": "Query 'TSLint' matches b-linting-v0 (superseded by v1, which is superseded by b-linting-current). Only terminal active belief surfaces.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-linting-current"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Supersession chain exclusion"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "relation-does-not-appear-as-entity",
+ "query": "Redis session backend dependency graph",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xddb9ebb721a3e31c9701c046d6d698c9f179eaee6f4a82b8c3eb39af4b5c1be8] \u2190 b-auth-depends-redis\nauth_service_depends_on_redis auth redis session backend auth Auth service depends on Redis for session storage. If Redis is down, auth fails open (denies all). Shapes failure-mode analysis \u2014 auth and Redis are coupled; cannot discuss auth resilience without addressing Redis availability.",
+ "context_tokens": 116,
+ "retrieve_time_ms": 1432.7,
+ "gold_answers": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Relation expansion",
+ "description": "Relation-type belief does not bleed into pinnedFacts despite alias overlap with pinned beliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Relation expansion"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ }
+ ],
+ "avg_retrieve_time_ms": 1157.0,
+ "avg_context_tokens": 91.5
+}
\ No newline at end of file
diff --git a/public/evidence/precisionmembench/2026-11-10-77/single-turn.json b/public/evidence/precisionmembench/2026-11-10-77/single-turn.json
new file mode 100644
index 0000000..c4fcbb1
--- /dev/null
+++ b/public/evidence/precisionmembench/2026-11-10-77/single-turn.json
@@ -0,0 +1,4887 @@
+{
+ "dataset": "precisionmembench",
+ "split": "single-turn",
+ "category": null,
+ "memory_provider": "proofable",
+ "run_name": "proofable-k2",
+ "mode": "retrieval",
+ "oracle": false,
+ "total_queries": 77,
+ "correct": 77,
+ "accuracy": 1.0,
+ "mean_precision": 0.9773,
+ "mean_recall": 1.0,
+ "active_passes": 43,
+ "active_total": 43,
+ "ingestion_time_ms": 232322.3,
+ "ingested_docs": 35,
+ "description": null,
+ "answer_llm": null,
+ "judge_llm": null,
+ "results": [
+ {
+ "query_id": "happy-stack-repository-layer",
+ "query": "How should I structure my repository layer?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 14.5,
+ "gold_answers": [
+ "b-db-decision"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "Repository layer query: pinned DB decision surfaces; scoped open question surfaces; relevant beliefs depend on query token overlap",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-db-decision"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "happy-alias-match-kubernetes",
+ "query": "what's my k8s deploy story?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xb9c30e137f3fe1ba706a7b66369dc2a6dc0e7dc1e4ddfa397bf3fd1c76551c1e] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.",
+ "context_tokens": 106,
+ "retrieve_time_ms": 1202.2,
+ "gold_answers": [
+ "b-kubernetes-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'k8s' resolves to Kubernetes entity",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-kubernetes-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-kubernetes-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-kube-resolves",
+ "query": "how does kube schedule pods across our CI pipeline nodes?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0xb9c30e137f3fe1ba706a7b66369dc2a6dc0e7dc1e4ddfa397bf3fd1c76551c1e] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.\n\n2. [0x3a922991324403cb75c69c1bee4b2129b89943f70e4df9a652672f53475b4efa] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.",
+ "context_tokens": 205,
+ "retrieve_time_ms": 3890.4,
+ "gold_answers": [
+ "b-ci-gha-entity",
+ "b-kubernetes-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'kube' drives text search retrieval",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ci-gha-entity",
+ "b-kubernetes-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-kubernetes-entity",
+ "b-ci-gha-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-gha-resolves",
+ "query": "GHA secrets config",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x3a922991324403cb75c69c1bee4b2129b89943f70e4df9a652672f53475b4efa] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.",
+ "context_tokens": 105,
+ "retrieve_time_ms": 2341.5,
+ "gold_answers": [
+ "b-ci-gha-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'GHA' (short form) drives text-search retrieval of b-ci-gha-entity",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ci-gha-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-ci-gha-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "long-query-rambling-kubernetes",
+ "query": "ok so I've been thinking about this for a while and I know we've talked about infrastructure before but I'm circling back because the team had some pushback in the last review and honestly I'm not sure what the right call is anymore. Basically the question is how we should handle rolling deploys for the landing page when it lands \u2014 should we do Kubernetes native rolling updates or something more custom, and does it matter that the service holds session state in memory?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xb9c30e137f3fe1ba706a7b66369dc2a6dc0e7dc1e4ddfa397bf3fd1c76551c1e] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.",
+ "context_tokens": 106,
+ "retrieve_time_ms": 2260.0,
+ "gold_answers": [
+ "b-kubernetes-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Verbose 400+ char query.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-kubernetes-entity"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-kubernetes-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "out-of-scope-coin-collecting",
+ "query": "what's a good way to grade a 1909-S VDB penny without sending it to PCGS?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.3,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Query in a domain the user has no beliefs about: personaPrelude only, no scope/relevant leakage",
+ "scope": [
+ "domain:hobby"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [],
+ "retrieved_questions": [],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "happy-writing-scope",
+ "query": "How should I open the next chapter with Mara?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.1,
+ "gold_answers": [
+ "b-protagonist-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Writing query surfaces writing beliefs only; no code leakage",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-protagonist-entity"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "conflict-scope-disambiguation-redis-writing",
+ "query": "What's Redis doing in chapter three?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xafd79eb0f1accec2143e53eb2a2245139091e22b4036bacc6f6ea9fd1b03d11c] \u2190 b-redis-writing\nredis_character Redis Redis is a supporting character in the novel \u2014 a jazz pianist Mara meets in Lisbon. Shapes interpretation of 'Redis' in the writing scope \u2014 always the jazz pianist character, not the database technology.",
+ "context_tokens": 100,
+ "retrieve_time_ms": 1699.4,
+ "gold_answers": [
+ "b-redis-writing"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "'Redis' in writing scope returns character, not datastore",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-writing"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-writing"
+ ],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "conflict-scope-disambiguation-redis-code",
+ "query": "What are we using Redis for?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x2b4e8c0824b86ca1ea97648343b1dcf188817dbc3451d44fbf8c310b8b15aa63] \u2190 b-redis-code\nredis_cache Redis Redis is used as a cache and rate-limiter in the backend. Single-node, not clustered. Shapes interpretation of 'Redis' in the code scope \u2014 always the caching datastore, not the fictional character.",
+ "context_tokens": 100,
+ "retrieve_time_ms": 1206.0,
+ "gold_answers": [
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "'Redis' in code scope returns datastore, not character",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-code"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "multi-scope-redis-both-surfaces",
+ "query": "What's Redis doing?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x2b4e8c0824b86ca1ea97648343b1dcf188817dbc3451d44fbf8c310b8b15aa63] \u2190 b-redis-code\nredis_cache Redis Redis is used as a cache and rate-limiter in the backend. Single-node, not clustered. Shapes interpretation of 'Redis' in the code scope \u2014 always the caching datastore, not the fictional character.\n\n2. [0xafd79eb0f1accec2143e53eb2a2245139091e22b4036bacc6f6ea9fd1b03d11c] \u2190 b-redis-writing\nredis_character Redis Redis is a supporting character in the novel \u2014 a jazz pianist Mara meets in Lisbon. Shapes interpretation of 'Redis' in the writing scope \u2014 always the jazz pianist character, not the database technology.",
+ "context_tokens": 194,
+ "retrieve_time_ms": 2483.1,
+ "gold_answers": [
+ "b-redis-code",
+ "b-redis-writing"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "With scope=['code','writing'], both Redis beliefs should surface \u2014 one per scope",
+ "scope": [
+ "domain:code",
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-code",
+ "b-redis-writing"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-code",
+ "b-redis-writing"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current",
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q",
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "conflict-mongodb-supersedes-sqlalchemy",
+ "query": "how's our SQLAlchemy setup these days?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 3.6,
+ "gold_answers": [
+ "b-db-decision"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Supersession chain exclusion",
+ "description": "Superseded SQLAlchemy belief never surfaces",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-db-decision"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Supersession chain exclusion"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "edge-empty-query",
+ "query": "",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.0,
+ "gold_answers": [
+ "b-db-decision"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "Empty query: persona + scope prelude + pinned facts injected, no relevant beliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-db-decision"
+ ],
+ "mustExclude": [
+ "b-auth-open-q",
+ "b-comm-response-length"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "edge-short-query-raw-passthrough",
+ "query": "fastify?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x474da1f52940ab49a0a252eb9e8d1b355d8c6e7d5ac83b90f3b97bbfaaf5170e] \u2190 b-fastify-pref\nframework_fastify Fastify express Uses Fastify for HTTP services rather than Express. Shapes server-side code examples toward Fastify and prevents suggesting Express as an alternative.",
+ "context_tokens": 93,
+ "retrieve_time_ms": 1081.7,
+ "gold_answers": [
+ "b-fastify-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Short query 'fastify?' passes raw to text search. Belief is pinned so verified in pinnedFacts.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-fastify-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-fastify-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "isolation-user-scoping",
+ "query": "what language do I write in?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xa15bcd6d98145e343e8278e64ed63e711104a54b9371506a680e775002941aa1] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.",
+ "context_tokens": 79,
+ "retrieve_time_ms": 996.8,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Cross-user isolation",
+ "description": "Other-user beliefs never leak",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "mustExclude": [
+ "b-other-user-leak-check"
+ ]
+ },
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-other-user-leak-check"
+ ]
+ },
+ "personaPrelude": {
+ "mustNotContain": [
+ "Haskell"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [
+ "b-ts-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 0.0,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Cross-user isolation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "questions-pinned-active-only",
+ "query": "any open questions?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.7,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "listPinnedOpenQuestions returns only pinned+active questions for the given scope",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ],
+ "mustExclude": [
+ "b-chapter-structure-q",
+ "b-hosting-resolved"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-ts-resolves",
+ "query": "TS strict mode question about unknown types",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xa15bcd6d98145e343e8278e64ed63e711104a54b9371506a680e775002941aa1] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.",
+ "context_tokens": 79,
+ "retrieve_time_ms": 949.6,
+ "gold_answers": [
+ "b-ts-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'TS' drives text-search retrieval.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ts-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-ts-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-mongo-resolves",
+ "query": "what's the Mongo pattern we're using for collections?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x75fb6600161c81a15cec4ca72e4f1c159fb15b1bfbe24e139cf9e77deecf82bc] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.\n\n2. [0x88bd805a7820c33f75ab2a17261095ac20e9af9985250418ff9b3f144e019d5c] \u2190 b-db-decision\ndatabase_mongodb database Decided on MongoDB after evaluating Postgres. Raw driver only. Shapes all data-layer recommendations \u2014 database choice is settled and prevents re-litigating Postgres as an alternative.",
+ "context_tokens": 183,
+ "retrieve_time_ms": 3048.5,
+ "gold_answers": [
+ "b-mongo-raw-driver"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'Mongo' drives text-search retrieval.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-mongo-raw-driver"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-mongo-raw-driver"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-reactjs-resolves",
+ "query": "how should I structure my ReactJS components here?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x96cf0916fcffef60d783c2cc764f1dfd1ba25c1f11d8942d84067580466eddd5] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.",
+ "context_tokens": 90,
+ "retrieve_time_ms": 966.3,
+ "gold_answers": [
+ "b-react-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'ReactJS' resolves to b-react-entity via text search \u2014 genuine retrieval proof",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-protagonist-resolves",
+ "query": "what does the protagonist want, and does their viewpoint stay consistent in act two?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x1781bd0176e222a7a92a9908b5ae201d2f0513156e84e224564d4a3784c1656a] \u2190 b-protagonist-entity\nmara protagonist Protagonist of the novel. Set in 1970s Lisbon. Does not reconcile with her father; ending is ambiguous. Shapes all plot suggestions \u2014 the father-daughter thread must remain unresolved; prevents any reconciliation arc.\n\n2. [0x91b8299a163189e8d0b675b648e474b82a159eeb40000c6fc4b007f614b7cd54] \u2190 b-pov-shift-entity\npov_shift POV point of view perspective shift viewpoint chapter 7 Chapter 7 uses a deliberate POV shift to an unnamed observer. This is the only chapter that breaks close-third. Shapes editorial feedback \u2014 prevents flagging the chapter 7 POV shift as an error; the break is intentional and structurally load-bearing.",
+ "context_tokens": 220,
+ "retrieve_time_ms": 2027.9,
+ "gold_answers": [
+ "b-pov-shift-entity",
+ "b-protagonist-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'protagonist' retrieves b-protagonist-entity (pinned). b-pov-shift-entity (non-pinned) proves text search fired.",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-protagonist-entity"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-pov-shift-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-pov-shift-entity"
+ ],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "canonical-name-match-mara",
+ "query": "tell me about Mara's backstory with her father",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.9,
+ "gold_answers": [
+ "b-protagonist-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Direct canonical_name 'Mara' retrieves protagonist. b-protagonist-entity is pinned.",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-protagonist-entity"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-pov-shift-resolves",
+ "query": "POV rules for chapter 7",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x91b8299a163189e8d0b675b648e474b82a159eeb40000c6fc4b007f614b7cd54] \u2190 b-pov-shift-entity\npov_shift POV point of view perspective shift viewpoint chapter 7 Chapter 7 uses a deliberate POV shift to an unnamed observer. This is the only chapter that breaks close-third. Shapes editorial feedback \u2014 prevents flagging the chapter 7 POV shift as an error; the break is intentional and structurally load-bearing.",
+ "context_tokens": 119,
+ "retrieve_time_ms": 1392.4,
+ "gold_answers": [
+ "b-pov-shift-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'POV' drives text-search retrieval of b-pov-shift-entity (non-pinned, writing scope)",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-pov-shift-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-pov-shift-entity"
+ ],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-base-class-retrieves-composition",
+ "query": "should I use a base class hierarchy for these domain objects?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xa36c887e7ded278f1fea1d59b1fff9e6368ed26d92817226f137f49b034f5994] \u2190 b-composition-inheritance\ncomposition_over_inheritance inheritance class hierarchies base class factory function Prefers composition over inheritance. Factory functions returning plain objects rather than class hierarchies. Shapes object design suggestions away from class hierarchies toward factory functions returning plain objects.",
+ "context_tokens": 102,
+ "retrieve_time_ms": 1128.7,
+ "gold_answers": [
+ "b-composition-inheritance"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Natural-language proxy 'base class' retrieves b-composition-inheritance (non-pinned) via alias",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-composition-inheritance"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-composition-inheritance"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-exceptions-retrieves-error-handling",
+ "query": "when should I throw exceptions versus return them?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x51428d969a3c4768136ea0ee473e045c0820114d3820c24faa5d141f3e76e883] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.",
+ "context_tokens": 104,
+ "retrieve_time_ms": 1010.7,
+ "gold_answers": [
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Natural-language proxy 'exceptions' retrieves b-error-handling (non-pinned) via alias",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "filter-resolved-at-excluded",
+ "query": "should we self-host the landing page or go with Fly.io?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 3.3,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "b-hosting-resolved has resolved_at set \u2014 query matching its aliases must not surface it",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ },
+ "openQuestions": {
+ "mustExclude": [
+ "b-hosting-resolved"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "scope-bleed-questions-code-excludes-writing",
+ "query": "any unresolved decisions on this sprint?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.3,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Code-scope query must not surface writing-scope open questions",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ],
+ "mustExclude": [
+ "b-chapter-structure-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "persona-prelude-content-present",
+ "query": "what's our stack?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.9,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Persona prelude content",
+ "description": "Persona prelude contains substantive content tokens derived from universal preferences",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true,
+ "contains": [
+ "direct",
+ "preamble"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Persona prelude content"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "edge-short-query-score-clears",
+ "query": "k8s?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xb9c30e137f3fe1ba706a7b66369dc2a6dc0e7dc1e4ddfa397bf3fd1c76551c1e] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.",
+ "context_tokens": 106,
+ "retrieve_time_ms": 1169.3,
+ "gold_answers": [
+ "b-kubernetes-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Short query 'k8s'",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-kubernetes-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-kubernetes-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "edge-whitespace-query",
+ "query": " ",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.0,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "Whitespace-only query is treated as empty by queryExpander \u2014 no relevant beliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "score-boundary-weak-match-filtered",
+ "query": "thinking about observability and monitoring dashboards",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 3.9,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "Weakly-related in-scope query does not surface tangential beliefs below minScore",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "user-edited-cross-scope-surfaces",
+ "query": "summarize what you know about my preferences",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.8,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "user_edited belief scoped to domain:writing does NOT surface in domain:code \u2014 scope isolation takes precedence over user_edited flag",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-prose-style-user-edited"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "user-edited-absent-without-flag",
+ "query": "what are my writing preferences?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.6,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Non-user-edited writing belief does NOT cross into code scope pinnedFacts",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-narrator-voice"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "ranking-canonical-name-over-content",
+ "query": "composition over inheritance \u2014 what's our pattern?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xa36c887e7ded278f1fea1d59b1fff9e6368ed26d92817226f137f49b034f5994] \u2190 b-composition-inheritance\ncomposition_over_inheritance inheritance class hierarchies base class factory function Prefers composition over inheritance. Factory functions returning plain objects rather than class hierarchies. Shapes object design suggestions away from class hierarchies toward factory functions returning plain objects.",
+ "context_tokens": 102,
+ "retrieve_time_ms": 1112.0,
+ "gold_answers": [
+ "b-composition-inheritance"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Ranking stability",
+ "description": "canonical_name match (weight 10) outranks content-only match (weight 1) in relevantBeliefs ordering",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-composition-inheritance"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-composition-inheritance"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Ranking stability"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-typo-blind-spot",
+ "query": "how do I configure typscript strict mode?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xa15bcd6d98145e343e8278e64ed63e711104a54b9371506a680e775002941aa1] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.",
+ "context_tokens": 79,
+ "retrieve_time_ms": 994.7,
+ "gold_answers": [
+ "b-ts-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Typo 'typscript' does NOT match 'TypeScript' \u2014 documents text search limitation",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ts-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-ts-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-all-caps-case-insensitive",
+ "query": "REACTJS component patterns",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x96cf0916fcffef60d783c2cc764f1dfd1ba25c1f11d8942d84067580466eddd5] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.",
+ "context_tokens": 90,
+ "retrieve_time_ms": 1811.4,
+ "gold_answers": [
+ "b-react-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "ALL-CAPS 'REACTJS' still resolves via case-insensitive text index",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-filler-heavy-extraction",
+ "query": "ugh so like I was wondering you know about how we should think about error handling in our codebase because honestly it's been bugging me for a while and I'm not really sure what the right approach is here",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x51428d969a3c4768136ea0ee473e045c0820114d3820c24faa5d141f3e76e883] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.",
+ "context_tokens": 104,
+ "retrieve_time_ms": 692.4,
+ "gold_answers": [
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Filler-heavy query \u2014 NLP extraction must cut through noise to find 'error handling'",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-negation-surfaces-topic",
+ "query": "I don't want class hierarchies in this codebase, what should I use instead?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xa36c887e7ded278f1fea1d59b1fff9e6368ed26d92817226f137f49b034f5994] \u2190 b-composition-inheritance\ncomposition_over_inheritance inheritance class hierarchies base class factory function Prefers composition over inheritance. Factory functions returning plain objects rather than class hierarchies. Shapes object design suggestions away from class hierarchies toward factory functions returning plain objects.",
+ "context_tokens": 102,
+ "retrieve_time_ms": 837.1,
+ "gold_answers": [
+ "b-composition-inheritance"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Negation 'I don't want class hierarchies' still surfaces the composition belief \u2014 text search has no semantic negation",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-composition-inheritance"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-composition-inheritance"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "messy-compound-surfaces-two-beliefs",
+ "query": "how should ReactJS components handle exceptions from async calls?",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x51428d969a3c4768136ea0ee473e045c0820114d3820c24faa5d141f3e76e883] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.\n\n2. [0x96cf0916fcffef60d783c2cc764f1dfd1ba25c1f11d8942d84067580466eddd5] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.",
+ "context_tokens": 188,
+ "retrieve_time_ms": 1594.4,
+ "gold_answers": [
+ "b-react-entity",
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Single query spanning two topics surfaces both non-pinned beliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity",
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling",
+ "b-react-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cap-stress-compound-six-entities",
+ "query": "When our React frontend catches error handling exceptions, how should the Kubernetes pods running MongoDB TypeScript services report to GitHub Actions?",
+ "answer": "7 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (7)\n\n1. [0x96cf0916fcffef60d783c2cc764f1dfd1ba25c1f11d8942d84067580466eddd5] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.\n\n2. [0x51428d969a3c4768136ea0ee473e045c0820114d3820c24faa5d141f3e76e883] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.\n\n3. [0xb9c30e137f3fe1ba706a7b66369dc2a6dc0e7dc1e4ddfa397bf3fd1c76551c1e] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.\n\n4. [0x75fb6600161c81a15cec4ca72e4f1c159fb15b1bfbe24e139cf9e77deecf82bc] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.\n\n5. [0x88bd805a7820c33f75ab2a17261095ac20e9af9985250418ff9b3f144e019d5c] \u2190 b-db-decision\ndatabase_mongodb database Decided on MongoDB after evaluating Postgres. Raw driver only. Shapes all data-layer recommendations \u2014 database choice is settled and prevents re-litigating Postgres as an alternative.\n\n6. [0xa15bcd6d98145e343e8278e64ed63e711104a54b9371506a680e775002941aa1] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.\n\n7. [0x3a922991324403cb75c69c1bee4b2129b89943f70e4df9a652672f53475b4efa] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.",
+ "context_tokens": 637,
+ "retrieve_time_ms": 2326.1,
+ "gold_answers": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Compound query mentions 6 entities \u2014 tests whether 12-token cap in buildSearchQuery drops non-pinned targets",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-ci-gha-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 7
+ },
+ "retrieved_count": 7
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cap-stress-kitchen-sink-exceeds-twelve",
+ "query": "Set up our React components to use composition pattern with explicit error handling, wire GitHub Actions CI to deploy Kubernetes pods serving the MongoDB TypeScript API with functional pipeline transforms",
+ "answer": "8 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (8)\n\n1. [0x96cf0916fcffef60d783c2cc764f1dfd1ba25c1f11d8942d84067580466eddd5] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.\n\n2. [0x51428d969a3c4768136ea0ee473e045c0820114d3820c24faa5d141f3e76e883] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.\n\n3. [0x3a922991324403cb75c69c1bee4b2129b89943f70e4df9a652672f53475b4efa] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.\n\n4. [0xb9c30e137f3fe1ba706a7b66369dc2a6dc0e7dc1e4ddfa397bf3fd1c76551c1e] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.\n\n5. [0x75fb6600161c81a15cec4ca72e4f1c159fb15b1bfbe24e139cf9e77deecf82bc] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.\n\n6. [0x88bd805a7820c33f75ab2a17261095ac20e9af9985250418ff9b3f144e019d5c] \u2190 b-db-decision\ndatabase_mongodb database Decided on MongoDB after evaluating Postgres. Raw driver only. Shapes all data-layer recommendations \u2014 database choice is settled and prevents re-litigating Postgres as an alternative.\n\n7. [0xa15bcd6d98145e343e8278e64ed63e711104a54b9371506a680e775002941aa1] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.\n\n8. [0x050f1e25b632f0054c0b1eec60f93a79ea976d539b9b4d9a7aef3e96fcdb642f] \u2190 b-functional-pipeline\nfunctional_pipeline pipe compose fp Uses functional composition patterns with pipe() to chain data transformations instead of imperative loops. Shapes data-processing code suggestions toward functional pipe composition and away from imperative loops or nested map/filter chains.",
+ "context_tokens": 731,
+ "retrieve_time_ms": 5459.6,
+ "gold_answers": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-functional-pipeline"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Dense 8+ entity query that deliberately exceeds the 12-token cap \u2014 tests which non-pinned beliefs survive extraction truncation",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-functional-pipeline"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity",
+ "b-error-handling",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-functional-pipeline"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 8
+ },
+ "retrieved_count": 8
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cap-stress-late-position-entity-at-risk",
+ "query": "Given our React frontend, the GitHub Actions CI pipeline, the Kubernetes deployment, and the MongoDB TypeScript backend, how should we approach exceptions in the data layer?",
+ "answer": "7 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (7)\n\n1. [0x51428d969a3c4768136ea0ee473e045c0820114d3820c24faa5d141f3e76e883] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.\n\n2. [0x75fb6600161c81a15cec4ca72e4f1c159fb15b1bfbe24e139cf9e77deecf82bc] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.\n\n3. [0xa15bcd6d98145e343e8278e64ed63e711104a54b9371506a680e775002941aa1] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.\n\n4. [0x88bd805a7820c33f75ab2a17261095ac20e9af9985250418ff9b3f144e019d5c] \u2190 b-db-decision\ndatabase_mongodb database Decided on MongoDB after evaluating Postgres. Raw driver only. Shapes all data-layer recommendations \u2014 database choice is settled and prevents re-litigating Postgres as an alternative.\n\n5. [0x96cf0916fcffef60d783c2cc764f1dfd1ba25c1f11d8942d84067580466eddd5] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.\n\n6. [0x3a922991324403cb75c69c1bee4b2129b89943f70e4df9a652672f53475b4efa] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.\n\n7. [0xb9c30e137f3fe1ba706a7b66369dc2a6dc0e7dc1e4ddfa397bf3fd1c76551c1e] \u2190 b-kubernetes-entity\nkubernetes k8s kube Deploys services to a Kubernetes cluster on EKS. Uses Helm charts checked into the monorepo. Shapes all deployment and infrastructure answers to assume Kubernetes on EKS rather than bare EC2 or Lambda.",
+ "context_tokens": 637,
+ "retrieve_time_ms": 2350.3,
+ "gold_answers": [
+ "b-ci-gha-entity",
+ "b-react-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Critical alias 'exceptions' appears after 6+ proper nouns \u2014 tests whether extraction order causes late-position tokens to fall past the 12-token cap",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ci-gha-entity",
+ "b-react-entity",
+ "b-kubernetes-entity",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling",
+ "b-mongo-raw-driver",
+ "b-ts-pref",
+ "b-react-entity",
+ "b-ci-gha-entity",
+ "b-kubernetes-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 7
+ },
+ "retrieved_count": 7
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-single-edit-fires-non-pinned",
+ "query": "eror handling patterns",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x51428d969a3c4768136ea0ee473e045c0820114d3820c24faa5d141f3e76e883] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.",
+ "context_tokens": 104,
+ "retrieve_time_ms": 1065.3,
+ "gold_answers": [
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "1-edit deletion 'eror' fuzzy-matches alias 'error' \u2014 prefix 'er' passes",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-scope-filter-discriminates-after-fuzzy-fires",
+ "query": "Reedis cache eviction strategy",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x2b4e8c0824b86ca1ea97648343b1dcf188817dbc3451d44fbf8c310b8b15aa63] \u2190 b-redis-code\nredis_cache Redis Redis is used as a cache and rate-limiter in the backend. Single-node, not clustered. Shapes interpretation of 'Redis' in the code scope \u2014 always the caching datastore, not the fictional character.",
+ "context_tokens": 100,
+ "retrieve_time_ms": 1483.6,
+ "gold_answers": [
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Typo 'Reedis' fuzzy-matches alias 'Redis' in both scopes \u2014 scope filter returns only code belief",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-code"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-prefix-blocks-transitive-match",
+ "query": "mango connection pooling",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.4,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "1-edit distance but prefix guard blocks \u2014 'mango' shares edit distance 1 with 'mongo' but 'ma' \u2260 'mo'",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-abbreviation-prefix-blind-spot",
+ "query": "k9s pod scheduling",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.7,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Alias 'k8s' \u2014 typo 'k9s' fails because prefix 'k9' \u2260 'k8' \u2014 single-char substitution in short aliases hits prefix guard",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-two-edit-transposition-blocked",
+ "query": "typsecript compilation config",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xa15bcd6d98145e343e8278e64ed63e711104a54b9371506a680e775002941aa1] \u2190 b-ts-pref\nlanguage_typescript TypeScript TS Writes TypeScript with strict mode and no implicit any. Shapes all code examples toward TypeScript with strict mode and no implicit any.",
+ "context_tokens": 79,
+ "retrieve_time_ms": 1018.5,
+ "gold_answers": [
+ "b-ts-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Transposition 'typsecript' IS matched by Atlas (Damerau-Levenshtein, 1 edit)",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ts-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-ts-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "fuzzy-single-term-only-signal",
+ "query": "mongod",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x75fb6600161c81a15cec4ca72e4f1c159fb15b1bfbe24e139cf9e77deecf82bc] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.\n\n2. [0x88bd805a7820c33f75ab2a17261095ac20e9af9985250418ff9b3f144e019d5c] \u2190 b-db-decision\ndatabase_mongodb database Decided on MongoDB after evaluating Postgres. Raw driver only. Shapes all data-layer recommendations \u2014 database choice is settled and prevents re-litigating Postgres as an alternative.",
+ "context_tokens": 183,
+ "retrieve_time_ms": 2017.7,
+ "gold_answers": [
+ "b-mongo-raw-driver"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Fuzzy matching and prefix guards",
+ "description": "Query is a single misspelled term \u2014 'mongod' is the only token, 1 edit from alias 'Mongo', prefix 'mo' passes",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-mongo-raw-driver"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-mongo-raw-driver"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Fuzzy matching and prefix guards"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "universal-scope-belief-content-in-persona-prelude",
+ "query": "what's a good way to grade a 1909-S VDB penny without sending it to PCGS?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.3,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Persona prelude content",
+ "description": "user:universal belief content appears in personaPrelude for a domain with no per-scope description",
+ "scope": [
+ "domain:hobby"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true,
+ "contains": [
+ "direct",
+ "preamble"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [],
+ "retrieved_questions": [],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Persona prelude content"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "type-isolation-open-question-not-in-relevant",
+ "query": "JWT authentication token strategy",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.9,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "open_question belief with directly matched aliases must appear in openQuestions only \u2014 not in pinnedFacts or relevantBeliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-auth-open-q"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "multi-hop-supersession-chain-all-excluded",
+ "query": "ESLint rules and TSLint configuration",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.7,
+ "gold_answers": [
+ "b-linting-current"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Supersession chain exclusion",
+ "description": "Three-hop chain: TSLint (b-linting-v0) \u2192 ESLint (b-linting-v1) \u2192 Biome (b-linting-current). Query terms matching both superseded hops must surface neither; only the active terminal belief surfaces via pinnedFacts",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-linting-current"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Supersession chain exclusion"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "cross-scope-one-empty-domain-does-not-degrade",
+ "query": "What are we using Redis for?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x2b4e8c0824b86ca1ea97648343b1dcf188817dbc3451d44fbf8c310b8b15aa63] \u2190 b-redis-code\nredis_cache Redis Redis is used as a cache and rate-limiter in the backend. Single-node, not clustered. Shapes interpretation of 'Redis' in the code scope \u2014 always the caching datastore, not the fictional character.",
+ "context_tokens": 100,
+ "retrieve_time_ms": 929.2,
+ "gold_answers": [
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Multi-scope query where one scope (domain:hobby) has zero beliefs \u2014 code-scope retrieval must be unaffected and no empty-scope error must occur",
+ "scope": [
+ "domain:code",
+ "domain:hobby"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-redis-code"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "ranking-orderedbefore-alias-beats-content",
+ "query": "explicit error returns in pipe transforms",
+ "answer": "2 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (2)\n\n1. [0x51428d969a3c4768136ea0ee473e045c0820114d3820c24faa5d141f3e76e883] \u2190 b-error-handling\nerror_handling_explicit error handling exceptions try-catch Result type error Go-style explicit error returns. Error tuples or Result types rather than thrown exceptions. Shapes error handling patterns toward explicit return values and away from thrown exceptions or try/catch wrappers.\n\n2. [0x050f1e25b632f0054c0b1eec60f93a79ea976d539b9b4d9a7aef3e96fcdb642f] \u2190 b-functional-pipeline\nfunctional_pipeline pipe compose fp Uses functional composition patterns with pipe() to chain data transformations instead of imperative loops. Shapes data-processing code suggestions toward functional pipe composition and away from imperative loops or nested map/filter chains.",
+ "context_tokens": 198,
+ "retrieve_time_ms": 1613.6,
+ "gold_answers": [
+ "b-functional-pipeline",
+ "b-error-handling"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Ranking stability",
+ "description": "Rare alias 'pipe' (high IDF, short field) outranks common alias 'error' \u2014 documents Atlas Search BM25 IDF weighting behavior",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-functional-pipeline",
+ "b-error-handling"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-error-handling",
+ "b-functional-pipeline"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 2
+ },
+ "retrieved_count": 2
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Ranking stability"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cold-start-user-no-beliefs",
+ "query": "how should I structure my repository layer?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 692.6,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Cold start behavior",
+ "description": "User with zero seeded beliefs returns fully empty context at every output tier without error",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "isNull": true
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [],
+ "retrieved_questions": [],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Cold start behavior"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "narrator-voice-pinned-in-own-scope",
+ "query": "",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.0,
+ "gold_answers": [
+ "b-narrator-voice",
+ "b-protagonist-entity",
+ "b-prose-style-user-edited"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Persona prelude content",
+ "description": "Empty query against domain:writing surfaces all three pinned writing-scope beliefs in pinnedFacts and the writing open question in openQuestions",
+ "scope": [
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-narrator-voice",
+ "b-protagonist-entity",
+ "b-prose-style-user-edited"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-chapter-structure-q"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Persona prelude content"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "multi-scope-prelude-uses-primary-scope-only",
+ "query": "give me an overview of what I'm working on",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.7,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Persona prelude content",
+ "description": "With scope=['code','writing'], personaPrelude is universal only \u2014 documents that scope-specific content is not injected",
+ "scope": [
+ "domain:code",
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current",
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q",
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Persona prelude content"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "why-it-matters-not-indexed-blind-spot",
+ "query": "Vue component architecture",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.9,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "'Vue' appears only in b-react-entity.why_it_matters \u2014 not in canonical_name, aliases, or content. The query must not surface the belief.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "budget-ceiling-evicts-relevant-beliefs",
+ "query": "How should I handle errors in my repository layer?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 3.5,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "maxBeliefs set to exact pinned count \u2014 no slots remain for relevantBeliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {
+ "maxBeliefs": 2
+ },
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 2,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "negative-guidance-mongoose",
+ "query": "suggest a good Mongoose schema for my user model",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x75fb6600161c81a15cec4ca72e4f1c159fb15b1bfbe24e139cf9e77deecf82bc] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.",
+ "context_tokens": 95,
+ "retrieve_time_ms": 7509.8,
+ "gold_answers": [
+ "b-mongo-raw-driver"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Counter-signal retrieval",
+ "description": "Query for Mongoose schema surfaces b-mongo-raw-driver \u2014 the raw-driver preference is the counter-signal",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-mongo-raw-driver"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-mongo-raw-driver"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Counter-signal retrieval"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "recency-tiebreak-reinforcement",
+ "query": "",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.0,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Ranking stability",
+ "description": "Among equally-scored pinned beliefs, higher last_reinforced_at sorts first in pinnedFacts",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Ranking stability"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "preference-flood-relevant-entity-survives",
+ "query": "how does our CI pipeline handle secrets?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x3a922991324403cb75c69c1bee4b2129b89943f70e4df9a652672f53475b4efa] \u2190 b-ci-gha-entity\ngithub_actions GHA CI pipeline github actions CI/CD runs on GitHub Actions. Merge to main triggers a deploy to staging; promotion to prod is manual. Shapes all CI/CD and pipeline answers to assume GitHub Actions rather than Jenkins or CircleCI.",
+ "context_tokens": 105,
+ "retrieve_time_ms": 1018.5,
+ "gold_answers": [
+ "b-ci-gha-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "Query about a specific entity is not drowned out by high-reinforcement preferences at budget ceiling",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {
+ "maxBeliefs": 8
+ },
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-ci-gha-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 8,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-ci-gha-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "cross-scope-non-user-edited-blocked",
+ "query": "tell me about Mara",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.6,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Writing-scope entity with strong name match does not surface in code-scope query",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "mustExclude": [
+ "b-protagonist-entity"
+ ],
+ "shouldOnlyInclude": []
+ },
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-protagonist-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-two-word-shingle-dead-letter",
+ "query": "what happens to dead letter messages that fail replay?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xc19dd61a618223aa624c53acf9ada2dc4783fa0e6a23308e4e7937c9206fc328] \u2190 b-dead-letter-queue\ndead_letter_queue dead letter DLQ failed messages Failed messages route to a MongoDB-backed dead letter queue with manual replay via admin CLI. Shapes error recovery and messaging answers to assume a DLQ exists rather than suggesting one from scratch.",
+ "context_tokens": 99,
+ "retrieve_time_ms": 994.3,
+ "gold_answers": [
+ "b-dead-letter-queue"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Two-word alias 'dead letter' fires via the shingle analyzer path on aliases.shingle multi-field",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-dead-letter-queue"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-dead-letter-queue"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "alias-dlq-abbreviation-resolves",
+ "query": "how do we replay items from the DLQ?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xc19dd61a618223aa624c53acf9ada2dc4783fa0e6a23308e4e7937c9206fc328] \u2190 b-dead-letter-queue\ndead_letter_queue dead letter DLQ failed messages Failed messages route to a MongoDB-backed dead letter queue with manual replay via admin CLI. Shapes error recovery and messaging answers to assume a DLQ exists rather than suggesting one from scratch.",
+ "context_tokens": 99,
+ "retrieve_time_ms": 1312.6,
+ "gold_answers": [
+ "b-dead-letter-queue"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Alias resolution",
+ "description": "Alias 'DLQ' (abbreviation) drives text-search retrieval",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-dead-letter-queue"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-dead-letter-queue"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Alias resolution"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "counter-signal-jest-surfaces-vitest",
+ "query": "should I configure Jest for this project?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x39980e81bcf0ec3bf21d08f9b74c0d74831572f49a7551df701059bd74dab5ac] \u2190 b-vitest-pref\ntesting_vitest Vitest vitest jest Uses Vitest for all testing. Replaced Jest. Shapes all testing examples toward Vitest and prevents suggesting Jest, Mocha, or Jasmine.",
+ "context_tokens": 93,
+ "retrieve_time_ms": 1778.3,
+ "gold_answers": [
+ "b-vitest-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Counter-signal retrieval",
+ "description": "Counter-signal alias 'jest' on b-vitest-pref surfaces when user queries about Jest",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-vitest-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-vitest-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Counter-signal retrieval"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "relation-type-expands-participants",
+ "query": "what are the auth service dependencies and failure modes?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x8c9e5eef6c5a7f159ad9219d6937974a01a8d89f2441547271c8875c6e735054] \u2190 b-auth-depends-redis\nauth_service_depends_on_redis auth redis session backend auth Auth service depends on Redis for session storage. If Redis is down, auth fails open (denies all). Shapes failure-mode analysis \u2014 auth and Redis are coupled; cannot discuss auth resilience without addressing Redis availability.",
+ "context_tokens": 114,
+ "retrieve_time_ms": 1542.1,
+ "gold_answers": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Relation expansion",
+ "description": "Relation belief 'auth depends on redis' surfaces via alias match AND pulls its participants (b-redis-code) into relevantBeliefs via one-hop join. b-auth-open-q goes to openQuestions since it's an open_question type.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Relation expansion"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "relation-participant-open-question-routes-correctly",
+ "query": "session backend dependency graph",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x8c9e5eef6c5a7f159ad9219d6937974a01a8d89f2441547271c8875c6e735054] \u2190 b-auth-depends-redis\nauth_service_depends_on_redis auth redis session backend auth Auth service depends on Redis for session storage. If Redis is down, auth fails open (denies all). Shapes failure-mode analysis \u2014 auth and Redis are coupled; cannot discuss auth resilience without addressing Redis availability.",
+ "context_tokens": 114,
+ "retrieve_time_ms": 1283.5,
+ "gold_answers": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Relation expansion",
+ "description": "Participant b-auth-open-q is type=open_question and pinned. Even though it's pulled in via relation expansion, it must appear in openQuestions, not relevantBeliefs.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ },
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Relation expansion"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "relation-expansion-respects-scope-filter",
+ "query": "auth redis failure modes",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x8c9e5eef6c5a7f159ad9219d6937974a01a8d89f2441547271c8875c6e735054] \u2190 b-auth-depends-redis\nauth_service_depends_on_redis auth redis session backend auth Auth service depends on Redis for session storage. If Redis is down, auth fails open (denies all). Shapes failure-mode analysis \u2014 auth and Redis are coupled; cannot discuss auth resilience without addressing Redis availability.",
+ "context_tokens": 114,
+ "retrieve_time_ms": 2630.8,
+ "gold_answers": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Relation expansion",
+ "description": "If a relation's participant is out of scope, it must NOT be pulled in by expansion. Hypothetical: b-auth-depends-redis has participant b-redis-writing (wrong scope). Scope filter in expandRelationParticipants blocks it.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "mustExclude": [
+ "b-redis-writing"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Relation expansion"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "subtype-expertise-retrieval",
+ "query": "what's my react depth \u2014 should you explain hooks to me?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x96cf0916fcffef60d783c2cc764f1dfd1ba25c1f11d8942d84067580466eddd5] \u2190 b-react-entity\nreact ReactJS react.js Frontend is React 18 with TanStack Query and Tailwind. Shapes all frontend code answers to assume React 18 with hooks rather than Vue or Svelte.",
+ "context_tokens": 90,
+ "retrieve_time_ms": 1019.2,
+ "gold_answers": [
+ "b-react-entity"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "Subtype 'expertise' belief surfaces on a query about React skill calibration",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-react-entity"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-react-entity"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "pinned-plus-resolved-excluded-from-questions",
+ "query": "API versioning strategy for v2 migration",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.0,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Type routing and open questions",
+ "description": "A belief that is both pinned AND has resolved_at set must NOT appear in openQuestions or pinnedFacts",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustExclude": [
+ "b-pinned-resolved-interaction"
+ ]
+ },
+ "openQuestions": {
+ "mustExclude": [
+ "b-pinned-resolved-interaction"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Type routing and open questions"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "zero-reinforcement-fresh-belief-surfaces",
+ "query": "how is our Turborepo monorepo structured?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0xf0cefa7162055d4a81a3794c63f1d04ed8398b18f681311c9b0657e8fcada382] \u2190 b-zero-reinforcement-fresh\nturborepo_monorepo Turborepo turbo monorepo Monorepo managed by Turborepo. Apps and packages in /apps and /packages directories. Shapes all build and workspace structure answers to assume Turborepo monorepo layout.",
+ "context_tokens": 112,
+ "retrieve_time_ms": 1084.8,
+ "gold_answers": [
+ "b-zero-reinforcement-fresh"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "Brand new belief with reinforcement_count: 0 still surfaces via alias match \u2014 recency/reinforcement must not suppress it",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-zero-reinforcement-fresh"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-zero-reinforcement-fresh"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "budget-zero-graceful-empty",
+ "query": "what's our stack?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.9,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "maxBeliefs: 0 returns empty arrays without throwing",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {
+ "maxBeliefs": 0
+ },
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": []
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 0,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "budget-one-pinned-wins-over-relevant",
+ "query": "MongoDB collection typing patterns",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x75fb6600161c81a15cec4ca72e4f1c159fb15b1bfbe24e139cf9e77deecf82bc] \u2190 b-mongo-raw-driver\nmongodb_raw_driver MongoDB Mongo moongoose Uses the MongoDB raw driver with explicit Collection types. Shapes all database code toward raw MongoDB driver calls and prevents ORM suggestions such as Mongoose or Prisma.",
+ "context_tokens": 95,
+ "retrieve_time_ms": 2036.1,
+ "gold_answers": [
+ "b-db-decision"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Budget eviction and capacity",
+ "description": "maxBeliefs: 1 with a query matching both a pinned belief and a non-pinned belief \u2014 pinned belief gets the slot",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {
+ "maxBeliefs": 1
+ },
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-db-decision"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 1,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Budget eviction and capacity"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "multi-scope-code-writing-dedup-no-double-surface",
+ "query": "what tools do we use?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 1.4,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "With scope=['domain:code','domain:writing'], a belief scoped to both does not appear twice",
+ "scope": [
+ "domain:code",
+ "domain:writing"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current",
+ "b-narrator-voice",
+ "b-prose-style-user-edited",
+ "b-protagonist-entity"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q",
+ "b-chapter-structure-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "empty-alias-belief-content-path-only",
+ "query": "should you give me one recommendation or a list of options?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.3,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "b-comm-recommendation has aliases: [] \u2014 can only be retrieved via content path. Since content is not a scored search path, it should NOT surface.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "trivially-empty",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "trivially-empty"
+ ]
+ }
+ },
+ {
+ "query_id": "universal-scope-explicit-query-no-relevant",
+ "query": "how do I want you to respond to me?",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 0.4,
+ "gold_answers": [],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Scope disambiguation",
+ "description": "Query scoped explicitly to user:universal \u2014 universal beliefs appear in personaPrelude only, not in relevantBeliefs",
+ "scope": [
+ "user:universal"
+ ],
+ "budget": {},
+ "expect": {
+ "personaPrelude": {
+ "nonEmpty": true,
+ "contains": [
+ "direct"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-comm-pushback",
+ "b-comm-response-length"
+ ],
+ "retrieved_questions": [],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Scope disambiguation"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "negation-query-still-surfaces-rejected-tool",
+ "query": "I do NOT want to use Express, what should I use instead?",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x474da1f52940ab49a0a252eb9e8d1b355d8c6e7d5ac83b90f3b97bbfaaf5170e] \u2190 b-fastify-pref\nframework_fastify Fastify express Uses Fastify for HTTP services rather than Express. Shapes server-side code examples toward Fastify and prevents suggesting Express as an alternative.",
+ "context_tokens": 93,
+ "retrieve_time_ms": 989.1,
+ "gold_answers": [
+ "b-fastify-pref"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Design boundary cases",
+ "description": "Query 'I do NOT want to use Express' still surfaces b-fastify-pref because text search has no negation awareness",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-fastify-pref"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-fastify-pref"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Design boundary cases"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ },
+ {
+ "query_id": "superseded-chain-query-terminal-only",
+ "query": "TSLint migration path",
+ "answer": "0 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (0)",
+ "context_tokens": 6,
+ "retrieve_time_ms": 2.5,
+ "gold_answers": [
+ "b-linting-current"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Supersession chain exclusion",
+ "description": "Query 'TSLint' matches b-linting-v0 (superseded by v1, which is superseded by b-linting-current). Only terminal active belief surfaces.",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "pinnedFacts": {
+ "mustInclude": [
+ "b-linting-current"
+ ]
+ },
+ "relevantBeliefs": {
+ "shouldOnlyInclude": []
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "structural",
+ "active_pass": false,
+ "relevant_beliefs": [],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": null,
+ "retrieval_recall": null,
+ "pinned_coverage": 1.0,
+ "failures": [],
+ "resolution": {},
+ "retrieved_count": 0
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Supersession chain exclusion"
+ ],
+ "Pass type": [
+ "structural"
+ ]
+ }
+ },
+ {
+ "query_id": "relation-does-not-appear-as-entity",
+ "query": "Redis session backend dependency graph",
+ "answer": "1 memories retrieved",
+ "reasoning": "",
+ "context": "## Retrieved memories (1)\n\n1. [0x8c9e5eef6c5a7f159ad9219d6937974a01a8d89f2441547271c8875c6e735054] \u2190 b-auth-depends-redis\nauth_service_depends_on_redis auth redis session backend auth Auth service depends on Redis for session storage. If Redis is down, auth fails open (denies all). Shapes failure-mode analysis \u2014 auth and Redis are coupled; cannot discuss auth resilience without addressing Redis availability.",
+ "context_tokens": 114,
+ "retrieve_time_ms": 1546.7,
+ "gold_answers": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "correct": true,
+ "judge_reason": "pass",
+ "score": null,
+ "meta": {
+ "category": "Relation expansion",
+ "description": "Relation-type belief does not bleed into pinnedFacts despite alias overlap with pinned beliefs",
+ "scope": [
+ "domain:code"
+ ],
+ "budget": {},
+ "expect": {
+ "relevantBeliefs": {
+ "shouldOnlyInclude": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ]
+ },
+ "openQuestions": {
+ "mustInclude": [
+ "b-auth-open-q"
+ ]
+ }
+ },
+ "retrieval_limit": 20,
+ "pass_type": "active",
+ "active_pass": true,
+ "relevant_beliefs": [
+ "b-auth-depends-redis",
+ "b-redis-code"
+ ],
+ "pinned_beliefs": [
+ "b-db-decision",
+ "b-linting-current"
+ ],
+ "retrieved_questions": [
+ "b-auth-open-q"
+ ],
+ "retrieval_precision": 1.0,
+ "retrieval_recall": 1.0,
+ "pinned_coverage": null,
+ "failures": [],
+ "resolution": {
+ "source_id": 1
+ },
+ "retrieved_count": 1
+ },
+ "raw_response": null,
+ "trajectory": null,
+ "git_history": null,
+ "category_axes": {
+ "Category": [
+ "Relation expansion"
+ ],
+ "Pass type": [
+ "active"
+ ]
+ }
+ }
+ ],
+ "avg_retrieve_time_ms": 1034.6,
+ "avg_context_tokens": 91.6
+}
\ No newline at end of file
diff --git a/styles.css b/styles.css
index 9833392..1260604 100644
--- a/styles.css
+++ b/styles.css
@@ -1147,19 +1147,6 @@ a.card:hover {
}
}
-/* Evidence charts ship a wide and a stacked variant; phones get the stacked one so labels stay legible. */
-@media (max-width: 639px) {
- .mdx-content img[src*="/images/evidence/"]:not([src*="-stacked-"]) {
- display: none !important;
- }
-}
-
-@media (min-width: 640px) {
- .mdx-content img[src*="/images/evidence/"][src*="-stacked-"] {
- display: none !important;
- }
-}
-
/*
Mobile docs shell (NEU-262).
Mintlify keeps a fixed two-row header (logo row + hamburger/title chip).
@@ -1353,24 +1340,16 @@ html {
background: var(--proofable-scrollbar-thumb-hover);
}
-/* Evidence charts: one cut per theme per viewport */
-/* Before this every variant carried the same mint-block / dark:mint-hidden pair, so both the
- wide (1200x460) and the tall (1200x1037) cut rendered at every width: each visitor saw each
- chart twice and a phone fetched four charts where it needed two.
- The .dark .evid-dark.evid-wide selector inside the media query is load-bearing: without it the
- dark-theme rule outside the query (0,3,0) outranks a plain .evid-dark.evid-wide (0,2,0) and
- both cuts show on a phone in dark mode. Verified in a browser at 1280 / 800 / 390 / 375. */
-.evid-wide, .evid-tall { display: none; }
-
-.evid-light.evid-wide { display: block; }
-.dark .evid-light.evid-wide { display: none; }
-.dark .evid-dark.evid-wide { display: block; }
-
-@media (max-width: 768px) {
- .evid-light.evid-wide,
- .evid-dark.evid-wide,
- .dark .evid-dark.evid-wide { display: none; }
- .evid-light.evid-tall { display: block; }
- .dark .evid-light.evid-tall { display: none; }
- .dark .evid-dark.evid-tall { display: block; }
+
+/* Tables on phones: tighter type and padding inside Mintlify's own horizontal scroll area. */
+@media (max-width: 639px) {
+ .mdx-content table {
+ font-size: 0.8125rem;
+ line-height: 1.45;
+ }
+ .mdx-content table th,
+ .mdx-content table td {
+ padding: 0.5rem 0.625rem;
+ min-width: 7.5rem;
+ }
}