diff --git a/benchmark.mdx b/benchmark.mdx index a93e6fd..8639eaa 100644 --- a/benchmark.mdx +++ b/benchmark.mdx @@ -7,13 +7,6 @@ description: "See how Proofable compares with the tools teams run today, on memo icon: chart-simple --- -Bar chart of PrecisionMemBench cases passed out of 77. Proofable 77 including all 43 cases that need the right facts, Hindsight 66, Supermemory 15, and Mem0, Zep and Cognee 9. -Bar chart of PrecisionMemBench cases passed out of 77. Proofable 77 including all 43 cases that need the right facts, Hindsight 66, Supermemory 15, and Mem0, Zep and Cognee 9. -Bar chart of PrecisionMemBench cases passed out of 77. Proofable 77 including all 43 cases that need the right facts, Hindsight 66, Supermemory 15, and Mem0, Zep and Cognee 9. -Bar chart of PrecisionMemBench cases passed out of 77. Proofable 77 including all 43 cases that need the right facts, Hindsight 66, Supermemory 15, and Mem0, Zep and Cognee 9. - -**Perfect scores on agent memory and governance.** Every result below is a public benchmark or a test you can rerun, and each one links to the record behind it. - The current fact every time, on all 43 cases that need it. Mem0 and Zep passed 9. @@ -43,60 +36,65 @@ icon: chart-simple ### Reuse instead of redo -Redoing a finished agent job took 2.79 s and 2,426 AI tokens. Checking its signed receipt took 3.7 ms and no tokens: 750× faster, which is 1 hour instead of 32 days at a million checks. [Time it yourself](#run-it-yourself). - Bar chart of the time to trust a finished result. Checking the Proofable receipt took 3.7 milliseconds and redoing the job took 2.79 seconds, about 750 times faster. Bar chart of the time to trust a finished result. Checking the Proofable receipt took 3.7 milliseconds and redoing the job took 2.79 seconds, about 750 times faster. Bar chart of the time to trust a finished result. Checking the Proofable receipt took 3.7 milliseconds and redoing the job took 2.79 seconds, about 750 times faster. Bar chart of the time to trust a finished result. Checking the Proofable receipt took 3.7 milliseconds and redoing the job took 2.79 seconds, about 750 times faster. -### Handoffs between agents +Redoing a finished agent job took 2.79 s and 2,426 AI tokens. Checking its signed receipt took 3.7 ms and no tokens: 750× faster, which is 1 hour instead of 32 days at a million checks. [Time it yourself](#run-it-yourself). -When one agent hands a finished result to another (LangGraph, CrewAI, AutoGen), the receiving agent normally either trusts the forwarded text blindly or pays to re-run the upstream work. With Proofable, the producing agent emits a 66-byte signed receipt (`qHash`) binding input, policy state and output, and the receiving agent checks it before acting. +### Handoffs between agents Bar chart of the cost per correct outcome. Handing off with a Proofable receipt cost $0.024, redoing the work $0.078, and trusting forwarded text $0.009. Bar chart of the cost per correct outcome. Handing off with a Proofable receipt cost $0.024, redoing the work $0.078, and trusting forwarded text $0.009. Bar chart of the cost per correct outcome. Handing off with a Proofable receipt cost $0.024, redoing the work $0.078, and trusting forwarded text $0.009. Bar chart of the cost per correct outcome. Handing off with a Proofable receipt cost $0.024, redoing the work $0.078, and trusting forwarded text $0.009. +When one agent hands a finished result to another (LangGraph, CrewAI, AutoGen), the receiving agent normally either trusts the forwarded text blindly or pays to re-run the upstream work. With Proofable, the producing agent emits a 66-byte signed receipt (`qHash`) binding input, policy state and output, and the receiving agent checks it before acting. + - **3.3× cheaper than redoing the work:** \$0.024 per correct outcome (33,240 tokens, 2 model calls per task) against \$0.078 (124,715 tokens, 3 to 5 model calls per task). - **Trusting forwarded text is cheap and wrong:** it costs \$0.009 on untouched input, but when the text was changed in transit it used the tampered value **3 of 3** times. Checking the receipt caught every change **3 of 3** times and still completed every task, where blind trust caught **0 of 3**. - **29× smaller handoffs:** across a 10-hop chain, passing the signed reference carried **9,900 bytes** per hop instead of **289,410 bytes** of full evidence. ### Memory that knows what is current -Most agent memory layers (Mem0, Zep, vector stores) return text chunks with no active or revoked status and no signer, so the agent keeps reading full bodies to work out which rule is current. Proofable stores typed, revocable, owner-signed records (`rules`, `instruction`, `receipts`), so the agent asks for active state by type and reads about 1 KB of verified metadata before opening a body. - Bar chart of the tokens to find the current rule on a 3,621-record vault. Typed Proofable state used 43,169 tokens and keyword search over record bodies used 479,623, about 11 times fewer. Bar chart of the tokens to find the current rule on a 3,621-record vault. Typed Proofable state used 43,169 tokens and keyword search over record bodies used 479,623, about 11 times fewer. Bar chart of the tokens to find the current rule on a 3,621-record vault. Typed Proofable state used 43,169 tokens and keyword search over record bodies used 479,623, about 11 times fewer. Bar chart of the tokens to find the current rule on a 3,621-record vault. Typed Proofable state used 43,169 tokens and keyword search over record bodies used 479,623, about 11 times fewer. +Most agent memory layers (Mem0, Zep, vector stores) return text chunks with no active or revoked status and no signer, so the agent keeps reading full bodies to work out which rule is current. Proofable stores typed, revocable, owner-signed records (`rules`, `instruction`, `receipts`), so the agent asks for active state by type and reads about 1 KB of verified metadata before opening a body. + On a 3,621-record production vault, the questions both methods answered correctly took **43,169 tokens in 66 seconds** (6 model calls) by typed lookup against **479,623 tokens in 178 seconds** (10 model calls) searching record bodies by keyword: **11.1× fewer tokens, 2.7× faster**, and 11.8× to 12.7× lower model cost across Claude Sonnet 5.5, Gemini 3.1 Pro and GPT-6 Astra. ### Memory that returns only the current fact +Bar chart of PrecisionMemBench cases passed out of 77. Proofable 77 including all 43 cases that need the right facts, Hindsight 66, Supermemory 15, and Mem0, Zep and Cognee 9. +Bar chart of PrecisionMemBench cases passed out of 77. Proofable 77 including all 43 cases that need the right facts, Hindsight 66, Supermemory 15, and Mem0, Zep and Cognee 9. +Bar chart of PrecisionMemBench cases passed out of 77. Proofable 77 including all 43 cases that need the right facts, Hindsight 66, Supermemory 15, and Mem0, Zep and Cognee 9. +Bar chart of PrecisionMemBench cases passed out of 77. Proofable 77 including all 43 cases that need the right facts, Hindsight 66, Supermemory 15, and Mem0, Zep and Cognee 9. + **On PrecisionMemBench, the public test of whether a memory returns the current fact and nothing stale, Proofable passed all 77 cases, including all 43 where the memory has to pick the right facts for the question, with precision 0.98 and recall 1.00.** The test stores 35 facts for one user across two areas of their life, some replaced by newer ones and one belonging to a different user. It then asks 77 questions and checks exactly which facts come back. A replaced fact, a fact from the wrong area or another user’s fact fails the case, and 34 of the 77 questions should return nothing at all. Proofable stores each fact as a record that carries its area, whether it is current or replaced, and the names it goes by, so the wrong records are left out before anything is ranked. The [Agent Memory Benchmark](https://agentmemorybenchmark.ai) toolkit scores the test without a model. The run is a [public record](https://proofable.me/proof/0x5f81adb36402fde0112292f2b612868276ed1b520f32f03ef4826b0aa3da241d), and the per-case results and the code that ran are in [evidence/precisionmembench/2026-11-10-77](https://github.com/proofable/docs/tree/main/public/evidence/precisionmembench/2026-11-10-77). -| Memory | Cases passed | Active cases | Precision | Run by and source | +| Memory | Cases passed | Active cases | Precision | Source | | --- | --- | --- | --- | --- | -| Proofable | **77 of 77** | **43 of 43** | 0.98 | Proofable, Agent Memory Benchmark toolkit: [results](https://github.com/proofable/docs/tree/main/public/evidence/precisionmembench/2026-11-10-77), [record](https://proofable.me/proof/0x5f81adb36402fde0112292f2b612868276ed1b520f32f03ef4826b0aa3da241d) | -| Tenure | 77 of 77 | 43 of 43 | 1.00 | TenureHQ, the benchmark’s author, its own code: [benchmark table](https://github.com/tenurehq/precisionMemBench) | -| Hindsight (cloud) | 66 of 77 | 33 of 43 | 0.85 | Vectorize, same toolkit: [published run](https://github.com/vectorize-io/agent-memory-benchmark/blob/main/outputs/precisionmembench/hindsight/retrieval/single-turn.json.gz) | -| Supermemory | 15 of 77 | 2 of 43 | 0.18 | TenureHQ, its own code: [benchmark table](https://github.com/tenurehq/precisionMemBench) | -| Mem0, Zep, Cognee | 9 of 77 | 0 of 43 | 0.06 | TenureHQ, its own code: [benchmark table](https://github.com/tenurehq/precisionMemBench) | +| Proofable | **77 of 77** | **43 of 43** | 0.98 | [Run results](https://github.com/proofable/docs/tree/main/public/evidence/precisionmembench/2026-11-10-77), [signed record](https://proofable.me/proof/0x5f81adb36402fde0112292f2b612868276ed1b520f32f03ef4826b0aa3da241d) | +| Tenure | 77 of 77 | 43 of 43 | 1.00 | [Benchmark author’s table](https://github.com/tenurehq/precisionMemBench) | +| Hindsight (cloud) | 66 of 77 | 33 of 43 | 0.85 | [Vectorize published run](https://github.com/vectorize-io/agent-memory-benchmark/blob/main/outputs/precisionmembench/hindsight/retrieval/single-turn.json.gz) | +| Supermemory | 15 of 77 | 2 of 43 | 0.18 | Author’s table above | +| Mem0, Zep, Cognee | 9 of 77 | 0 of 43 | 0.06 | Author’s table above | Proofable and Hindsight ran on the same toolkit, so those two rows compare directly. The other rows are the benchmark author’s own results with its own code, and Tenure is the author’s own product. High recall with low precision means the right fact came back buried among unrelated ones. ### Research with a cost ceiling -Live pages change between requests: two fetches of the same page 221 seconds apart returned 96,713 characters, then 1,129, handing two agents different evidence with no way to prove what the page said when the decision was made. On public questions about standards, agent protocols and regulation, reading signed Proofable records **halved the cost per correct answer (\$0.020 vs \$0.039)**, held the worst case to **16,301 tokens** where open-web search spiked to **50,554 tokens** and 104 seconds (**3.1× lower**), and cited a checkable source on **every answer** (6 of 6, against 5 of 6 on the open web). - Bar chart of the worst-case tokens on six public questions. Typed Proofable records used 16,301 and the open web 50,554, about 3 times lower. Bar chart of the worst-case tokens on six public questions. Typed Proofable records used 16,301 and the open web 50,554, about 3 times lower. Bar chart of the worst-case tokens on six public questions. Typed Proofable records used 16,301 and the open web 50,554, about 3 times lower. Bar chart of the worst-case tokens on six public questions. Typed Proofable records used 16,301 and the open web 50,554, about 3 times lower. +Live pages change between requests: two fetches of the same page 221 seconds apart returned 96,713 characters, then 1,129, handing two agents different evidence with no way to prove what the page said when the decision was made. On public questions about standards, agent protocols and regulation, reading signed Proofable records **halved the cost per correct answer (\$0.020 vs \$0.039)**, held the worst case to **16,301 tokens** where open-web search spiked to **50,554 tokens** and 104 seconds (**3.1× lower**), and cited a checkable source on **every answer** (6 of 6, against 5 of 6 on the open web). + ### Identity checks paid for once A person or business completes an interactive check once, and downstream apps and agents verify the portable proof instead of paying the vendor to run the flow again. Priced from published vendor list prices and Proofable's \$0.0025 per check: @@ -112,13 +110,13 @@ A person or business completes an interactive check once, and downstream apps an ### Governance coverage -[AgentGovBench](https://github.com/agentic-control-plane/agentgovbench) is the open library of 48 agent-governance scenarios in eight areas: audit completeness, cross-tenant isolation, delegation provenance, fail-mode discipline, identity propagation, per-user policy, rate limits and scope inheritance. Each scenario is tagged with the NIST AI RMF controls it tests, and nothing passes on a technicality. - Bar chart of AgentGovBench scenarios handled out of 48. Proofable 48, Claude Code or Codex with the ACP gateway 43, Cursor with ACP 37, and agent tools on their own 13. Bar chart of AgentGovBench scenarios handled out of 48. Proofable 48, Claude Code or Codex with the ACP gateway 43, Cursor with ACP 37, and agent tools on their own 13. Bar chart of AgentGovBench scenarios handled out of 48. Proofable 48, Claude Code or Codex with the ACP gateway 43, Cursor with ACP 37, and agent tools on their own 13. Bar chart of AgentGovBench scenarios handled out of 48. Proofable 48, Claude Code or Codex with the ACP gateway 43, Cursor with ACP 37, and agent tools on their own 13. +[AgentGovBench](https://github.com/agentic-control-plane/agentgovbench) is the open library of 48 agent-governance scenarios in eight areas: audit completeness, cross-tenant isolation, delegation provenance, fail-mode discipline, identity propagation, per-user policy, rate limits and scope inheritance. Each scenario is tagged with the NIST AI RMF controls it tests, and nothing passes on a technicality. + | Setup | Scenarios handled | |---|---| | Proofable policy engine | **48/48** | @@ -161,6 +159,11 @@ Reviewers in the AgenTrust integrations project [verified the records offline](h ### Prompt injection on AgentDojo +Bar chart of AgentDojo prompt injections stopped across 949 attacked task pairs. Proofable stopped 93.8 percent with 87.0 percent of tasks still finished, against 83.6 percent without it and 92.0 percent for an injection filter that finished 21 percent of tasks. +Bar chart of AgentDojo prompt injections stopped across 949 attacked task pairs. Proofable stopped 93.8 percent with 87.0 percent of tasks still finished, against 83.6 percent without it and 92.0 percent for an injection filter that finished 21 percent of tasks. +Bar chart of AgentDojo prompt injections stopped across 949 attacked task pairs. Proofable stopped 93.8 percent with 87.0 percent of tasks still finished, against 83.6 percent without it and 92.0 percent for an injection filter that finished 21 percent of tasks. +Bar chart of AgentDojo prompt injections stopped across 949 attacked task pairs. Proofable stopped 93.8 percent with 87.0 percent of tasks still finished, against 83.6 percent without it and 92.0 percent for an injection filter that finished 21 percent of tasks. + **On [AgentDojo](https://agentdojo.spylab.ai/), the open prompt-injection benchmark from ETH Zurich, putting an owner-signed, per-task permission in front of every tool call cut the share of attacks that landed from 16.4% to 6.2% across all four suites, while the agent still completed 94.9% of its tasks (95.9% without the permission).** Same model, same 949 attacked task pairs, same attack template, scored by AgentDojo's own checkers; the only change is Proofable's authority engine deciding each call against the permission the owner signed for that task. | | Task done, no attack | Task done under attack | Attacks that landed | @@ -172,13 +175,13 @@ For scale, these are AgentDojo's published results on other models. With no defe ### A hijacked agent cannot send your data out -**When an agent was taken over, 0 of 6 attempts to send private data out got past a scoped permission. On an all-purpose key, 6 of 6 got out.** Five of the six were refused by the permission before they ran and one was never attempted. The scoped agent finished every task and never used a tool outside it (0 of 42 runs); under the all-purpose key, 7 of 42 runs did, with a signed record for each run. [Run the test on your own agent](https://github.com/proofable/mcp/tree/main/examples/injection-test). - Bar chart of attempts to send private data out of a hijacked agent. 6 of 6 got out on an all-purpose key and 0 of 6 on an owner-scoped permission. Bar chart of attempts to send private data out of a hijacked agent. 6 of 6 got out on an all-purpose key and 0 of 6 on an owner-scoped permission. Bar chart of attempts to send private data out of a hijacked agent. 6 of 6 got out on an all-purpose key and 0 of 6 on an owner-scoped permission. Bar chart of attempts to send private data out of a hijacked agent. 6 of 6 got out on an all-purpose key and 0 of 6 on an owner-scoped permission. +**When an agent was taken over, 0 of 6 attempts to send private data out got past a scoped permission. On an all-purpose key, 6 of 6 got out.** Five of the six were refused by the permission before they ran and one was never attempted. The scoped agent finished every task and never used a tool outside it (0 of 42 runs); under the all-purpose key, 7 of 42 runs did, with a signed record for each run. [Run the test on your own agent](https://github.com/proofable/mcp/tree/main/examples/injection-test). + ### Records nobody can fake **150 of 150 forged records caught.** Each of 15 real records was changed ten ways, such as editing the content, swapping the owner or flipping one bit of the signature. Every changed copy failed offline signature verification and every real record passed. A check that only reads the format accepted all 150. diff --git a/images/evidence/stop-agentdojo-dark.svg b/images/evidence/stop-agentdojo-dark.svg new file mode 100644 index 0000000..af8579f --- /dev/null +++ b/images/evidence/stop-agentdojo-dark.svg @@ -0,0 +1,6 @@ + +Proofable stopped 93.8% of AgentDojo prompt injections while the agent still finished 87% of its tasksAgentDojo (ETH Zurich SPY Lab, 949 attacked task pairs across four suites). Putting an owner-signed, per-task permission in front of every tool call cut the share of attacks that landed from 16.4% to 6.2% (93.8% stopped vs 83.6% on the same model without it), while task completion under attack held at 87.0% (88.1% without it). AgentDojo's published prompt-injection detector on gpt-4o stopped 92.0% of attacks while task completion under attack fell to 21.1%. + +93.8%of AgentDojo attacks stopped,87% of tasks still doneInjection filter92.0%21% of tasks done under attackSame agent, no defense83.6%88.1% of tasks done under attackWith Proofable93.8%87.0% of tasks done under attack + + \ No newline at end of file diff --git a/images/evidence/stop-agentdojo-light.svg b/images/evidence/stop-agentdojo-light.svg new file mode 100644 index 0000000..0a4a21b --- /dev/null +++ b/images/evidence/stop-agentdojo-light.svg @@ -0,0 +1,6 @@ + +Proofable stopped 93.8% of AgentDojo prompt injections while the agent still finished 87% of its tasksAgentDojo (ETH Zurich SPY Lab, 949 attacked task pairs across four suites). Putting an owner-signed, per-task permission in front of every tool call cut the share of attacks that landed from 16.4% to 6.2% (93.8% stopped vs 83.6% on the same model without it), while task completion under attack held at 87.0% (88.1% without it). AgentDojo's published prompt-injection detector on gpt-4o stopped 92.0% of attacks while task completion under attack fell to 21.1%. + +93.8%of AgentDojo attacks stopped,87% of tasks still doneInjection filter92.0%21% of tasks done under attackSame agent, no defense83.6%88.1% of tasks done under attackWith Proofable93.8%87.0% of tasks done under attack + + \ No newline at end of file diff --git a/images/evidence/stop-agentdojo-stacked-dark.svg b/images/evidence/stop-agentdojo-stacked-dark.svg new file mode 100644 index 0000000..56e4eef --- /dev/null +++ b/images/evidence/stop-agentdojo-stacked-dark.svg @@ -0,0 +1,6 @@ + +Proofable stopped 93.8% of AgentDojo prompt injections while the agent still finished 87% of its tasksAgentDojo (ETH Zurich SPY Lab, 949 attacked task pairs across four suites). Putting an owner-signed, per-task permission in front of every tool call cut the share of attacks that landed from 16.4% to 6.2% (93.8% stopped vs 83.6% on the same model without it), while task completion under attack held at 87.0% (88.1% without it). AgentDojo's published prompt-injection detector on gpt-4o stopped 92.0% of attacks while task completion under attack fell to 21.1%. + +93.8%of AgentDojo attacks stopped,87% of tasks still doneInjection filter92.0%21% of tasks done under attackSame agent, no defense83.6%88.1% of tasks done under attackWith Proofable93.8%87.0% of tasks done under attack + + \ No newline at end of file diff --git a/images/evidence/stop-agentdojo-stacked-light.svg b/images/evidence/stop-agentdojo-stacked-light.svg new file mode 100644 index 0000000..f5ac55e --- /dev/null +++ b/images/evidence/stop-agentdojo-stacked-light.svg @@ -0,0 +1,6 @@ + +Proofable stopped 93.8% of AgentDojo prompt injections while the agent still finished 87% of its tasksAgentDojo (ETH Zurich SPY Lab, 949 attacked task pairs across four suites). Putting an owner-signed, per-task permission in front of every tool call cut the share of attacks that landed from 16.4% to 6.2% (93.8% stopped vs 83.6% on the same model without it), while task completion under attack held at 87.0% (88.1% without it). AgentDojo's published prompt-injection detector on gpt-4o stopped 92.0% of attacks while task completion under attack fell to 21.1%. + +93.8%of AgentDojo attacks stopped,87% of tasks still doneInjection filter92.0%21% of tasks done under attackSame agent, no defense83.6%88.1% of tasks done under attackWith Proofable93.8%87.0% of tasks done under attack + + \ No newline at end of file diff --git a/styles.css b/styles.css index 6d707a2..706c609 100644 --- a/styles.css +++ b/styles.css @@ -430,6 +430,22 @@ body { max-width: 17rem !important; } +/* Mintlify sets w-[19rem] pl-10 on #table-of-contents-layout and w-[16.5rem] on #table-of-contents, + totalling 20.5rem inside a 17rem rail and pushing the right 32px of every TOC label out the window. */ +#table-of-contents-layout, +#table-of-contents { + width: 100% !important; + max-width: 100% !important; + min-width: 0 !important; + padding-left: 0 !important; + box-sizing: border-box !important; +} + +#table-of-contents a { + overflow-wrap: break-word !important; + white-space: normal !important; +} + #content-area > #header, #content-area > #content, #content-area > #pagination,