Structured record of the Week 4 evaluation: what is measured, the fixture and question set, how to run it, and the results. The narrative reasoning behind the design choices lives in decision-log.md; this file holds the data.
The project's thesis is about agency, not eloquence. Both limitations found so far (journey-5's fixed top-K, journey-6's tool selection) share the same shape: the model reasoned correctly over what it was given, but the right note never entered its context. Neither was a quality failure.
So the primary metric is context recall — did the target notes reach the model — plus which tool the model chose. Both are mechanically checkable and language-independent. No LLM-as-judge is involved.
This is a deliberate scope reduction from the roadmap's original Week 4 plan (faithfulness/relevancy scored by a second LLM, requiring responseSchema). Rationale in decision-log.md. Practical consequence: the primary run makes zero cloud API calls, so free-tier rate limits are a non-issue.
Substring matching against generated prose was rejected as a metric: the model paraphrases and translates freely (journey-6 logged it rendering a Turkish note into an English sentence), which produces false negatives that measure phrasing rather than retrieval.
| Arm | Class | Orchestration |
|---|---|---|
| agentic | OnDeviceAgent |
model picks among search_notes / add_note / count_notes / get_all_notes |
| non-agentic | OnDeviceRagAnswerer |
fixed pipeline: retrieveRelevant(question, topK = 3) then generate |
Same model (Gemma-4-E2B-it via LiteRT-LM, CPU backend), same device, same retrieval infrastructure, same fixture. The only difference is orchestration — which is precisely the journey-5 vs. journey-6 delta.
AgentSelector is bypassed; both arms are constructed directly, so a silent cloud fallback cannot contaminate an on-device measurement.
The non-agentic arm's ceiling is structural, and that is the point. With topK = 3 it cannot exceed 3 retrieved notes, so every broad question whose answer spans four or more notes is capped below full recall before the model runs. The measurement's value is not that ceiling — it is how often the agentic arm clears it.
1 buy olive oil
2 pick up laundry detergent
3 get cat food
4 buy avocados
5 football match tomorrow night
6 ballet show next week
7 dentist appointment on Friday
8 started reading Sapiens
9 review Kotlin coroutines
10 sign up for the gym
11 call mom
12 pay the electricity bill
Two properties are deliberate:
- The "event" concept is spread across three vocabularies (match / show / appointment). No single embedding query pulls all three into the top 3 — this reproduces journey-6's miss on purpose instead of by luck.
- The shopping notes avoid the few-shot's items.
OnDeviceAgent's few-shot example uses "buy milk / buy eggs / buy bread". Reusing those would test memorization of the worked example rather than generalization, which the project's own test-rigor rule rejects.
The fixture is seeded into a separate ObjectBox store (eval-notes), so a run never touches real notes in the app's store.
A. Direct command / count — expected tool count_notes / add_note
| id | question | expectation |
|---|---|---|
| A1 | how many notes do I have? | 12 |
| A2 | what's the total number of notes I've saved? | 12 (paraphrase robustness) |
| A3 | add a note: renew my passport | note containing "passport" exists |
| A4 | note that I have a meeting tomorrow | note containing "meeting" exists |
B. Narrow search — expected tool search_notes, single target note
| id | question | target |
|---|---|---|
| B1 | do I have a note about Sapiens? | #8 |
| B2 | when is my dentist appointment? | #7 |
| B3 | what did I note about Kotlin? | #9 |
| B4 | is there anything about the electricity bill? | #12 |
C. Broad / relational — expected tool get_all_notes, more than three target notes
| id | question | targets |
|---|---|---|
| C1 | do I have any events coming up? | #5, #6, #7 |
| C2 | what's on my shopping list? | #1–4 |
| C3 | what should I pick up from the store? | #1–4 (paraphrase of C2) |
| C4 | what do I need to do this week? | #7, #10, #11, #12 |
| C5 | what topics are in my notes? | all 12 |
| C6 | what personal errands do I have? | #10, #11, #12 |
C2 is the weakest test in this group and is kept deliberately: its wording ("what's on my shopping list?") is a near-paraphrase of the few-shot's "What's in my grocery list?", even though its target items are not. The generalization signal therefore lives in C3, C4 and C6. If C2 succeeds while C3 fails on the same four target notes, that contrast is itself the finding — the model would be following the worked example rather than the lesson.
X. Cross-language side check — reported separately, not part of the primary tables
| id | question | targets |
|---|---|---|
| X1 | yaklaşan etkinliklerim var mı? | #5, #6, #7 (Turkish twin of C1) |
| X2 | alışveriş listemde ne var? | #1–4 (Turkish twin of C2) |
The embedding model is universal_sentence_encoder.tflite (6.1 MB, 100-dim) — the English-only MediaPipe USE variant. X1/X2 ask in Turkish against English notes to isolate how much of a cross-language failure is retrieval rather than generation. See the confound note in decision-log.md.
Every agentic measurement runs 3 times and is reported as 3/3, 2/3, 0/3. Journey-6 recorded the failure as intermittent ("sometimes picks the wrong tool"); a single run cannot measure an intermittent behaviour, and a published single-run table would not reproduce for a reader.
Non-agentic rows that involve no generation (retrieval recall) are deterministic and run once, marked det. Non-agentic add_note rows are marked structurally unsupported — that pipeline can only read.
The model must already be downloaded in the app (the run skips itself otherwise).
./gradlew installDebug installDebugAndroidTest
adb logcat -c && adb logcat -s NotesAiEval > eval-run.log &
adb shell am instrument -w \
-e class com.caner.notesai.eval.OnDeviceEvalRun \
com.caner.notesai.test/androidx.test.runner.AndroidJUnitRunner
adb pull /storage/emulated/0/Android/data/com.caner.notesai/files/eval-results.md .Do not run this with ./gradlew connectedDebugAndroidTest. That task uninstalls both APKs when the run finishes, and uninstalling wipes /Android/data/com.caner.notesai — which takes the report and the 2.6 GB downloaded model with it. Learned the hard way on the first run: the run itself succeeded, and every result was deleted seconds later. am instrument runs the same test against the installed APKs and uninstalls nothing.
The logcat redirect is deliberate redundancy: the report on the device and the local log are written from the same emit() call, so either one alone is sufficient evidence.
Each row is appended to eval-results.md on the device as it completes, so a mid-run failure does not discard the minutes already spent. Expected wall-clock: roughly 14 minutes (16 questions × 3 agentic repetitions at ~15 s, plus two non-agentic count questions).
The report opens with a dump of the tool schema the model actually receives (ToolManager.getToolsDescription()), and the run aborts if it is not exactly the four expected tools. LiteRT-LM builds that schema by reflection over NotesToolSet, so the invocation recorder added for this evaluation could in principle leak into it — which would silently change tool-selection behaviour and make the numbers describe a different agent than journey-6 documented. The dump is also the artifact that lets a reader confirm the model saw the same four tools with the same descriptions.
It closes with an aggregate block: correct-tool rate and pooled context recall per category and arm.
This is a measurement run, not a test. It asserts nothing and cannot fail the build; it uses the instrumentation runner purely as a way to execute code on the device against the real model. The real unit test (AskViewModelTest) stays where it belongs.
Run of 2026-08-07, Pixel 10 Pro XL, Gemma-4-E2B-it on CPU. 12 min 31 s. Raw log: eval-run-2026-08-07.log. The tool-schema check passed — the model was shown exactly the four expected tools.
| category | arm | correct tool chosen | context recall |
|---|---|---|---|
| DIRECT_COMMAND | agentic | 12/12 | - |
| DIRECT_COMMAND | non-agentic | n/a | - |
| NARROW_SEARCH | agentic | 12/12 | 12/12 (100%) |
| NARROW_SEARCH | non-agentic | n/a | 4/4 (100%) |
| BROAD_RELATIONAL | agentic | 6/18 | 63/90 (70%) |
| BROAD_RELATIONAL | non-agentic | n/a | 10/30 (33%) |
| CROSS_LANGUAGE | agentic | 0/6 | 6/21 (28%) |
| CROSS_LANGUAGE | non-agentic | n/a | 1/7 (14%) |
The thesis in two numbers. On broad questions, agency roughly doubles context recall — 70% vs. 33%. And it does so while choosing the right tool only 6 times out of 18. Both layers are visible at once: giving the model tools helps a lot; teaching it which tool to reach for remains largely unsolved.
Tool choice and recall were identical across all three repetitions of every question, so each row summarises three runs.
| id | agentic tool | agentic recall | non-agentic recall |
|---|---|---|---|
| A1 how many notes | count_notes |
- | - (answered "4", expected 12) |
| A2 total number saved | count_notes |
- | - (answered "4", expected 12) |
| A3 add: renew passport | add_note, written |
- | cannot write |
| A4 note: meeting tomorrow | add_note, written |
- | cannot write |
| B1 note about Sapiens | search_notes |
1/1 | 1/1 |
| B2 dentist appointment | search_notes |
1/1 | 1/1 |
| B3 note about Kotlin | search_notes |
1/1 | 1/1 |
| B4 electricity bill | search_notes |
1/1 | 1/1 |
| C1 events coming up | get_all_notes |
3/3 | 0/3 |
| C2 what's on my shopping list | search_notes ✗ |
2/4 | 2/4 |
| C3 pick up from the store | search_notes ✗ |
2/4 | 2/4 |
| C4 what to do this week | search_notes ✗ |
1/4 | 2/4 |
| C5 what topics are in my notes | get_all_notes |
12/12 | 3/12 |
| C6 personal errands | search_notes ✗ |
1/3 | 1/3 |
| X1 (tr) yaklaşan etkinliklerim | search_notes ✗ |
0/3 | 0/3 |
| X2 (tr) alışveriş listemde ne var | search_notes ✗ |
2/4 | 1/4 |
1. Three repetitions produced no metric-level variance. For every question, all three runs chose the same tool and reached the same recall. (Answer text is truncated at 180 characters in the report, so full-text identity is not claimed.) This reframes journey-6's "sometimes picks the wrong tool": the variance is across phrasings, not across runs of the same phrasing. The model is stable and wrong in the same way every time, which makes the failure a property of the question, not of chance.
2. Agency is not uniformly better — C4 is the counterexample. On "what do I need to do this week?", the agentic arm scored 1/4 against the fixed pipeline's 2/4. Its self-chosen search_notes query pulled event notes instead of errands. A table where agency wins everywhere would be the less credible artifact.
3. Narrow retrieval doesn't only omit — it injects. C2 and C3 answered that the shopping list includes "a call to mom", because search_notes returned call mom inside the top 3 and the model reported what it was handed as fact. The failure mode is not just missing items; it is confidently wrong items.
4. The few-shot fix has a cost: it leaks fabricated notes into answers. C5 reached full recall (12/12 — every real note entered context) yet the answer lists "Milk, eggs, bread" under Groceries. Those three strings appear nowhere in the fixture; they are verbatim from FEW_SHOT_TOOL_USE_EXAMPLE in OnDeviceAgent. The journey-6 remediation that taught the model to call tools also taught it three notes that do not exist.
This is invisible to the recall metric by construction — recall measures what entered the context, not what the answer claims about it. It was found by reading the answer column. Worth stating plainly: a metric chosen for being mechanically checkable will miss failure modes it was not designed to see.
5. The count failure is worse than journey-5 recorded. journey-5 documented the non-agentic arm reporting "3" — the size of the retrieved set. Here it reports "4" for a 12-note store, three times out of three, with three notes in context. The number is not anchored to the retrieved set at all.
Derived from the report's own timestamps (the gap between consecutive rows). Every measurement re-seeds the 12-note fixture first, so these are upper bounds — but the overhead is identical across arms, so the comparison is fair.
| arm | path | n | median | min | max |
|---|---|---|---|---|---|
| agentic | count_notes |
5 | 9.1 s | 8.7 s | 9.4 s |
| agentic | add_note |
6 | 10.7 s | 10.2 s | 11.0 s |
| agentic | search_notes |
30 | 12.7 s | 11.0 s | 43.7 s |
| agentic | get_all_notes |
6 | 16.5 s | 12.7 s | 42.3 s |
| non-agentic | retrieval + generation | 6 | 3.0 s | 2.7 s | 3.3 s |
| non-agentic | retrieval only (no model) | 12 | 1.5 s | 1.2 s | 3.6 s |
One caveat on the base before reading anything into it: the non-agentic median comes from the two count questions only (n=6) — the only ones where that arm ran the model at all, since its recall elsewhere is deterministic and needs no generation. Its prompt is three notes for every question regardless, so the figure should generalise, but it rests on a narrow base and should be read as an estimate rather than a measurement across the question set.
Two things fall out of this.
Agency costs roughly 4× the wall-clock. 12.7 s median against the fixed pipeline's 3.0 s, same device, same model. The tool round-trip is two model turns instead of one — emit the call, then synthesise the result — and each turn pays full on-device inference. The recall table shows what agency buys; this is what it costs. Reporting one without the other would be half a result.
The correct tool is also the expensive tool. get_all_notes puts all 12 notes into the context window and runs ~30% slower at the median than search_notes, with a much heavier tail. So the model's wrong choice on C2/C3/C4/C6 was also its fast choice. Fixing tool selection would not be a free win — a version of this app that always scanned broadly would be materially slower to answer.
The two outliers (42.3 s and 43.7 s) are consecutive, both around C5 — the question with the largest context. Sustained CPU inference on a phone throttles; single worst-case numbers from a run this size should not be read as representative, which is why the median is the column that matters.
C1 and X1 are the same question in two languages, against the same English notes and the same three target notes.
| tool | recall | |
|---|---|---|
| C1 (en) "do I have any events coming up?" | get_all_notes 3/3 |
3/3 |
| X1 (tr) "yaklaşan etkinliklerim var mı?" | search_notes 0/3 |
0/3 |
X1 degraded on both axes, and the non-agentic arm scored 0/3 on it too — which isolates retrieval as definitively broken for Turkish queries against English notes, exactly as an English-only embedder predicts.
X2 complicates the picture usefully: Turkish "alışveriş listemde ne var?" scored 2/4, identical to its English twin C2. So the language effect is large on the events question and mild on the shopping one.
What this means for journey-6. Journey-6's decisive test was Turkish, and the note it missed was Turkish. This run shows the same question in English succeeding perfectly with the right tool. The honest conclusion is not that journey-6's finding was wrong — the tool-selection weakness is real and reproduced here on six English questions — but that its magnitude was substantially a language artifact, not solely a top-K one.
- 12 notes, 16 questions, one device, one model, three repetitions. This is a reproducible illustration, not a benchmark.
- Target-note matching is substring containment of the seeded note text inside the tool result — reliable here because the fixture is fixed and short.
- Only the on-device arms are measured in this phase. A cloud arm (
NotesAgent) over the identical fixture is a possible second phase, deliberately separated so that rate-limit risk cannot interfere with the primary result.