A small experiment framework for testing whether writing quality for human readers affects AI agent intelligibility.
The core question: if you take the same documentation and degrade it in a specific, measurable way — adding pronoun ambiguity, removing actors with passive voice, hedging specific instructions, or leaving key terms undefined — does a language model's ability to answer questions from that document get worse? By how much? Does it vary by model size?
This is a controlled-degradation design: start from a well-written document (the control), produce isolated degraded versions that each introduce exactly one type of writing problem, and run the same query set against all versions across multiple models. Each version differs from the control in one dimension only, so you know what caused any accuracy drop.
This is a personal project built by a technical writer trying to understand how writing quality affects AI tools — not a peer-reviewed study. The experiments use synthetic documents, a small query set, and automated scoring. The results are directionally interesting but should not be cited as rigorous research. Treat this as a structured way to think about the problem, not as proof of anything.
If you run your own experiments and find different results, that's useful too.
This framework was built to support research for a book on technical documentation practices. The underlying hypothesis is supported by arXiv:2507.07847 ("From Ambiguity to Accuracy"), which demonstrated that resolving coreference ambiguity in source documents improved RAG QA accuracy by 7–25% depending on model size. luciddocs extends that finding to additional ambiguity types using synthetic documents designed to isolate specific writing problems.
luciddocs tests prose quality — the writing inside the document. It does not test whether your docs site is structurally accessible to AI agents in the first place.
For that, see the AFDocs spec — an open standard covering llms.txt discovery, markdown availability, page size, URL stability, and content structure. The afdocs CLI tests any docs site against the spec.
- Node.js 18+
- API keys for the agents you want to test (see
.env.example) - OpenAI API key for RAG modes (used for
text-embedding-3-smallembeddings by default) - Voyage API key if using
--embedder voyageor--embedder voyage-3.5 - Google API key if using
--embedder google
cp .env.example .env
# Add your API keys to .env
npm installnode run.js --experiment <name> [options]| Flag | Default | Description |
|---|---|---|
--experiment |
required | Experiment directory name under experiments/ |
--mode |
context |
Retrieval mode (see below) |
--agents |
claude-sonnet |
Comma-separated agent names (see below) |
--versions |
all | Comma-separated version IDs to run |
--k |
3 |
Number of chunks to retrieve in RAG modes |
--embedder |
openai |
Embedding model: openai, openai-large, google, voyage, voyage-3.5 |
Anthropic
| Alias | Model |
|---|---|
claude-opus |
claude-opus-4-8 |
claude-sonnet |
claude-sonnet-4-6 |
claude-haiku |
claude-haiku-4-5-20251001 |
OpenAI
| Alias | Model |
|---|---|
gpt-5.5 |
gpt-5.5 |
gpt-5.4 |
gpt-5.4 |
gpt-5.4-mini |
gpt-5.4-mini |
gpt-5.4-nano |
gpt-5.4-nano |
gpt-4o |
gpt-4o |
gpt-4o-mini |
gpt-4o-mini |
| Alias | Model |
|---|---|
gemini-3.5-flash |
gemini-3.5-flash |
gemini-3.1-pro |
gemini-3.1-pro-preview |
gemini-2.5-pro |
gemini-2.5-pro |
gemini-2.5-flash |
gemini-2.5-flash |
gemini-2.5-flash-lite |
gemini-2.5-flash-lite |
# Quick sanity check — one agent, context mode, all versions
node --env-file=.env run.js --experiment api-auth-short --mode context --agents claude-sonnet
# Compare Opus vs Sonnet vs Haiku — does model size change sensitivity to vague writing?
node --env-file=.env run.js --experiment api-auth-short --mode context --agents claude-opus,claude-sonnet,claude-haiku
# Compare context stuffing vs RAG on the same agent
node --env-file=.env run.js --experiment api-auth-short --mode context --agents claude-sonnet
node --env-file=.env run.js --experiment api-auth-short --mode rag --agents claude-sonnet
# Full battery — all degradation types, two agents, both modes
node --env-file=.env run.js --experiment api-sdk-complex --mode context --agents claude-sonnet,gpt-4o
node --env-file=.env run.js --experiment api-sdk-complex --mode rag --agents claude-sonnet,gpt-4o
# Isolate one degradation type
node --env-file=.env run.js --experiment api-auth-short --mode context --agents claude-sonnet --versions original,degraded-coreference
# Compare all RAG strategies on the same agent — does retrieval method change the results?
node --env-file=.env run.js --experiment api-sdk-complex --mode rag --agents claude-haiku
node --env-file=.env run.js --experiment api-sdk-complex --mode rag-hyde --agents claude-haiku
node --env-file=.env run.js --experiment api-sdk-complex --mode rag-multiquery --agents claude-haiku
node --env-file=.env run.js --experiment api-sdk-complex --mode rag-parentdoc --agents claude-haiku
node --env-file=.env run.js --experiment api-sdk-complex --mode rag-contextual --agents claude-haiku
node --env-file=.env run.js --experiment api-sdk-complex --mode rag-agent --agents claude-haiku
# Test whether HyDE recovers undefined-terms failures (the vocabulary gap hypothesis)
node --env-file=.env run.js --experiment api-sdk-complex --mode rag --agents claude-haiku --versions original,degraded-undefined
node --env-file=.env run.js --experiment api-sdk-complex --mode rag-hyde --agents claude-haiku --versions original,degraded-undefined
# Compare embedding models on the same strategy
node --env-file=.env run.js --experiment api-sdk-complex --mode rag --agents claude-haiku --embedder openai
node --env-file=.env run.js --experiment api-sdk-complex --mode rag --agents claude-haiku --embedder voyage-3.5
node --env-file=.env run.js --experiment api-sdk-complex --mode rag --agents claude-haiku --embedder openai-large
# Cross-validate a finding across models
node --env-file=.env run.js --experiment api-sdk-complex --mode rag-agent --agents claude-haiku
node --env-file=.env run.js --experiment api-sdk-complex --mode rag-agent --agents gpt-4o-mini
node --env-file=.env run.js --experiment api-sdk-complex --mode rag-agent --agents gemini-2.5-flashResults land in experiments/<name>/results/<run-name>/ as results.json and report.md, and print as a markdown table to stdout.
| Mode | Description |
|---|---|
context |
Whole document in the system prompt. Isolates comprehension failure from retrieval failure. |
rag |
Naive RAG: chunk by heading, embed with the selected embedder, retrieve top-k by cosine similarity. |
rag-hyde |
HyDE: generate a hypothetical documentation excerpt for the question, embed that instead of the raw query. Closes the vocabulary gap between question phrasing and document phrasing. |
rag-multiquery |
Multi-query: generate 3 rephrased versions of the question, retrieve for each, union the results. Increases the chance that at least one phrasing matches the document's vocabulary. |
rag-contextual |
Contextual retrieval: before embedding, prepend a one-sentence document-level summary to each chunk. Gives the embedding model context it would otherwise lack. Costs one LLM call per chunk per document version (cached within a run). Anthropic technique. |
rag-parentdoc |
Parent-document retrieval: index small paragraph chunks for similarity search, but pass the full parent heading section to the LLM when a chunk is retrieved. Small chunks for finding; large chunks for reading. Helps most with undefined abbreviations and pronouns that resolve in adjacent text. |
rag-agent |
Agent retrieval: the LLM receives a search(query) tool it can call up to four times per question. It reads results and can refine its phrasing before answering. Most deployed systems do not use agent retrieval. |
All RAG modes use OPENAI_API_KEY for embeddings by default. Use --embedder to switch to a different model.
Create a directory under experiments/:
experiments/
my-experiment/
original.md ← the well-written control document
degraded-coreference.md ← degraded version(s)
experiment.json ← version list + query set
experiment.json schema:
{
"name": "Human-readable name",
"description": "What this experiment tests",
"versions": [
{ "id": "original", "file": "original.md", "description": "..." },
{ "id": "degraded-coreference", "file": "degraded-coreference.md", "description": "..." }
],
"queries": [
{
"id": "q1",
"question": "The question to ask the agent",
"canary": "unique phrase embedded in the doc, or null",
"expected_facts": ["fact one", "fact two"]
}
]
}Two experiments ship with the repo. Results from a full battery run across context mode, all seven RAG strategies, five embedding models, and three LLM providers are included under each experiment's results/ directory.
A single-topic API authentication guide (~400 words). Used to establish baseline behavior and validate the degradation types. Because the document is short and simple, models can partially compensate for poor writing using context — making this a conservative test. Results are stored in experiments/api-auth-short/results/.
A multi-section payments SDK integration guide (~800 words) covering two distinct credentials, multi-step webhook validation, specific numeric rate limits, multiple HTTP error codes, and idempotency. Facts are spread across sections, which means RAG chunking splits related content and degradation effects are amplified. This is the primary experiment for the research findings. Results are stored in experiments/api-sdk-complex/results/.
A full summary of findings across both experiments is in experiments/SUMMARY.md. RAG strategy and embedding model findings are in experiments/RAG-STRATEGIES.md.
Both experiments use the same five degradation types. Each degraded version changes exactly one dimension from the clean control, so any accuracy drop can be attributed to that specific writing problem.
The clean baseline. Specific nouns, clear instructions, exact values, all terms defined on first use. Every other version is measured against this.
The Acme API uses API keys for authentication. Every request must include
your API key in the `Authorization` header using Bearer token format.
What it is: Using "it," "this," or "they" instead of naming the thing you're talking about.
Why writers do it: Avoids repetition. Sounds less robotic. Totally fine when there's only one thing the pronoun could refer to.
Why it's a problem: When a document has multiple things — two credentials, two systems, two steps — "it" becomes ambiguous. A human reader uses context from the whole page to figure out which "it" you mean. An AI retrieving a single chunk may not have that context.
It uses API keys for this purpose. Every request must include it in the
header using the right format.
What the AI does wrong: Confidently answers about the wrong "it," or gives a vague answer that sounds correct but isn't specific enough to act on.
Real example (gpt-4o-mini, api-auth-short, degraded-coreference):
You should include your API key in the request header using the format:
[it]: [right format] YOUR_KEYFor example, using curl:
curl -H "[it]: [right format] YOUR_KEY" https://api.acme.io/v1/status
The AI parroted unresolved pronoun placeholders — [it] and [right format] — directly into the answer as working syntax. A developer would copy-paste this and get an authentication error with no obvious debugging path.
What it is: Writing sentences where no one is doing anything. "The key must be included" instead of "include your key."
Why writers do it: Passive voice sounds formal and objective. It's common in technical and legal writing. Sometimes the actor genuinely doesn't matter.
Why it's a problem: Instructions need an actor. "The key must be included" doesn't tell you who includes it — you, the system, the server? For a human, context makes it obvious. For an AI answering "what do I need to do?", the missing actor makes every step ambiguous.
API keys are used for authentication. An API key must be included in the
`Authorization` header of every request using Bearer token format.
What the AI does wrong: Echoes the passive voice back ("the key should be included") without clarifying who does it, or hedges when it should be definitive.
Real example (gpt-4o-mini, api-auth-short, degraded-passive, asked for exact rotation steps):
- A new API key must be created in the Acme dashboard.
- Environment variables must be updated to use the new key.
- A test request must be made to verify the new key works.
- The old key must be deleted from the Acme dashboard.
Every step is passive — "must be created," "must be updated," "must be deleted." There's no actor. The reader cannot tell whether they do these things, or the system does, or something else.
What it is: Wrapping clear instructions in qualifiers — "generally," "where applicable," "in most cases," "depending on your configuration."
Why writers do it: To avoid being wrong. If an edge case exists, a hedge protects you. It's also common when instructions are being written for a broad audience and the writer isn't sure what applies to everyone.
Why it's a problem: A hedge turns a fact into a maybe. "Every request must include X" is actionable. "Requests should generally include X where applicable" is not. The AI has no way to know whether the hedge applies to the user asking the question, so it echoes the hedge back — and the user gets a non-answer.
Every request should include your API key in the appropriate header using
the correct format where applicable.
What the AI does wrong: Mirrors the vagueness — "in standard configurations, you may want to include your key" — rather than giving a definitive answer.
Real example (claude-haiku, api-auth-short, degraded-scope, asked how often to rotate):
It is generally recommended to rotate your API key every 90 days or as required by your security policy. The documentation states that rotation steps may vary by environment... you may want to consult your dashboard or Acme support for environment-specific instructions if needed.
The source document had precise, unambiguous steps. The degraded version introduced vague scope language, and the AI amplified it — adding "generally recommended," "steps may vary," and a redirect to "consult Acme support" for information that was already explicitly in the document.
What it is: Replacing specific values with approximations. "Every 90 days" becomes "periodically." "401 Unauthorized" becomes "an error." "Settings → API Keys" becomes "the appropriate settings area."
Why writers do it: Specific values go stale. If the UI changes or the interval gets updated, a vague instruction is easier to maintain. Sometimes the writer just doesn't know the exact value.
Why it's a problem: AI assistants are asked for specific answers. If a user asks "how often should I rotate my key?" and the doc says "periodically," the AI will say "periodically." The specific value — the only thing the user actually needed — is gone.
You should probably rotate your API key every so often, roughly every 90
days or thereabouts. Look for something like Settings or similar.
What the AI does wrong: Reproduces the vague language ("rotate periodically") or invents a specific value that wasn't in the doc — both leave the user without the answer they needed.
Real example (claude-haiku, api-auth-short, degraded-hedging, asked how often to rotate):
According to the documentation, quarterly is a common approach, though your specific needs may vary... The documentation notes that things should generally continue working during the transition, more or less.
The source document said "every 90 days" and described precise transition behavior. The degraded version softened it to vague hedges, and the AI converted the specific value to "quarterly" and the clear transition guarantee to "things should generally continue working... more or less."
What it is: Using abbreviations, acronyms, or internal terminology without ever explaining what they mean.
Why writers do it: Abbreviations save space. Within a team, everyone knows what "BT" means. Jargon signals expertise. After you've written "API key" fifty times, "AK" feels obvious.
Why it's a problem: An AI has no shared context with your team. If your doc says "include your BT in the AC-H," the AI has to guess — and it will guess, often confidently and wrong. Worse, in RAG mode the AI can't even find the answer because the search is looking for "API key" and the doc says "BT." The content exists but is effectively invisible.
Every request must include your BT in the AC-H using BF. Log in to the
ADP and navigate to S → BT-M.
What the AI does wrong: Either parrots the abbreviations back ("include your BT in the AC-H"), refuses to answer because it can't decode the terms, or hallucinates a definition that sounds plausible but isn't in the doc.
Real example (claude-haiku, api-auth-short, degraded-undefined, asked where to put the API key):
According to the documentation, you should include your API key (called a "BT" in the Acme API) in the AC-H header using the BF format.
AC-H: BF YOUR_BTHere's an example using curl:
curl -H "AC-H: BF YOUR_BT" https://api.acme.io/v1/status
The AI treated unexplained abbreviations as authoritative technical terms and built a complete — and completely wrong — code example around them, with no signal that these are meaningless placeholders. A developer would copy-paste this and get an authentication error with nothing to go on.
These are the findings from the full battery run on api-sdk-complex in naive RAG mode (claude-haiku baseline).
| Degradation type | Score | Drop from control |
|---|---|---|
| original (control) | 100% | — |
| passive voice | 93% | −7 pp |
| pronoun ambiguity | 69% | −31 pp |
| undefined terms | 65% | −35 pp |
| quantifier vagueness | 56% | −44 pp |
| scope hedging | 54% | −46 pp |
Key findings from the full experiment set:
- Passive voice had almost no effect — never dropped below 88% across any model, mode, or strategy. The null result.
- Quantifier vagueness and scope hedging resist every fix — both hold near 54–58% regardless of model size, RAG strategy, or embedding model. Agent retrieval gains +2pp on vague quantities (noise) and +19pp on scope hedging (partial recovery). The vague-quantities floor is a confabulation risk: the model synthesizes a plausible specific value and states it confidently.
- Undefined jargon is fixable — but the fix depends on the model. Parent-document retrieval recovers +11–20pp across all three providers. Agent retrieval recovers +30pp for claude-haiku but only +6pp for gpt-4o-mini and −4pp for gemini-2.5-flash. Always cross-validate RAG strategy gains across models before deploying.
- Embedding model effects are also model-dependent. Voyage 3.5 gains +10pp on jargon for haiku, +7pp for gemini, +4pp for gpt-4o-mini. OpenAI large gains +6pp on jargon for haiku but loses −6pp for gemini and −10pp for gpt-4o-mini. Optimizing embedder and strategy separately then combining them does not reliably produce the expected sum.
- Contextual retrieval makes pronoun ambiguity worse across all three providers tested (−2 to −13pp). The summarization step resolves pronouns and can get them wrong.
- Document complexity amplifies every effect — the same degradation types that cause moderate drops on the short doc cause much larger drops on the complex multi-section doc.
Full results are in experiments/SUMMARY.md and experiments/RAG-STRATEGIES.md.
The tool scores every response automatically using signals defined in experiment.json:
- Canary: a unique phrase embedded in the source document. Binary — either the agent surfaced it or didn't. Useful for confirming the agent read a specific passage rather than hallucinating an answer from prior training.
- Expected facts: strings that should appear in the response (case-insensitive). Partial credit — 3 of 4 facts found = 75%. Useful for confirming specific values (status codes, header names, time intervals) made it through.
Automated scoring catches fact omission and hallucination of specific values. It does not catch confident-but-wrong answers, appropriate hedging that the tool penalizes, or cases where the agent answered correctly but used different phrasing.
After each response is generated, the tool automatically sends it to claude-haiku for qualitative scoring. The judge receives the source document, the question, and the agent's answer, and rates the response on four dimensions (0–2 each) for a total of 0–8.
| Dimension | 0 | 1 | 2 |
|---|---|---|---|
| Factual accuracy | States something false | Partially correct; omits or conflates key facts | All stated facts are correct |
| Specificity | Vague or hedged ("you may want to...") | Somewhat specific but missing key values | Precisely answers the question with correct specifics |
| Completeness | Major gaps; question not meaningfully answered | Answers the core question but misses supporting context | Fully answers the question including relevant caveats |
| Hallucination | Adds false information not in the source doc | Adds plausible but unverifiable information | Stays within the source doc; admits gaps if present |
Rubric scores appear in results.json alongside automated scores and are included in report.md. Use them alongside the automated score, not instead of it — a response can score 100% on automated facts but only 4/8 on the rubric if the answer is technically correct but so hedged it would confuse a real user. This is exactly what happens with scope hedging failures.
Why a separate judge model matters: the automated score catches fact omission ("did the word '90 days' appear in the answer?") but misses quality failures that string matching can't see. The rubric catches cases where the agent echoed vague source language, parroted abbreviations, or gave an answer that sounds complete but isn't. Running the rubric through a cheap fast model (haiku) keeps the cost low while adding qualitative signal the string-match scorer can't produce.
Human review: the rubric is a starting point, not a final verdict. For findings you intend to publish, rate a sample of responses manually — especially scope hedging failures, where AI-judged rubric scores tend to be more generous than a human reviewer would be.
Calibration tip: compare rubric scores on original-version responses first to establish a per-agent baseline, then look at the delta on degraded versions. The delta is the finding.
The automated score tells you that accuracy dropped. The rubric tells you how it failed:
- Coreference failures tend to show up as wrong-referent answers — the agent picks the wrong entity to describe. Specificity score drops; accuracy score drops.
- Passive voice failures tend to produce actor-free instructions ("the key should be included") that are technically accurate but operationally useless. Specificity drops; accuracy may hold.
- Scope hedging failures are mirrored back — the agent echoes the hedge. Completeness drops; hallucination stays clean.
- Quantifier vagueness failures show up two ways: the agent echoes the vague language (specificity drops) or invents a specific value (hallucination rises).
- Undefined terms failures are the most visible: either the agent parrots the abbreviations, refuses to answer, or hallucinates definitions. All four rubric dimensions tend to drop together.
Results are saved under the experiment directory: experiments/<name>/results/<run-name>/
Each run folder contains:
results.json— array of result objects, one per version × agent × query. Each includes the full agent response, retrieved chunks (RAG mode), automated scores, and AI-judged rubric scores.report.md— auto-generated human-readable report with degradation deltas, response previews, and the rubric table for manual review.
A markdown summary table is also printed to stdout after each run and can be pasted directly into a research document.