Skip to content

Surface AskBill sources in search_documentation MCP tool#29

Merged
phoenixy1 merged 3 commits intomainfrom
ah-mcp-surface-doc-sources
May 5, 2026
Merged

Surface AskBill sources in search_documentation MCP tool#29
phoenixy1 merged 3 commits intomainfrom
ah-mcp-surface-doc-sources

Conversation

@phoenixy1
Copy link
Copy Markdown
Contributor

@phoenixy1 phoenixy1 commented May 5, 2026

Summary

  • The search_documentation MCP tool was discarding the sources array returned by the AskBill websocket and only forwarding answer.
  • Without citations, hallucinated URLs in the answer text slip through unchallenged — I hit this in practice while researching Plaid trial plans and got a plausible-but-broken plaid.com/docs/account/plans/#trial URL with no way to verify.
  • Append a ## Sources section to the response, formatted from the SourceMetadata shape (title, url) defined in plaid/finn-mvp:message_types.py. URLs are deduplicated; sources with neither URL nor title are skipped.
  • Title text is escaped (\, [, ]) and URLs use the <url> autolink form so titles like Auth [v2] and URLs containing parens (e.g. Plaid_(company)) render correctly. The answer is rstrip()ed before the Sources section to avoid stacked blank lines.

Drive-by: pytest config fix

CI was failing on a pre-existing latent bug in sandbox/pyproject.toml: [tool.pytest] should be [tool.pytest.ini_options], and python_files must be a list, not a bare string. Older pytest tolerated both; pytest 9 hard-fails:

TypeError: pyproject.toml: config option 'python_files' expects a list for type 'args', got str: 'test_*.py'

Last green CI was Oct 2025 (pytest 8.x); this PR was the first run since pytest 9 entered the environment, so it tripped here. Fix is in its own commit (Fix pytest config in pyproject.toml) so it's reviewable on its own — happy to split into a separate PR if preferred.

Verification

Ran the local checkout end-to-end against the real AskBill websocket (Claude Code MCP config swapped from uvx mcp-server-plaid to uv run --directory <local-clone> mcp-server-plaid):

  • Same query that previously hallucinated plaid.com/docs/account/plans/#trial now appends a ## Sources section with four real support.plaid.com/hc/en-us/articles/... URLs returned by AskBill's retrieval — confirming the prior link was an LLM hallucination with no source backing it.
  • Sources render as clickable markdown autolinks; no markdown breakage observed.
  • Existing 21 unit tests pass locally with the pytest config fix.

Test plan

  • Manually invoke the tool from an MCP client and confirm the ## Sources section appears with clickable links
  • Existing test suite passes locally
  • Confirm a query that produces no sources still returns just the answer (no empty Sources section) — relies on AskBill returning []; not yet observed in practice

🤖 Generated with Claude Code

phoenixy1 and others added 3 commits May 5, 2026 16:12
The AskBill websocket service returns both `answer` and `sources`, but the
handler was only forwarding `answer`. Without citations, callers (and the
LLMs consuming this tool) have no way to verify URLs the answer text mentions
— in practice, hallucinated links slip through unchallenged.

Append a `## Sources` section to the response, formatted from the
`SourceMetadata` shape (title, url) defined in finn-mvp.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…g whitespace

- Escape `\`, `[`, `]` in source titles so titles like "Auth [v2]" don't
  break the rendered markdown link.
- Wrap URLs in angle brackets (`<url>`) so URLs containing parens (e.g.
  Wikipedia's `Plaid_(company)`) still render correctly.
- `rstrip()` the answer before appending the Sources section so trailing
  newlines in the AskBill response don't produce 3+ blank lines.
- Add a docstring to `_format_sources` to match the rest of the file.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `[tool.pytest]` section was non-canonical (should be
`[tool.pytest.ini_options]`) and `python_files` was a bare string instead
of a list. Older pytest tolerated both; newer pytest (9.x) hard-fails:

  TypeError: pyproject.toml: config option 'python_files' expects a
  list for type 'args', got str: 'test_*.py'

Last green CI run was October 2025 against pytest 8.x; this PR is the
first run since pytest 9 made it into the environment.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@phoenixy1 phoenixy1 marked this pull request as ready for review May 5, 2026 23:48
@phoenixy1 phoenixy1 merged commit 0369426 into main May 5, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant