Skip to content

filesystem: realistic examples in search_files' description stop small models refusing to search (measured; follow-up to #896 / #3419) #5553

Open

Description

@Tanishq67m

Which server?

filesystem (@modelcontextprotocol/server-filesystem)

Protocol era

Both, or not era-specific

MCP client

No response

The problem

Server: @modelcontextprotocol/server-filesystem, measured on 2.0.0 and 2.2.0 (checked: 2.3.0 ships the same search_files description and the same empty result). Run with npx -y @modelcontextprotocol/server-filesystem@<version> <dir>, stdio, Node 24, macOS.

Thanks for the rewrite from #896 / #3419. We measured it, and it clearly fixes how agents search: a small model that does search now finds files reliably.

What it doesn't fix: on questions like "How many orders are there in total?", the model usually doesn't search at all. It answers on step 1, "I don't have access to any database or system that would contain order information", while the data sits one tool call away. In a controlled test the example patterns in the description decided this:

search_files description (server 2.0.0, 4 tasks x 3 runs, answers checked in code) Pass
2.0.0 original 2/12
2.2.0 / 2.3.0 text 2/12
precise rules + generic examples '**/*.ext', '**/name' 3/12
precise rules + concrete examples '**/*.json', '**/*invoice*' 9/12 (reproduced in 3 separate runs)

The only difference between the last two rows is the two example patterns. On the two affected tasks they went from 0/6 to 6/6, and with the concrete examples the model searched **/*order* and found the data.

Case study (method, all tables, caveats): https://github.com/Tanishq67m/AgentGrade/blob/main/docs/CASE_STUDY.md
Raw reports: https://github.com/Tanishq67m/AgentGrade/tree/main/results

Solution you have in mind (optional)

  1. Add one or two realistic examples next to the generic ones, e.g. '**/*.json', '**/*report*'. Small models seem to read abstract placeholders as "this is a file utility", not "the user's data is here".
  2. Replace the bare No matches found with a hint, e.g. "No matches for '' under ''. Patterns are relative to path; use '**/' to search subdirectories." In earlier runs agents repeated non-recursive patterns up to 9 times after a bare empty result.

Which MCP feature does it demonstrate? (optional)

No response

Alternatives or workarounds you have tried

  • Agent-side prompt advice ("Explore with the tools before saying information is unavailable"): 3/12 vs 1/12, still mostly refusals.
  • Advising "look at the folder layout first" in the description: no effect (1/12).
  • Removing search_files entirely: no effect (1/12); the refusal happens before any tool is chosen.
  • A larger model (gpt-oss-120b) solved 11/12 even on 2.0.0, so this mainly affects small, cheap models. One small model and synthetic data, 12 runs per row: indicative, not universal.

Already built it locally?

No server code was changed. Each description was tested by having the client show the model only a different search_files description, with the server binary unchanged and the same tasks and repeats. Verified by code-checked answers, 12 runs per variant. The exact texts are in the case study.

Before you submit

  • I searched existing issues and this is not a duplicate.
  • This is a request for this repository or one of its servers, not a new server, the MCP specification or an SDK.

Activity

  1. shivamm-gupta commented on Nov 18, 2026

    @shivamm-gupta

    Put together a patch for this (and the related #5552 for git_log):
    shivamm-gupta@b3c9386b

    Summary of changes

    • In server.ts and README.md, added realistic glob examples ('**/*.json', '**/package.json', '**/*config*') to both the tool-level description and the pattern parameter schema.
    • In mcp_server_git/server.py and README.md, clarified git_log description to state results are in reverse chronological order (newest first) and clarified that start_timestamp/end_timestamp are optional filters.
    • Updated src/git/tests/snapshots/list_tools.json snapshot and added a patch changeset for @modelcontextprotocol/server-filesystem.

    Prompt used

    Per repo contribution guidelines, here is the prompt that produced the change:

    In @modelcontextprotocol/server-filesystem (src/filesystem/server.ts and README.md), update the search_files tool description and pattern parameter description to include realistic examples ('**/*.json', '**/package.json', '**/*config*') alongside generic ones (#5553). Also in mcp-server-git (src/git/src/mcp_server_git/server.py and README.md), update git_log tool description to specify reverse chronological order (newest first) and clarify that start_timestamp and end_timestamp are optional date range filters (#5552). Update src/git/tests/snapshots/list_tools.json snapshot and add a patch changeset for the filesystem server.

  2. added
    server-filesystemReference implementation for the Filesystem MCP server - src/filesystem
    on Nov 18, 2026
  3. cliffhall commented on Nov 18, 2026

    @cliffhall
    Member

    Triage: Priority Low (total 4)

    • Severity 2: small models skip search_files and answer without searching; the description change is a measured improvement, and agent-side prompting is a partial workaround
    • Urgency 2: wanted eventually, nothing blocked on it
    • Bonuses: none

    Board: Servers V2 (#43), Status Incoming, awaiting maintainer review (no milestone yet).

  4. added this to the v2.6.0 milestone on Nov 18, 2026
  5. Tanishq67m commented on Nov 18, 2026

    @Tanishq67m
    Author

    Thanks @shivamm-gupta for putting this together. Before it lands I measured the patch on real builds (your commit b3c9386b vs its parent e1d154ee, filesystem 2.3.0). I also need to correct the scope of this issue.

    Setup: qwen3:4b-instruct (Ollama, temperature 0.7), the same 4 file-lookup tasks as in the issue, 3 repeats each (12 runs per row), every answer checked in code.

    search_files on 2.3.0 Passed
    main e1d154ee 1/24 (two runs)
    patch b3c9386b, as built 2/12
    patch, with the user-data examples from this issue instead 2/12
    patch, first sentence softened 0/12
    main + the full description from this issue 4/12

    Almost every failure is the same as before: the model answers "I don't have access to any database" without calling a tool.

    Correction: the 3/12 → 9/12 result in this issue was measured on 2.0.0. It still holds there (7/12 in a re-run today). On 2.3.0, realistic examples alone no longer change the behavior, whether they're the patch's examples or ours.

    To find what changed between 2.0.0 and 2.3.0, I ran a 2×2. Every cell uses the description from this issue:

    2.0.0 descriptions on the other tools 2.3.0 descriptions on the other tools
    2.0.0 input schemas 7/12 6/12
    2.3.0 input schemas 5/12 4/12

    The new parameter descriptions (e.g. "…it must resolve inside the allowed directories" on every path) and the reworded list_directory / directory_tree / get_file_info descriptions each seem to cost a little. With 12 runs per cell I can't separate them further.

    So my honest read: the patch is a harmless clarity improvement, but I wouldn't expect it to fix the refusals, and I don't have a measured fix for 2.3.0 yet. I'm happy to run any candidate wording through the same A/B before it's merged; it takes about 15 minutes per variant.

    Suite: https://github.com/Tanishq67m/AgentGrade/blob/main/suites/pr-5553-filesystem.yaml
    Reports: https://github.com/Tanishq67m/AgentGrade/tree/main/results (files 2026-11-18-pr5553-*)

  6. self-assigned this
    on Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestserver-filesystemReference implementation for the Filesystem MCP server - src/filesystemv2

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions