From 1ab5242c05b3b45c72d4d4a93dd5f8f18e0fa147 Mon Sep 17 00:00:00 2001 From: Koichi ITO Date: Tue, 3 Nov 2026 05:13:31 +0900 Subject: [PATCH] Run every referee scenario in the server conformance leg ## Motivation and Context The server conformance leg ran the referee's default `active` suite, which leaves out the extension-tagged scenarios, while the client leg has selected every scenario with `--suite all`. The Skills server implementation therefore never met the referee's `sep-2640-skills-*` server scenarios, although the conformance server already serves the `conformance/skills` directory for them, and the `all` suite's other additions (stateless server, SSE polling, caching, standard header validation, `input-required` results) were not run either. The server leg now selects `--suite all` the way the client leg does, with a requirement set or a single scenario still replacing the selection. Of the 34 scenarios this adds, 23 pass. The rest reach three areas the SDK does not implement and are listed in `expected_failures.yml` the way the TypeScript and Python SDKs list theirs: the Tasks extension (SEP-2663); the server side of SEP-2243 custom headers, where the server does not bind `Mcp-Param-*` headers to `x-mcp-header` tool parameters and the conformance server exposes no such tool, so the referee reports the scenario's checks as untestable; and SEP-1699 resumability, where a POST SSE stream carries no priming event or `retry` field, which the referee reports as warnings that its baseline counts as failures. `tasks-status-notifications` is left out of the baseline because the referee runs no check in it, and the baseline check would flag an entry for a passing scenario as stale. ## How Has This Been Tested? test/conformance/server_runner_test.rb, and `bundle exec rake conformance:test`: the server leg runs 65 scenarios, the 11 baselined failures reported as expected and the three `sep-2640-skills-*` scenarios passing, where it ran 31 before the change. `REQUIREMENTS=2026-07-28` still runs the frozen set without a suite. ## Breaking Changes None. --- .github/workflows/conformance.yml | 2 +- conformance/README.md | 12 ++-- conformance/expected_failures.yml | 22 ++++++- conformance/server_runner.rb | 12 +++- test/conformance/server_runner_test.rb | 88 +++++++++++++++++++++++++- 5 files changed, 126 insertions(+), 10 deletions(-) diff --git a/.github/workflows/conformance.yml b/.github/workflows/conformance.yml index 6d54ec2d..2fc3fa7e 100644 --- a/.github/workflows/conformance.yml +++ b/.github/workflows/conformance.yml @@ -29,7 +29,7 @@ jobs: - run: bundle exec rake conformance:test # Frozen requirement sets for SEP-1730 tier scoring: each spec revision's scenarios run at - # that revision's own wire version, unlike the moving active suite above. + # that revision's own wire version, unlike the moving full suite above. requirements-conformance: runs-on: ubuntu-latest continue-on-error: true diff --git a/conformance/README.md b/conformance/README.md index c600ec04..79bf6cba 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -15,10 +15,12 @@ Validates the Ruby SDK's conformance to the MCP specification using [`@modelcont bundle exec rake conformance:test ``` -Runs both server and client conformance tests in sequence. Server conformance starts the -conformance server and tests it. Client conformance spawns test servers for each scenario -and invokes `conformance/client.rb` against them. Scenarios listed in `expected_failures.yml` -are allowed to fail without affecting the exit code. +Runs both server and client conformance tests in sequence. Server conformance starts the conformance server and tests it. +Client conformance spawns test servers for each scenario and invokes `conformance/client.rb` against them. +By default both legs select every scenario the referee ships (`--suite all`), the extension-tagged ones included, +so the Skills server and client implementations are checked; `SCENARIO` and `REQUIREMENTS` below replace that selection. +Scenarios listed in `expected_failures.yml`, which cover the areas the SDK does not implement, +are allowed to fail without affecting the exit code; the referee flags an entry as stale once its scenario passes. ### Environment variables @@ -55,7 +57,7 @@ bundle exec rake conformance:server PORT=3000 bundle exec rake conformance:server # Terminal 2: run all scenarios -npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp +npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp --suite all # Terminal 2: run a single scenario npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp --scenario ping diff --git a/conformance/expected_failures.yml b/conformance/expected_failures.yml index 93258420..c6c76cc7 100644 --- a/conformance/expected_failures.yml +++ b/conformance/expected_failures.yml @@ -1,4 +1,24 @@ -server: [] +server: + # SEP-1699 resumability: the server sends no priming event or `retry` field on a POST SSE stream, and + # the referee warns about each of the two; the baseline counts a warning as a failure. The resume on + # `Last-Event-ID` is not exercised, since the server does not close the stream mid-call. + - server-sse-polling + # SEP-2243 custom headers: the server does not bind `Mcp-Param-*` headers to `x-mcp-header` tool parameters + # (the client side sends them), so the conformance server exposes no such tool and the referee reports + # the scenario's checks as untestable. + - http-custom-header-server-validation + # SEP-2663 (io.modelcontextprotocol/tasks): the server does not implement the tasks extension. + # `tasks-status-notifications` is intentionally absent: the referee runs no check in it, and a baseline entry + # for a scenario with no failing check is flagged stale. + - tasks-lifecycle + - tasks-capability-negotiation + - tasks-wire-fields + - tasks-request-state-removal + - tasks-mrtr-input + - tasks-request-headers + - tasks-dispatch-and-envelope + - tasks-required-task-error + - tasks-mrtr-composition client: # Unimplemented authorization extensions. - auth/dpop diff --git a/conformance/server_runner.rb b/conformance/server_runner.rb index 94b1a83a..bdf61bcb 100644 --- a/conformance/server_runner.rb +++ b/conformance/server_runner.rb @@ -33,9 +33,17 @@ def build_command expected_failures_yml = File.expand_path("expected_failures.yml", __dir__) npx_command = ["npx", "--yes", "@modelcontextprotocol/conformance@alpha", "server", "--url", "http://localhost:#{@port}/mcp"] - npx_command += ["--scenario", @scenario] if @scenario + # A requirement set already fixes which scenarios run, so it replaces the suite selection. + # The referee's default `active` suite leaves out the extension-tagged scenarios (Skills, Tasks), which `all` selects, + # as the client leg does; the areas the SDK does not implement are listed in expected_failures.yml. + npx_command += if @requirements + ["--requirements", @requirements] + elsif @scenario + ["--scenario", @scenario] + else + ["--suite", "all"] + end npx_command += ["--spec-version", @spec_version] if @spec_version - npx_command += ["--requirements", @requirements] if @requirements npx_command += ["--verbose"] if @verbose npx_command += ["--expected-failures", expected_failures_yml] npx_command diff --git a/test/conformance/server_runner_test.rb b/test/conformance/server_runner_test.rb index dc52cd2b..a0ed7a09 100644 --- a/test/conformance/server_runner_test.rb +++ b/test/conformance/server_runner_test.rb @@ -7,7 +7,7 @@ module Conformance class ServerRunnerTest < ActiveSupport::TestCase EXPECTED_FAILURES_YML = File.expand_path("../../conformance/expected_failures.yml", __dir__) - test "build_command runs the active suite by default" do + test "build_command runs the full suite by default" do command = ServerRunner.new.send(:build_command) assert_equal( @@ -18,6 +18,92 @@ class ServerRunnerTest < ActiveSupport::TestCase "server", "--url", "http://localhost:#{Server::DEFAULT_PORT}/mcp", + "--suite", + "all", + "--expected-failures", + EXPECTED_FAILURES_YML, + ], + command, + ) + end + + test "build_command keeps the single scenario selection" do + command = ServerRunner.new(scenario: "ping").send(:build_command) + + assert_equal( + [ + "npx", + "--yes", + "@modelcontextprotocol/conformance@alpha", + "server", + "--url", + "http://localhost:#{Server::DEFAULT_PORT}/mcp", + "--scenario", + "ping", + "--expected-failures", + EXPECTED_FAILURES_YML, + ], + command, + ) + end + + test "build_command pins the full suite to a spec version" do + command = ServerRunner.new(spec_version: "2025-11-25").send(:build_command) + + assert_equal( + [ + "npx", + "--yes", + "@modelcontextprotocol/conformance@alpha", + "server", + "--url", + "http://localhost:#{Server::DEFAULT_PORT}/mcp", + "--suite", + "all", + "--spec-version", + "2025-11-25", + "--expected-failures", + EXPECTED_FAILURES_YML, + ], + command, + ) + end + + test "build_command pins a single scenario to a spec version" do + command = ServerRunner.new(scenario: "ping", spec_version: "2025-11-25").send(:build_command) + + assert_equal( + [ + "npx", + "--yes", + "@modelcontextprotocol/conformance@alpha", + "server", + "--url", + "http://localhost:#{Server::DEFAULT_PORT}/mcp", + "--scenario", + "ping", + "--spec-version", + "2025-11-25", + "--expected-failures", + EXPECTED_FAILURES_YML, + ], + command, + ) + end + + test "build_command lets a requirement set replace a scenario selection" do + command = ServerRunner.new(requirements: "2026-07-28", scenario: "ping").send(:build_command) + + assert_equal( + [ + "npx", + "--yes", + "@modelcontextprotocol/conformance@alpha", + "server", + "--url", + "http://localhost:#{Server::DEFAULT_PORT}/mcp", + "--requirements", + "2026-07-28", "--expected-failures", EXPECTED_FAILURES_YML, ],