Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/conformance.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ jobs:
- run: bundle exec rake conformance:test

# Frozen requirement sets for SEP-1730 tier scoring: each spec revision's scenarios run at
# that revision's own wire version, unlike the moving active suite above.
# that revision's own wire version, unlike the moving full suite above.
requirements-conformance:
runs-on: ubuntu-latest
continue-on-error: true
Expand Down
12 changes: 7 additions & 5 deletions conformance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,12 @@ Validates the Ruby SDK's conformance to the MCP specification using [`@modelcont
bundle exec rake conformance:test
```

Runs both server and client conformance tests in sequence. Server conformance starts the
conformance server and tests it. Client conformance spawns test servers for each scenario
and invokes `conformance/client.rb` against them. Scenarios listed in `expected_failures.yml`
are allowed to fail without affecting the exit code.
Runs both server and client conformance tests in sequence. Server conformance starts the conformance server and tests it.
Client conformance spawns test servers for each scenario and invokes `conformance/client.rb` against them.
By default both legs select every scenario the referee ships (`--suite all`), the extension-tagged ones included,
so the Skills server and client implementations are checked; `SCENARIO` and `REQUIREMENTS` below replace that selection.
Scenarios listed in `expected_failures.yml`, which cover the areas the SDK does not implement,
are allowed to fail without affecting the exit code; the referee flags an entry as stale once its scenario passes.

### Environment variables

Expand Down Expand Up @@ -55,7 +57,7 @@ bundle exec rake conformance:server PORT=3000
bundle exec rake conformance:server

# Terminal 2: run all scenarios
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp --suite all

# Terminal 2: run a single scenario
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp --scenario ping
Expand Down
22 changes: 21 additions & 1 deletion conformance/expected_failures.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,24 @@
server: []
server:
# SEP-1699 resumability: the server sends no priming event or `retry` field on a POST SSE stream, and
# the referee warns about each of the two; the baseline counts a warning as a failure. The resume on
# `Last-Event-ID` is not exercised, since the server does not close the stream mid-call.
- server-sse-polling
# SEP-2243 custom headers: the server does not bind `Mcp-Param-*` headers to `x-mcp-header` tool parameters
# (the client side sends them), so the conformance server exposes no such tool and the referee reports
# the scenario's checks as untestable.
- http-custom-header-server-validation
# SEP-2663 (io.modelcontextprotocol/tasks): the server does not implement the tasks extension.
# `tasks-status-notifications` is intentionally absent: the referee runs no check in it, and a baseline entry
# for a scenario with no failing check is flagged stale.
- tasks-lifecycle
- tasks-capability-negotiation
- tasks-wire-fields
- tasks-request-state-removal
- tasks-mrtr-input
- tasks-request-headers
- tasks-dispatch-and-envelope
- tasks-required-task-error
- tasks-mrtr-composition
client:
# Unimplemented authorization extensions.
- auth/dpop
Expand Down
12 changes: 10 additions & 2 deletions conformance/server_runner.rb
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,17 @@ def build_command
expected_failures_yml = File.expand_path("expected_failures.yml", __dir__)

npx_command = ["npx", "--yes", "@modelcontextprotocol/conformance@alpha", "server", "--url", "http://localhost:#{@port}/mcp"]
npx_command += ["--scenario", @scenario] if @scenario
# A requirement set already fixes which scenarios run, so it replaces the suite selection.
# The referee's default `active` suite leaves out the extension-tagged scenarios (Skills, Tasks), which `all` selects,
# as the client leg does; the areas the SDK does not implement are listed in expected_failures.yml.
npx_command += if @requirements
["--requirements", @requirements]
elsif @scenario
["--scenario", @scenario]
else
["--suite", "all"]
end
npx_command += ["--spec-version", @spec_version] if @spec_version
npx_command += ["--requirements", @requirements] if @requirements
npx_command += ["--verbose"] if @verbose
npx_command += ["--expected-failures", expected_failures_yml]
npx_command
Expand Down
88 changes: 87 additions & 1 deletion test/conformance/server_runner_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ module Conformance
class ServerRunnerTest < ActiveSupport::TestCase
EXPECTED_FAILURES_YML = File.expand_path("../../conformance/expected_failures.yml", __dir__)

test "build_command runs the active suite by default" do
test "build_command runs the full suite by default" do
command = ServerRunner.new.send(:build_command)

assert_equal(
Expand All @@ -18,6 +18,92 @@ class ServerRunnerTest < ActiveSupport::TestCase
"server",
"--url",
"http://localhost:#{Server::DEFAULT_PORT}/mcp",
"--suite",
"all",
"--expected-failures",
EXPECTED_FAILURES_YML,
],
command,
)
end

test "build_command keeps the single scenario selection" do
command = ServerRunner.new(scenario: "ping").send(:build_command)

assert_equal(
[
"npx",
"--yes",
"@modelcontextprotocol/conformance@alpha",
"server",
"--url",
"http://localhost:#{Server::DEFAULT_PORT}/mcp",
"--scenario",
"ping",
"--expected-failures",
EXPECTED_FAILURES_YML,
],
command,
)
end

test "build_command pins the full suite to a spec version" do
command = ServerRunner.new(spec_version: "2025-11-25").send(:build_command)

assert_equal(
[
"npx",
"--yes",
"@modelcontextprotocol/conformance@alpha",
"server",
"--url",
"http://localhost:#{Server::DEFAULT_PORT}/mcp",
"--suite",
"all",
"--spec-version",
"2025-11-25",
"--expected-failures",
EXPECTED_FAILURES_YML,
],
command,
)
end

test "build_command pins a single scenario to a spec version" do
command = ServerRunner.new(scenario: "ping", spec_version: "2025-11-25").send(:build_command)

assert_equal(
[
"npx",
"--yes",
"@modelcontextprotocol/conformance@alpha",
"server",
"--url",
"http://localhost:#{Server::DEFAULT_PORT}/mcp",
"--scenario",
"ping",
"--spec-version",
"2025-11-25",
"--expected-failures",
EXPECTED_FAILURES_YML,
],
command,
)
end

test "build_command lets a requirement set replace a scenario selection" do
command = ServerRunner.new(requirements: "2026-07-28", scenario: "ping").send(:build_command)

assert_equal(
[
"npx",
"--yes",
"@modelcontextprotocol/conformance@alpha",
"server",
"--url",
"http://localhost:#{Server::DEFAULT_PORT}/mcp",
"--requirements",
"2026-07-28",
"--expected-failures",
EXPECTED_FAILURES_YML,
],
Expand Down
Loading