Repository navigation
Add aitools mcp login, logout and list --check - #7551
Conversation
f4c096c to
62d20d0
Compare
|
Commit: Regression test reportSelected 6 changed tests (limit=20, 0 not selected) 6 tests: regression=6 unreleased=0 coverage=0 failing=0
TestAccept/cmd/aitools/mcp/add-text: branch ✅ | main (cb94103) ❌ | latest (v1.27.0) ❌main (cb94103): latest (v1.27.0): TestAccept/cmd/aitools/mcp/help: branch ✅ | main (cb94103) ❌ | latest (v1.27.0) ❌main (cb94103): latest (v1.27.0): TestAccept/cmd/aitools/mcp/hidden: branch ✅ | main (cb94103) ❌ | latest (v1.27.0) ❌main (cb94103): latest (v1.27.0): TestAccept/cmd/aitools/mcp/list-and-remove: branch ✅ | main (cb94103) ❌ | latest (v1.27.0) ❌main (cb94103): latest (v1.27.0): 2 more failure details omitted, see the full report in the run. |
✅ Integration test reportCommit: e6ae53a
Top 28 slowest tests (at least 2 minutes):
|
eec9d1a to
5b1d279
Compare
62d20d0 to
d7c9aec
Compare
5b1d279 to
9187003
Compare
d7c9aec to
d2dd977
Compare
9187003 to
48fac6f
Compare
0aa7987 to
a08bc42
Compare
7064bbc to
10450c9
Compare
a08bc42 to
4a8c08c
Compare
10450c9 to
9ec98a0
Compare
8ef469f to
d779ae1
Compare
9ec98a0 to
1d09e53
Compare
d779ae1 to
43bb551
Compare
1d09e53 to
01b4249
Compare
43bb551 to
e441c85
Compare
01b4249 to
11e6e00
Compare
e441c85 to
60bf035
Compare
2a9b99f to
c7d75cb
Compare
109bc73 to
661f21e
Compare
c7d75cb to
0b9a961
Compare
1eef59f to
f2e0cd4
Compare
a4d1a39 to
14af751
Compare
f2e0cd4 to
e41d98d
Compare
9abcc11 to
79d56db
Compare
anton-107
left a comment
There was a problem hiding this comment.
Approving: I found nothing that blocks the merge. I reviewed only what this PR adds on top of aitools-mcp/06b-serve (c5aa08b). At 79d56db, the touched packages pass with -race (libs/aitools/mcp, libs/browser, libs/exec, cmd/aitools, libs/testserver), and so does TestAccept/cmd/aitools/mcp.
The change is reversible. aitools mcp is still hidden, and the state file format doesn't change. The one change everyone sees is that the default browser.Open now returns when the launcher exits, and that is easy to revert.
Non-blocking notes:
PathSafeNamerefuses more names than the API does. Unity Catalog names may contain%,#and?; only., space,/and control characters are forbidden.serviceResourceNamealready escapes the name withurl.PathEscape, sologin,logoutand nowserverefuse names the request would handle correctly. Thesystem.aiservices don't use these characters, so this only matters for a custom schema. Refusing just/and control characters would be enough.- A DEGRADED credential keeps waiting without being reset.
SignIndeletes a FAILED credential before opening the page, but a DEGRADED one is kept and the wait starts.list --checkreports DEGRADED as "sign-in needed". Is there a check against the real API that the provider page turns a DEGRADED credential back to ACTIVE without a delete first? If it can't, the user waits the full 5 minutes. add --loginreports the not-required state inconsistently. InsignInInstalled, a server that doesn't need sign-in getssign_in: {state: NOT_REQUIRED}even when it reached no agent. A server that needs sign-in and reached no agent gets nosign_inat all. CheckinginstalledAnywherebefore the not-required branch would make the two cases match.- A failed sign-in appears twice in the
add --loginJSON: inservers[].sign_in.errorand in the top-levelerror.TestTopLevelFailureKeepsErrorsOutsideAgentsshows this is intended. Mentioning it in the JSON contract would help consumers know they can read either.
|
Thanks!
|
49ed447 to
57552af
Compare
171374b to
ad9ad88
Compare
One commit for the restack onto main. The individual commits, up to 49ed447, are in the pull request's history. Co-authored-by: Isaac <no-reply@databricks.com>
57552af to
e6ae53a
Compare
❌ Integration test reportCommit: 4285bf9
1256 interesting tests: 1231 MISS, 9 RECOVERED, 6 PANIC, 6 KNOWN, 4 FAIL
Top 50 slowest tests (at least 2 minutes):
|
…databricks#7553) This pull request is part of a stack that adds the hidden `databricks aitools mcp` command group, on top of databricks#7551. A service that needs provider sign-in rejects `tools/list` until the user signs in, so the agent has no tool to start it. The proxy now offers one `authenticate` tool that runs the sign-in. ## Changes - A challenge makes `tools/list` return only `authenticate`. A challenge is a JSON-RPC `-32006`, a `-32042` answer with a sign-in URL, or an error result that names a sign-in URL. A `-32042` URL is accepted on the profile's host, or on any host whose `o=` or `w=` query names the bound workspace. - On a server that needs sign-in, a 401 is a challenge only when its body asks for sign-in. Any other 401 gets the `databricks auth login` hint. A server that needs no sign-in has no such handling, so its 401 passes through unchanged. - A challenged `tools/call` names `authenticate` and the login command. If `tools/list` still passes, `authenticate` is added to its first page until a call succeeds or the sign-in completes. - If the service still challenges after an "Already signed in" answer, a challenged `tools/call` is answered with the `mcp logout` command to run first, since neither `authenticate` nor `login` can fix it then. - `authenticate` runs `login`'s sign-in, sends progress naming the page, and sends `tools/list_changed` when signed in. A timeout names the URL, the last failed check and the login command. - The known-service note is printed by `mcp add`, by `mcp login` and by the authenticate tool's sign-in. `mcp list` does not read it. - The proxy never deletes an active credential. `SignIn` deletes a FAILED or DEGRADED one before it shows the page. - The page opens with no stdin, since stdin carries the protocol, and stays open when the request is cancelled. - A server with an install record takes its workspace ID from the record, so its sign-in makes no request for it. - A server with no install record reads the service at start. A rejected token there exits 2 with the login hint. ## Why Discovery fails before sign-in, so the agent can only start a sign-in through a tool that exists before it. A rejected Databricks token is not fixed by a provider sign-in, so it gets the login hint instead. ## Decisions to review - No automatic recovery when a credential reads as active but the service keeps rejecting it. Deleting and re-creating an active credential would also delete working ones, for example while the gateway catches up with a sign-in just made in a terminal. Cost: a stale provider grant needs one `mcp logout`. - A 401 that does not ask for sign-in is an auth failure, whatever its body. Cost: a gateway that answers a missing provider sign-in with a bare 401 gets the login hint instead of a sign-in. - A page is not reopened after a cancelled attempt that showed it, so a retry does not open a second tab. ## Tests - `libs/aitools/mcp/proxy/challenge_test.go` covers the challenge bodies, 401s, call-only challenges, the logout answer, page reopening, queued and cancelled calls, and progress. `TestSignInRequiredNamesTheLogoutWhileStillChallenged` covers the logout answer after "Already signed in". - `TestServeSignInUsesTheRecordedWorkspaceID` covers the workspace ID taken from the record. - Acceptance `serve-authenticate-tool` runs the stdio flow with `BROWSER=none`. - `go test -race ./cmd/aitools/ ./libs/aitools/mcp/...`, the `TestAccept/cmd/aitools/mcp` acceptance tests and `GOOS=windows go vet` on the same packages pass. - Not tested: a live `-32042` challenge, and a real browser. ## How to review Refactors that keep behavior: the sign-in path that `mcp login` and the proxy share (`libs/aitools/mcp/login.go`, `cmd/aitools/mcp_serve.go`), and the shared JSON-RPC response envelope in `libs/aitools/mcp/proxy/`. Read closely: `libs/aitools/mcp/proxy/challenge.go` (new), its `Result` method and the 401 rule, and `serveChallenge` in `cmd/aitools/mcp_serve.go`. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
Part of a stack that adds the hidden
databricks aitools mcpcommand group, on #7550 (aitools mcp serve). It adds provider sign-in for external MCP services such as GitHub, which each user signs in to before the service's tools work.Changes
mcp login <service>prints the sign-in URL, opens it throughlibs/browser, and waits up to 5 minutes. Server and network errors while polling are retried until the timeout, which names the last one. An already signed-in caller gets "Already signed in" and no URL.mcp logout <service>deletes the caller's credential, even when its service is no longer visible.loginandlogoutrefuse a changed profile before any request./or a control character. Other URL syntax, such as?,#and%, is escaped in the request and is allowed.mcp add --loginsigns in to each sign-in-required server that reached an agent. Its JSON gains a per-serversign_inobject, given only to servers that reached an agent; a server that does not need sign-in getsNOT_REQUIRED. A failed sign-in is reported in bothservers[].sign_in.errorand the top-levelerror, which is intended, so consumers can read either. The command exits 1 and keeps the installs.mcp list --checkchecks each profile's token, then each server. A drifted server or one that cannot be read is skipped, and says why. Each check request ends after 5 seconds. A needed sign-in names the login command.DATABRICKS_CONFIG_FILE.libs/browsergainsOpenOutputToStderr, and its default opener no longer waits for a browser its launcher leaves running.Why
A sign-in is judged on the attempt that runs, so a FAILED or DEGRADED credential left by an earlier attempt is deleted before the page opens. A failed poll does not mean the user has left the provider's page, so it does not end the wait. One unreachable workspace held
list --checkfor the SDK's five-minute retry budget.Decisions to review
SignInreturns(alreadySignedIn, urlShown bool, err error), not a state. The command derives the JSON state and URL from them.?,#or%are accepted, rather than refused locally. Cost: such names reach the API.list --checksetsDATABRICKS_CONFIG_FILEin the process environment while it checks a profile, because the token store reads the process environment.logoutsends the DELETE before it looks up the service, so a credential for a deleted service can be removed. Cost: a mistyped name costs one request.Tests
TestOpenDoesNotWaitForLauncherChildinlibs/browserfailed before the fix.cmd/aitools:TestAddChecksBoundsAProfileThatNeverAnswershung past five minutes before the fix.libs/aitools/mcp.TestSignInDeletesDegradedCredentialcovers the DEGRADED deletion.TestPathSafeNameandTestServiceNameAllowsEscapedURLSyntaxcover the name check.TestSignInInstalledRecordsTheSignInOfEachServercoverssign_infor a server that reached no agent.list-checkandlogin-flips-activepass.go test -race ./cmd/aitools/ ./libs/aitools/mcp/...andGOOS=windows go veton the same packages pass.How to review
Refactors that keep behavior: the sign-in helpers shared by
login,logoutandadd --login(libs/aitools/mcp/login.go,cmd/aitools/mcp_login.go). Read closely: the poll loop inlibs/aitools/mcp/login.go, the bounded checks incmd/aitools/mcp_list.go, and the default opener inlibs/browser/browser.go, which changes howbrowser.Openwaits.This pull request and its description were written by Isaac.