Repository navigation
Log AitoolsMcpEvent from aitools mcp and the serve proxy - #7554
simonfaltum wants to merge 1 commit into
Conversation
|
Commit: Regression test reportSelected 4 changed tests (limit=20, 0 not selected) 4 tests: regression=4 unreleased=0 coverage=0 failing=0
TestAccept/cmd/aitools/telemetry/mcp-add: branch ✅ | main (ae033ef) ❌ | latest (v1.27.0) ❌main (ae033ef): latest (v1.27.0): TestAccept/cmd/aitools/telemetry/mcp-remove: branch ✅ | main (ae033ef) ❌ | latest (v1.27.0) ❌main (ae033ef): latest (v1.27.0): TestAccept/cmd/aitools/telemetry/mcp-serve: branch ✅ | main (ae033ef) ❌ | latest (v1.27.0) ❌main (ae033ef): latest (v1.27.0): TestAccept/cmd/aitools/telemetry/mcp-sign-in: branch ✅ | main (ae033ef) ❌ | latest (v1.27.0) ❌main (ae033ef): latest (v1.27.0): |
✅ Integration test reportCommit: 482c318
Top 23 slowest tests (at least 2 minutes):
|
2dd54b7 to
a1fd867
Compare
62533cc to
a1d0d64
Compare
a1fd867 to
b90afff
Compare
a1d0d64 to
55e2a1d
Compare
b90afff to
0aa2ee4
Compare
55e2a1d to
a603d10
Compare
0aa2ee4 to
2dbdfe3
Compare
a603d10 to
1391f64
Compare
1fff1da to
e9bad5b
Compare
1391f64 to
c3e93cc
Compare
b30caa0 to
43a64fd
Compare
f6918d2 to
48880f0
Compare
43a64fd to
b12f2bf
Compare
8fe9c49 to
d705658
Compare
b12f2bf to
1ee83b9
Compare
d705658 to
b3a08ad
Compare
1ee83b9 to
29624cd
Compare
b3a08ad to
d206004
Compare
a7c5766 to
52a1885
Compare
d206004 to
34ee3fb
Compare
52a1885 to
e652cdf
Compare
e1a2287 to
a06230b
Compare
793107d to
a3a8757
Compare
8e7258c to
211c894
Compare
a3a8757 to
27ecba1
Compare
211c894 to
c77fc98
Compare
27ecba1 to
8047e0c
Compare
c77fc98 to
e309481
Compare
8047e0c to
828e6f7
Compare
d4abf00 to
b8bc227
Compare
a1bf5bf to
f1ee623
Compare
7bde116 to
5f95215
Compare
201f9a7 to
ffc760f
Compare
5f95215 to
9910702
Compare
anton-107
left a comment
There was a problem hiding this comment.
One blocking finding, inline on cmd/aitools/mcp_serve.go: serve's PROXY_STARTED is lost for any session that sits idle longer than the access token lasts.
I ran the unit tests for the changed packages (including -race on libs/telemetry, proxy and cmd/aitools) and the cmd/aitools/telemetry and cmd/aitools/mcp acceptance tests locally. All pass.
Non-blocking notes:
- Two paths depend on there being no
cmdctx.ConfigUsed.dropUploadIfReboundcallsSetUploadConfig(ctx, nil), and a profile that fails to resolve insetUploadConfignever sets a config. Either wayUploadConfigfalls back tocmdctx.ConfigUsed. That's safe today only because nomcpcommand sets it. If the funnel PR wireswithTelemetryAuthinto these commands, "drop the upload" turns into "upload with ConfigUsed". An explicit no-upload state in the logger would keep the intent. removeuploads nothing when the selected servers belong to more than one profile, for exampleremove --allwith servers on two profiles (sharedProfilereturns the zero key). "removegroups its servers by profile and config file" in the description reads as if it uploads per group. No test covers a remove that spans several profiles.signInHooks.Resultnow names itsctxparameter but doesn't use it (libs/aitools/mcp/proxy/challenge.go:162).KnownLeavesdoesn't need//deadcode:allow:deadcode -testalready counts callers in tests.
This review was written by Isaac.
|
The notes are in too:
The start upload's execution context is built by |
anton-107
left a comment
There was a problem hiding this comment.
Round 2 (9910702..a688fa4): the blocking finding is fixed. serve now uploads PROXY_STARTED right after its startup checks. That upload reads the token Bind has just refreshed into the cache, and Upload removes the events the service accepted, so the exit upload doesn't send them again.
I checked the shared changes for effects on other commands:
- Every other command uploads once, so removing accepted events changes nothing for them. The one exception is the ssh agent shim's exec-failure path. That path no longer resends the success event the shim already uploaded, which is what its
launchedcomment intended. - Partial accepts behave as before: the batch is retried whole and kept on failure. Only serve's exit upload could resend it, and at start the batch is just
PROXY_STARTED. SkipUploadis only called fromaitools mcppaths.
Handshake delay: the start upload adds one POST before the proxy reads stdin, and at most 3 s if /telemetry-ext hangs. The agent's initialize waits in the pipe and isn't dropped, so this fits within agent startup timeouts.
Tests: the touched packages and TestAccept/cmd/aitools/telemetry pass.
b32f61f to
9cb89d5
Compare
a688fa4 to
4f9607b
Compare
9cb89d5 to
be36588
Compare
One commit for the restack onto main. The individual commits, up to a688fa4, are in the pull request's history. Co-authored-by: Isaac <no-reply@databricks.com>
4f9607b to
482c318
Compare
This pull request is part of the stack that adds the hidden
databricks aitools mcpcommand group, on top of #7552. It adds typedAitoolsMcpEventtelemetry for the management commands and forserve's start and sign-in prompts. It depends on a telemetry schema change that lives outside this repository and is not merged yet, so this pull request stays a draft.Changes
AitoolsMcpEvent, its server result and their enums inlibs/telemetry/protos. The MCP error categories joinAitoolsErrorCategory; their JSON strings are unchanged.addlogsADDandSIGN_IN;loginandlogoutlogSIGN_INandSIGN_OUT;removelogsREMOVEwhen it touched a server;servelogsPROXY_STARTEDandSIGN_IN_PROMPTED. Whenadd --logincannot build a server's sign-in URL, itsSIGN_INevent is FAILED with sign-in statePENDING.removeuploads only when every server it removes is bound to one profile and config file, and that profile still matches them; otherwise it logs its event and uploads nothing. A server pinned to another config file uploads with that file, and a same-named profile in another file no longer stops the upload. An unpinned server uploads with the default config file, the one its binding is checked against.servebinds the upload to its profile once startup succeeds and uploadsPROXY_STARTEDright then, while the token is known to work, so a killed or long-idleservekeeps it.SIGN_IN_PROMPTEDuploads at exit, with the profile bound at start. At exit, serve checks the binding again, and skips the upload if the profile no longer names the host it bound.libs/telemetrygets an upload slot (SetUploadConfigandUploadConfig) andSkipUpload, which no config overrides.Uploadremoves the events the service accepted, so a later upload doesn't resend them.installis unchanged.cmd/rootexportsTelemetryExecutionContext, which its exit upload andserve's start upload share.system.aiisCUSTOM, and its name is not sent.Why
The MCP commands and the proxy have no telemetry, so their use cannot be measured. Background auth resolution, as
installuses, could send a refreshed token or another profile's credential, or upload under a profile the command did not bind.Decisions to review
PROFILE_NOT_FOUNDandPROFILE_HOST_CHANGEDare never sent, since the token must not reach a host the record does not name, andAUTH_FAILEDwould repeat the rejected request. These failures are undercounted.removemay wait for the upload after printing its output.servewaits for itsPROXY_STARTEDupload before it serves, bounded by the 3 s upload timeout. That upload's execution context has exit code 0 and the time to start, and shares the run's exec ID with the exit upload.SIGN_IN_PROMPTEDuploads at exit with credentials that never refresh, so a session whose access token has expired by then loses it. These events are undercounted for long idle sessions.Tests
aitools_mcp_test.gopins the JSON keys.mcp_telemetry_test.gocovers each service, outcome, profile count and rejected credentials.TestMcpRemoveUploadIgnoresEnvironmentTokenchecks that an upload never uses another workspace's token.TestSignInInstalledURLFailureLogsFailedSignIncovers the URL failure,TestMcpRemoveUploadUsesConfigFileOfServerandTestMcpRemoveUploadIgnoresSameNamedProfileInOtherFilethe remove grouping,TestMcpServeUploadUsesProfileBoundAtStartthe bound profile,TestMcpServeDropsUploadAfterRepointthe drop after a re-point, andTestMcpRemoveUploadSkipsServersOfSeveralProfilesa remove across two profiles.TestMcpServeUploadsProxyStartedAtStartchecks thatPROXY_STARTEDreaches the service while serve runs and isn't sent again at exit.TestUploadSendsEachAcceptedEventOnce,TestUploadKeepsEventsNotAcceptedandTestUploadSkippedDespiteConfigUsedcover the logger.TestLogMcpEventLogsRejectedCredentialchecks that an event naming a rejected credential is logged and uploads with the run.TestSetUploadConfigLogsNoWarningForFailedHostMetadatacovers the metadata warning.acceptance/cmd/aitools/telemetry/mcp-add,mcp-remove,mcp-sign-inandmcp-serveprint the events each command uploaded.How to review
Refactors that keep behavior:
debug_only.gomoved tolibs/log/handler/, and the enum mirrors inlibs/telemetry/protos/. Read closely:cmd/aitools/mcp_telemetry.gofor what each command sends,libs/telemetry/logger.gofor the upload slot and its locking,cmd/aitools/mcp_serve.gofordropUploadIfRebound, andlibs/telemetry/protos/aitools_mcp.gofor the enums.This pull request and its description were written by Isaac.