Repository navigation
aitools: add install funnel telemetry - #7469
Conversation
0d29efd to
a15150a
Compare
5637f13 to
21d6b41
Compare
|
Commit: Regression test reportSelected 2 changed tests (limit=20, 0 not selected) 2 tests: regression=2 unreleased=0 coverage=0 failing=0
TestAccept/cmd/aitools/telemetry/install: branch ✅ | main (e11d32c) ❌ | latest (v1.26.0) ❌main (e11d32c): latest (v1.26.0): TestAccept/cmd/aitools/telemetry/install-failure: branch ✅ | main (e11d32c) ❌ | latest (v1.26.0) ❌main (e11d32c): latest (v1.26.0): |
✅ Integration test reportCommit: 5a3cc25
Top 19 slowest tests (at least 2 minutes):
|
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at 21d6b41. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 1 open design question.
This PR adds an aitools_command_event funnel to every aitools install run (STARTED, the steps reached, then FINISHED with an outcome and error category) and tags failures with a category where they happen. I traced every way the command can exit and found exactly one FINISHED event each time; the fields and enum values match the Universe proto, and CI is green. Nothing blocking. The main suggestion is to report "no skills delivered" the same way in --path mode and the other modes; the rest are nits on outcome semantics, small simplifications and test conventions. There is also one question about the extra upload latency for runs that stop early.
Questions
- Runs that stop early (a flag error, a cancelled prompt, no agent detected, a
--pathdump) used to upload no telemetry. With the funnel, each of them now makes an upload, so against a slow but reachable workspace a run that fails instantly can wait up to the upload timeout (3 s by default) on top of the auth wait. The Merge Danger section mentions this. Is that latency acceptable for these runs? Or would you rather use a shorter upload timeout for runs that end before PLAN_CONFIRMED, or upload their events only when auth has already resolved without waiting?
Comments outside the changed lines
- [nit]
.github/PULL_REQUEST_TEMPLATE.md:.agents/skills/pr-checklist/SKILL.mdasks PR bodies to "Follow.github/PULL_REQUEST_TEMPLATE.mdexactly", using## Changes,## Whyand## Testsand no invented sections such as## Summary. It also asks for a concise summary. This description uses## Summary/## Evidence/## Merge Danger. Map the content onto Changes / Why / Tests and trim it; the flow diagram and the "not covered" list could stay, shortened, under those headings.
| for _, o := range outcomes { | ||
| if o.status == outcomeInstalled { | ||
| return "", false | ||
| } | ||
| } |
There was a problem hiding this comment.
[nice to have] When every skill in the manifest is filtered out (e.g. only experimental skills, no --experimental), the outcome depends on the mode. A --path dump finishes FAILURE / UNCATEGORIZED (install.go:159). --skills-only --agents=cursor (or the default mode with a skills agent) finishes SKILLS_INSTALLED plus SUCCESS. That happens because InstallSkillsForAgents returns nil, executePlan marks the agents outcomeInstalled, and this check lets it through. The description lists this as a known gap. But the funnel now reports the same "nothing delivered" condition as both a failure and a success, and the --path side files a known, non-error cause under UNCATEGORIZED, the bucket meant to show what the classifier misses. Pick one outcome for both modes. Either carry the resolved skill count from the installer into the outcomes and report the zero case the way --path does, or, until per-agent delivery is reported, make --path match the other modes. In both cases use a specific outcome (e.g. NOTHING_TO_DO) instead of FAILURE / UNCATEGORIZED.
There was a problem hiding this comment.
Done in 61fb77f, partly. A --path dump that writes no skill now finishes NOTHING_TO_DO instead of FAILURE / UNCATEGORIZED (TestInstallFunnelPathDumpWithNoSkillToWrite). The other modes still can't see that case: InstallSkillsForAgents returns only an error. Carrying the count would change its signature and the installSkillsForAgentsFn seam that the PRs stacked above mock, so it's not in this PR. The description lists it under "Not covered".
There was a problem hiding this comment.
Still only the --path part, as of 54514a6. Reporting zero skills for agent installs needs InstallSkillsForAgents to return how many skills it resolved. That changes the signature of its installSkillsForAgentsFn seam, and the PRs stacked above this one add more mocks of that seam, so the change isn't contained to this PR. For now it stays listed under "Not covered" in the description: an agent install where every skill is filtered out still finishes SUCCESS.
There was a problem hiding this comment.
Our review found this too. Note: #7481 fixes it later in the stack. There, InstallSkillsForAgents returns ErrNoSkillsResolved when no skill is left (libs/aitools/installer/installer.go:398), and finish() maps it to NOTHING_TO_DO. So the "Not covered" item in this description is resolved once #7481 lands.
There was a problem hiding this comment.
Thanks, confirmed: in #7481 InstallSkillsForAgents returns installer.ErrNoSkillsResolved and finish() maps it to NOTHING_TO_DO. I updated the "Not covered" item in this description to point there.
| if _, err := resolveScope(projectFlag, globalFlag); err != nil { | ||
| return withCategory(err, protos.AitoolsErrorCategoryInvalidFlags) | ||
| } | ||
| funnel.step(ctx, protos.AitoolsCommandStepFlagsValidated) |
There was a problem hiding this comment.
[nit] funnel.step(FLAGS_VALIDATED) runs before resolveAgentNames (line 206). So --agents , or --agents bogus logs FLAGS_VALIDATED and SCOPE_RESOLVED, then FINISHED FAILURE INVALID_FLAGS / UNKNOWN_AGENT. The step event says the flags were valid, and the final category says they were not. resolveAgentNames does not depend on the scope, so you could validate --agents before this step. Calling it there would also fail an invalid name before the interactive scope prompt instead of after it. If you'd rather keep the prompt order unchanged in this PR, note on AitoolsCommandStepFlagsValidated that --agents is checked after SCOPE_RESOLVED.
There was a problem hiding this comment.
Done in 61fb77f. resolveAgentNames now runs before FLAGS_VALIDATED, so --agents , / --agents bogus finish STARTED -> FAILURE INVALID_FLAGS / UNKNOWN_AGENT with no step events, and fail before the scope prompt. The description notes the order change.
| return "", false | ||
| } | ||
| } | ||
| return firstFailureCategory(outcomes), true |
There was a problem hiding this comment.
[nit] Suppose every detected agent is skipped for the chosen scope, e.g. --skills-only --scope=project with only Cursor detected. On the interactive path, agentChoices drops the skipped agents, selectAgents returns none, and the run finishes NOTHING_TO_DO (install.go:335). The non-interactive path (e.g. CI) keeps the skipped agents in the plan, so this function reports FAILURE / UNSUPPORTED_SCOPE. Both runs exit 0, but the funnel splits one situation into two outcomes depending on the TTY. If that is unintended, report an all-skipped plan with no named agents as NOTHING_TO_DO, or document the split.
There was a problem hiding this comment.
Documented in 61fb77f on nothingInstalledCategory and in the description. Keeping FAILURE / UNSUPPORTED_SCOPE for the non-interactive run is deliberate: it acted on the detected agents and set up none of them. The interactive picker never offers a skipped agent, so that run has nothing to choose and finishes NOTHING_TO_DO before the plan. #7481, further up this stack, makes the non-interactive run exit 1, so its outcome will match its exit code.
| case deliverySkills: | ||
| skills = skills || o.status == outcomeInstalled | ||
| case deliveryPlugin: | ||
| plugins = plugins || o.status == outcomeInstalled |
There was a problem hiding this comment.
[nit] The CLI and the Universe proto now describe three values differently. PLUGINS_INSTALLED: the proto says "The plugin install loop ran (individual agents may still have failed)", but deliverySteps emits it only when at least one plugin install succeeded. FAILURE: the proto says "The command returned an error", but the CLI also emits it for exit-0 runs (a --path dump that writes nothing, a detected agent whose plugin install failed). CANCELLED: the proto says "The user declined the confirm prompt or aborted a prompt", but the CLI also uses it for an empty picker selection, which exits 1. Anyone building dashboards from the proto will misread these runs. Update the Universe enum.proto comments to the CLI's semantics (which the Go comments in aitools_events.go already state), or change the CLI to match.
There was a problem hiding this comment.
The Go comments in aitools_events.go describe the CLI's behavior; I'll align the schema descriptions separately.
| scope, err := resolveScopeWithPrompt(ctx, projectFlag, globalFlag) | ||
| if err != nil { | ||
| return err | ||
| return promptError(err) |
There was a problem hiding this comment.
[nit] promptError wraps everything resolveScopeWithPrompt returns. But defaultPromptScopeSelection calls env.UserHomeDir and os.Getwd before any prompt opens. An interactive run in a deleted working directory therefore finishes FAILURE / PROMPT_FAILED instead of FILESYSTEM. Tag only the error from the huh ... Run() call (inside defaultPromptScopeSelection, and likewise for the picker and confirm prompts), so the earlier errors keep their stdlib classification.
There was a problem hiding this comment.
Done in 61fb77f. promptError now wraps only the huh ... Run() error inside defaultPromptScopeSelection, defaultPromptAgentSelection and defaultPromptProceed. The call sites return errors as they are, so an env.UserHomeDir / os.Getwd failure keeps its stdlib category (FILESYSTEM for the deleted cwd).
| // setScope records the resolved scope (installer.ScopeGlobal/ScopeProject). | ||
| func (f *commandFunnel) setScope(scope string) { f.scope = scopeType(scope) } | ||
|
|
||
| // setAgents records the targeted agents, reported on the FINISHED event. |
There was a problem hiding this comment.
[nit] setAgents wraps each agent in an agentPlanItem only so it can call agentsField. buildPlan has no side effects and runs on the next line of install.go. Build the plan first and record agentsField(plan) (e.g. funnel.setAgents(plan) taking []agentPlanItem). That drops the extra loop, and the funnel and aitools_install_event then derive agents from the same value.
There was a problem hiding this comment.
Done in 61fb77f. The plan is built first, then funnel.setAgents(plan) records agentsField(plan), the same value aitools_install_event uses.
| } | ||
| // --project with --global is a flag error; check it here so it is not | ||
| // reported as a scope prompt failure below. | ||
| if _, err := resolveScope(projectFlag, globalFlag); err != nil { |
There was a problem hiding this comment.
[nit] This resolveScope(projectFlag, globalFlag) pre-check throws its result away, and resolveScopeWithPrompt runs the same check again two lines later. That leaves its conflict error unreachable from install. Keep this tagged check and use its scope when a flag is set, calling the prompt only when neither flag is set. Then the validation and the tagging happen once.
There was a problem hiding this comment.
Done in 61fb77f. Install calls resolveScope once, only when a scope flag is set, tags its error INVALID_FLAGS and uses its scope. resolveScopeWithPrompt(ctx) no longer takes the flags: it prompts when interactive and defaults to global otherwise. Install was its only caller.
| errcode $CLI aitools install --skills-only --path out | ||
| trace print_telemetry | ||
|
|
||
| title "unknown agent fails after the scope step\n" | ||
| errcode $CLI aitools install --skills-only --scope=global --agents=no-such-agent | ||
| trace print_telemetry | ||
|
|
||
| title "a skill missing from the manifest fails after the plan is confirmed\n" | ||
| errcode $CLI aitools install --skills-only --scope=global --agents=cursor --skills=nonexistent | ||
| trace print_telemetry | ||
|
|
||
| title "a manifest the server does not have is MANIFEST_NOT_FOUND\n" | ||
| errcode env DATABRICKS_SKILLS_REF=missing-ref $CLI aitools install --skills-only --scope=global --agents=cursor | ||
| trace print_telemetry | ||
|
|
||
| title "any other HTTP error status from the manifest fetch is not classified yet (the error is untyped)\n" | ||
| errcode env DATABRICKS_SKILLS_REF=broken-ref $CLI aitools install --skills-only --scope=global --agents=cursor | ||
| trace print_telemetry | ||
|
|
||
| title "a named agent skipped for the scope takes its category from the agent's outcome\n" | ||
| errcode $CLI aitools install --skills-only --scope=project --agents=cursor |
There was a problem hiding this comment.
[nit] .agents/rules/testing.md:150 says: "Assert an expected failure with musterr, not ! cmd or a bare errcode". errcode is for tolerated failures. All six failure cases in this script must fail. Also, without trace (testing.md:148), output.txt shows each Error: ... with no >>> line naming the command. Use musterr trace $CLI ... for each case, as other scripts in the repo do.
There was a problem hiding this comment.
Done in 61fb77f. Each case is now musterr trace $CLI .... The cases that end before PLAN_CONFIRMED (invalid flags, unknown agent, empty --path dump) are no longer in this script: those runs now upload only if auth has already resolved, so whether their events appear depends on timing. TestInstallFunnelEarlyExits and TestInstallFunnelPathDumpWithNoSkillToWrite cover them.
| func isolateAgents(t *testing.T, kv ...string) { | ||
| t.Helper() | ||
| testutil.SetAgentEnv(t, kv...) | ||
| for _, v := range []string{"CLAUDE_CONFIG_DIR", "PI_CODING_AGENT_DIR", "GEMINI_CLI_HOME", "GOOSE_PATH_ROOT", "APPDATA"} { |
There was a problem hiding this comment.
[nit] testutil.SetAgentEnv already clears the whole environment (NullEnvironment calls os.Clearenv). So this loop that blanks CLAUDE_CONFIG_DIR, GEMINI_CLI_HOME, APPDATA etc. does nothing, and the helper's comment suggests the loop is what isolates the test. Drop the loop and keep SetAgentEnv. Similarly, runInstall (line 73) takes t but never uses it. Remove the parameter, or call t.Helper().
There was a problem hiding this comment.
Done in 61fb77f. Dropped the loop and fixed the comment. isolateAgents now only calls testutil.SetAgentEnv; I kept it as a thin wrapper because tests in the PRs stacked above call it. runInstall now calls t.Helper() (kept the parameter for the same reason).
| assert.Equal(t, protos.AitoolsInstallModeSkillsOnly, e.InstallMode) | ||
| assert.False(t, e.Interactive) | ||
| } | ||
| finished := events[len(events)-1] |
There was a problem hiding this comment.
[nit] events[len(events)-1] here, and events[0] at lines 274 and 331, follow only an assert on the event sequence. If the funnel emits nothing, the test panics with index out of range instead of failing with the diff. .agents/rules/testing.md says to "Use require for preconditions that would make the rest of the test meaningless on failure", so switch those shape checks to require.Equal.
There was a problem hiding this comment.
Done in 61fb77f. The shape checks that come before events[len(events)-1] / events[0] now use require.Equal.
a15150a to
86362f1
Compare
3147276 to
920dbc0
Compare
21d6b41 to
61fb77f
Compare
920dbc0 to
95aeec8
Compare
|
Thanks for the review. Every inline comment except one has a reply in its thread. Question 1 (upload latency for runs that stop early): changed in 61fb77f. A run that ends before PLAN_CONFIRMED (a flag error, a cancelled prompt, no agent detected, a
PR template nit: I'd rather keep the current body format. Every PR in this stack uses the same layout, and restructuring all of them would be churn. Happy to change it if you feel strongly. |
61fb77f to
8fcfd3e
Compare
95aeec8 to
e503a62
Compare
d5ef5c6 to
3b60b60
Compare
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at 3b60b60. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 2 open design questions.
This PR adds an aitools_command_event funnel to databricks aitools install: STARTED, a STEP_COMPLETED for each step reached, and one FINISHED with the outcome and error category. It also tags failures where they happen and stops early exits from waiting for telemetry auth. The fixes from the last round are in and work as described: every exit path emits exactly one FINISHED, CI is green, and the funnel and auth tests pass locally (removing the post-plan wait makes the wait test fail 298 of 300 runs). Nothing is blocking. There are two questions about trade-offs (timing skew in early-exit data, and the bundled --agents reorder, which has no changelog entry) and a few small nits.
Questions
- Runs that end before the plan is confirmed now upload their events only if the background auth has already resolved. A flag error, an unknown
--agentsname or a non-interactive run with no agent detected ends within milliseconds, before the auth resolution (config plus a host-metadata request) can finish, so those outcomes will rarely reach the table. Slower early exits, such as a cancelled prompt, usually will. The funnel will therefore undercount the fastest drop-offs, and STARTED, relative to the slower ones. Is that skew acceptable for the dashboards you plan to build? Options: keep it and note the bias where the dashboards are defined; give early exits a short bounded wait (for example a few hundred milliseconds); or let them use the normal bounded wait. - Validating
--agents(and the--project/--globalconflict) before the scope prompt changes what an interactive user sees: an unknown agent name now errors before the prompt instead of after it. That is a small user-visible change inside a telemetry PR, and the diff has no.nextchanges/clifragment, which the PR checklist asks for when command or flag behavior changes. Would you rather keep it here and add a fragment, or move the reorder into its own small PR?
Agreeing with existing threads
- [nice to have] Zero-skill agent installs finish SUCCESS while --path finishes NOTHING_TO_DO: #7469 (comment)
| if explicit { | ||
| targetAgents, err = resolveAgentNames(ctx, agentsFlag) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| } else { | ||
| } |
There was a problem hiding this comment.
[nit] With --agents, the targets are known here, but funnel.setAgents runs only after buildPlan (line 235). An interactive databricks aitools install --agents cursor with no --scope that is aborted (or fails) at the scope prompt finishes CANCELLED / USER_ABORTED (or FAILURE / PROMPT_FAILED) with an empty agents, although AitoolsCommandEvent.Agents says it is set "once the targets are known". A scratch test with the scope prompt returning huh.ErrUserAborted gives [STARTED FLAGS_VALIDATED FINISHED:CANCELLED:USER_ABORTED] with agents=[]. Record the explicit agents right after resolveAgentNames succeeds (e.g. a setAgents variant that takes []*agents.Agent), or reword the field comment to say it is set once the plan is built.
There was a problem hiding this comment.
Done in 54514a6. Agents named via --agents are now recorded right after resolveAgentNames succeeds (funnel.setTargets), so a run aborted at the scope prompt reports them. TestInstallFunnelAbortedScopePromptKeepsNamedAgents covers it (STARTED, FLAGS_VALIDATED, FINISHED CANCELLED USER_ABORTED with agents: [CURSOR]). The plan still sets them after buildPlan for the picker path; for --agents that is the same set.
| Env.DATABRICKS_SKILLS_BASE_URL = "$DATABRICKS_HOST" | ||
| Env.DATABRICKS_CLI_GITHUB_API_URL = "$DATABRICKS_HOST" | ||
| Env.DATABRICKS_SKILLS_REF = "test-ref" |
There was a problem hiding this comment.
[nit] This parent config now sets DATABRICKS_SKILLS_BASE_URL / DATABRICKS_SKILLS_REF and serves GET /test-ref/manifest.json, but install/test.toml still sets the same two variables and serves its own manifest on the same route. The leaf stub wins (acceptance/internal/prepare_server.go:248-251), so install runs against test-stable while its siblings use databricks-test, and a change to the parent manifest never reaches install. .agents/rules/testing.md asks to put common test.toml options in the parent. Drop the duplicated env lines and stubs from install/test.toml (its output only says "Installed 1 skill."), or keep the new stubs in the new leaf directories so hint/ and apps-init-tip/ don't inherit them.
There was a problem hiding this comment.
Done in 54514a6. Removed install/test.toml: install now uses the parent's env and manifest stubs like its siblings. Its output is unchanged ("Installed 1 skill.").
| mkdir -p "${USERPROFILE:-$HOME}/.cursor" | ||
|
|
||
| title "install: STARTED, one event per step reached, FINISHED with SUCCESS" | ||
| trace $CLI aitools install --skills-only --scope=global --agents=cursor |
There was a problem hiding this comment.
[nit] This is the same run as the first case of install/ (--skills-only --scope=global --agents=<one agent>), whose updated output.txt already pins STARTED through FINISHED SUCCESS; only the agent differs. .agents/rules/testing.md says to extend the existing acceptance directory when coverage overlaps. Drop install-success/ and rely on install/, or fold anything extra into install/script.
There was a problem hiding this comment.
Done in 54514a6. Dropped install-success/. The first case of install/ pins the same STARTED through FINISHED SUCCESS sequence.
| if err == nil { | ||
| return nil | ||
| } |
There was a problem hiding this comment.
[nit] Every caller passes a non-nil error: string literals at the flag checks, and values inside if err != nil at install.go:172/197/422/443 and scope.go:97. So the nil branches here and in promptError (line 43) run only in TestWithCategoryKeepsMessage. AGENTS.md asks not to add speculative fallbacks or defensive nil checks for values callers always provide. Drop both nil branches and the two NoError(...(nil)) asserts, or keep them and say on the doc comment why nil must be accepted.
There was a problem hiding this comment.
Done in 54514a6. Dropped the nil branches from withCategory and promptError and the two NoError(...(nil)) asserts. Both doc comments now say they take a non-nil error.
| }) | ||
| } | ||
|
|
||
| func (f *commandFunnel) log(ctx context.Context, e protos.AitoolsCommandEvent) { |
There was a problem hiding this comment.
[nit] .agents/rules/comments.md asks for a doc comment on Go functions, starting with the name and ending with a period. This is the only new function without one. For example: // log stamps e with the run's command, interactivity, install mode and scope, and buffers it for upload.
There was a problem hiding this comment.
Done in 54514a6, with the doc comment you suggested.
3b60b60 to
54514a6
Compare
|
Thanks. Every inline comment has a reply in its thread. Changes are in 54514a6. Question 1 (early exits under-counted): I'm keeping the no-wait rule. Early exits stay instant by design, and a run that fails in milliseconds shouldn't wait seconds for telemetry. The resulting under-count of the fastest early exits (and their STARTED events) relative to slower ones is now stated in the code comment next to the rule ( Question 2 ( Zero-skill agent installs: answered in that thread. Only the |
e503a62 to
76d9ee3
Compare
54514a6 to
f6a8dd1
Compare
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at 8ac74ab. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 3 open design questions.
This PR adds an aitools_command_event funnel to databricks aitools install (STARTED, each step reached, one FINISHED with an outcome and error category). It also tags failures with a category where they happen, lets early exits skip the telemetry auth wait, and validates --agents before the scope prompt. Every exit path logs exactly one FINISHED, JSON mode classifies the real error, and CI is green. Nothing blocking was found. The open items are three design questions (early-exit sampling bias, the bundled --agents reorder, and two event families that define the outcome differently), a few missing funnel-sequence tests, and small test and convention nits.
Questions
- You've kept the no-wait rule for runs that end before the plan is confirmed and documented the under-count. That drops most fast flag errors, unknown agents and quick successful
--pathdumps (and their STARTED events), but keeps slow prompt cancels. Will the dashboards built on this funnel note or correct for that? Or would you consider a short bounded wait, for example a few hundred milliseconds, or waiting for a--pathdump, since it does network I/O anyway? - AGENTS.md asks to keep each PR focused on one change, and validating
--agentsbefore the scope prompt is a user-visible change with its own changelog entry. Would you land it as a small PR below this one, with a test that the scope prompt isn't shown for an unknown name, or do you prefer to keep it here? - The same run now gets two outcome definitions. A named agent skipped for the scope reports
aitools_install_event.error_category = TYPE_UNSPECIFIEDwithagent_results, while the funnel says FINISHEDFAILURE/UNSUPPORTED_SCOPE, and an exit-0 run that installed nothing is FAILURE only in the funnel. The outcome logic also lives in three places (the funnel switch,logInstallEvent,buildInstallOutput). Did you consider computing one run result afterexecutePlanfor all three? Alternatively,aitools_install_eventcould be logged on every run, as its doc says, withinstall_modeand a last-step field instead of a separate event family, since all funnel rows upload together at exit without timestamps.
Agreeing with existing threads
- [nice to have] An agent install where every skill is filtered out finishes SUCCESS, while --path finishes NOTHING_TO_DO: #7469 (comment)
- [nit] PR description does not follow the repo PR template: #7469 (review)
| funnel.finishWith(ctx, protos.AitoolsCommandOutcomeNothingToDo, protos.AitoolsErrorCategoryUnspecified) | ||
| return nil | ||
| } | ||
| funnel.step(ctx, protos.AitoolsCommandStepSkillsDumped) |
There was a problem hiding this comment.
[nice to have] No test asserts the sequence for a successful --path dump (FLAGS_VALIDATED, SKILLS_DUMPED, FINISHED SUCCESS). The same goes for a successful plugin install (PLUGINS_INSTALLED is checked only through deliverySteps) and for an accepted interactive confirm (TestInstallFunnelInteractiveCancel has no 'confirmed' row). If one of these branches stopped logging its step or logged the wrong outcome, no test would fail. Add a success case next to TestInstallFunnelPathDumpWithNoSkillToWrite with a non-experimental skill whose SKILL.md is served, a plugin-install run using setupPluginMock, and a confirmed row in TestInstallFunnelInteractiveCancel.
There was a problem hiding this comment.
Done in a8819ec. Added TestInstallFunnelPathDumpSuccess (local server with a stable skill and its SKILL.md: FLAGS_VALIDATED, SKILLS_DUMPED, FINISHED SUCCESS), TestInstallFunnelPluginSuccess (setupPluginMock, --agents codex: ... PLAN_CONFIRMED, PLUGINS_INSTALLED, FINISHED SUCCESS), and a confirmed row in the interactive test, now TestInstallFunnelInteractiveConfirm (PLAN_CONFIRMED, SKILLS_INSTALLED, FINISHED SUCCESS). Dropping the SKILLS_DUMPED or PLUGINS_INSTALLED step makes the matching test fail.
| opts.Scope = scope | ||
|
|
||
| // Resolve the agents to act on. | ||
| // Validate --agents with the other flags, before the scope prompt. |
There was a problem hiding this comment.
[nice to have] The .nextchanges fragment says an unknown --agents name now fails before the scope prompt. The only tests of an unknown name (TestInstallFunnelEarlyExits, TestInstallUnknownAgentErrors) run with cmdio.MockDiscard, where the prompt is never shown, so moving resolveAgentNames back below the prompt would not fail any test. Add a test with prompts supported, --agents bogus and a promptScopeSelection stub that fails the test if it is called.
There was a problem hiding this comment.
Done in a8819ec. TestInstallUnknownAgentFailsBeforeScopePrompt runs with prompts supported, --agents bogus and a promptScopeSelection stub that fails the test if called. It fails when resolveAgentNames is moved back below the prompt.
| start := time.Now() | ||
| require.ErrorContains(t, cmd.Execute(), "cannot use --skills-only with --path") | ||
| assert.Less(t, time.Since(start), 5*time.Second) | ||
| assert.False(t, cmdctx.HasConfigUsed(cmd.Context()), "an unfinished resolution leaves no config, so nothing uploads") |
There was a problem hiding this comment.
[nit] The blocked resolution is released only at test cleanup and has a 10 s budget. So with the wait put back, p.wait() times out and still leaves no config, and assert.False(t, cmdctx.HasConfigUsed(...)) passes either way. Only the < 5s timing check on the line above catches the regression. Drop this assertion, or reword its message so it doesn't read as a second check.
There was a problem hiding this comment.
Done in a8819ec. Dropped that assertion; the < 5s check is the one that catches the regression.
| return protos.AitoolsInstallScopeUnspecified | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
[nit] .agents/rules/testing.md says each source file should have a corresponding test file. The funnel code (commandFunnel, installModeType, deliverySteps, nothingInstalledCategory, firstFailureCategory) is in telemetry.go, but its tests are in the new funnel_test.go. Move the funnel code into cmd/aitools/funnel.go, or move the tests into telemetry_test.go.
There was a problem hiding this comment.
Done in a8819ec. Moved the funnel code (commandFunnel, installModeType, deliverySteps, nothingInstalledCategory, firstFailureCategory) unchanged from telemetry.go into cmd/aitools/funnel.go, next to funnel_test.go.
| cmd.PreRunE = func(cmd *cobra.Command, args []string) error { | ||
| pending = startTelemetryAuth(cmd, args) | ||
| auth.pending = startTelemetryAuth(cmd, args) | ||
| auth.waitAtExit = true |
There was a problem hiding this comment.
[nit] Install is the only command that uses withTelemetryAuth, and its RunE sets auth.waitAtExit = false before doing anything (install.go:129), so the PreRunE assignment and the field comment's 'True at the start of every run' never take effect. Drop the PreRunE assignment and default to no-wait, so install only sets it to true at PLAN_CONFIRMED, or note why other commands need the default.
There was a problem hiding this comment.
Done in a8819ec. Dropped the PreRunE assignment. waitAtExit now defaults to false (no wait), and install sets it to true only at PLAN_CONFIRMED. The comment about the early-exit bias moved next to that line. withTelemetryAuth's own tests ask for the wait explicitly.
| }, funnelShape(funnelEvents(ctx))) | ||
| } | ||
|
|
||
| func TestInstallFunnelPathDump(t *testing.T) { |
There was a problem hiding this comment.
[nit] isolateAgents clears the whole environment, and the comment on the next test says Windows sockets need SystemRoot. On Windows the dial to 127.0.0.1:1 therefore fails during Winsock initialisation, not with connection refused. That is still a *net.OpError, so the test passes for the wrong reason. Set only HOME/USERPROFILE as TestInstallFunnelPathDumpWithNoSkillToWrite does, or keep SystemRoot.
There was a problem hiding this comment.
Done in a8819ec. TestInstallFunnelPathDump no longer clears the environment. It shares a setupPathDump helper with the other --path tests that sets only a temp HOME/USERPROFILE and the skills env, and stubs the telemetry auth. SystemRoot stays set, so the dial to 127.0.0.1:1 reaches the connect step on Windows too.
| trace print_telemetry | ||
|
|
||
| title "a manifest the server does not have is MANIFEST_NOT_FOUND" | ||
| musterr trace env DATABRICKS_SKILLS_REF=missing-ref $CLI aitools install --skills-only --scope=global --agents=cursor |
There was a problem hiding this comment.
[nit] .agents/rules/testing.md documents that trace exports leading KEY=value arguments for the command, and that form is the common one in the repo. Use musterr trace DATABRICKS_SKILLS_REF=missing-ref $CLI ... here and on line 19.
There was a problem hiding this comment.
Done in a8819ec. Both cases now use musterr trace DATABRICKS_SKILLS_REF=... $CLI ....
8ac74ab to
a8819ec
Compare
|
Thanks. Every inline comment has a reply in its thread. Changes are in a8819ec. Question 1 (early-exit sampling): I'm keeping the no-wait rule as documented. A run that ends before the plan is confirmed never waits for telemetry auth, so it stays instant. The resulting bias is stated in the comment next to the rule (where install sets Question 2 ( Question 3 (one run result): Computing one run result after Every skill filtered out in an agent install: same answer as in that thread. It needs PR template: I'm keeping the stack's single body format, as before. |
7e03112 to
d25f879
Compare
a8819ec to
9f73e89
Compare
d25f879 to
fa4413b
Compare
9f73e89 to
2241742
Compare
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Approval from simonfaltum's review agent at 2241742, on Simon's decision.
The design questions from the earlier automated reviews no longer block this PR: your answers are accepted, and any still-open ones are non-blocking. The changes since the last automated review (8ac74ab → 2241742) address its comments.
rugpanov
left a comment
There was a problem hiding this comment.
Also found by our reviewers, already asked in simonfaltum's review (Question 1): runs that end before the plan is confirmed rarely upload, so the fastest early exits and their STARTED events are under-counted. Two of our reviewers found the same, and one also suggested a short bounded wait for those runs.
| // install run that gets past flag parsing emits a STARTED event, a | ||
| // STEP_COMPLETED event per step reached and a FINISHED event. All events of one | ||
| // run share ExecutionContext.CmdExecID. | ||
| // The calling agent is not a field: the User-Agent of the telemetry upload |
There was a problem hiding this comment.
[Question] This comment and libs/telemetry/protos/README.md say that the calling agent is not a field, because the User-Agent carries it. The Universe proto defines calling_agent = 6 and calling_agent_tooling = 7 on AitoolsCommandEvent. It documents calling_agent_tooling as "Set on STARTED and FINISHED events", and the User-Agent does not carry the tooling state. Will the dashboards leave both columns empty? If yes, write that the CLI leaves these fields empty on purpose, not that they are not fields. The answer matters because queries on these columns get only empty values.
There was a problem hiding this comment.
The CLI does not set a calling-agent field on this event. It reports the calling agent through the User-Agent of the telemetry upload: agent/<name> from the SDK, plus the calling agent's own Databricks tooling as aitools-agent/<plugin|skills|none> (installer.WithCallingAgentToolingInUserAgent, #7442), next to the same cmd-exec-id every event of the run carries. In be1f960 I reworded the comment here and in libs/telemetry/protos/README.md to say the CLI sets no such field, rather than that there is none.
| interactive: cmdio.IsPromptSupported(ctx), | ||
| installMode: installMode, | ||
| } | ||
| f.log(ctx, protos.AitoolsCommandEvent{Stage: protos.AitoolsCommandStageStarted}) |
There was a problem hiding this comment.
[Question] All events stay in memory, and cmd/root uploads them in one batch at exit. So a STARTED event always arrives together with its FINISHED event, and the STEP_COMPLETED events show only the last step reached. What does STARTED tell the dashboards that FINISHED does not? The answer matters because STARTED adds one row to every run of all five commands (#7470), and FINISHED with the last step would carry the same data.
There was a problem hiding this comment.
With the current upload, nothing: every run that logs STARTED also logs exactly one FINISHED, and cmd/root uploads them together at exit. So STARTED adds no information the FINISHED event (with the last STEP_COMPLETED before it) doesn't already carry. It makes each run's event list self-describing, and it would matter if events were ever uploaded before the process exits. Dropping it, or folding the last step into FINISHED, would change the event shape that the PRs stacked above this one also emit, so I'd leave that to a follow-up rather than change it here.
|
|
||
| // setTargets records agents known before the plan is built (named via | ||
| // --agents), so a run that stops at the scope prompt still reports them. | ||
| func (f *commandFunnel) setTargets(targets []*agents.Agent) { |
There was a problem hiding this comment.
[nit] setTargets makes agentPlanItem values with only agent set, so that it can reuse agentsField. If agentsField later reads another field of the plan item, this path reports wrong data. Map the agents to AitoolsAgentType in a small helper, and use that helper in setTargets and in agentsField.
There was a problem hiding this comment.
Done in be1f960. Added agentTypes([]*agents.Agent), which dedupes and sorts the enums. agentsField now collects the plan's agents and calls it, and setTargets calls it directly, so no partial agentPlanItem is built.
| tmp := t.TempDir() | ||
| t.Setenv("HOME", tmp) | ||
| t.Setenv("USERPROFILE", tmp) | ||
| t.Setenv("XDG_CONFIG_HOME", tmp+"/.config") |
There was a problem hiding this comment.
[nit] This line and line 470 build the path with tmp+"/.config". setupTestAgents uses filepath.Join(tmp, ".config"), and AGENTS.md asks for platform-agnostic paths. Use filepath.Join here too.
There was a problem hiding this comment.
Done in be1f960. Both lines now use filepath.Join(tmp, ".config").
2241742 to
be1f960
Compare
fa4413b to
0b368c6
Compare
be1f960 to
e7f6e53
Compare
|
@rugpanov thanks for the review. Every inline comment has a reply in its thread; the code changes are in be1f960. On the early-exit under-count (review body): this is the same point as Question 1 in simonfaltum's review, answered in #7469 (comment). The no-wait rule stays. The bias is stated in the comment next to it (install sets |
0b368c6 to
03fa288
Compare
e7f6e53 to
f4aaa10
Compare
03fa288 to
d13de92
Compare
f4aaa10 to
508718b
Compare
Every `databricks aitools install` run now logs an `aitools_command_event` funnel next to the existing `aitools_install_event`, so the telemetry shows where runs stop:
```text
install RunE
STARTED (every run past flag parsing)
validate flags, --agents names
--> FINISHED FAILURE INVALID_FLAGS | UNKNOWN_AGENT
STEP FLAGS_VALIDATED
--path dump --> STEP SKILLS_DUMPED, FINISHED SUCCESS
(0 skills written: FINISHED NOTHING_TO_DO)
scope prompt --> FINISHED CANCELLED USER_ABORTED | FAILURE PROMPT_FAILED
STEP SCOPE_RESOLVED
picker --> FINISHED CANCELLED (nothing checked)
no agent detected: FINISHED NOTHING_TO_DO
STEP AGENTS_SELECTED
confirm prompt --> FINISHED CANCELLED (declined)
STEP PLAN_CONFIRMED
executePlan
STEP SKILLS_INSTALLED (skills group installed)
STEP PLUGINS_INSTALLED (at least one plugin installed)
FINISHED SUCCESS | FAILURE <category>
```
Each event carries `command`, `stage`, `interactive`, `install_mode` (DEFAULT / SKILLS_ONLY / PATH) and, once known, `scope`. FINISHED also carries `outcome`, `error_category` and the targeted `agents`. Every event of a run shares the entry's `cmd_exec_id`. The CLI sets no calling-agent field on these events: the upload's User-Agent already has `agent/<name>`, the calling agent's Databricks tooling (`aitools-agent/...`) and `cmd-exec-id`.
`error_category` comes from the existing `classifyInstallError`. Failures in flag validation, agent lookup and prompts are tagged with a category where they happen (a prompt's own failure is PROMPT_FAILED whatever its type, except a Ctrl-C abort; an error before the prompt opens keeps its own category). `Error()` / `Unwrap()` pass through, so the user still sees the same message:
```diff
classifyInstallError(err)
*installer.BlockedError / *installer.SkillError -> their category
+ *categorizedError (withCategory, promptError) -> the tagged category
stdlibErrorCategory(err)
```
Some runs get an explicit outcome instead of the one their return value implies:
- Returns nil but installed nothing (e.g. the only detected agent's plugin install failed, which is a warning): FAILURE with the first failing agent's category.
- Fails only because a named agent failed or was skipped (`--agents`): the category comes from that agent's outcome (e.g. UNSUPPORTED_SCOPE), not UNCATEGORIZED.
- `--output json` failure: classifies the real error, not the `ErrAlreadyPrinted` sentinel.
- `--project --global` and `--agents ","`: INVALID_FLAGS, not PROMPT_FAILED / UNKNOWN_AGENT.
Runs that end before PLAN_CONFIRMED (a flag error, a cancelled prompt, no agent detected, a `--path` dump) upload their events only if the background telemetry auth has already resolved; they never wait for it, so they stay instant. The funnel therefore under-counts the fastest early exits relative to slower ones. Runs past PLAN_CONFIRMED keep the bounded wait.
An unknown `--agents` name now fails before the scope prompt instead of after it (changelog fragment `.nextchanges/cli/aitools-install-agents-before-scope-prompt.md`). Otherwise no change to output, prompts, exit codes or error text.
Co-authored-by: Isaac <no-reply@databricks.com>
508718b to
5a3cc25
Compare
…atabricks#7470) Stacked on databricks#7469. Stacked on: aitools: add install funnel telemetry ## Summary `aitools update`, `list`, `version` and `uninstall` now log the same `aitools_command_event` funnel as `aitools install` (STARTED, then FINISHED with the outcome and error category) and resolve auth best-effort in the background so the events can upload at exit. `list` and `version` never wait for that resolution at exit, and neither do `update` and `uninstall` runs that end before their work (flag and scope errors, a declined confirm, nothing installed in the scope, a `--skills` name that is not installed), the same rule databricks#7469 applies to `install` before its plan is confirmed; their events upload only if auth has already resolved. ```diff NewUpdateCmd / NewListCmd / NewVersionCmd / NewUninstallCmd - return cmd + return instrumentCommand(cmd, protos.AitoolsCommand<X>) + cmd.RunE = withCommandFunnel(...) // STARTED; FINISHED from RunE's error + withTelemetryAuth(cmd) // background auth; no wait at exit unless + // RunE calls waitForTelemetryAuth (update and + // uninstall, once a scope has state) ``` ```text finish(err) nil -> SUCCESS errors.Is(err, ErrNothingInstalled) -> NOTHING_TO_DO (still exits 1 with the same text) prompt aborted / no agents picked -> CANCELLED otherwise -> FAILURE, classifyInstallError(err) (scope flag errors tagged INVALID_FLAGS, as in install) RunE returns nil, but uninstall confirm answered "no" -> CANCELLED uninstall removed nothing and nothing failed -> NOTHING_TO_DO every plugin/skill update or removal attempted -> FAILURE with the first error's category failed and nothing else succeeded (still exits 0) (each via funnelFrom(ctx).finishWith) ``` Installer additions: `ErrNothingInstalled` (same error text as before), `PluginUpdate.Err` (failed plugins are returned after the updated ones; `update` prints only the updated ones), and `UpdateResult.CheckErr` (the manifest fetch error a `--check` run only warns about), and `UninstallSkillsWithResult` returning `UninstallResult{PluginsRemoved, SkillsRemoved, PluginFailures, SkillFailures}`, which replaces `UninstallSkills` and `UninstallSkillsOpts`. A plugin record for an agent this CLI does not know counts as a failed update or removal (`ErrUnknownAgent`, UNKNOWN_AGENT), and a pre-state (legacy) install returns `ErrLegacyInstall` (same text, STATE_INVALID). `uninstall --skills` with a skill that is not installed returns a `SkillError` (same text). Full uninstall now removes skills in sorted order. Changelog: `.nextchanges/cli/aitools-telemetry-funnel.md`. ## Evidence - **Before** (`return instrumentCommand(cmd, protos.AitoolsCommandList)` in `list.go` reverted to `return cmd`): ```text --- FAIL: TestInstrumentedCommandsResolveAuth Expected value not to be nil (list has no PreRunE) --- FAIL: TestListFunnelFailure expected [STARTED FINISHED:FAILURE:NETWORK], actual [] ``` **After** (current head): ```text ok cmd/aitools TestInstrumentedCommandsResolveAuth, TestListFunnelFailure, TestUninstallFunnelWhenRemovalsFail, TestUpdateFunnelWhenPluginsFail, TestNothingInstalledFinishesNothingToDo, TestInstrumentedCommandsDoNotWaitForTelemetryAuth, TestUpdateAndUninstallWorkWaitsForTelemetryAuth ``` - **No wait**: with `instrumentCommand` forced to wait, `TestInstrumentedCommandsDoNotWaitForTelemetryAuth` fails for version, list, the update/uninstall flag errors and update/uninstall with nothing installed (10 s each, budget set to 10 s); with `waitForTelemetryAuth` a no-op, `TestUpdateAndUninstallWorkWaitsForTelemetryAuth` fails for both commands. With `PreRunE` cleared in `instrumentCommand`, `TestInstrumentedCommandsResolveAuth` fails for update, list, version and uninstall. - Built CLI, fake token, `DATABRICKS_HOST=https://127.0.0.1:1` (refused): `aitools version` 18 ms, `update --scope=bogus` 17 ms, `update` and `uninstall` with nothing installed 17-18 ms, `uninstall` with one skill recorded 3.0 s. - Acceptance `cmd/aitools/telemetry/update-uninstall` pins the uploaded STARTED/FINISHED events of `update` and `uninstall` after an install. `list`, `version`, and `update`/`uninstall` with nothing installed upload only if auth has already resolved, which depends on timing, so their events are pinned by unit tests instead. `workspace-unreachable` checks that the background resolution prints nothing on install, update and uninstall runs that wait for it (with the handler's level clamp removed, it shows the hostmetadata warning). - At this head: `go build ./...`, `./task test-exp-aitools`, `cmd/aitools`, `libs/aitools` and `libs/telemetry` unit tests, the `cmd/aitools` acceptance tests, `./task lint-q`, `./task fmt-q`, `./task checks`, and `go vet` for windows and darwin: all pass. ## Merge Danger **Door:** two-way **Blast Radius:** latency - `update` and `uninstall` runs on a scope with something installed now contact the configured workspace: up to about 6 s longer on a slow workspace (3 s auth wait + 3 s upload), at least 3 s per run while connections are refused or time out. - `list`, `version`, and `update`/`uninstall` runs that end at a flag or scope error, a declined confirm, with nothing installed or at a `--skills` name that is not installed never wait for auth. When the resolution has already finished (e.g. a PAT or a valid cached token, with host metadata cached), they still upload and can take up to 3 s longer (the upload timeout) on a slow workspace; otherwise their events are dropped. - Exit codes and output are unchanged; uninstall and update failures that were only warnings still exit 0 and are reported only in telemetry. - The background resolution never refreshes an OAuth token (databricks#7468); with an expired cached token the events of these commands are dropped, as for `install`. - Not tested here: Windows and macOS (run on Linux only); a real workspace upload. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
## Release v1.27.0 ### Notable Changes * `databricks aitools install`, `update` and `uninstall` now exit with code 1 when they do nothing they were asked to. `install` fails when it sets up nothing: no coding agent is detected, every agent is skipped or fails to install, or no skill in the release can be installed (also with `--path`). `update` fails when every update or check it attempts fails, and `uninstall` when every removal it attempts fails. Scripts that ran these commands as a best-effort step and relied on exit code 0 should handle the new exit code. ([#7481](#7481)) * Add `file:<path>` selectors to `pipelines run` and pipeline `bundle run`. `--refresh` and `--full-refresh` now use space-separated lists, such as `--refresh "orders payments"`, instead of comma-separated lists. Single names containing spaces now require backticks, e.g. ``--refresh 'table:`order details`'``. ([#7184](#7184)) * The Databricks CLI is now available on PyPI as `databricks`: run it with `uvx databricks`, or install it with `uv tool install databricks`, `pipx install databricks` or `pip install databricks`. The package contains only the prebuilt CLI binary. Uninstall the legacy `databricks-cli` package first, since both install a `databricks` command. ([#7704](#7704), [#7777](#7777)) ### CLI * Add `databricks sandbox fs` commands (`cp`, `ls`, `cat`, `mkdir`, `rm`) to transfer files between your local filesystem and sandboxes using `<sandbox-id>:/path`. ([#7372](#7372)) * `databricks fs` commands retry reads (stat, list, open) and deletes on DBFS and Volumes when the server returns a transient error or does not respond within the HTTP timeout. ([#7556](#7556)) * Retry UC Volumes file operations (`fs cp`, `fs ls`, `fs rm`, bundle uploads to Volumes) when the Files API returns HTTP 408, 429, 502, 503 or 504 (for example "AWS S3 is throttling requests"), or on a transient network error; this regressed in v1.10.0. ([#7559](#7559)) * Stop warning when the host metadata request (`/.well-known/databricks-config`) times out; the CLI already falls back to your config, and the message is still shown with `--debug`. ([#7752](#7752)) * Add `databricks aitools auto-install disable` and `enable` to turn off (or back on) the automatic installation of Databricks AI tools on CLI upgrades. A global-scope `databricks aitools uninstall` also turns it off. ([#7701](#7701)) * Stop printing the Databricks AI tooling install hint on `auth`, `configure`, `current-user me`, `version`, `help`, and shell-completion commands. ([#7452](#7452)) * When Codex, Cursor, GitHub Copilot CLI, OpenCode, Gemini CLI, Antigravity, Pi, Goose or Kiro run the CLI without Databricks skills installed for that agent, the CLI now prints a one-line hint to install them with `databricks aitools install --agents <agent> --scope global`, previously shown only to Claude Code. ([#7586](#7586)) * `databricks aitools install` now rejects an unknown `--agents` name before it shows the scope prompt, instead of after. ([#7469](#7469)) * `databricks aitools install` no longer asks "Proceed?" after the agent picker; the picker selection confirms the plan. The picker title now shows the install scope, its hint says Enter installs, `--skills-only` runs label plugin agents "skills" and pre-check every agent whose config dir exists, and the plan summary after it is reduced to the scope line. ([#7478](#7478)) * `databricks aitools install --output json` reports more specific `error_category` values: `SKILLS_FETCH_FAILED`, `RELEASE_LOOKUP_FAILED`, `RATE_LIMITED`, `MANIFEST_INVALID`, `STATE_INVALID` and `PLUGIN_CLI_TIMEOUT`. A coding agent CLI that does not answer in time is now reported as `timed out after <duration>`, followed by any stderr it wrote before it was stopped. ([#7475](#7475)) * `databricks aitools install` no longer prompts when a coding agent runs it, even in a pseudo-terminal; it uses the non-interactive defaults instead and prints a note saying so. ([#7477](#7477)) * `databricks aitools install` now prints the directory it installed skills into (for example `Installed 1 skill in ~/.databricks/aitools/skills.`), so agents can open the installed `SKILL.md` files without searching for them. ([#7668](#7668)) * `databricks aitools install` now ends a failed install with a step to recover, in the text output and in a new `recovery` field of `--output json`. ([#7480](#7480)) * `databricks aitools install` now installs plugins for the other coding agents even when installing skill files fails. ([#7479](#7479)) * `databricks aitools install` now installs skill files for a detected coding agent whose CLI is not on PATH (for example a Claude Code desktop or IDE install), unless it already has the plugin, says how to switch to the plugin later, and warns when the current project also has the plugin. `databricks aitools update` no longer tries to install the plugin for such an agent, and no longer writes skill files next to a plugin that such an agent already has. ([#7482](#7482)) * Fix `databricks aitools install` waiting a long time on an unreachable or slow workspace while it resolves authentication for telemetry; that wait is now limited to a few seconds. ([#7468](#7468)) * `databricks aitools install`, `update` and `list` now refuse a skills manifest that names a skill or file outside the skills directory, before writing any skill files. A skill whose name or file paths are not portable is skipped with a warning, and `update` keeps its installed copy. ([#7722](#7722)) * `databricks aitools install` and `databricks aitools update` refresh only the marketplace the Databricks plugin is listed in, so another unreachable marketplace no longer blocks them. When that refresh fails, both commands warn, install or update the plugin from the local copy of the marketplace, and record the plugin version as unknown. ([#7473](#7473)) * `databricks aitools install` and `databricks aitools update` now wait up to 20 seconds instead of 5 for a coding agent's CLI to answer the plugin capability check, so a slow first start is less likely to report the agent as unavailable. ([#7472](#7472)) * `databricks aitools install` no longer fails a plugin-only install when the lookup of the latest skills release on GitHub fails or is rate-limited. The plugin is installed and its version is recorded as unknown. ([#7474](#7474)) * `databricks aitools` commands now send usage telemetry, and the CLI records when it recommends installing Databricks AI tools. When the configured workspace is slow or unreachable, `aitools install`, `update` and `uninstall` can take up to about 6 seconds longer, and `list`, `version` and runs that stop early up to about 3 seconds. Set `DATABRICKS_CLI_DISABLE_TELEMETRY=1` to opt out. ([#7467](#7467), [#7469](#7469), [#7470](#7470)) * `databricks aitools install` and `databricks aitools update` now report a coding agent CLI shim (`.cmd` or `.bat`) on Windows that prints its plugin help and then exits with an error as `shim-exited-with-error`, with the command to run to see the error, instead of reporting the CLI as not on PATH. ([#7476](#7476)) * `apps init` no longer suggests installing skills when every detected coding agent already has the Databricks plugin. ([#7467](#7467)) * `databricks auth login` now gives up on the optional workspace lookup after about 5 seconds instead of retrying for about 5 minutes, and explains the fallback to the manual workspace ID prompt. ([#7522](#7522)) * Throttle Claude Code's native plugin install hint separately from the text hint: it prints at most once per 10 minutes per session, so the install dialog still comes up when a subagent, a hook, or a command outside a Claude Code subprocess runs the session's first `databricks` command. ([#7692](#7692)) * Print Claude Code's native plugin install hint alongside the text hint, so Claude Code offers a one-click install of the Databricks plugin. Claude Code now counts as having the tooling only when it loads it: skills only in `~/.databricks/aitools/skills` don't count, and a project- or local-scope plugin counts only when the CLI runs inside that project. `auth token`, `auth docker token` and CLI subprocesses no longer print install hints. ([#7585](#7585)) * Interactive prompts and pagers now work in Git for Windows' bash and other MSYS2 shells when they run under ConPTY (the default since Git for Windows 2.47) or directly in a Windows console. ([#7699](#7699)) * Add the `databricks feedback` command to send feedback about the CLI to Databricks. ([#7724](#7724)) * Fix shell completion in `databricks fs` commands for local paths typed with a leading `./` or repeated separators, which previously offered no suggestions. ([#7679](#7679)) * Fix a panic in `databricks labs` commands when a non-string SDK setting such as `DATABRICKS_DEBUG_HEADERS` or `DATABRICKS_RATE_LIMIT` is set in the environment. ([#7673](#7673)) * Fix a panic in `databricks labs install` for projects whose `labs.yml` has no `install` section. ([#7673](#7673)) * `NO_COLOR` no longer disables prompts, spinners or the list pager; they run without color. ([#7700](#7700)) * Add the Beta `databricks pipelines test` command, which runs the tests of a pipeline, and the `testing` property of pipelines in bundles. ([#7429](#7429)) * Add `databricks sandbox forward`, which serves an app running in a sandbox at a local URL. ([#7395](#7395)) * Bound the whole telemetry upload at exit, including config resolution and authentication, by its timeout, so a stalled auth or telemetry endpoint no longer keeps a command from exiting. ([#7467](#7467)) * `databricks version --check` now suggests `databricks aitools install` on stderr when a coding agent is detected without the Databricks plugin or skills. ([#7471](#7471)) ### AI Runtime * `air get -o json` now reports `PENDING` while the run waits for compute, matching `air list`, and includes the run's `status_message`. ([#7455](#7455)) * Wrap long failure reasons and log lines in the `air list` info and logs panes, keep the MLflow link (labeled with its retry) when a run's last retry failed before starting, and point `air logs` at earlier retries when the last one has no logs. ([#7456](#7456)) * `databricks air logs --node` now reports an out-of-range node index when it can determine the run's node count, instead of looking like a successful run with no logs. ([#7516](#7516)) * Add experiment, duration, accelerator, and dashboard metadata to `air list --output json`. For retried runs, `started_at` now reports the latest attempt, matching `air get`. ([#7568](#7568)) * `air list` shortens MLflow run names and unnamed-run IDs by keeping their end behind a leading `…`, and its columns no longer overflow on wide (for example CJK) text; `air get` shows an unnamed run's full MLflow run ID. ([#7582](#7582)) * Shrink the Experiment and User columns of the interactive `air list` table to fit the terminal width, keeping the end of a shortened experiment name. ([#7583](#7583)) * Show the experiment, accelerators and configuration in the `air list` info pane for runs where it showed `N/A`. ([#7587](#7587)) * Strip terminal control characters from the MLflow run, experiment and user names shown by `air list` and `air get`, so a run name cannot clear or retitle the terminal of whoever lists it. ([#7601](#7601)) * `air logs`, `air get`, and `air run --watch` no longer make an extra credential-check request before fetching the run, and `air get` reuses the configured workspace ID instead of looking it up. ([#7613](#7613)) * `air list` no longer waits on extra current-user requests before listing runs, which makes the default list about 150ms faster. ([#7615](#7615)) * `air list --all-status` requests larger index pages, so it makes far fewer round trips once the server supports `page_size`. ([#7616](#7616)) * Fit the interactive `air list` table to narrow terminals: when the Experiment and User columns are at their floors, shorten Started to `MM-DD HH:MM` and size the MLflow column to its labels, then hide Accelerators, User, MLflow and Progress in that order, naming the hidden columns in the hint line. ([#7629](#7629)) * `air get -o json` now includes `mlflow_run_id` and `mlflow_experiment_id`, so a run can be handed to MLflow-based steps without parsing `mlflow_url`. They identify the MLflow run `mlflow_url` links to, including one from an earlier retry that `mlflow_retry` names. ([#7636](#7636)) * `air get` and the `air list` info pane now show the environment of runs on the Databricks AI environment (`databricks_ai_v*`) instead of `N/A`. ([#7637](#7637)) * `air run` now rejects a `code_source.include_paths` path that selects no files instead of uploading a snapshot without it; glob patterns and pathspec magic are left to git. ([#7643](#7643)) * `air convert-to-dabs` now suggests the `--output-dir` that converts a working-tree workload whose `code_source.root_path` is outside the bundle directory, instead of suggesting `sync.paths`. ([#7644](#7644)) * `air run` now lists a `code_source.root_path` outside a git repository relative to `root_path`, so running from a subdirectory of `root_path` no longer uploads an empty or partial snapshot. ([#7646](#7646)) * `air list --all-status` now fills `--limit` past purged or inaccessible indexed runs. ([#7743](#7743)) * `air run` now warns about missing PyTorch when a pinned snapshot's `include_paths` selects a Python file. ([#7744](#7744)) * `air cancel JOB_RUN_ID` now rejects run IDs that are not AI Runtime runs instead of cancelling unrelated Jobs runs. In `-o json` output, the `failed[].error` text for a run that does not exist or that you cannot view now starts with `failed to get run <id>:` instead of the cancel API error. ([#7569](#7569)) * `air get -o json` now reports `mlflow_retry` when the MLflow link comes from an earlier retry because the run's last retry never reached MLflow. ([#7620](#7620)) * `databricks air run --help` now shows how to install the databricks-ai-runtime coding agent skill. ([#7652](#7652)) * `air run --watch -o json` now reports each lifecycle `STATUS` transition once and never reports a non-terminal state after the run has ended, including when logs are served from MLflow artifacts. ([#7621](#7621)) * `air convert-to-dabs` now explains the `--output-dir` workaround when `code_source.root_path` resolves to the bundle root, and notes when the generated job carries the default `max_retries`. ([#7579](#7579)) * `air run` now refuses to upload a `code_source` snapshot that selects no files (for example when `root_path` is git-ignored) and explains the cause, and `--dry-run` runs the snapshot file selection and reports the file count. ([#7570](#7570)) * Repeat the Jobs run link after live log streaming ends. ([#7571](#7571)) * `air get -o json` includes run details including flat `accelerator_type` and `num_accelerators`. Use `--include-config` to include stored config, which may contain plaintext `env_variables` and parameters; downloads over 1 MiB are omitted, and config-dependent `max_retries` is otherwise null. ([#7624](#7624)) * `air get` and the `air list` info pane no longer show `Max Retries 0` when the Jobs API leaves the task's retry limit out. For a run submitted with `air run`, the limit now comes from the run's uploaded config, shown as `3 (CLI default)` when the config doesn't set `max_retries`; it shows `N/A` when the limit can't be known (job runs, multi-container runs, legacy runs with no explicit value, or an unreadable config). ([#7623](#7623)) * Reorganize `databricks air` help: subcommands are grouped by task, `air run` has examples, and YAML fields are listed under headings, with pool fields marked optional and `command`, `compute.num_accelerators`, and `compute.accelerator_type` correctly marked required. ([#7512](#7512)) * `air run -h config` now lists the environment variables AI Runtime sets on every node (such as `MLFLOW_RUN_ID` and `CODE_SOURCE_PATH`), and `air logs --help` notes that `--download-to` cannot be combined with `--tail` or `--minutes`. ([#7655](#7655)) * Support full workspace accelerator types in `air list` filters while preserving family matching, and reject invalid filter values. ([#7515](#7515)) * `databricks air list --all-status` now warns when the AI Runtime run index is unavailable and it falls back to scanning Jobs runs. That fallback scan is capped at 250 runs, and a stuck index request fails after 15 seconds instead of retrying for minutes. `--all-status --filter user=<your email>` now uses the run index instead of scanning the workspace's full run history. ([#7619](#7619)) * Return structured JSON errors for invalid `air list --filter` values when `-o json` is used. ([#7612](#7612)) * `air list --all-status` no longer fails when a listed run has been purged. ([#7523](#7523)) * **Behavior change:** `databricks air logs` now prints the logs available now and exits, also for an active run, matching `kubectl logs`, `docker logs` and `databricks apps logs`. Pass `--follow` (`-f`) to stream new lines until the run completes; `--tail N --follow` prints the last N lines first. `databricks air run --watch` still submits and streams. ([#7658](#7658)) * `databricks air logs -o json` and `databricks air run --watch -o json` now emit a `WAITING` event, for example `{"type":"WAITING","message":"Waiting for accelerator compute capacity..."}`, when the reported waiting status changes. ([#7695](#7695)) * `databricks air logs` on a run that ended in the last two minutes now waits for the log index to settle before printing, instead of sometimes dropping lines that were indexed late. ([#7663](#7663)) * `air logs` and `air run --watch` now print the waiting status (for example `Waiting for accelerator compute capacity...`) as a line on stderr when stderr is not a terminal, instead of printing nothing until the run produces logs. ([#7657](#7657)) * Route AIR log links to the MLflow Logs page instead of the hidden artifact path. ([#7491](#7491)) * Invalid `compute.accelerator_type` errors and `air run -h config.compute` no longer list the CLI's built-in accelerator types, which could differ from the workspace's. ([#7520](#7520)) * `databricks air --help` now notes that GPU pools are not required to run workloads. ([#7581](#7581)) * `air run` now prints the job run URL with the `?o=<workspace-id>` suffix, matching `air get` and `air list`; `dashboard_url` in `air run -o json` and the `--watch` events gains it too. When `air list` cannot resolve the workspace ID, its links now omit `?o=` instead of ending in `?o=0`. ([#7573](#7573)) * `databricks air run` now reports an unknown workload YAML key by its field path, suggests the closest valid field, and points to `-h config` instead of naming an internal Go type. ([#7574](#7574)) * `air run` (including `--dry-run`) now warns before submitting when the `code_source` imports `torch` but PyTorch is not declared: the environment is the Standard environment (version 5 or later) with no `torch` dependency, so the run may fail after it waits for a GPU. ([#7656](#7656)) ### Bundles * Add `isolation_mode` to the `catalogs` resource, so a catalog can be deployed as `OPEN` or `ISOLATED`. ([#7463](#7463)) * `bundle validate` no longer warns when resource permissions and top-level or target permissions name the same principal. ([#7291](#7291)) * Retry bundle file uploads, reads and deletes when the server does not respond within the HTTP timeout ("request timed out after ... of inactivity"). ([#7521](#7521)) * Show how to recover when `bundle deploy` fails because an app's service principal does not exist or was deleted. ([#7533](#7533)) * Retry transient server errors when creating Lakebase (`postgres_*`) resources in the direct deployment engine. ([#7557](#7557)) * Fix `bundle plan` reporting the `spark_conf` and `custom_tags` the backend derives from `is_single_node: true` as drift on clusters. ([#7577](#7577)) * Stop planning a spurious update for Lakeflow Connect pipelines whose `ingestion_definition.data_staging_options` is filled in by the backend. ([#7603](#7603)) * Retry the root path check in `bundle destroy` on transient errors, so a dropped connection no longer fails the destroy. ([#7597](#7597)) * Use the Lakebase branch when its create request times out but the branch was created, instead of failing the deploy, in the direct deployment engine. ([#7602](#7602)) * Retry transient server errors when deleting resources in the direct deployment engine. ([#7708](#7708)) * Use the Lakebase project when its create request times out but the project was created, instead of failing the deploy with "already exists", in the direct deployment engine. ([#7747](#7747)) * Use the feature, model serving endpoint or dashboard that a create request created despite timing out, instead of failing the deploy, in the direct deployment engine. ([#7757](#7757)) * Use the Lakebase role when its create request fails with a transient error but the role was created, instead of failing the deploy, in the direct deployment engine. ([#7767](#7767)) * Use the MCP service, model service or model provider service that a create request created despite failing with a timeout, a 499 or a reset stream, instead of failing the deploy, in the direct deployment engine. Create recovery for features, serving endpoints and dashboards now also covers 499 and reset streams. ([#7782](#7782)) * Retry app updates and deletes in the direct deployment engine when they fail with 400 BAD_REQUEST because the app compute is `UPDATING`, after waiting for the compute to settle. ([#7606](#7606)) * Enable drift detection for `source_code_path` and `git_source` on apps. ([#6399](#6399)) * `bundle destroy` and `bundle deploy` no longer fail with "internal error, missing in state" when the state holds an empty-ID entry left by an interrupted recreate in an older CLI. ([#7676](#7676)) * Fix a feature whose `full_name` contains uppercase letters being recreated on every deploy. ([#7398](#7398)) * Fix `bundle deployment bind` (and `bundle generate --bind`) failing with `no such node "resources.internal_immutable_snapshots.immutable"` when `workspace.sync_mode` is `immutable`, as reported in [#7306](#7306). ([#7329](#7329)) * Retry creating the deployment lock (without `--force-lock`) when the request fails with a transient server error, times out ("request timed out after ... of inactivity") or loses its connection ("unexpected EOF"). ([#7594](#7594)) * Fix a materialization set member whose `feature_name` contains uppercase letters failing to deploy, or recreating the set on every deploy. ([#7398](#7398)) * Delete the old resource when its name follows a renamed schema. ([#7718](#7718)) * Fix `pipelines dry-run` starting a real run when given a job KEY, and a crash in its shell completion. ([#7672](#7672)) * Rename schemas in place; schemas and volumes follow a renamed catalog or schema. ([#7343](#7343)) * Fix a panic when validating or deploying jobs with local libraries and serverless job clusters. ([#7725](#7725)) * Fix `tgz` artifacts with `git` set producing an empty tarball or failing when the bundle is in a subdirectory of the git repository. ([#7674](#7674)) * Apply `presets.trigger_pause_status` and the development mode pause default to `triggers[]` entries on jobs. ([#7450](#7450)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([#7750](#7750)) * Update the Go SDK to [v0.198.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.198.0). ([#7530](#7530)) * Update the Go SDK to [v0.199.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.199.0). ([#7584](#7584)) * Update the Go SDK to [v0.200.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.200.0). ([#7653](#7653)) * Update the Go SDK to [v0.201.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.201.0). ([#7769](#7769))
Stacked on #7468.
Stacked on: aitools: bound best-effort telemetry auth
Summary
Every
databricks aitools installrun now logs anaitools_command_eventfunnel next to the existingaitools_install_event, so the telemetry shows where runs stop:Each event carries
command,stage,interactive,install_mode(DEFAULT / SKILLS_ONLY / PATH) and, once known,scope. FINISHED also carriesoutcome,error_categoryand the targetedagents. Every event of a run shares the entry'scmd_exec_id. The CLI sets no calling-agent field on these events: the upload's User-Agent already hasagent/<name>, the calling agent's Databricks tooling (aitools-agent/...) andcmd-exec-id.error_categorycomes from the existingclassifyInstallError. Failures in flag validation, agent lookup and prompts are tagged with a category where they happen (a prompt's own failure is PROMPT_FAILED whatever its type, except a Ctrl-C abort; an error before the prompt opens keeps its own category).Error()/Unwrap()pass through, so the user still sees the same message:classifyInstallError(err) *installer.BlockedError / *installer.SkillError -> their category + *categorizedError (withCategory, promptError) -> the tagged category stdlibErrorCategory(err)Some runs get an explicit outcome instead of the one their return value implies:
--agents): the category comes from that agent's outcome (e.g. UNSUPPORTED_SCOPE), not UNCATEGORIZED.--output jsonfailure: classifies the real error, not theErrAlreadyPrintedsentinel.--project --globaland--agents ",": INVALID_FLAGS, not PROMPT_FAILED / UNKNOWN_AGENT.Runs that end before PLAN_CONFIRMED (a flag error, a cancelled prompt, no agent detected, a
--pathdump) upload their events only if the background telemetry auth has already resolved; they never wait for it. Runs past PLAN_CONFIRMED keep the bounded wait.An unknown
--agentsname now fails before the scope prompt instead of after it (changelog fragment.nextchanges/cli/aitools-install-agents-before-scope-prompt.md). Otherwise no change to output, prompts, exit codes or error text.Evidence
cmd/aitools/telemetry/installfails; the upload has onlyaitools_install_event:install-failurepins the failure paths past PLAN_CONFIRMED: missing skill, manifest 404 / 500, named agent skipped for the scope. Runs that end earlier upload only if auth already resolved, which depends on timing, so they are covered by unit tests instead.cmd/aitools/funnel_test.go(for the funnel code infunnel.go) cover every early return and each success path (e.g.TestInstallFunnelEarlyExits,TestInstallFunnelAbortedScopePromptKeepsNamedAgents,TestInstallFunnelPathDumpSuccess,TestInstallFunnelPathDumpWithNoSkillToWrite,TestInstallFunnelPluginSuccess,TestInstallFunnelInteractiveConfirm,TestInstallFunnelDetectedPluginFailureIsNotSuccess,TestInstallFunnelSkillsFailureThroughJSON); they andTestAitoolsEventsWireKeyspass with-count=1.TestInstallUnknownAgentFailsBeforeScopePrompt: with prompts supported,--agents bogusfails without showing the scope prompt (fails ifresolveAgentNamesmoves back below the prompt).TestInstallEarlyExitDoesNotWaitForTelemetryAuth(a flag error returns at once while auth is still resolving; fails after 10 s with the wait put back) andTestInstallPastPlanConfirmedWaitsForTelemetryAuth(auth that finishes only after the install step still reaches the upload; fails without the wait).go build ./...,./task test-exp-aitools,go test ./libs/telemetry/... ./cmd/root/...,./task lint-q,./task fmt-q,./task checks, andgo vetwithGOOS=windows/GOOS=darwinpass.Merge Danger
Door: two-way
Telemetry only. Reverting removes the events; nothing on disk or in the install state changes.
Blast Radius: telemetry
aitools installupload (about eight instead of one), sent at exit and only when auth resolved. Runs that stop before the plan is confirmed (flag errors, a cancelled prompt, no agent detected) and--pathdumps logged no event before, so they made no telemetry request. They now upload only if the background auth has already resolved when they end, and never wait for it, so an unreachable or slow-to-authenticate workspace adds nothing to them. When auth has resolved, the upload itself can still take up to its timeout (DATABRICKS_CLI_TELEMETRY_TIMEOUT, default 3 s) against a slow workspace. The flip side: a run that ends in milliseconds (e.g. a flag error) usually ends before auth resolves, so its events are often dropped; a run that stopped at a prompt usually had time to resolve auth. The funnel therefore under-counts the fastest early exits, and their STARTED events, relative to slower ones; the code comment next to the rule says the same. Runs past the plan confirmation keep the bounded auth wait.--output jsonandaitools_install_eventreport these runs as installed today too. The--pathdump knows how many skills it wrote, so a dump that writes none finishes NOTHING_TO_DO. A run killed by Ctrl-C outside a prompt, or rejected by flag parsing, uploads no funnel events.This pull request and its description were written by Isaac.