Repository navigation
aitools: install AI tools after the first command of a new CLI version - #7702
Conversation
|
Commit: Regression test reportSelected 4 changed tests (limit=20, 0 not selected) 4 tests: regression=4 unreleased=0 coverage=0 failing=0
TestAccept/cmd/aitools/auto-install/first-run: branch ✅ | main (1a68ed9) ❌ | latest (v1.26.0) ❌main (1a68ed9): latest (v1.26.0): TestAccept/cmd/aitools/auto-install/retries: branch ✅ | main (1a68ed9) ❌ | latest (v1.26.0) ❌main (1a68ed9): latest (v1.26.0): TestAccept/cmd/aitools/auto-install/skipped: branch ✅ | main (1a68ed9) ❌ | latest (v1.26.0) ❌main (1a68ed9): latest (v1.26.0): TestAccept/cmd/aitools/auto-install/upgrade: branch ✅ | main (1a68ed9) ❌ | latest (v1.26.0) ❌main (1a68ed9): latest (v1.26.0): |
✅ Integration test reportCommit: 3344168
Top 4 slowest tests (at least 2 minutes):
|
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at 019d40c. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 2 blocking issue(s), tagged [blocking]; 1 open question about a change users could be hurt by.
This PR installs or updates Databricks AI tools after the first successful command of each new CLI version. It is gated off behind cli-compat and has opt-out, CI, holdout, retry and lock handling. The decision logic, locking and tests are careful, and CI is green. Two regressions ship even with the gate off: apps init drops the DATABRICKS_CLI_SELF_EXEC marker on its children, and the new process group stops Ctrl-C from reaching agent CLIs in the explicit aitools install/update. Several gaps should be closed before the gate is turned on: the skip list (auth docker token, CI providers, DBR/non-interactive runs), the gate-failure settle, a stuck partial first install, plugin migration hiding edits, and telemetry outcomes.
Questions
- [blocking] Auto-install skips only CI (
CI=true), but the version-check notice andMaybeHintalso skip Databricks Runtime, non-interactive runs and-o json. Scripts, IDE extensions and ephemeral homes (containers, DBR clusters) pay the gate lookup now: up to 3 s, once per version per fresh home. Once the gate is on, they also pay up to about 20 s, plus retries, after their command's output. Should auto-install reuse those skips, or is running there intended? If it is intended, a comment saying why would help.
Accepted answers
- Changelog fragment for this change
| cmd.Stdout = os.Stdout | ||
| cmd.Stderr = os.Stderr | ||
| cmd.Stdin = os.Stdin | ||
| cmd.Env = autoinstall.ChildEnv() |
There was a problem hiding this comment.
[blocking] cmd.Env = autoinstall.ChildEnv() replaces the cmd.Env = agents.SelfExecEnv(os.Environ()) set four lines above (line 1968). The same happens in runPostCreateDev (lines 1998 and 2002). The apps deploy / apps dev-remote children therefore lose DATABRICKS_CLI_SELF_EXEC=1, and MaybeHint (libs/aitools/agents/hint.go:196) no longer suppresses the coding-agent hint in them. Under Claude Code and other agents, the child prints the hint again inside the parent's output and logs a second hint event. This ships even while the gate is off. Let ChildEnv take the environ slice and compose the two, e.g. cmd.Env = agents.SelfExecEnv(autoinstall.ChildEnv(os.Environ())), at both call sites. That also stops it from reading os.Environ() in library code. A test that checks both markers would cover these call sites, which currently have none.
There was a problem hiding this comment.
Fixed in 879027f. ChildEnv now takes the environ, and both call sites use a new selfExecEnv(environ) in cmd/apps/init.go = agents.SelfExecEnv(autoinstall.ChildEnv(environ)), so the children keep DATABRICKS_CLI_SELF_EXEC=1. TestSelfExecEnvCarriesBothMarkers checks both markers, before and after a "no".
| out, err := process.Background(cctx, argv) | ||
| // A timed-out agent CLI is stopped with its descendants: one left running | ||
| // would hold the output pipes open and block past the timeout. | ||
| out, err := process.Background(cctx, argv, process.WithProcessGroup()) |
There was a problem hiding this comment.
[blocking] WithProcessGroup moves the agent CLI out of the terminal's foreground process group (here and in libs/aitools/agents/detect.go:115), so Ctrl-C no longer reaches it. Nothing in cmd/root or cmd/aitools catches SIGINT. Ctrl-C kills the CLI, the context is never cancelled, and claude plugin install / marketplace add keep running as orphans that modify ~/.claude. Before this PR, they were interrupted too. This changes the explicit aitools install / update today, for a feature that is off. cmd/environments/sync.go:92-101 documents the same effect for uv and adds a signal handler for that reason. With the gate on, it also releases the run lock while the orphan is still installing, which is the hazard used to exclude Windows. Either cancel the context on SIGINT/SIGTERM while agent CLIs run (see cmd/air/run.go:350 for restoring the default disposition), or use the process group only on the auto-install path.
There was a problem hiding this comment.
Fixed in 879027f, with the signal route so the explicit commands and auto-install both get it. New process.BackgroundInGroup (libs/process/interrupt_unix.go) runs the child with WithProcessGroup and, while it runs, catches the first SIGINT/SIGTERM. It calls signal.Stop first, so a second signal kills at once (the same order as cmd/air/run.go), then cancels, which stops the group, and then raises the signal again in the CLI. The CLI therefore ends as it did before the group, and nothing is orphaned or keeps running past the run lock. runAgentCmd and ProbePlugin use it. On Windows the child stays in the console group, so nothing is forwarded there. TestBackgroundInGroupForwardsInterrupt interrupts a helper process mid-run, then checks that it died by SIGINT and that its grandchild was reaped.
| return true | ||
| } | ||
| } | ||
| return slices.Equal(path, []string{"auth", "token"}) |
There was a problem hiding this comment.
[nice to have] isSkippedCommand exempts only the exact path auth token. docker-credential-databricks runs databricks auth docker token (libs/dockercredentials/shim.go:68) on pulls and pushes. So auto-install runs inside Docker's credential lookup: up to 3 s for the gate now, and an install plus a stderr line once the gate is on. agents.isCredentialHelperCommand (hint.go:268-279) already covers both paths, and versioncheck.isExemptCommand covers the rest of this list. Reuse them instead of a parallel list. That also replaces the hard-coded "__complete" strings with cobra's constants.
There was a problem hiding this comment.
Fixed in 879027f. The command list is gone from Decide; it takes a SkippedCommand bool. skipsAutoInstall(cmd) in cmd/aitools is versioncheck.IsExemptCommand || agents.IsCredentialHelperCommand || in an aitools tree. Both helpers are now exported, so auth docker token is covered and the cobra constants are used. Test: TestSkipsAutoInstall.
| Command: path, | ||
| CacheDisabled: autoinstall.CacheDisabled(ctx), | ||
| Version: version, | ||
| CI: versioncheck.InCI(ctx), |
There was a problem hiding this comment.
[nice to have] versioncheck.InCI is env.GetBool(ctx, "CI"). It misses CI=<provider name>, which GetBool maps to false. It also misses providers that don't set CI, such as Azure Pipelines (TF_BUILD), Jenkins and CodeBuild, which useragent.CiCdProvider() detects. libs/aitools/agents/hint.go:253-266 (inCI) handles both cases, and its comment explains why GetBool can't be used. Such build agents are treated as user machines: the gate lookup today, and installs once the gate is on. Reuse inCI.
There was a problem hiding this comment.
Fixed in 879027f. It now uses agents.InCI (exported), so CI=<provider> and SDK-detected providers such as TF_BUILD count. versioncheck.InCI is reverted, so notice.go only has the IsExemptCommand export now.
| } | ||
| // Skills without install state have no checksums to tell the user's edits | ||
| // from ours, and an install would replace them. | ||
| if installer.HasUntrackedSkills(ctx, installer.ScopeGlobal) { |
There was a problem hiding this comment.
[nice to have] InstallSkillsForAgents writes skill directories one by one and saves .state.json only after the loop (installer.go:478). A first attempt that hits the 10 s installTimeout partway leaves skills on disk with no state, and counts as FAILED. On the next run, HasUntrackedSkills is true, so planTools takes the keep path and the run settles INSTALLED. Every later version takes the same path. The machine keeps a partial skill set forever and reports success. Tell the CLI's own interrupted install apart from a legacy one (for example, save state incrementally, or write a marker before the first skill), and retry it.
There was a problem hiding this comment.
Fixed in 879027f with a marker. Before a fresh install, the state gets installing: true, which only settling clears. planTools takes the keep path for untracked skills only when that marker is not set, so an install cut short by the timeout is finished (and retried) on a later run, while hand-made or legacy skills are still left alone. Tests: TestPlanToolsKeepsSkillsWithoutInstallState (second half) and TestRecordClearsInstallingWhenSettled; the retries golden shows the marker.
| return installResult{}, err | ||
| } | ||
| result := installResult{updated: true} | ||
| if after != nil && toolsChanged(before, after) { |
There was a problem hiding this comment.
[nit] With KeepModified, UpdateSkills still advances state.Release to the new tag (update.go:225) even when every outdated skill was kept. toolsChanged then sees a different Release, and the notice says the tools were updated when no file changed. It also lists every agent with a global install, not just the ones that changed. Compare skills and plugin versions only, or have UpdateSkills report whether anything changed.
There was a problem hiding this comment.
Fixed in 879027f. toolsChanged became changedAgents, which ignores Release and names an agent only if it has skills and a skill version changed, or if its own plugin version changed. With no changed agent, the run reports UP_TO_DATE and prints nothing. Test: TestChangedAgents.
| // explicit choice matters more than a broken record. | ||
| func SetOptedOut(ctx context.Context, optedOut bool) error { | ||
| s, err := Load(ctx) | ||
| unlock, err := lock(ctx, stateLockFileName, stateLockWait) |
There was a problem hiding this comment.
[nit] SetOptedOut now requires the state lock. On a home directory whose filesystem rejects flock (some NFS/FUSE mounts), lock fails at once, so databricks aitools auto-install disable exits 1. In the base it just wrote the file. Auto-install can't run there either, because Load fails. Consider letting the explicit opt-out write without the lock when locking is unsupported (not merely held).
There was a problem hiding this comment.
Done in 879027f. SetOptedOut still returns errLockHeld when another process holds the lock. On any other lock error it writes without the lock (logged at debug), since auto-install cannot run there anyway. Test: TestSetOptedOutWithoutLocks.
| } | ||
| // Commands and builds that never auto-install skip before the state is | ||
| // read, and without a log line: they are most runs. | ||
| if reason := autoinstall.Decide(in); reason == autoinstall.ReasonSkippedCommand || reason == autoinstall.ReasonDevelopmentBuild { |
There was a problem hiding this comment.
[nit] The early return covers only skipped commands and development builds. With the cache disabled, the run still calls Load, which creates ~/.databricks/aitools/ and auto-install.json.lock. Add ReasonCacheDisabled to the early return.
There was a problem hiding this comment.
Fixed in 879027f. The early return now covers skipped command, Databricks Runtime, cache disabled and development build (stateFreeSkips), so none of them touches ~/.databricks/aitools/.
| state, err := autoinstall.Load(ctx) | ||
| require.NoError(t, err) | ||
| assert.True(t, state.OptedOut) | ||
| assert.True(t, autoinstall.Declined()) |
There was a problem hiding this comment.
[nit] This test leaves the package-global autoinstall.declined flag set, and the agents package can't reset it. Separately, TestChildEnvCarriesTheRefusal (libs/aitools/autoinstall/run_test.go:102) fails when the developer has DATABRICKS_AITOOLS_AUTO_INSTALL=false set, because ChildEnv reads os.Environ(). Either export a test reset or pass the environ in, and clear EnvVar in that test.
There was a problem hiding this comment.
Fixed in 879027f. Added autoinstall.ResetDeclined(), which TestRecommendSkillsInstallInteractiveDecline and the new apps test pass to t.Cleanup. ChildEnv takes the environ, so TestChildEnvCarriesTheRefusal no longer reads the developer's environment.
| $CLI completion bash > /dev/null 2>&1 | ||
| $CLI __complete jobs > /dev/null 2>&1 | ||
| $CLI help jobs > /dev/null 2>&1 |
There was a problem hiding this comment.
[nit] .agents/rules/testing.md asks to route noisy output to LOG.<name> rather than /dev/null, and to capture a tolerated non-zero exit with errcode rather than || true. cursor_skills in script.prepare:63 uses || true; a [ -d ... ] check, as auto_install_state already does, avoids it.
There was a problem hiding this comment.
Fixed in 879027f. cursor_skills checks [ -d ] instead of using || true, and the completion/__complete/help runs in skipped/script go to LOG.* instead of /dev/null.
First of three changes that install Databricks AI tools automatically on the first run of a new CLI version. This one stores the state and the opt-out and adds the remote gate; nothing installs yet. - ~/.databricks/aitools/auto-install.json holds last_version, attempts and opted_out. It is a dedicated file, not a libs/cache entry, so an opt-out survives upgrades and `databricks cache clear`. - `databricks aitools auto-install disable|enable` toggles the opt-out; enable warns when DATABRICKS_AITOOLS_AUTO_INSTALL=false is set. - A global-scope `databricks aitools uninstall` (including --skills and --agents) opts out. - cli-compat.json entries get an optional auto_install field, resolved by clicompat.ResolveAutoInstall. No entry sets it, and TestManifestLeavesAutoInstallOff keeps it off. Co-authored-by: Isaac <no-reply@databricks.com>
258418b to
c5eef9c
Compare
After a command succeeds, the root PersistentPostRunE installs (or updates) Databricks AI tools for the coding agents on the machine, once per new CLI version. It ships off: no cli-compat entry sets the gate. - autoinstall.Decide is a pure decision over the command, version, state, CI, opt-out, gate and holdout; skipped commands and development builds return before any state read. - A fresh install runs as `aitools install --scope=global` without prompts for agents whose config dir exists; an existing global install runs `aitools update --scope=global`, keeping edited skills. The run has a 10s budget, discards installer output and prints one notice line. - State is updated under OS file locks that are only tried, never waited on beyond 1s. Attempts are retried up to 10 times per version. - Agent CLIs run in their own process group so a timeout stops their descendants. Windows cannot do that, so auto-install skips plugin installs there. - Answering "no" at the `apps init` skills prompt opts out and is passed to the CLI children `apps init` starts. - Telemetry: aitools_install_event.trigger = AUTO_FIRST_RUN and a new aitools_auto_install_event. - Refactor: `aitools update`'s RunE body moves to runUpdate, versioncheck.InCI is exported, and the print_telemetry acceptance helper moves up a directory. Co-authored-by: Isaac <no-reply@databricks.com>
019d40c to
e5662ad
Compare
- A global uninstall opts out only when it removed something, so a mistyped --skills or an empty scope no longer turns auto-install off. - `aitools auto-install <typo>` now fails instead of printing help and exiting 0. - Document auto_install in internal/build/README.md (format, the copy-forward step for new entries, the trust model). - Say that any non-true DATABRICKS_AITOOLS_AUTO_INSTALL value opts out, and move EnvVar next to the other constant. - Add unit tests for the state file; pin the env var in the acceptance test and log the reset steps instead of discarding them. Co-authored-by: Isaac <no-reply@databricks.com>
- apps init children keep DATABRICKS_CLI_SELF_EXEC: ChildEnv takes the environ and composes with agents.SelfExecEnv. - Agent CLIs run through process.BackgroundInGroup, which forwards the first SIGINT/SIGTERM to the child's process group and then re-raises it, so Ctrl-C stops `claude plugin install` again in aitools install/update. - Skips reuse versioncheck.IsExemptCommand, agents.IsCredentialHelperCommand (covers `auth docker token`) and agents.InCI (CI=<provider>, TF_BUILD...), and add Databricks Runtime. Cache-disabled runs no longer read the state. - A gate lookup that only reaches the embedded manifest fails instead of reading as off; it uses up an attempt and does not settle the version. - An unfinished first install is marked in the state, so its skills are finished on a later run instead of being kept as untracked forever. - Auto-install updates no longer migrate or remove raw skills. - New outcomes UP_TO_DATE, KEPT and SKIPPED_PLATFORM; the update notice names only the agents whose skills or plugin changed. - Saying "no" in apps init says that auto-install is off too. - The explicit opt-out writes without the lock where locks are unsupported. - Tests: ResetDeclined, environ passed in; acceptance helpers use LOG.* and no `|| true`. Co-authored-by: Isaac <no-reply@databricks.com>
|
Thanks for the review. All 14 threads are addressed in 879027f; there's a reply on each one. The PR description is updated too. Re question 1 [blocking], the skips:
Once the gate is on, the cost after a script's first command of a version is still there: up to about 20 s, plus retries. I've left that as is, because a script can't be told apart from an agent here. Also: the lumberjack proto needs the new outcomes |
…nstall-first-run Co-authored-by: Isaac <no-reply@databricks.com>
…ests agents.InCI also detects providers such as GITHUB_ACTIONS, so on CI runners every case reported SKIPPED_CI. Co-authored-by: Isaac <no-reply@databricks.com>
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at eafb251. Each finding was verified against the code; simonfaltum may follow up.
No blocking issues found, approving. 13 non-blocking comment(s) and 1 question(s) below.
This PR installs or updates Databricks AI tools after the first successful command of each new CLI version, behind a cli-compat gate that ships off. It also runs agent CLIs in their own process group and relays Ctrl-C to them. Both blocking issues from the last round are fixed, and nothing blocks merging while the gate stays off. The acceptance cases that failed CI on 15cf94d pass locally with the new unset list, but CI hasn't run on this head because the branch conflicts with main now that #7701 merged. Most comments below should be fixed before the gate is turned on for a version that includes this code: skills in agent folders get moved aside, some interrupted or partial installs never finish, and the gate can read off from a cache an older CLI rewrote.
Questions
- Each
Reasonis classified in three places that have to matchDecide's order:Reason.Due,stateFreeSkips(which relies onDecidereturning those reasons for a zeroState) andskipOutcomes. Would one table per reason (due, state-free, reported outcome) be simpler, so a new reason can't be added to one place and missed in another? Or is the split intended? (non-blocking)
Accepted answers
- Skip non-interactive, -o json, DBR and CI runs?: #7702 (comment)
- Process group changes Ctrl-C for the explicit install/update: #7702 (comment)
Agreeing with existing threads
- [nice to have] A first install that only keeps existing plugins is still reported as INSTALLED: #7702 (comment)
| // Skills without install state have no checksums to tell the user's edits | ||
| // from ours, and an install would replace them. The ones an unfinished | ||
| // install of auto-install left are its own. | ||
| if !prior.Installing && installer.HasUntrackedSkills(ctx, installer.ScopeGlobal) { |
There was a problem hiding this comment.
[nice to have] HasUntrackedSkills only looks in ~/.databricks/aitools/skills and ~/.databricks/agent-skills, so this guard misses Databricks skills that live only in an agent's own folder. Once the gate is on, suppose a user copied or edited databricks-* skills into ~/.cursor/skills, by hand or with another skills tool. After an unrelated command, the fresh install's backupThirdPartySkill (installer.go:666) moves them to $TMPDIR/databricks-skill-backup-* and logs only a debug line. Also check each detected agent's global skills folder for databricks* entries that aren't symlinks to the CLI's copy, and settle as KEPT. Alternatively, skip the backup on the auto-install path.
| continue | ||
| } | ||
|
|
||
| if wasInstalled && opts.KeepModified && !skillPrunable(ctx, baseDir, name, scope, cwd, state) { |
There was a problem hiding this comment.
[nice to have] UpdateSkills writes each changed skill's files and saves the state once, at the end. Say an automatic update writes skill A, then runs out the 10 s installTimeout on skill B. It returns before SaveState, so A's new files sit next to A's old version and checksums. On every later automatic update, skillPrunable sees that A's files differ from the record, and KeepModified keeps A as "modified" with a warning. A is never updated automatically again. Save the state after each skill, or record the written skills before returning the error, so a retry recognizes its own writes.
| if err != nil { | ||
| return toolsPlan{}, err | ||
| } | ||
| if existing != nil { |
There was a problem hiding this comment.
[nice to have] A first install's skills batch can fail before it writes anything (the skills manifest returns 5xx, say) while a plugin install succeeds. executePlan then records the plugin, so the state holds plugins and no skills. On the retry existing != nil, so this takes the update path even though prior.Installing is set. runUpdate skips the skills reconcile for a plugins-only state, so the skills agents (Cursor here) never get skills and the version settles as UP_TO_DATE. The description says the installing marker makes the next run finish an install; that does not hold here. When prior.Installing is set, plan a fresh install for the agents that have no tools yet. Alternatively, don't record plugins on the auto-install path until the skills step succeeds.
| // embedded manifest is available: that one predates any remote change of the | ||
| // gate, so it cannot tell off from not known yet. | ||
| func ResolveAutoInstall(ctx context.Context, cliVersion string) (bool, error) { | ||
| m, fallback, err := fetchManifest(ctx) |
There was a problem hiding this comment.
[nice to have] The gate trusts a fresh cached manifest. But writeLocalManifest re-marshals the parsed Manifest, and CLIs from before #7701 have no AutoInstall field, so they write compat-manifest.json without auto_install. With the gate on: a user runs databricks apps init or aitools update on the old CLI, upgrades, and runs any command within the 1 h TTL. Tier 1 returns the old CLI's copy, the gate reads off, and the version settles silently: no install and no event. The lookup runs once per version, so fetch the remote manifest for the gate first, and fall back to the cache only when the fetch fails.
| // Runs without a terminal and runs with JSON output are not skipped, unlike | ||
| // for the notice: coding agents run the CLI that way, and they are what the | ||
| // tools are for. Auto-install writes nothing to stdout. | ||
| func skipsAutoInstall(cmd *cobra.Command) bool { |
There was a problem hiding this comment.
[nice to have] Auto-install runs after any command that returns nil, including one the user stopped with Ctrl-C. apps dev-remote catches SIGINT, shuts down and returns nil (cmd/apps/dev.go:219), and never calls signal.Stop. With the gate on, its first run of a version then runs the gate lookup and an install of up to 10 s. Further Ctrl-C presses, and BackgroundInGroup's re-raised SIGINT, land in that stale channel, so the user can't stop it. Under apps init the parent already died from the Ctrl-C, so the orphaned child prints "Installed Databricks AI tools…" after the shell prompt is back. Skip auto-install when DATABRICKS_CLI_SELF_EXEC is set, as MaybeHint does; the parent's own post-run still sees Declined(). Also let auto-install cancel its own context on SIGINT/SIGTERM for its whole run, not only while an agent CLI runs.
| } | ||
|
|
||
| // isExemptCommand suppresses the notice for commands where it would be noise or | ||
| // IsExemptCommand reports whether cmd gets no notice, as it for commands where it would be noise or |
There was a problem hiding this comment.
[nit] The rename also rewrote this doc comment, which now reads "reports whether cmd gets no notice, as it for commands where…". .agents/rules/comments.md asks not to rewrite existing comments. Change only the identifier: // IsExemptCommand suppresses the notice for commands where it would be noise or….
|
|
||
| `skills` is an exact pin for its CLI range or `"latest"` to install the skills repository's latest published GitHub release. When the latest-release lookup fails or the selected skills manifest is missing, ranges tracking latest use `AgentSkillsFallbackVersion` from `internal/build/clicompat.go`. Pinned ranges use their embedded manifest's skills version as the fallback. Changing either fallback takes effect in the next CLI build. Released binaries keep the fallback they shipped with. | ||
|
|
||
| An entry can also set `"auto_install": true` to turn on the automatic installation of Databricks AI tools on the first run of a new CLI version in its range (`aitools auto-install`). It is off when absent. A change reaches a machine on its first run of a CLI version after the change; machines that resolve to a cached or embedded manifest keep the value they have. |
There was a problem hiding this comment.
[nit] This sentence says machines that resolve to the embedded manifest keep the value they have. ResolveAutoInstall now fails on the embedded fallback, and the run counts an attempt; please update it. The PR description also has two errors. The hook is in cmd/root/root.go, wired in main.go, not cmd/cmd.go. And a Windows machine whose agents all install the plugin through their CLI settles as SKIPPED_PLATFORM (autoinstall_run.go:341), not SKIPPED_NO_AGENTS.
| // On the first run of a new CLI version, install Databricks AI tools. | ||
| // Cobra runs post-run hooks only when the command succeeded. | ||
| if autoinstall.Run != nil { | ||
| autoinstall.Run(cmd) |
There was a problem hiding this comment.
[nit] Auto-install runs inside ExecuteContextC, after startTime. So the gate lookup (up to 3 s now) and, once the gate is on, the install (up to about 20 s) count toward the command's execution_time_ms. The first run of each version then skews per-command latency after every release. Report the auto-install time on aitools_auto_install_event and leave it out of the command's time.
| for dir in $PATH; do | ||
| keep=1 | ||
| for bin in claude cursor-agent codex opencode copilot pi gemini goose kiro; do | ||
| if [ -x "$dir/$bin" ]; then keep=; fi | ||
| for ext in $exts; do | ||
| if [ -f "$dir/$bin$ext" ]; then keep=; fi | ||
| done | ||
| done | ||
| if [ -n "$keep" ]; then filtered_path="$filtered_path:$dir"; fi | ||
| done | ||
| export PATH="${filtered_path#:}" |
There was a problem hiding this comment.
[nit] use_home drops every PATH directory that holds an agent CLI, but print_telemetry needs jq (and print_requests.py needs python3) from PATH. On a machine where jq comes only from the same directory as an agent CLI, every auto-install case fails. An example is Homebrew's /opt/homebrew/bin with an npm-installed claude. Resolve these first, as is done for mkdir/rm/ls. Line 9 also has a run of spaces between CIRCLECI and TRAVIS.
| s.LastVersion = version | ||
| s.Attempts = attempts | ||
| if attempts == 0 { | ||
| s.Installing = false |
There was a problem hiding this comment.
[nit] Installing is cleared only when a version settles. After a first install runs out of attempts, the marker stays set for later versions, and planTools then skips the untracked-skills guard. If skills without install state show up in ~/.databricks/aitools/skills after that (copied by hand, say), the next version's auto-install replaces them. Clear the marker when the attempts run out, or store the version that set it and trust it only for that version.
#7701 landed on main as a squash of this branch's base; its conflicts resolve to this branch's side, as main has not changed those files since. Co-authored-by: Isaac <no-reply@databricks.com>
❌ Integration test reportCommit: b5bc9ed
24 interesting tests: 9 FAIL, 9 RECOVERED, 5 flaky, 1 KNOWN
Top 50 slowest tests (at least 2 minutes):
|
…s#7703) Stacked on databricks#7702, the PR that installs AI tools after the first command of a new CLI version. When a person runs the CLI in a terminal, auto-install asks before its first install. ## Summary ```diff attempt(prior state) plan := planTools() # local only: install state, detected agents + if a person may be at the terminal && no "yes" stored for this version (accepted_version) + no existing install && some agent would get tools + cannot ask -> skip; save the machine ID, nothing installed, + version not settled (a later run asks) + discard type-ahead (cannot -> cannot ask) + settle the version # asked once per version, answered or not + "Install Databricks AI tools for Claude Code, Cursor? [Y/n]: " (1 min) + Enter / y / yes -> store accepted_version (its retries don't ask), continue + n / no -> store opted_out, report DECLINED, + "Skipped. You won't be asked again; to install Databricks AI tools later, run: databricks aitools install" + anything else -> report UNANSWERED, "Skipped. To install ... later, run: databricks aitools install" + no answer -> report UNANSWERED (stdin closed, Ctrl+C, or 1 min passed); not opted out record attempt; install within 10s + failed after a "yes" -> "Could not install Databricks AI tools. To install them, run: databricks aitools install" ``` - "A person may be at the terminal": stdin, stdout or stderr is a terminal and no coding agent is detected. The question is asked only when it can be seen and answered (`cmdio.IsLinePromptSupported`): all three streams are terminals (a redirected stream would hide the question or take the answer from elsewhere, and a pager reading stdout would share the terminal's keystrokes), the process is in the terminal's foreground process group (a background job would stop on SIGTTIN), and no goroutine still reads stdin (`apps dev-remote`, via `cmdio.MarkStdinReadInBackground`). Runs with no terminal at all keep the parent's non-interactive install. - "Asked" is recorded explicitly: a "yes" stores `accepted_version`, and the question and the early skip check it, not the attempt count. That count also advances on a failed gate lookup, which never asks. A planning failure is an ordinary failed attempt whose retry asks. - A run that cannot ask returns before the lock and the gate lookup, and saves the machine ID it drew. The locked path saves a drawn ID before deciding, and `autoinstall.SaveID` returns the stored ID, so concurrent runs agree on the holdout bucket. Holdout machines and gate-off versions wait for a run that can ask, like treatment machines. - A CLI process that the CLI started itself (`DATABRICKS_CLI_SELF_EXEC`, now `agents.IsSelfExec`) does not ask; the parent asks after its own command. - Input typed while the command ran is discarded before asking (`cmdio.DiscardInput`: tcflush on Linux and macOS, FlushConsoleInputBuffer on Windows), and Ctrl+C at the question counts as no answer, so the command keeps its exit status. Where input can't be discarded (a pipe that Git Bash's mintty counts as a terminal), the question is skipped. A timed-out or interrupted question also turns off later line prompts in the process, since its read stays blocked on stdin. - The question is a plain-text line (`cmdio.AskCancellable`), so `NO_COLOR`, `TERM=dumb` and Git Bash, which turn off the TUI prompts, still get it. `cmdio.HasTerminal` and the new `cmdio.IsLinePromptSupported` express the two checks. - Only a fresh install asks. Updates of an existing installation and retries after a "yes" do not. - The question names only the agents that will get something (skills, or a plugin whose CLI is on PATH); on Windows that excludes agents installed through their CLI, which the plan already filters out. With none, there is no question. - An agent in a pseudo-terminal that the SDK does not recognize waits at most a minute, once per version. - `DECLINED` and `UNANSWERED` added to the auto-install outcome enum; the Universe proto for the whole event is https://github.com/databricks-eng/universe/pull/3127275. `auto-install --help` mentions the prompt. Review order: `cmd/aitools/autoinstall_run.go` (`autoInstall`'s early skip and ID save, `attempt`, `canAsk`, `skipUnaskable`, `ask`, `decline`), `libs/aitools/autoinstall` (`AcceptedVersion`, `Accept`, `SaveID`), then `libs/cmdio/io.go` and `libs/cmdio/terminal_*.go`, then `cmd/aitools/autoinstall_run_test.go`. ## Evidence The prompt needs a terminal, which acceptance tests lack, so `TestAutoInstallPrompt` (cmd/aitools) drives the hook through a test terminal and checks the prompt and notice on stderr, the skill in Cursor's dir, the stored state and the event: ```text enter accepts -> installed, INSTALLED, interactive=true yes accepts -> installed no declines and opts out -> no skill, opted_out=true, DECLINED no answer settles the version without opting out -> no skill, last_version=1.30.0, UNANSWERED another answer settles the version without opting out -> no skill, not opted out, UNANSWERED a planning failure is a failed attempt, retried later -> no prompt, FAILED, attempts=1 a failed install after yes says so -> "Could not install ...", FAILED, accepted_version=1.30.0 NO_COLOR still asks -> prompt, Enter installs TERM=dumb still asks -> prompt, "n" opts out, DECLINED redirected stderr skips without installing -> no prompt, no skill, no last_version, no event piped stdin skips without installing -> no prompt, no skill, no last_version, no event stdout to a pager skips without installing -> no prompt, no skill, no last_version, no event no terminal at all installs without asking -> no prompt, installed a retry after yes is not asked again -> no prompt, installed a retry that was never asked asks -> (attempts=1 from a gate failure) prompt, installed a retry that was never asked skips when it cannot ask -> no install, attempts=1 kept a CLI process started by the CLI does not ask -> no prompt, no skill, version not settled a holdout machine that cannot ask waits too -> no event, version not settled (every case: stdout empty) ``` `TestAutoInstallKeepsTheIDWhenItCannotAsk` runs a partial-terminal hook twice on a machine with no stored ID: the ID is saved and stays the same, and the version is not settled. `TestToolsPlanNewAgents` covers which agents are named. `TestCapabilities_SupportsLinePrompt` and `TestIsLinePromptSupported` (libs/cmdio) cover the stream checks and the background stdin reader. On a real terminal (tmux, Linux, `databricks auth profiles` with the gate cached on): ```text cli auth profiles & -> job ends "Done", no prompt, no "Stopped (tty input)", version not settled sleep 2; cli ... (typed "n⏎" in sleep) -> prompt still waits: the type-ahead was discarded Ctrl+C at the prompt -> "^C", newline, EXIT=0, last_version=1.30.0, not opted out state {last_version:1.30.0, attempts:1} -> the next foreground run asks (after the restack onto eafb251) ``` With Enter mapped to "no" (a temporary edit, not committed) `enter accepts` fails, so the test catches the default. Locally on Linux at 957cbf0 (after merging the base at eafb251): unit tests of `cmd/aitools` (also shuffled), `cmd/apps`, `cmd/root`, `libs/cmdio` (also shuffled), `libs/aitools/...`, `libs/apps/...`, `libs/telemetry/...` and `libs/process/...` pass, and so do the `cmd/aitools` and `cmd/apps` acceptance tests. golangci-lint, whitespace and deadcode pass. `GOOS=windows`/`darwin` build and vet pass, and so does a `GOOS=freebsd` vet of `libs/cmdio`. I did not run the full unit and acceptance suites. ## Merge Danger **Door:** two-way Behind the same off gate as the parent. **Blast Radius:** small Once the gate is on, a person in a terminal sees one prompt after the first command of a version that has nothing installed; the command waits for the answer (at most a minute). A run with only some streams on a terminal installs nothing until a run in a full terminal. On the same first run, a Claude Code user without skills also sees the existing install hint (unchanged) above the "Installed ..." line. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…cks#7802) ## Changes `TestRecordKeepsConcurrentOptOut` starts 21 writers (20 `Record` + 1 `SetOptedOut`) that take the state lock one at a time. Each writer waits at most `stateLockWait` (1s) for the lock. On slow runners, mostly Windows, each locked read-and-rename takes long enough that the last writers time out with `held by another CLI process`. The failing Windows attempt ran for 3.83s, about 180ms per writer. The test was added in databricks#7702. This PR makes `stateLockWait` a package var and raises it to a minute in this test. Production behavior is unchanged: the wait is still 1s, which is plenty for the few CLI processes that contend for the lock in real use. Decision: I kept the 20 concurrent `Record` calls. Using fewer writers would make the flake rarer without fixing it, and it would make the test less likely to catch a lost opt-out. ## Why Flaky on main and in unrelated PRs: 12 failures in 3 days, 9 of them on Windows. ## Tests - A temporary local test (not committed) wrapped `tryLock` so each holder slept 60ms. It failed every run before this change (5 "held by another CLI process" errors per run) and passed after. - `go test -race -count=3 ./libs/aitools/autoinstall/` passes. ## Dashboard - [error/3669](http://go/cli/dashboard/error/3669?main=1&pr=1&end=2026-11-19T12:00:00Z&last=3d) - [libs/aitools/autoinstall.TestRecordKeepsConcurrentOptOut](http://go/cli/dashboard/test/libs/aitools/autoinstall.TestRecordKeepsConcurrentOptOut?main=1&pr=1&end=2026-11-19T12:00:00Z&last=3d) ``` Error: Received unexpected error: [PATH]/.databricks/aitools/auto-install.json.lock: held by another CLI process Test: TestRecordKeepsConcurrentOptOut --- FAIL: TestRecordKeepsConcurrentOptOut (3.83s) ``` This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
Stacked on #7701, the PR that stores the auto-install state and adds the opt-out commands and the cli-compat gate.
After a command succeeds, the CLI installs (or updates) Databricks AI tools for the coding agents on the machine, once per new CLI version. Ships off: nothing sets the gate yet.
Summary
last_version = current, attempts = 0and clearsinstalling: each machine reports at most one outcome per version, plus one per retry. A new version resets the retry budget.flock/LockFileEx) on files that are never removed. They are only tried, never waited on. The state lock is retried for up to 1 s while another process holds it; any other lock error returns at once. A busy run lock skips the attempt.process.BackgroundInGroup), so a timeout stops their descendants too. That takes them out of the terminal's foreground group, so while one runs, the first SIGINT/SIGTERM is caught, stops the group, and is raised again in the CLI: Ctrl-C endsaitools install/updateas before, without orphans. Windows has no such stop, so there auto-install runs no agent CLI: a fresh install drops its plugin installs (skills are still installed), and an existing install whose update would touch plugins is left alone. A Windows machine with only plugin-capable agents settles asSKIPPED_NO_AGENTS.apps initskills prompt opts out, and says so with theauto-install enablecommand. It is stored, honored in the process even if the write fails, and passed asDATABRICKS_AITOOLS_AUTO_INSTALL=falseto theapps deploy/apps dev-remotechildren thatapps initstarts, next to theirDATABRICKS_CLI_SELF_EXEC=1.version,help, completion), the credential helpers (auth token,auth docker token) and theaitoolstrees. Non-interactive and-o jsonruns are not skipped, unlike for the notice: coding agents run the CLI that way, and auto-install writes nothing to stdout. Databricks Runtime is skipped (ephemeral home, platform-managed CLI).clicompat.ResolveAutoInstallfails): the version is not settled, and the lookup uses up one of the version's 10 attempts, so an offline machine stops paying for it.HOLDOUT/TREATMENT) is reported.DATABRICKS_AITOOLS_AUTO_INSTALL_CLI_VERSIONto simulate releases and upgrades.aitools_install_event.trigger = AUTO_FIRST_RUN(unset for the command's own installs, so existing goldens do not change) and a newaitools_auto_install_event {outcome, holdout_arm, interactive, error_category}, documented inlibs/telemetry/protos/README.md. Outcomes:INSTALLED,UP_TO_DATE,KEPT,SKIPPED_CI,SKIPPED_OPTOUT,SKIPPED_HOLDOUT,SKIPPED_NO_AGENTS,SKIPPED_PLATFORM,FAILED,RETRIES_EXHAUSTED; the lumberjack proto needs the same values.aitools update's RunE body moves torunUpdate;agents.InCI,agents.IsCredentialHelperCommandandversioncheck.IsExemptCommandare exported for reuse; the acceptance helperprint_telemetrymoves up a directory.Known limitations
Both must be fixed before the gate is turned on:
aitools uninstallthat runs while an automatic install is in flight can see that attempt recreate the tools. Running uninstall again removes them.INSTALLED, and later updates never add the missing agent. The same applies to the agents the Windows rule above leaves out. One fix is to record the targeted agents inauto-install.json. (An install cut short by the timeout is different: itsinstallingmarker makes the next run finish it.)Review order
libs/aitools/autoinstall/decide.go: the decision, pure.cmd/aitools/autoinstall_run.go: the hook,planTools, reporting.libs/aitools/autoinstall/run.go,lock_unix.go,lock_windows.go,state.go: locks, attempts,Decline/ChildEnv.cmd/cmd.go,cmd/apps/init.go,libs/aitools/agents/recommend.go: the wiring.cmd/aitools/update.go(refactor),libs/aitools/installer/update.go(KeepModified,HasUntrackedSkills),libs/aitools/installer/plugin.go(process group),libs/telemetry/protos/.Evidence
New acceptance tests in
acceptance/cmd/aitools/auto-install/rundatabricks jobs listagainst the fake server and a fake skills source, with a temporary HOME and agent dirs:first-run:upgrade: 1.30.0 installs skills release 0.3.0; 1.31.0 with release 0.4.0 printsUpdated Databricks AI tools for Cursor., updates one skill and keeps a skill the user edited; 1.32.0 while the latest release is 0.2.0 refuses the downgrade (files unchanged, nothing printed, settled asKEPT, not retried); a 1.31.0 CLI run afterwards does nothing.skipped:CI=true(reported once per version),DATABRICKS_AITOOLS_AUTO_INSTALL=false,auto-install disable,version/aitools/completion/__complete/help,DATABRICKS_CACHE_ENABLED=false, gate off (nothing reported, version settled), holdout machine, no agent config dir.retries: nineFAILEDattempts,RETRIES_EXHAUSTEDon the tenth, nothing on the eleventh, a fresh install on the next version.TestDecide(table-driven over every input);TestSkipsAutoInstall;TestPlanToolsRunsNoAgentCLIWhereTimeoutsCannotStopThem,TestPlanToolsReportsWhyNothingIsInstalledandTestPlanToolsKeepsSkillsWithoutInstallState(also: an unfinished install of ours is finished);TestChangedAgents;TestTryLockExcludesOtherRuns,TestLoadWaitsForStateLock,TestLoadFailsWhenStateLockStaysHeld,TestLockReturnsOtherErrorsAtOnce,TestRecordKeepsConcurrentOptOut,TestSetOptedOutWithoutLocks,TestRecordClearsInstallingWhenSettled;TestChildEnvCarriesTheRefusal,TestSelfExecEnvCarriesBothMarkers(theapps initchildren's env);TestResolveAutoInstallFailsOnTheEmbeddedFallback;TestBackgroundInGroupForwardsInterrupt(a helper process interrupted mid-run dies by SIGINT and its grandchild is reaped);TestRunAgentCmdTimeoutStopsDescendants;TestUpdateKeepModifiedKeepsEditedSkill.cmd/...andlibs/..., the aitools and apps acceptance suites, lint and whitespace checks pass on Linux;GOOS=windows/darwinbuild and vet pass. The gate-lookup failure has no acceptance test: the manifest URL has no override, so it is covered by the clicompat and state unit tests.Merge Danger
Door: two-way while the gate is off
With the gate off it reads one small file and settles the version on the first run of each release, sends
SKIPPED_CI/SKIPPED_OPTOUTevents when those apply, and does nothing else. Turning the gate on (acli-compat.jsonedit on main) is the one-way step: installs it causes are not rolled back by turning it off again.Blast Radius: every command (post-run)
aitools install/updatetoday: agent CLIs run in their own process group, with Ctrl-C forwarded to it.~/.databricks/aitools/auto-install.jsonand two lock files next to it.This pull request and its description were written by Isaac.