Repository navigation
aitools: ask before auto-installing AI tools in a terminal - #7703
Conversation
✅ Integration test reportCommit: 1fbc949
Top 5 slowest tests (at least 2 minutes):
|
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at a139569. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 1 blocking issue(s), tagged [blocking].
This PR adds a [Y/n] question before auto-install's first install of a version, asked only when stdin, stdout and stderr are all terminals and no coding agent is detected. The approach is sound, but the new skip path for partial-terminal runs returns before the machine ID is saved, so new machines get a new holdout bucket on every such run and drift into the control group. That one needs a fix before merge. Smaller issues: typed-ahead input can answer the question and opt the user out for good, background jobs and Ctrl+C at the prompt behave badly, and partial-terminal runs repeat the gate lookup on every run. CI is green.
Questions
DECLINEDis a new value ofAitoolsAutoInstallOutcome, which mirrors the Universe lumberjack proto inproto/logs/frontend/databricks_cli. Neither this PR nor #7702 links a Universe change. Is there one that addsDECLINED(or the wholeaitools_auto_install_event)? Until the backend knows the value, declines may be dropped or read as unspecified. (non-blocking)
Accepted answers
- Minute-long wait for unrecognized agents in a pseudo-terminal
- Skip instead of install when only some streams are terminals
Comments outside the changed lines
- [nice to have]
cmd/aitools/autoinstall_run.go:123: When only some streams are terminals,attemptreturns at line 175 without recording anything. Every later run of that kind repeats the two state loads, the machine-wide lock, this gate lookup andplanTools, until the user runs one command with all three streams on a terminal. Before this PR that work happened once per version. The gate lookup refetches the manifest from GitHub whenever the 1-hour cache is stale, bounded by the 3sgateTimeout. A failed fetch reuses the stale file without refreshing its time. So on a network that drops the request, every such run after the first hour waits up to 3s at exit. Check the streams before taking the lock and resolving the gate: ifautoInstallAsks(ctx) && !cmdio.IsLinePromptSupported(ctx)and there is no global install state, save the ID (see the comment on line 175) and return. - [nice to have]
libs/telemetry/protos/aitools_events.go:218: The auto-install event's docs no longer match what the code sends.Interactiveis documented ascmdio.IsPromptSupportedwith no agent, but it is nowcmdio.HasTTYwith no agent. It is true for runs that never prompt (updates, retries, NO_COLOR), so it can't tell a run that asked from one that didn't.FAILED(line 187) says a later run of the same version retries, butsettleFailedreportsFAILEDand settles the version. An unanswered prompt (timeout or EOF) settles the version with no event at all, while the protos README says the event is logged once per version and machine, so a prompt nobody answered doesn't show up next to the holdout arm. Update the two comments and the README. Also consider reporting an outcome for an unanswered prompt, or a field that says the user was asked.
| log.Debugf(ctx, "auto-install: skipped: cannot ask, stdin, stdout or stderr is not a terminal") | ||
| return |
There was a problem hiding this comment.
[blocking] On a machine with no stored ID, this early return throws away the ID that autoInstall just generated (NewID(), line 111). Only Record writes State.ID, and this path never calls it. So each later run with a piped or redirected stream draws a new ID and a new bucket. 90% of the draws return here again. The first draw that lands in the 10% holdout settles as SKIPPED_HOLDOUT and saves that ID for good. A new machine whose first 10 eligible runs look like $(databricks …) or databricks … | jq therefore ends up in holdout about 65% of the time instead of 10%. That skews the control group, and those users are never asked in any later version. The tests for this path seed {"id": "treatment-machine"}, so they miss it. Save the ID here without settling the version, for example with a small autoinstall.SaveID that sets s.ID only when it is empty. Also add a test case with no seeded state.
There was a problem hiding this comment.
Fixed in cdf01c0. A run that cannot ask now saves the ID it drew (new autoinstall.SaveID, which sets the ID only when none is stored and leaves the version unsettled), so later runs keep the bucket. Added TestAutoInstallKeepsTheIDWhenItCannotAsk, which runs twice with no seeded ID and checks that the ID stays the same and the version is not settled. I also took the out-of-diff suggestion: the check runs before the lock and the gate lookup (autoInstall, right after the first Decide), so these runs now cost one state read plus the local planTools, with no manifest fetch. The check under the lock stays, because the state can change between the two reads.
| switch strings.ToLower(strings.TrimSpace(ans)) { | ||
| case "", "y", "yes": | ||
| return true | ||
| } | ||
| r.decline(ctx) |
There was a problem hiding this comment.
[nice to have] The answer is read straight after a command that may have run for minutes, and nothing discards what was typed while it ran. Type-ahead therefore answers the question before the user sees it. Say the user types the next command during a bundle deploy, e.g. git status⏎. The prompt reads it as the answer. It is neither "" nor y/yes, so it stores a permanent opt-out and prints "You won't be asked again", and the shell never runs git status. A typo like yess has the same effect, and a stray Enter installs. Treat only n/no as a decline, and treat anything else as no answer (or ask again once). Also consider discarding pending input before asking: tcflush with TCIFLUSH on Unix, FlushConsoleInputBuffer on Windows. The doc comment on ask ("Only "no" stores the opt-out") already describes the safer rule.
There was a problem hiding this comment.
Fixed in cdf01c0. Only n/no decline now. Any other answer settles the version without opting out, reports the new UNANSWERED outcome and prints "Skipped. To install Databricks AI tools later, run: databricks aitools install". Pending input is also discarded before asking (cmdio.DiscardInput: TCFLSH/TCIFLUSH on Linux, TIOCFLUSH/FREAD on macOS, FlushConsoleInputBuffer on Windows), which covers the stray Enter too. Checked on a real terminal in tmux: sleep 2; databricks auth profiles with n⏎ typed during the sleep still waits at the prompt.
| } | ||
| actx, cancel := context.WithTimeout(ctx, promptTimeout) | ||
| defer cancel() | ||
| ans, err := cmdio.AskCancellable(actx, "Install Databricks AI tools for "+strings.Join(names, ", ")+"? [Y/n]", "") |
There was a problem hiding this comment.
[nice to have] The prompt doesn't handle job control or Ctrl+C, so two common terminal actions now end badly after a command that succeeded. (1) databricks jobs list & from an interactive shell has the terminal on all three streams, so it prompts. A background process group that reads the terminal gets SIGTTIN and stops ("Stopped (tty input)"), and promptTimeout can't fire while it is stopped. (2) The CLI installs no SIGINT handler, so pressing Ctrl+C to dismiss the question kills the process. The shell sees exit status 130 for a command that succeeded, so a make target or && chain stops. The command's own telemetry upload in root.Execute also never runs. Ask only when the process is in the terminal's foreground process group (tcgetpgrp on stdin equals getpgrp). Consider treating SIGINT during the prompt as no answer, e.g. with signal.NotifyContext around AskCancellable, so the command keeps its exit status.
There was a problem hiding this comment.
Fixed in cdf01c0. (1) cmdio.IsLinePromptSupported now also requires the process to be in the terminal's foreground process group (TIOCGPGRP == getpgrp; always true on Windows). A background run takes the skip path, which saves the ID and doesn't settle the version. In tmux, databricks auth profiles & now ends as Done, with no prompt and no Stopped (tty input). (2) The prompt runs under signal.NotifyContext(..., os.Interrupt), so Ctrl+C counts as no answer (UNANSWERED, not opted out). In tmux, Ctrl+C at the prompt prints ^C and a newline, and the command exits 0.
| // per version even when it is never answered: the process is killed, stdin | ||
| // is closed, or nobody answers within promptTimeout (an unrecognized agent in | ||
| // a pseudo-terminal). Only "no" stores the opt-out. | ||
| func (r autoInstallRun) ask(ctx context.Context, plan []agentPlanItem) bool { |
There was a problem hiding this comment.
[nit] The prompt assumes nothing else is reading stdin. Without --auto-approve, apps dev-remote starts readStdinLines on os.Stdin and returns nil on Ctrl+C, and Bridge.Stop doesn't end that read. When it is the first eligible command of a version, the prompt runs while that goroutine is still blocked on stdin, so the first line the user types can go to the bridge's reader instead. Its signal.Notify also stays registered, so Ctrl+C does nothing at the prompt and the user waits out the minute. Skip the prompt after commands that read stdin in the background, or have the bridge stop its reader before returning.
There was a problem hiding this comment.
Fixed in cdf01c0, using the first option. New cmdio.MarkStdinReadInBackground(), which the bridge calls before it starts readStdinLines. After that, IsLinePromptSupported is false for the rest of the process, so the run takes the skip path and a later command asks.
| if err := autoinstall.SetOptedOut(ctx, true); err != nil { | ||
| log.Warnf(ctx, "Could not save the answer, so a later CLI version asks again: %v", err) | ||
| } | ||
| r.settle(ctx, protos.AitoolsAutoInstallOutcomeDeclined) | ||
| cmdio.LogString(ctx, "Skipped. You won't be asked again; to install Databricks AI tools later, run: databricks aitools install") |
There was a problem hiding this comment.
[nit] decline calls autoinstall.SetOptedOut instead of autoinstall.Decline, the helper apps init uses, which also sets the in-process flag. When saving fails, stderr shows Could not save the answer, so a later CLI version asks again at Warn, which is the default log level, followed by Skipped. You won't be asked again. Call autoinstall.Decline(ctx), and print the "won't be asked again" line only when saving succeeded.
There was a problem hiding this comment.
Fixed in cdf01c0. It now calls autoinstall.Decline. When saving fails, it prints the warning and "Skipped. To install Databricks AI tools later, run: databricks aitools install", without "won't be asked again". To run several declines in one test process I added autoinstall.ResetDeclined for t.Cleanup.
|
|
||
| // HasTTY reports whether stdin, stdout or stderr is a terminal, so that a | ||
| // person may be running the command. | ||
| func HasTTY(ctx context.Context) bool { |
There was a problem hiding this comment.
[nit] HasTTY duplicates cmdio.HasTerminal (line 121 of this file, used by libs/aitools/agents/hint.go), which already reports whether stdin, stdout or stderr is a terminal. Likewise, NewTestIOWithInputTTY in testing.go does what cmdio.NewIO(ctx, flags.OutputText, cmdio.FakeTTYReader(in), out, err, "", "") already does. Use the existing helpers and drop the new ones, including Capabilities.HasTTY.
There was a problem hiding this comment.
Fixed in cdf01c0. I switched to cmdio.HasTerminal and NewIO with FakeTTYReader, and dropped HasTTY, Capabilities.HasTTY, their test and NewTestIOWithInputTTY.
| names := make([]string, len(plan)) | ||
| for i, it := range plan { | ||
| names[i] = it.agent.DisplayName | ||
| } |
There was a problem hiding this comment.
[nit] The question names every agent in the plan, including items that will get nothing. An agent whose CLI is not on PATH but that already has the plugin is planned as deliveryPlugin with cliMissing, and executePlan only records it as kept. If that is the only agent, the user is asked "Install Databricks AI tools for Claude Code?" for something already installed, and "n" opts them out for good. Name only the items that install something (skills, or a plugin whose CLI is present), and skip the question when none do.
There was a problem hiding this comment.
Fixed in cdf01c0. New toolsPlan.newAgents names only skills items and plugins whose CLI is on PATH. The question lists those, and when it is empty there is no question. Covered by TestToolsPlanNewAgents.
|
|
||
| // The interactive prompt needs a terminal, which acceptance tests do not | ||
| // have, so these cases drive the hook the root command runs with stdin as a | ||
| // test terminal; the rest of the terminal is detected as in a real run. Like the acceptance tests in acceptance/cmd/aitools/auto-install, |
There was a problem hiding this comment.
[nit] This comment line is 138 columns long: "Like the acceptance tests in acceptance/cmd/aitools/auto-install," was appended without rewrapping. Rewrap the paragraph.
First of three changes that install Databricks AI tools automatically on the first run of a new CLI version. This one stores the state and the opt-out and adds the remote gate; nothing installs yet. - ~/.databricks/aitools/auto-install.json holds last_version, attempts and opted_out. It is a dedicated file, not a libs/cache entry, so an opt-out survives upgrades and `databricks cache clear`. - `databricks aitools auto-install disable|enable` toggles the opt-out; enable warns when DATABRICKS_AITOOLS_AUTO_INSTALL=false is set. - A global-scope `databricks aitools uninstall` (including --skills and --agents) opts out. - cli-compat.json entries get an optional auto_install field, resolved by clicompat.ResolveAutoInstall. No entry sets it, and TestManifestLeavesAutoInstallOff keeps it off. Co-authored-by: Isaac <no-reply@databricks.com>
After a command succeeds, the root PersistentPostRunE installs (or updates) Databricks AI tools for the coding agents on the machine, once per new CLI version. It ships off: no cli-compat entry sets the gate. - autoinstall.Decide is a pure decision over the command, version, state, CI, opt-out, gate and holdout; skipped commands and development builds return before any state read. - A fresh install runs as `aitools install --scope=global` without prompts for agents whose config dir exists; an existing global install runs `aitools update --scope=global`, keeping edited skills. The run has a 10s budget, discards installer output and prints one notice line. - State is updated under OS file locks that are only tried, never waited on beyond 1s. Attempts are retried up to 10 times per version. - Agent CLIs run in their own process group so a timeout stops their descendants. Windows cannot do that, so auto-install skips plugin installs there. - Answering "no" at the `apps init` skills prompt opts out and is passed to the CLI children `apps init` starts. - Telemetry: aitools_install_event.trigger = AUTO_FIRST_RUN and a new aitools_auto_install_event. - Refactor: `aitools update`'s RunE body moves to runUpdate, versioncheck.InCI is exported, and the print_telemetry acceptance helper moves up a directory. Co-authored-by: Isaac <no-reply@databricks.com>
When a person runs the CLI in a terminal, auto-install asks before its first install of a version: "Install Databricks AI tools for <agents>? [Y/n]" (1 minute). - The question is asked only when stdin, stdout and stderr are all terminals and no coding agent is detected. If only some streams are terminals, nothing is installed and the version is not settled, so a later run asks. Runs with no terminal keep the non-interactive install. - Enter/y/yes installs; any other answer stores the opt-out and reports DECLINED; no answer stops without opting out. - Only a fresh install asks; updates and retries after an accepted, failed attempt do not. A planning failure on the first attempt settles the version and reports FAILED. - The question is a plain-text line (cmdio.AskCancellable), so NO_COLOR, TERM=dumb and Git Bash still get it. New cmdio.IsLinePromptSupported and cmdio.HasTTY express the stream checks. Co-authored-by: Isaac <no-reply@databricks.com>
019d40c to
e5662ad
Compare
a139569 to
12bfe17
Compare
- A global uninstall opts out only when it removed something, so a mistyped --skills or an empty scope no longer turns auto-install off. - `aitools auto-install <typo>` now fails instead of printing help and exiting 0. - Document auto_install in internal/build/README.md (format, the copy-forward step for new entries, the trust model). - Say that any non-true DATABRICKS_AITOOLS_AUTO_INSTALL value opts out, and move EnvVar next to the other constant. - Add unit tests for the state file; pin the env var in the acceptance test and log the reset steps instead of discarding them. Co-authored-by: Isaac <no-reply@databricks.com>
Address review on #7703: - A run that cannot ask saves the machine ID it drew (autoinstall.SaveID), so partial-terminal runs no longer redraw the holdout bucket. It now skips before the lock and the gate lookup. - Only "n"/"no" declines; any other answer, EOF, a timeout or Ctrl+C is reported as UNANSWERED and settles the version without opting out. - Discard type-ahead before asking, ask only in the terminal's foreground process group, and treat Ctrl+C at the prompt as no answer. - Skip the prompt when a goroutine still reads stdin (apps dev-remote). - Use autoinstall.Decline and promise "won't be asked again" only when the opt-out was saved. - Name only agents that would get something; reuse cmdio.HasTerminal and FakeTTYReader instead of new helpers. - Fix the auto-install event docs. Co-authored-by: Isaac <no-reply@databricks.com>
|
@simonfaltum thanks for the review. Every inline comment is addressed in cdf01c0, with a reply on each thread. On the review body:
Real-terminal checks (tmux) are in the PR description under Evidence. |
- apps init children keep DATABRICKS_CLI_SELF_EXEC: ChildEnv takes the environ and composes with agents.SelfExecEnv. - Agent CLIs run through process.BackgroundInGroup, which forwards the first SIGINT/SIGTERM to the child's process group and then re-raises it, so Ctrl-C stops `claude plugin install` again in aitools install/update. - Skips reuse versioncheck.IsExemptCommand, agents.IsCredentialHelperCommand (covers `auth docker token`) and agents.InCI (CI=<provider>, TF_BUILD...), and add Databricks Runtime. Cache-disabled runs no longer read the state. - A gate lookup that only reaches the embedded manifest fails instead of reading as off; it uses up an attempt and does not settle the version. - An unfinished first install is marked in the state, so its skills are finished on a later run instead of being kept as untracked forever. - Auto-install updates no longer migrate or remove raw skills. - New outcomes UP_TO_DATE, KEPT and SKIPPED_PLATFORM; the update notice names only the agents whose skills or plugin changed. - Saying "no" in apps init says that auto-install is off too. - The explicit opt-out writes without the lock where locks are unsupported. - Tests: ResetDeclined, environ passed in; acceptance helpers use LOG.* and no `|| true`. Co-authored-by: Isaac <no-reply@databricks.com>
…nstall-first-run Co-authored-by: Isaac <no-reply@databricks.com>
…ests agents.InCI also detects providers such as GITHUB_ACTIONS, so on CI runners every case reported SKIPPED_CI. Co-authored-by: Isaac <no-reply@databricks.com>
|
Universe proto for the auto-install event (including |
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at cdf01c0. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 1 blocking issue(s), tagged [blocking].
This PR asks before auto-install's first install of a CLI version, and only when a person can see and answer the question. Every fix from the earlier review is in and works, and CI is green. The PR now conflicts with its moved base branch, though. On that base a failed gate lookup uses up an attempt, so after the restack a run would install without asking, because the question is gated on attempt == 1. That needs an explicit "asked" record before merge. The other findings (Git Bash type-ahead, prompts in child CLI processes, a silent failure after "yes", an ID race outside the lock) are non-blocking.
Accepted answers
- Does the Universe proto know DECLINED/UNANSWERED?: #7703 (comment)
- Why does a planning failure before the prompt settle the version?
Comments outside the changed lines
- [nice to have]
cmd/aitools/autoinstall_run.go:220: After the user answers yes, a failed install shows nothing:attemptonly writes the debug log and the event, andquietContextdiscards the installer's output. Scenario: a user presses Enter on a machine where the GitHub download fails. They wait up toinstallTimeout(10s), then get their shell back with no message, so they can't tell that nothing was installed. When the run asked, print one line on failure, e.g. "Could not install Databricks AI tools. To install them, run: databricks aitools install".
| // A terminal user is asked before the first install of a version, once: | ||
| // a retry was answered "yes" (or ran non-interactively), so it is not | ||
| // asked again. | ||
| if r.interactive && attempt == 1 { |
There was a problem hiding this comment.
[blocking] The base branch has moved to eafb251 and this PR now conflicts with it, so it needs a restack. On the new base, a failed gate lookup stores attempts = NextAttempt(...) without asking. So attempt == 1 here, and in.State.LastVersion != version in the early skip at line 102, no longer mean "not asked yet". Scenario: the first run of a version is offline, or the manifest fetch passes the 3s gateTimeout, so attempts=1 is stored. The next run in a terminal is attempt 2: it skips the question and installs. A run with piped stdout also gets past the early skip and installs. Record explicitly that this version was asked, e.g. a state field that ask sets before the question (or a "yes" answer), and gate the question and the early skip on that instead of the attempt count. A planning error could then also retry instead of settling the version.
There was a problem hiding this comment.
Fixed in 957cbf0: I merged the new base (eafb251) into the branch and resolved the conflicts. Added State.AcceptedVersion. A "yes" at the prompt records it (autoinstall.Accept), and both the question and the early skip now check AcceptedVersion != version instead of attempt == 1 or LastVersion != version. So a retry after a failed gate lookup asks, and a retry after a "yes" doesn't. The version is still settled right before the question, so a process killed at the prompt doesn't install later. I also took the follow-on: a planning error is now an ordinary failed attempt (FAILED, attempts=1) whose retry asks, and settleFailed is gone. New cases in TestAutoInstallPrompt: "a retry that was never asked asks", "a retry that was never asked skips when it cannot ask" (piped stdout, attempts=1, no install), and "a retry after yes is not asked again". I also checked on a real terminal: with {"last_version":"1.30.0","attempts":1} stored, the next foreground run shows the question.
| if err := cmdio.DiscardInput(ctx); err != nil { | ||
| log.Debugf(ctx, "auto-install: could not discard the pending input: %v", err) | ||
| } |
There was a problem hiding this comment.
[nice to have] In Git Bash under mintty, stdin is a pipe that isatty.IsCygwinTerminal counts as a terminal, so the question is asked. But FlushConsoleInputBuffer fails on a pipe handle, and the error is only logged. A stray Enter typed while the command ran is then read as the default "yes" and installs without the user seeing the question; a typed-ahead n opts out for good. Do the discard before Record, and treat a failure as "cannot ask" (skipUnaskable). Alternatively, on Windows, drain a non-console handle with PeekNamedPipe + ReadFile.
There was a problem hiding this comment.
Fixed in 957cbf0, using the first option. ask now discards pending input before it settles the version, and if the discard fails it takes skipUnaskable (ID saved, version not settled, no question). So in mintty with a pipe stdin, the question is skipped and never answered by type-ahead. I didn't do the PeekNamedPipe drain: x/sys/windows doesn't wrap it, and I can't test a hand-rolled kernel32 call here. Happy to add it in a follow-up if you'd rather mintty users get the question.
| func autoInstallAsks(ctx context.Context) bool { | ||
| return cmdio.HasTerminal(ctx) && useragent.AgentProvider() == "" | ||
| } |
There was a problem hiding this comment.
[nice to have] autoInstallAsks does not skip CLI processes that the CLI starts itself, unlike MaybeHint, which checks DATABRICKS_CLI_SELF_EXEC. Scenario: on the first run of a version, databricks apps init --deploy runs in flags mode (or in a NO_COLOR or Git Bash terminal, where RecommendSkillsInstall only prints the tip). It runs the child apps deploy on the same terminal. When the deploy succeeds, the child asks "Install Databricks AI tools for …? [Y/n]" and waits up to a minute, before the parent prints "Deploy complete" and moves on to the run step. Skip auto-install, or at least the question, in a self-exec child, and leave it to the parent's own run. (On this PR's base, autoinstall.ChildEnv() also overwrites SelfExecEnv at those apps init call sites; the current base tip fixes that.)
There was a problem hiding this comment.
Fixed in 957cbf0, and it covers the question as well as the install: a self-exec child doesn't ask and doesn't install. canAsk now requires !agents.IsSelfExec(ctx), a new exported check beside SelfExecEnv, which MaybeHint now uses too. The child takes the cannot-ask skip: ID saved, version not settled, nothing installed. The parent then asks after its own command. New case: "a CLI process started by the CLI does not ask". The base's ChildEnv/SelfExecEnv fix comes in with the merge.
| // every such run until a run in a terminal asks. | ||
| if reason == autoinstall.ReasonGateOff && in.State.LastVersion != version && autoInstallAsks(ctx) && !cmdio.IsLinePromptSupported(ctx) { | ||
| if tools, err := planTools(quietContext(ctx)); err == nil && len(tools.newAgents()) > 0 { | ||
| skipUnaskable(ctx, in.State.ID, autoinstall.NewID()) |
There was a problem hiding this comment.
[nice to have] skipUnaskable saves the ID here without holding the run lock. Scenario: on a machine with no stored ID, another process holds the lock. It has drawn its own ID at lines 121–125 and decided the holdout arm with it. This SaveID stores a different ID first, and that run's Record keeps it, because s.ID is no longer empty. The run installs and reports as treatment, but every later run uses the saved ID's bucket, which can be holdout, so an installed machine counts as holdout. The window lasts as long as that run's question, up to a minute. In the locked path, persist the drawn ID with SaveID before deciding, and use the ID that is actually stored (for example, have SaveID return it).
There was a problem hiding this comment.
Fixed in 957cbf0. SaveID now returns the stored ID. In the locked path, a machine with no ID saves the ID it draws with SaveID before deciding anything, and uses the ID that comes back for the bucket, the event and Record. The early skip also goes through SaveID, so whichever writes first, both runs end up with the same ID. TestSaveIDKeepsTheFirstID checks the returned value.
| // A run that would ask before installing but cannot skips before the lock | ||
| // and the gate lookup: it would skip after them too, and it repeats on | ||
| // every such run until a run in a terminal asks. |
There was a problem hiding this comment.
[nit] The comment says a run that cannot ask "would skip after them too". But after the gate lookup, a holdout machine settles as SKIPPED_HOLDOUT and a gate-off version settles silently, so the early return changes the outcome of those runs. That is probably intended: holdout and treatment machines both wait for a run that can ask, which keeps the arms comparable. The comment should say so. A test with a holdout-arm ID in a partial terminal (no event, version not settled) would pin it.
There was a problem hiding this comment.
Fixed in 957cbf0. The comment now says that holdout machines and gate-off versions wait for a run that can ask, like treatment machines, so the arms are counted alike. New case "a holdout machine that cannot ask waits too" (holdout-machine, redirected stderr) checks that there's no event and the version is not settled.
| // succeeded. | ||
| actx, stop := signal.NotifyContext(actx, os.Interrupt) | ||
| defer stop() | ||
| ans, err := cmdio.AskCancellable(actx, "Install Databricks AI tools for "+strings.Join(names, ", ")+"? [Y/n]", "") |
There was a problem hiding this comment.
[nit] When the question times out or gets Ctrl+C, AskCancellable leaves its read blocked on the terminal until the process exits, and root.Execute then uploads telemetry for up to 3s. A command the user types right after Ctrl+C is read by the abandoned goroutine and lost. Consider a read that stops on ctx.Done() (muesli/cancelreader is already in the module graph through bubbletea). At least have AskCancellable call MarkStdinReadInBackground when it returns on cancellation, so later prompts in the process know about the leftover read.
There was a problem hiding this comment.
Fixed in 957cbf0, with the minimum: when AskCancellable returns on cancellation, it calls MarkStdinReadInBackground, so no later line prompt in the process can lose its answer to the abandoned read. TestAskCancellable_Cancelled checks this. I didn't switch to cancelreader. It would mean changing cmdio's unbuffered stdin handling, which other prompts share, and that's more than this PR should take on. The line typed right after Ctrl+C is still lost until the process exits; I can follow up if you think it's worth it.
| } | ||
| log.Debugf(ctx, "auto-install: skipped: the answer was neither yes nor no") | ||
| r.settle(ctx, protos.AitoolsAutoInstallOutcomeUnanswered) | ||
| cmdio.LogString(ctx, "Skipped. To install Databricks AI tools later, run: databricks aitools install") |
There was a problem hiding this comment.
[nit] "Skipped. To install Databricks AI tools later, run: databricks aitools install" is written out at lines 299 and 309. .agents/rules/style-guide-go.md asks for a package-level constant when a value is used in more than one place, as this file already does for autoInstallDisable.
There was a problem hiding this comment.
Fixed in 957cbf0: added a skippedNotice constant.
| func inForeground(f *os.File) bool { | ||
| pgrp, err := unix.IoctlGetInt(int(f.Fd()), unix.TIOCGPGRP) | ||
| return err == nil && pgrp == unix.Getpgrp() | ||
| } |
There was a problem hiding this comment.
[nit] inForeground is identical in terminal_darwin.go and terminal_linux.go. Move it to one file with //go:build linux || darwin, and keep only discardInput per OS.
There was a problem hiding this comment.
Fixed in 957cbf0. inForeground is now in terminal_foreground.go (//go:build linux || darwin), and only discardInput is per OS.
…ll-prompt Restack onto the moved base and address the second review of #7703: - Record a "yes" at the prompt (State.AcceptedVersion) and gate the question and the early skip on it instead of the attempt count, which a failed gate lookup now advances without asking. A planning failure is a failed attempt whose retry asks, instead of settling the version. - Save a drawn machine ID before deciding (SaveID returns the stored ID), so a concurrent run that cannot ask cannot store a different one. - Discard pending input before settling the version, and do not ask when that fails (a pipe that mintty counts as a terminal). - Do not ask in a CLI process the CLI started itself (agents.IsSelfExec). - Say so when the install fails after a "yes". - A cancelled AskCancellable turns off later line prompts in the process. - Share inForeground between Linux and macOS, name the repeated notice. Co-authored-by: Isaac <no-reply@databricks.com>
|
@simonfaltum second round is in 957cbf0: I merged the moved base (eafb251) and replied on each thread. Blocking item: a "yes" now stores |
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at 957cbf0. Each finding was verified against the code; simonfaltum may follow up.
No blocking issues found, approving. 13 non-blocking comment(s) and 2 question(s) below.
This PR asks [Y/n] before auto-install's first install of a CLI version, and only when a person can see and answer the question. A "yes" is stored per version, only "no" opts out, and runs that cannot ask wait for one that can. Every fix from the earlier rounds is in place, and CI is green. Three non-blocking issues remain. A self-exec child with no terminal (e.g. air images push's auth docker configure) still installs without asking. Settling before the question clears the Installing marker. In mintty, the cannot-ask skip happens only after the gate lookup. There are also a few nits and two questions: whether the question should default to yes, and whether discarding type-ahead is acceptable.
Questions
- The question defaults to yes (
[Y/n]), so an Enter pressed after it appears installs into the agents' config directories. The CLI's other consent prompts for installs,apps init's skills prompt (RecommendSkillsInstallinlibs/aitools/agents/recommend.go) andconfirmUvInstallinlibs/localenv/uv.go, useAskYesOrNo, which defaults to No. So Enter means opposite things in two prompts for the same tools. Was default-yes chosen on purpose? An alternative is[y/N], where Enter counts as no answer (UNANSWERED, no opt-out) and only an explicit yes installs. (non-blocking) DiscardInputthrows away whatever the user typed while the command ran. That is usually their next shell command (e.g.git push⏎typed during a longbundle deploy), and nothing tells them it was dropped. Is losing that input acceptable once per version? An alternative is to check for pending input (FIONREAD/TIOCINQ on Unix, pending console input events on Windows). When there is some, skip the question without settling the version, so the shell still receives it and a later run asks. (non-blocking)
Accepted answers
- PeekNamedPipe drain for mintty pipes: #7703 (comment)
- cancelreader for the read that Ctrl+C or the timeout abandons: #7703 (comment)
Comments outside the changed lines
- [nit]
cmd/aitools/autoinstall_run.go:221:askalready settled the version withRecord(v, 0). If thisRecord(v, attempt)then fails (the state lock is held for more than 1s, or a disk error),attemptreturns with only a debug log. The state stays{last_version: v, attempts: 0}, so the version is never retried, and the user who just said yes sees nothing. Whenaskedis set, print the "Could not install…" notice on this path too. - [nit]
.github/PULL_REQUEST_TEMPLATE.md: The description uses## Summary,## Evidenceand## Merge Danger. The repo'spr-checklistskill (.agents/skills/pr-checklist/SKILL.md) asks agent-written PRs to use the template's## Changes,## Whyand## Testsheadings in that order, and to keep the summary concise. Restructure the description under those headings.
| // version: a "yes" covers that version's retries. A planning failure is | ||
| // a failed attempt, and its retry asks. | ||
| asked := false | ||
| if err == nil && r.interactive && prior.AcceptedVersion != r.version && len(tools.newAgents()) > 0 { |
There was a problem hiding this comment.
[nice to have] agents.IsSelfExec is checked only in canAsk, which runs only when r.interactive is true. A self-exec child with no terminal therefore installs without asking. Scenario: databricks air images push runs auth docker configure as a self-exec child (configureImageDocker in cmd/air/image.go), with stdout and stderr in a buffer and stdin unset. That child has no terminal, so it installs and settles the version before the parent's hook runs. The user at the terminal is never asked, and the parent then prints the child's "Installed Databricks AI tools…" line. This contradicts the earlier answer that a self-exec child doesn't ask and doesn't install. Skip auto-install in a self-exec child before anything else, without settling the version, for example next to the state-free skips in autoInstall. (in reply to #7703 (comment))
| skipUnaskable(ctx, r.id) | ||
| return false | ||
| } | ||
| if err := autoinstall.Record(ctx, r.version, 0, r.id); err != nil { |
There was a problem hiding this comment.
[nice to have] This Record with 0 attempts settles the version before the question, and it also clears Installing. Scenario: a first install that ran without asking (no terminal, e.g. an agent) times out after writing some skills, leaving Installing=true and no install state. A later run in a terminal plans a fresh install, because planTools skips the untracked-skills check while Installing is set, and asks. The user presses Ctrl+C, waits out the minute, or types something else. With Installing cleared, every later version finds untracked skills and settles as KEPT. The machine is never asked again and the partial install is never finished, contrary to UNANSWERED's "the next CLI version asks again". Keep Installing when settling before the question and for UNANSWERED/DECLINED, e.g. with a settle variant that leaves Installing alone.
| // no. "Yes" is recorded so that the version's retries do not ask again; only | ||
| // "no" stores the opt-out. | ||
| func (r autoInstallRun) ask(ctx context.Context, names []string) bool { | ||
| if err := cmdio.DiscardInput(ctx); err != nil { |
There was a problem hiding this comment.
[nice to have] In Git Bash's mintty, stdin is a Cygwin pipe that isTTY counts as a terminal. So canAsk is true and the early skip at line 118 does not fire. On every command, a treatment machine takes the lock, resolves the gate (a manifest fetch of up to 3s once the hourly cache is stale), plans, and only then fails here in DiscardInput and takes skipUnaskable. The version is never settled and nothing is reported. A holdout machine on the same setup settles as SKIPPED_HOLDOUT after the gate lookup, so the arms are not counted alike, against the intent of the early skip. Make IsLinePromptSupported false on Windows when stdin is not a real console (for example, isatty.IsTerminal is false but IsCygwinTerminal is true, or GetConsoleMode fails), so the pre-lock skip catches it. Keep this error path as a fallback.
| // no. "Yes" is recorded so that the version's retries do not ask again; only | ||
| // "no" stores the opt-out. | ||
| func (r autoInstallRun) ask(ctx context.Context, names []string) bool { | ||
| if err := cmdio.DiscardInput(ctx); err != nil { |
There was a problem hiding this comment.
[nit] Pending input is discarded before Record, which can wait up to 1s for the state lock and then rewrites the file, and only then is the question printed. An Enter pressed in that window (e.g. tapping Enter while a deploy finishes) is read as the default "yes" before the question appears. Keep the discard here as the can-ask check, and discard again right before AskCancellable.
| if err != nil { | ||
| log.Debugf(ctx, "auto-install: skipped: the prompt was not answered: %v", err) | ||
| // End the question's line. | ||
| cmdio.LogString(ctx, "") |
There was a problem hiding this comment.
[nit] On a timeout, Ctrl+C or EOF, the user sees the question followed by an empty line, with no hint that it was skipped or how to install later. The "neither yes nor no" path prints skippedNotice, and this version won't ask again. Print skippedNotice here too.
| // version: a "yes" covers that version's retries. A planning failure is | ||
| // a failed attempt, and its retry asks. | ||
| asked := false | ||
| if err == nil && r.interactive && prior.AcceptedVersion != r.version && len(tools.newAgents()) > 0 { |
There was a problem hiding this comment.
[nit] tools.newAgents() is built here and again on line 213. The condition interactive && AcceptedVersion != version && len(newAgents()) > 0 is also written out again at line 118, and the two copies must stay in sync by hand. Compute the names once, and consider one helper for the "would ask" predicate used by both places.
| if err == nil && r.interactive && prior.AcceptedVersion != r.version && len(tools.newAgents()) > 0 { | ||
| // autoInstall skips this case before the lock; the state it decided | ||
| // on may have changed since. | ||
| if !canAsk(ctx) { |
There was a problem hiding this comment.
[nit] No test reaches this second canAsk check, because the early skip catches every current case. The timeout and Ctrl+C paths of ask and the DiscardInput failure path are also untested, and promptTimeout is a const, so a test can't shorten it. The new libs/cmdio/terminal_*.go files have no tests either (.agents/rules/testing.md), and every test uses FakeTTYReader, so discardInput and inForeground never run. An os.Pipe test would check that inForeground is false and discardInput fails on a pipe. Consider making the timeout a package variable and adding cases for a cancelled question and a failed discard.
| setupAutoInstallHome(t, "{}") | ||
| t.Setenv("TERM", "xterm") | ||
| var stderr bytes.Buffer | ||
| run := func() autoinstall.State { |
There was a problem hiding this comment.
[nit] run := func() autoinstall.State {...} names a repeated operation as an inline closure, which .agents/rules/style-guide-go.md asks to extract as a regular named function. Make it a helper that takes t and the stderr buffer.
| installTimeout = 10 * time.Second | ||
|
|
||
| // promptTimeout bounds the wait for an answer at the install prompt. | ||
| promptTimeout = time.Minute |
There was a problem hiding this comment.
[nit] ExecutionTimeMs in cmd/root/root.go is measured around ExecuteContextC, which includes the post-run hook. So the first command of each version can now report up to this 1-minute wait as command time. Before this PR it already included up to about 13s of install. Consider excluding the hook's time, or noting it where command latency is analyzed.
| version to a later run that can ask; a process killed at the prompt logs nothing either. An install | ||
| with no tools installed yet also logs an `aitools_install_event` with `trigger` `AUTO_FIRST_RUN`; | ||
| installs by `databricks aitools install` leave `trigger` unset. The holdout arm comes from a random | ||
| machine id kept in |
There was a problem hiding this comment.
[nit] The reflow left "machine id kept in" on its own line before the backticked path. Rewrap the paragraph.
#7701 landed on main as a squash of this branch's base; its conflicts resolve to this branch's side, as main has not changed those files since. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com> # Conflicts: # cmd/aitools/autoinstall.go # cmd/aitools/autoinstall_run.go # libs/aitools/autoinstall/run.go # libs/aitools/autoinstall/run_test.go # libs/aitools/autoinstall/state.go # libs/telemetry/protos/README.md # libs/telemetry/protos/aitools_events.go
❌ Integration test reportCommit: 304522b
24 interesting tests: 9 FAIL, 9 RECOVERED, 5 flaky, 1 KNOWN
Top 50 slowest tests (at least 2 minutes):
|
Stacked on #7702, the PR that installs AI tools after the first command of a new CLI version.
When a person runs the CLI in a terminal, auto-install asks before its first install.
Summary
cmdio.IsLinePromptSupported): all three streams are terminals (a redirected stream would hide the question or take the answer from elsewhere, and a pager reading stdout would share the terminal's keystrokes), the process is in the terminal's foreground process group (a background job would stop on SIGTTIN), and no goroutine still reads stdin (apps dev-remote, viacmdio.MarkStdinReadInBackground). Runs with no terminal at all keep the parent's non-interactive install.accepted_version, and the question and the early skip check it, not the attempt count. That count also advances on a failed gate lookup, which never asks. A planning failure is an ordinary failed attempt whose retry asks.autoinstall.SaveIDreturns the stored ID, so concurrent runs agree on the holdout bucket. Holdout machines and gate-off versions wait for a run that can ask, like treatment machines.DATABRICKS_CLI_SELF_EXEC, nowagents.IsSelfExec) does not ask; the parent asks after its own command.cmdio.DiscardInput: tcflush on Linux and macOS, FlushConsoleInputBuffer on Windows), and Ctrl+C at the question counts as no answer, so the command keeps its exit status. Where input can't be discarded (a pipe that Git Bash's mintty counts as a terminal), the question is skipped. A timed-out or interrupted question also turns off later line prompts in the process, since its read stays blocked on stdin.cmdio.AskCancellable), soNO_COLOR,TERM=dumband Git Bash, which turn off the TUI prompts, still get it.cmdio.HasTerminaland the newcmdio.IsLinePromptSupportedexpress the two checks.DECLINEDandUNANSWEREDadded to the auto-install outcome enum; the Universe proto for the whole event is https://github.com/databricks-eng/universe/pull/3127275.auto-install --helpmentions the prompt.Review order:
cmd/aitools/autoinstall_run.go(autoInstall's early skip and ID save,attempt,canAsk,skipUnaskable,ask,decline),libs/aitools/autoinstall(AcceptedVersion,Accept,SaveID), thenlibs/cmdio/io.goandlibs/cmdio/terminal_*.go, thencmd/aitools/autoinstall_run_test.go.Evidence
The prompt needs a terminal, which acceptance tests lack, so
TestAutoInstallPrompt(cmd/aitools) drives the hook through a test terminal and checks the prompt and notice on stderr, the skill in Cursor's dir, the stored state and the event:TestAutoInstallKeepsTheIDWhenItCannotAskruns a partial-terminal hook twice on a machine with no stored ID: the ID is saved and stays the same, and the version is not settled.TestToolsPlanNewAgentscovers which agents are named.TestCapabilities_SupportsLinePromptandTestIsLinePromptSupported(libs/cmdio) cover the stream checks and the background stdin reader.On a real terminal (tmux, Linux,
databricks auth profileswith the gate cached on):