Skip to content

aitools: only auto-install without a terminal for coding agent calls - #7834

Open
anton-107 wants to merge 6 commits into
mainfrom
aitools-auto-install-consent-without-terminal
Open

anton-107 wants to merge 6 commits into
mainfrom
aitools-auto-install-consent-without-terminal

Conversation

@anton-107

Copy link
Copy Markdown
Contributor

Changes

Automatic installation of Databricks AI tools now asks before a first install whenever the caller is not a coding agent, whether or not a terminal is attached. Before this change it asked only when stdin, stdout or stderr was a terminal. A run with no terminal at all installed without asking.

  • autoInstallAsks now depends only on whether a coding agent was detected. It no longer checks for a terminal.
  • A run that is not a coding agent's call and cannot show the question (canAsk is false) is skipped, the same way runs that have a terminal but cannot prompt are skipped today. It installs nothing, sends no telemetry and leaves the version unsettled, so a later run that can ask, or a coding agent's call, still installs. This skip happens before the lock and the gate lookup, so cron jobs and job pods don't wait on the lookup every time they run.
  • Coding agents still get the tools without a question, with or without a terminal.
  • The interactive field on the auto-install telemetry event is still true only when a terminal is attached and no coding agent is detected. It is now computed at the call site.

Why

Commands run over ssh without a terminal, under nohup, from cron or from an IDE extension have no terminal. Installing in those cases changed the user's agent configuration without asking a person who may have started the command. Asking is not possible there, so the run waits for one where it is. A coding agent's call is the case that is meant to install without a question, and it keeps doing so.

Tests

  • Unit tests in cmd/aitools/autoinstall_run_test.go:
    • The "no terminal at all" case now expects a skip with the version left unsettled.
    • New cases check that a coding agent installs without asking, both with no terminal and in a terminal.
    • A new test checks that a run with no terminal and no coding agent skips before the gate lookup.
    • Existing tests that expect an install with no terminal now run as a coding agent's call.
  • New acceptance test acceptance/cmd/aitools/auto-install/no-terminal. A run with no terminal and no coding agent installs nothing and leaves the version unsettled. A later coding agent call then installs.
  • The shared script.prepare for the other auto-install acceptance cases sets AGENT. Those cases have no terminal, so they now run as a coding agent's call and their outputs stay the same.
  • Changelog fragment added under .nextchanges/cli/.

## Changes

Automatic installation of Databricks AI tools now asks before a first install whenever the caller is not a coding agent, whether or not a terminal is attached. Before this change it asked only when stdin, stdout or stderr was a terminal. A run with no terminal at all installed without asking.

- `autoInstallAsks` now depends only on whether a coding agent was detected. It no longer checks for a terminal.
- A run that is not a coding agent's call and cannot show the question (`canAsk` is false) is skipped, the same way runs that have a terminal but cannot prompt are skipped today. It installs nothing, sends no telemetry and leaves the version unsettled, so a later run that can ask, or a coding agent's call, still installs. This skip happens before the lock and the gate lookup, so cron jobs and job pods don't wait on the lookup every time they run.
- Coding agents still get the tools without a question, with or without a terminal.
- The `interactive` field on the auto-install telemetry event is still true only when a terminal is attached and no coding agent is detected. It is now computed at the call site.

## Why

Commands run over ssh without a terminal, under `nohup`, from cron or from an IDE extension have no terminal. Installing in those cases changed the user's agent configuration without asking a person who may have started the command. Asking is not possible there, so the run waits for one where it is. A coding agent's call is the case that is meant to install without a question, and it keeps doing so.

## Tests

- Unit tests in `cmd/aitools/autoinstall_run_test.go`:
  - The "no terminal at all" case now expects a skip with the version left unsettled.
  - New cases check that a coding agent installs without asking, both with no terminal and in a terminal.
  - A new test checks that a run with no terminal and no coding agent skips before the gate lookup.
  - Existing tests that expect an install with no terminal now run as a coding agent's call.
- New acceptance test `acceptance/cmd/aitools/auto-install/no-terminal`. A run with no terminal and no coding agent installs nothing and leaves the version unsettled. A later coding agent call then installs.
- The shared `script.prepare` for the other auto-install acceptance cases sets `AGENT`. Those cases have no terminal, so they now run as a coding agent's call and their outputs stay the same.
- Changelog fragment added under `.nextchanges/cli/`.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 requested review from a team and lennartkats-db as code owners November 20, 2026 01:20
@github-actions github-actions Bot added the aitools AI tools (databricks aitools) related changes label Nov 20, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 38e0e85cb | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

2 tests: regression=1 unreleased=0 coverage=1 failing=0
test branch main (8b7e509) latest (v1.27.0)
TestAccept/cmd/aitools/auto-install/no-terminal ✅ ❌ ❌
TestAccept/cmd/aitools/auto-install/upgrade ✅ ✅ ✅
TestAccept/cmd/aitools/auto-install/no-terminal: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌

main (8b7e509):

acceptance_test.go:1270: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:36549" ("[DATABRICKS_HOST]")
    acceptance_test.go:1270: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:36549" ("[DATABRICKS_HOST]")
    acceptance_test.go:1402: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1402
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1143
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:686
        	Error:      	Not equal: 
        	            	expected: "\n=== no terminal and no coding agent: cannot ask, so nothing is installed or reported, and the version is not settled\n\n>>> [CLI] jobs list\n\n>>> print_telemetry\n\n>>> cursor_skills\n(no skills)\n\n>>> auto_install_state\n{\"id\": \"treatment-machine\"}\n\n=== the version is still due: a coding agent's call installs without asking\n\n>>> AGENT=test-agent [CLI] jobs list\nInstalled Databricks AI tools for Cursor. Disable: databricks aitools auto-install disable\n\n>>> print_telemetry\n{\"command\":\"jobs_list\",\"exit_code\":0,\"aitools_install_event\":{\"agents\":[\"CURSOR\"],\"scope\":\"GLOBAL\",\"error_category\":\"TYPE_UNSPECIFIED\",\"trigger\":\"AUTO_FIRST_RUN\"}}\n{\"command\":\"jobs_list\",\"exit_code\":0,\"aitools_auto_install_event\":{\"outcome\":\"INSTALLED\",\"holdout_arm\":\"TREATMENT\",\"interactive\":false}}\n\n>>> cursor_skills\ndatabricks-test\n\n>>> auto_install_state\n{\n  \"last_version\": \"1.30.0\",\n  \"id\": \"treatment-machine\"\n}\n"
        	            	actual  : "\n=== no terminal and no coding agent: cannot ask, so nothing is installed or reported, and the version is not settled\n\n>>> [CLI] jobs list\nInstalled Databricks AI tools for Cursor. Disable: databricks aitools auto-install disable\n\n>>> print_telemetry\n{\"command\":\"jobs_list\",\"exit_code\":0,\"aitools_install_event\":{\"agents\":[\"CURSOR\"],\"scope\":\"GLOBAL\",\"error_category\":\"TYPE_UNSPECIFIED\",\"trigger\":\"AUTO_FIRST_RUN\"}}\n{\"command\":\"jobs_list\",\"exit_code\":0,\"aitools_auto_install_event\":{\"outcome\":\"INSTALLED\",\"holdout_arm\":\"TREATMENT\",\"interactive\":false}}\n\n>>> cursor_skills\ndatabricks-test\n\n>>> auto_install_state\n{\n  \"last_version\": \"1.30.0\",\n  \"id\": \"treatment-machine\"\n}\n\n=== the version is still due: a coding agent's call installs without asking\n\n>>> AGENT=test-agent [CLI] jobs list\n\n>>> print_telemetry\n\n>>> cursor_skills\ndatabricks-test\n\n>>> auto_install_state\n{\n  \"last_version\": \"1.30.0\",\n  \"id\": \"treatment-machine\"\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,14 +4,2 @@
        	            	 >>> [CLI] jobs list
        	            	-
        	            	->>> print_telemetry
        	            	-
        	            	->>> cursor_skills
        	            	-(no skills)
        	            	-
        	            	->>> auto_install_state
        	            	-{"id": "treatment-machine"}
        	            	-
        	            	-=== the version is still due: a coding agent's call installs without asking
        	            	-
        	            	->>> AGENT=test-agent [CLI] jobs list
        	            	 Installed Databricks AI tools for Cursor. Disable: databricks aitools auto-install disable
        	            	@@ -31 +19,16 @@
        	            	 
        	            	+=== the version is still due: a coding agent's call installs without asking
        	            	+
        	            	+>>> AGENT=test-agent [CLI] jobs list
        	            	+
        	            	+>>> print_telemetry
        	            	+
        	            	+>>> cursor_skills
        	            	+databricks-test
        	            	+
        	            	+>>> auto_install_state
        	            	+{
        	            	+  "last_version": "1.30.0",
        	            	+  "id": "treatment-machine"
        	            	+}
        	            	+
        	Test:       	TestAccept/cmd/aitools/auto-install/no-terminal
        	Messages:   	cmd/aitools/auto-install/no-terminal/output.txt vs /tmp/TestAcceptcmdaitoolsauto-installno-terminal2292131866/002/output.txt
FAIL

latest (v1.27.0):

acceptance_test.go:1270: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:34633" ("[DATABRICKS_HOST]")
    acceptance_test.go:1270: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:34633" ("[DATABRICKS_HOST]")
    acceptance_test.go:1402: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1402
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1143
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:686
        	Error:      	Not equal: 
        	            	expected: "\n=== no terminal and no coding agent: cannot ask, so nothing is installed or reported, and the version is not settled\n\n>>> [CLI] jobs list\n\n>>> print_telemetry\n\n>>> cursor_skills\n(no skills)\n\n>>> auto_install_state\n{\"id\": \"treatment-machine\"}\n\n=== the version is still due: a coding agent's call installs without asking\n\n>>> AGENT=test-agent [CLI] jobs list\nInstalled Databricks AI tools for Cursor. Disable: databricks aitools auto-install disable\n\n>>> print_telemetry\n{\"command\":\"jobs_list\",\"exit_code\":0,\"aitools_install_event\":{\"agents\":[\"CURSOR\"],\"scope\":\"GLOBAL\",\"error_category\":\"TYPE_UNSPECIFIED\",\"trigger\":\"AUTO_FIRST_RUN\"}}\n{\"command\":\"jobs_list\",\"exit_code\":0,\"aitools_auto_install_event\":{\"outcome\":\"INSTALLED\",\"holdout_arm\":\"TREATMENT\",\"interactive\":false}}\n\n>>> cursor_skills\ndatabricks-test\n\n>>> auto_install_state\n{\n  \"last_version\": \"1.30.0\",\n  \"id\": \"treatment-machine\"\n}\n"
        	            	actual  : "\n=== no terminal and no coding agent: cannot ask, so nothing is installed or reported, and the version is not settled\n\n>>> [CLI] jobs list\nInstalled Databricks AI tools for Cursor. Disable: databricks aitools auto-install disable\n\n>>> print_telemetry\n{\"command\":\"jobs_list\",\"exit_code\":0,\"aitools_install_event\":{\"agents\":[\"CURSOR\"],\"scope\":\"GLOBAL\",\"error_category\":\"TYPE_UNSPECIFIED\",\"trigger\":\"AUTO_FIRST_RUN\"}}\n{\"command\":\"jobs_list\",\"exit_code\":0,\"aitools_auto_install_event\":{\"outcome\":\"INSTALLED\",\"holdout_arm\":\"TREATMENT\",\"interactive\":false}}\n\n>>> cursor_skills\ndatabricks-test\n\n>>> auto_install_state\n{\n  \"last_version\": \"1.30.0\",\n  \"id\": \"treatment-machine\"\n}\n\n=== the version is still due: a coding agent's call installs without asking\n\n>>> AGENT=test-agent [CLI] jobs list\n\n>>> print_telemetry\n\n>>> cursor_skills\ndatabricks-test\n\n>>> auto_install_state\n{\n  \"last_version\": \"1.30.0\",\n  \"id\": \"treatment-machine\"\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,14 +4,2 @@
        	            	 >>> [CLI] jobs list
        	            	-
        	            	->>> print_telemetry
        	            	-
        	            	->>> cursor_skills
        	            	-(no skills)
        	            	-
        	            	->>> auto_install_state
        	            	-{"id": "treatment-machine"}
        	            	-
        	            	-=== the version is still due: a coding agent's call installs without asking
        	            	-
        	            	->>> AGENT=test-agent [CLI] jobs list
        	            	 Installed Databricks AI tools for Cursor. Disable: databricks aitools auto-install disable
        	            	@@ -31 +19,16 @@
        	            	 
        	            	+=== the version is still due: a coding agent's call installs without asking
        	            	+
        	            	+>>> AGENT=test-agent [CLI] jobs list
        	            	+
        	            	+>>> print_telemetry
        	            	+
        	            	+>>> cursor_skills
        	            	+databricks-test
        	            	+
        	            	+>>> auto_install_state
        	            	+{
        	            	+  "last_version": "1.30.0",
        	            	+  "id": "treatment-machine"
        	            	+}
        	            	+
        	Test:       	TestAccept/cmd/aitools/auto-install/no-terminal
        	Messages:   	cmd/aitools/auto-install/no-terminal/output.txt vs /tmp/TestAcceptcmdaitoolsauto-installno-terminal274325453/002/output.txt
FAIL

Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 38e0e85

Run: 44759683295

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 21 7:17
💚​ aws-windows-latest-4core-16gb 1 274 19 5:53
💚​ azure linux-2core-8gb 1 271 21 6:08
💚​ azure-windows-latest-4core-16gb 1 273 19 5:06
💚​ gcp linux-2core-8gb 1 272 21 6:03
💚​ gcp-windows-latest-4core-16gb 1 274 19 4:17
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:13 aws-windows-latest-4core-16gb TestAccept
4:10 gcp-windows-latest-4core-16gb TestAccept
3:57 azure-windows-latest-4core-16gb TestAccept

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review from simonfaltum's review agent (pr-review skill) at 96994c5. Each finding was verified against the code; simonfaltum may follow up.

No blocking issues found, approving. 4 non-blocking comment(s) and 2 question(s) below.

This PR makes auto-install ask before a first install whenever no coding agent is detected, terminal or not. Runs that cannot ask now skip before the lock and the gate lookup, and coding agents still install without a question. The logic matches the description, the interactive telemetry field keeps its meaning, the new tests catch the old behavior, and CI passes. Nothing is blocking. The main follow-ups are wording: the help text and the changelog don't say that headless updates of an existing install still run. I also left two non-blocking questions, on per-run planning cost and on telemetry for runs that wait for consent.

Questions

  1. With this change, a run with no terminal and no coding agent never settles the version. Every cron, nohup or IDE-extension call on a machine with an agent config dir therefore runs planTools again after its command: it reads the install state, scans for untracked skills, detects agents and looks up agent CLIs on PATH. Before, such a machine settled after one run. The cost is local I/O, but it now repeats on every call. Could the skip check the cheap conditions first (autoInstallAsks(), AcceptedVersion != version, !canAsk(ctx)) and read only installer.LoadState to tell an update from a first install? Or is the current per-run cost acceptable? (non-blocking)
  2. Headless runs without a coding agent used to report INSTALLED with interactive=false. They now take skipUnaskable, which emits nothing, so machines that wait for a run that can ask (IDE-only or cron-only users) disappear from the auto-install events. Dashboards then can't tell "waiting for consent" from "never ran". Is that intended, or should an unsettled skip outcome be reported once per version so this group stays measurable? (non-blocking)

Comments outside the changed lines

  • [nice to have] cmd/aitools/autoinstall.go:23: The help still says only "In a terminal it asks before the first install." With this PR, that reads as if the tools install without asking when there is no terminal. In fact a run with no terminal and no coding agent now skips and waits for a run that can ask, and a coding agent is never asked. Consider rewording it, for example: "It asks before the first install and skips when it cannot ask; a coding agent's call installs without a question."

Comment thread cmd/aitools/autoinstall_run.go Resolved
anton-107 and others added 3 commits November 20, 2026 12:13
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review from simonfaltum's review agent (pr-review skill) at 38e0e85. Each finding was verified against the code; simonfaltum may follow up.

No blocking issues found, approving. 5 non-blocking comment(s) and 1 question(s) below.

This PR makes auto-install skip a first install when there is no terminal and no coding agent, leaving the version for a run that can ask. Coding agents still install without a question, and headless updates still run. The logic matches the description and CI passes. I found nothing blocking. Open items: the auto-install help text still describes the old behavior, the headless retry after a "yes" has no test, there are a few test and comment nits, and one question about agents the SDK doesn't detect.

Questions

  1. Without a terminal, a first install now happens only when the SDK recognizes the coding agent from an environment variable (its known list, or AGENT/AI_AGENT). An agent that sets none of these, such as Aider, Continue, Zed, Junie or Roo Code, used to get the tools when it ran the CLI without a terminal. Now it never does, unless a person later runs the CLI in a foreground terminal, and those skips are not reported. Is that trade-off intended? Or should the skip be counted, or the change wait for data on how many interactive=false installs had no detected agent? (non-blocking)

Comments outside the changed lines

  • [nit] cmd/aitools/autoinstall_run.go:88: This comment says runs without a terminal are not skipped because coding agents run the CLI that way. That is still true for this command-level check. But a headless run with no coding agent now skips its first install later, in autoInstall, so a reader can come away thinking headless non-agent runs install. Consider adding a pointer, for example: "A first install without a terminal still needs a coding agent's call (see autoInstallAsks)."
  • [nit] .agents/skills/pr-checklist/SKILL.md: The commits carry Co-authored-by: Isaac, but the PR body has no closing disclosure line. .agents/skills/pr-checklist/SKILL.md asks for one on the last line when an agent wrote or helped write the PR. Consider adding a single italic line at the end of the description. Separately, "Changelog fragment added" is listed under Tests and could move to Changes.

Agreeing with existing threads

  • [nice to have] auto-install help text doesn't describe the headless skip: #7834 (review)

wantOutcome: protos.AitoolsAutoInstallOutcomeInstalled,
},
{
name: "a retry after yes is not asked again",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nice to have] After this PR, the only first install a run with no terminal and no coding agent still makes is a retry of a version the user already accepted (accepted_version equals the version, so the checks at autoinstall_run.go:151 and :248 pass). No test covers it. This case runs with a TTY stdin, and the new headless cases have no accepted version. A later change that made the accepted-version check depend on canAsk would stop headless retries after a "yes" without failing any test. Consider adding a table case with stdinPiped, stdoutRedirected and stderrRedirected and the accepted state, expecting INSTALLED with interactive=false.


// setCodingAgent makes the run the call of the coding agent whose env var is
// name. setupAutoInstallHome resets the SDK's cached agent after the test.
func setCodingAgent(t *testing.T, name string) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] setCodingAgent sets the env var with t.Setenv and clears the SDK cache, but it depends on running after setupAutoInstallHome. That call's isolateAgents clears the whole environment, and its cleanup resets the cache. Called earlier, the variable is wiped. Called without it, a cached agent can leak into later tests. All current callers run it in the right order. The existing pattern, passing the agent pair to isolateAgents as setupAgentCallerInstall in install_test.go does, would avoid the ordering dependency: for example, let setupAutoInstallHome accept extra env pairs.


// A run with no terminal and no coding agent cannot ask, so it skips before the
// gate lookup: a cron job or a job pod does not wait for it on every run.
func TestAutoInstallSkipsTheGateLookupWhenItCannotAsk(t *testing.T) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] TestAutoInstallSkipsTheGateLookupWhenItCannotAsk repeats TestAutoInstallSkipsTheGateLookupWithoutAgents almost line for line: same setup, same failing gateOn fake, same assertions. The only differences are agent versus no agent and whether .cursor exists. .agents/rules/testing.md prefers a table for similar cases, and a table would make that difference explicit. It would also be a place to say why setCodingAgent is now needed in ...WithoutAgents: it keeps that test on the no-agents skip instead of the cannot-ask skip.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

aitools AI tools (databricks aitools) related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants