Skip to content

aitools: run aitools-dependent cmd tests and goldens in test-exp-aitools - #7848

Open
anton-107 wants to merge 1 commit into
mainfrom
aitools-ci-dependent-tests
Open

anton-107 wants to merge 1 commit into
mainfrom
aitools-ci-dependent-tests

Conversation

@anton-107

Copy link
Copy Markdown
Contributor

Changes

task test-exp-aitools now also runs the tests that depend on aitools but live outside the aitools directories:

  • Unit tests: adds the cmd, cmd/apps, cmd/experimental, cmd/root and cmd/version packages and the root package (.) to the gotestsum package list.
  • Acceptance tests: extends the -run pattern with TestAccept/help$, TestAccept/cmd/version, TestAccept/quickstart, TestAccept/air/help, TestAccept/air/config-help and TestAccept/air/image-push.

The task's sources: stay the same. Comments in Taskfile.yml explain why each addition is there.

Why

CI uses the sources: of test-exp-aitools to decide which job to run. A change that touches only aitools code runs test-exp-aitools and skips the general test job. Some code outside the aitools directories still depends on aitools:

  • cmd/root runs aitools hooks on every command.
  • cmd, cmd/apps and cmd/version print aitools text, and the root package imports aitools.
  • The goldens for help, cmd/version, quickstart and air help contain aitools text.
  • air/image-push starts a child process with the aitools self-exec environment.

So an aitools-only change could break these tests and still pass CI. Running them inside test-exp-aitools catches that on the PR.

cmd/air unit tests are left out on purpose. The only aitools call there is already covered by the air/image-push golden, and its unit tests take over a minute without touching aitools.

Tests

  • Ran ./task test-exp-aitools locally with no DATABRICKS_* variables set. Both the unit and acceptance steps pass, and the new packages and goldens run.
  • Checked that every added -run prefix matches an existing directory under acceptance/.

## Changes

`task test-exp-aitools` now also runs the tests that depend on aitools but live outside the aitools directories:

- Unit tests: adds the `cmd`, `cmd/apps`, `cmd/experimental`, `cmd/root` and `cmd/version` packages and the root package (`.`) to the `gotestsum` package list.
- Acceptance tests: extends the `-run` pattern with `TestAccept/help$`, `TestAccept/cmd/version`, `TestAccept/quickstart`, `TestAccept/air/help`, `TestAccept/air/config-help` and `TestAccept/air/image-push`.

The task's `sources:` stay the same. Comments in `Taskfile.yml` explain why each addition is there.

## Why

CI uses the `sources:` of `test-exp-aitools` to decide which job to run. A change that touches only aitools code runs `test-exp-aitools` and skips the general `test` job. Some code outside the aitools directories still depends on aitools:

- `cmd/root` runs aitools hooks on every command.
- `cmd`, `cmd/apps` and `cmd/version` print aitools text, and the root package imports aitools.
- The goldens for `help`, `cmd/version`, `quickstart` and `air` help contain aitools text.
- `air/image-push` starts a child process with the aitools self-exec environment.

So an aitools-only change could break these tests and still pass CI. Running them inside `test-exp-aitools` catches that on the PR.

`cmd/air` unit tests are left out on purpose. The only aitools call there is already covered by the `air/image-push` golden, and its unit tests take over a minute without touching aitools.

## Tests

- Ran `./task test-exp-aitools` locally with no `DATABRICKS_*` variables set. Both the unit and acceptance steps pass, and the new packages and goldens run.
- Checked that every added `-run` prefix matches an existing directory under `acceptance/`.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 requested a review from a team as a code owner November 20, 2026 15:32

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 71d96e4. Each finding was verified against the code.

No blocking issues found. 2 non-blocking comment(s) below.

This PR extends the test-exp-aitools task to run unit tests and acceptance goldens for packages outside the aitools directories that still depend on aitools at compile or output time. The change is well-motivated and the list of additions is reasonable. Two minor comment/consistency issues survive verification.

Comment thread Taskfile.yml
- acceptance/experimental/aitools/**
- "{{.EMBED_SOURCES}}"
cmds:
# cmd/root runs aitools hooks on every command, cmd, cmd/apps and

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The comment lists every added package and explicitly justifies excluding cmd/air, but ./cmd/experimental (line 721) is silently included. It belongs here because cmd/experimental/experimental.go imports experimental/aitools/cmd—worth a short clause so a future reader doesn't wonder why it's there or remove it.

Comment thread Taskfile.yml
--no-summary=skipped \
--packages ./acceptance/... \
-- -timeout=${LOCAL_TIMEOUT:-60m} -run "TestAccept/apps|TestAccept/cmd/aitools|TestAccept/experimental/aitools"
-- -timeout=${LOCAL_TIMEOUT:-60m} -run "TestAccept/apps|TestAccept/cmd/aitools|TestAccept/experimental/aitools|TestAccept/help$|TestAccept/cmd/version|TestAccept/quickstart|TestAccept/air/help|TestAccept/air/config-help|TestAccept/air/image-push"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] TestAccept/help$ is correctly anchored with $, but the other new patterns (TestAccept/air/help, TestAccept/cmd/version, TestAccept/quickstart, etc.) are not. Go's -run is an unanchored regex, so TestAccept/air/help would also select a future TestAccept/air/help-debug. Either anchor all the new patterns with $ or drop the anchor on help so the style is consistent.

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 71d96e4

Run: 44764382201

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 7:02
💚​ aws-windows-latest-4core-16gb 1 274 17 4:58
💚​ azure linux-2core-8gb 1 271 19 6:25
💚​ azure-windows-latest-4core-16gb 1 273 17 4:30
💚​ gcp linux-2core-8gb 1 272 19 5:44
💚​ gcp-windows-latest-4core-16gb 1 274 17 5:52
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 4 slowest tests (at least 2 minutes):
duration env testname
4:08 azure-windows-latest-4core-16gb TestAccept
3:58 aws-windows-latest-4core-16gb TestAccept
3:57 gcp-windows-latest-4core-16gb TestAccept
2:08 gcp linux-2core-8gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants