Skip to content

tools: add aitools-quick and aitools-full verification tasks - #7825

Merged
anton-107 merged 5 commits into
mainfrom
aitools-quick-verification
Nov 20, 2026
Merged

anton-107 merged 5 commits into
mainfrom
aitools-quick-verification

Conversation

@anton-107

Copy link
Copy Markdown
Contributor

Changes

Adds two verification tasks for the aitools command group (cmd/aitools, libs/aitools, experimental/aitools), plus a benchmark for the faster one:

  • ./task aitools-quick runs three steps in parallel: golangci-lint and the unit tests of the aitools packages, and go build of the packages that import aitools (the root package, cmd, cmd/root, cmd/apps and others). Packages with Go changes since the merge base with main (--ref picks another ref) are also linted and tested, so editing a shared package such as libs/telemetry runs its tests too. Every step gets a one-line summary, and failed steps also print their output. With -- --json <path> it also writes a machine-readable report with file:line findings.
  • ./task aitools-full runs the same lint, then the unit tests of the importers, the acceptance tests that exercise aitools (cmd/aitools, experimental/aitools, apps, the help and version goldens, quickstart, air help) and tools/check_deadcode.py.
  • Unit and acceptance tests run with coding-agent detection variables and DATABRICKS_* variables removed, and with HOME pointing at a fixed directory under the user cache dir. This way the agent a developer happens to run in can't change aitools behavior, and Go's test cache stays valid between runs. The Go cache, module cache and go env file are pinned so moving HOME doesn't force a cold build.
  • ./task aitools-quick-bench measures the quick tier. time makes a small edit in one file per aitools package and reports p50/p95 wall time, failing if p50 exceeds --budget. detect applies source mutants one at a time and reports what share of the mutants caught by the full tier the quick tier also catches. Both modes work in a temporary git worktree and never write to the checkout.
  • An .agents/rules file (with its .cursor/rules symlink), scoped to the aitools paths, tells coding agents to run aitools-quick after every edit there and aitools-full before opening a PR.

Why

To check an aitools edit today you run either test-exp-aitools or the whole suite. Both take long enough that you don't run them after each edit. The quick tier covers the common failures (lint, unit tests, and broken importers such as cmd/root, which calls aitools from the root command's pre-run) in a time short enough to run after every edit. The full tier catches the output and wiring changes the quick tier misses. The bench script lets anyone re-measure that trade-off when the package lists change.

Tests

  • ./task aitools-quick passes on this branch. Its three steps ran in parallel and the run took about 25s, with the test step the slowest.
  • The doctests in tools/aitools_quick.py and tools/aitools_quick_bench.py pass (python3 -m doctest).
  • No Go code changes, so this PR doesn't affect the existing unit and acceptance tests.

## Changes

Adds two verification tasks for the aitools command group (`cmd/aitools`, `libs/aitools`, `experimental/aitools`), plus a benchmark for the faster one:

- `./task aitools-quick` runs three steps in parallel: golangci-lint and the unit tests of the aitools packages, and `go build` of the packages that import aitools (the root package, `cmd`, `cmd/root`, `cmd/apps` and others). Packages with Go changes since the merge base with `main` (`--ref` picks another ref) are also linted and tested, so editing a shared package such as `libs/telemetry` runs its tests too. Every step gets a one-line summary, and failed steps also print their output. With `-- --json <path>` it also writes a machine-readable report with `file:line` findings.
- `./task aitools-full` runs the same lint, then the unit tests of the importers, the acceptance tests that exercise aitools (`cmd/aitools`, `experimental/aitools`, `apps`, the help and version goldens, `quickstart`, `air` help) and `tools/check_deadcode.py`.
- Unit and acceptance tests run with coding-agent detection variables and `DATABRICKS_*` variables removed, and with `HOME` pointing at a fixed directory under the user cache dir. This way the agent a developer happens to run in can't change aitools behavior, and Go's test cache stays valid between runs. The Go cache, module cache and `go env` file are pinned so moving `HOME` doesn't force a cold build.
- `./task aitools-quick-bench` measures the quick tier. `time` makes a small edit in one file per aitools package and reports p50/p95 wall time, failing if p50 exceeds `--budget`. `detect` applies source mutants one at a time and reports what share of the mutants caught by the full tier the quick tier also catches. Both modes work in a temporary git worktree and never write to the checkout.
- An `.agents/rules` file (with its `.cursor/rules` symlink), scoped to the aitools paths, tells coding agents to run `aitools-quick` after every edit there and `aitools-full` before opening a PR.

## Why

To check an aitools edit today you run either `test-exp-aitools` or the whole suite. Both take long enough that you don't run them after each edit. The quick tier covers the common failures (lint, unit tests, and broken importers such as `cmd/root`, which calls aitools from the root command's pre-run) in a time short enough to run after every edit. The full tier catches the output and wiring changes the quick tier misses. The bench script lets anyone re-measure that trade-off when the package lists change.

## Tests

- `./task aitools-quick` passes on this branch. Its three steps ran in parallel and the run took about 25s, with the test step the slowest.
- The doctests in `tools/aitools_quick.py` and `tools/aitools_quick_bench.py` pass (`python3 -m doctest`).
- No Go code changes, so this PR doesn't affect the existing unit and acceptance tests.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 requested a review from a team as a code owner November 19, 2026 19:37

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 63b26a4. Each finding was verified against the code.

No blocking issues found. 2 non-blocking comment(s) below.

This PR adds two verification tasks (aitools-quick and aitools-full) for the aitools command group plus a benchmark script. The code is well-structured with parallel step execution, hermetic test environments, and JSON reporting. Two small issues: the agent-env sanitization list is already out of sync with its stated source of truth, and the bench script's percentile helper crashes on an empty input that argparse accepts.

Comment thread tools/aitools_quick.py Resolved
Comment thread tools/aitools_quick_bench.py Resolved
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 0ac0390

Run: 44735025445

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 3:18
💚​ aws-windows-latest-4core-16gb 1 274 17 3:53
💚​ azure linux-2core-8gb 1 271 19 3:05
💚​ azure-windows-latest-4core-16gb 1 273 17 3:55
💚​ gcp linux-2core-8gb 1 272 19 3:21
💚​ gcp-windows-latest-4core-16gb 1 274 17 3:52
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:53 azure-windows-latest-4core-16gb TestAccept
3:51 gcp-windows-latest-4core-16gb TestAccept
3:51 aws-windows-latest-4core-16gb TestAccept

anton-107 and others added 2 commits November 19, 2026 20:24
Co-authored-by: Isaac <no-reply@databricks.com>
… the bench

The agent-env list now matches the one in acceptance/acceptance_test.go.
With --rounds 0 the time mode had no samples and percentile() raised
IndexError; argparse now reports the bad value instead.

Co-authored-by: Isaac <no-reply@databricks.com>

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review from simonfaltum's review agent (pr-review skill) at 63b26a4. Each finding was verified against the code; simonfaltum may follow up.

No blocking issues found, approving. 17 non-blocking comment(s) and 2 question(s) below.

This PR adds ./task aitools-quick and ./task aitools-full as fast and pre-PR verification tiers for aitools, plus a benchmark and an agent rule. The tiers are a useful idea and nothing blocking turned up. Some gaps undercut their promises, though: package-level test failures (timeouts, init panics) are dropped from the summary, moving HOME for the acceptance run hides uv's Python installs from the harness, and the importer and golden lists miss acceptance/internal/fakeagent and air/image-push. The only red CI job failed on a JFrog 503, which this PR didn't cause.

Questions

  1. PACKAGES and ACCEPTANCE in tools/aitools_quick.py duplicate the --packages/-run lists of test-exp-aitools and add more goldens (help, cmd/version, quickstart, air/*) and the importer tests. testmask sends an aitools-only PR to test-exp-aitools alone, so CI never runs those extras on such PRs. Should they also go into test-exp-aitools? Or should the two tasks share one list, so the local tier and CI don't drift? (non-blocking)
  2. The description explains why the bench exists (re-measuring the quick/full trade-off). Does it need to land with the two tiers, or could tools/aitools_quick_bench.py and the aitools-quick-bench task go in a follow-up PR, given AGENTS.md's one-change-per-PR rule? The tiers and the agent rule work without it. (non-blocking)

Agreeing with existing threads

Comment thread tools/aitools_quick.py Resolved
Comment thread tools/aitools_quick.py Resolved
Comment thread tools/aitools_quick.py Resolved
Comment thread tools/aitools_quick.py Resolved
Comment thread tools/aitools_quick.py Resolved
Comment thread tools/aitools_quick.py Resolved
Comment thread tools/aitools_quick.py Resolved
Comment thread .agents/rules/aitools-verification.md Resolved
Comment thread Taskfile.yml Resolved
Comment thread tools/aitools_quick.py Resolved
anton-107 and others added 2 commits November 20, 2026 12:14
Co-authored-by: Isaac <no-reply@databricks.com>
aitools-quick:
- Read the agent-detection variables from acceptance/acceptance_test.go
  instead of copying them, so the two lists can't drift; also remove
  CLAUDE_CODE_CHILD_SESSION and COPILOT_HOME.
- Report packages that fail without a failing test (timeout, init panic,
  TestMain exit) with their output.
- Run the acceptance step with the host environment; the harness isolates
  each test and finds uv's cache through the real HOME.
- Build acceptance/internal/fakeagent and run all of TestAccept/air.
- Default --ref to origin/main when it exists.
- Share changed_go_dirs with tools/lintdiff.py, which now passes
  --no-renames and keeps only directories that still hold a Go file.
- Pass the Taskfile's acceptance timeout to the full tier.
- Prefix test-log findings with the package directory.

aitools-quick-bench:
- Reject --json and --budget with --cold.
- Require both tiers to pass on the unmodified tree; run the full tier
  only on mutants the quick tier misses.
- Walk directories in a fixed order and skip import blocks.
- Edit one file in every aitools package.
- Pin the diff format used to copy uncommitted changes.

Also drop the timing claim from the agent rule and move the tasks next
to lint-q, away from the testmask comment.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit f68fa4b Nov 20, 2026
37 checks passed
@anton-107
anton-107 deleted the aitools-quick-verification branch November 20, 2026 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants