Repository navigation
tools: add aitools-quick and aitools-full verification tasks - #7825
Conversation
## Changes Adds two verification tasks for the aitools command group (`cmd/aitools`, `libs/aitools`, `experimental/aitools`), plus a benchmark for the faster one: - `./task aitools-quick` runs three steps in parallel: golangci-lint and the unit tests of the aitools packages, and `go build` of the packages that import aitools (the root package, `cmd`, `cmd/root`, `cmd/apps` and others). Packages with Go changes since the merge base with `main` (`--ref` picks another ref) are also linted and tested, so editing a shared package such as `libs/telemetry` runs its tests too. Every step gets a one-line summary, and failed steps also print their output. With `-- --json <path>` it also writes a machine-readable report with `file:line` findings. - `./task aitools-full` runs the same lint, then the unit tests of the importers, the acceptance tests that exercise aitools (`cmd/aitools`, `experimental/aitools`, `apps`, the help and version goldens, `quickstart`, `air` help) and `tools/check_deadcode.py`. - Unit and acceptance tests run with coding-agent detection variables and `DATABRICKS_*` variables removed, and with `HOME` pointing at a fixed directory under the user cache dir. This way the agent a developer happens to run in can't change aitools behavior, and Go's test cache stays valid between runs. The Go cache, module cache and `go env` file are pinned so moving `HOME` doesn't force a cold build. - `./task aitools-quick-bench` measures the quick tier. `time` makes a small edit in one file per aitools package and reports p50/p95 wall time, failing if p50 exceeds `--budget`. `detect` applies source mutants one at a time and reports what share of the mutants caught by the full tier the quick tier also catches. Both modes work in a temporary git worktree and never write to the checkout. - An `.agents/rules` file (with its `.cursor/rules` symlink), scoped to the aitools paths, tells coding agents to run `aitools-quick` after every edit there and `aitools-full` before opening a PR. ## Why To check an aitools edit today you run either `test-exp-aitools` or the whole suite. Both take long enough that you don't run them after each edit. The quick tier covers the common failures (lint, unit tests, and broken importers such as `cmd/root`, which calls aitools from the root command's pre-run) in a time short enough to run after every edit. The full tier catches the output and wiring changes the quick tier misses. The bench script lets anyone re-measure that trade-off when the package lists change. ## Tests - `./task aitools-quick` passes on this branch. Its three steps ran in parallel and the run took about 25s, with the test step the slowest. - The doctests in `tools/aitools_quick.py` and `tools/aitools_quick_bench.py` pass (`python3 -m doctest`). - No Go code changes, so this PR doesn't affect the existing unit and acceptance tests. Co-authored-by: Isaac <no-reply@databricks.com>
There was a problem hiding this comment.
Automated review at 63b26a4. Each finding was verified against the code.
No blocking issues found. 2 non-blocking comment(s) below.
This PR adds two verification tasks (aitools-quick and aitools-full) for the aitools command group plus a benchmark script. The code is well-structured with parallel step execution, hermetic test environments, and JSON reporting. Two small issues: the agent-env sanitization list is already out of sync with its stated source of truth, and the bench script's percentile helper crashes on an empty input that argparse accepts.
✅ Integration test reportCommit: 0ac0390
Top 3 slowest tests (at least 2 minutes):
|
Co-authored-by: Isaac <no-reply@databricks.com>
… the bench The agent-env list now matches the one in acceptance/acceptance_test.go. With --rounds 0 the time mode had no samples and percentile() raised IndexError; argparse now reports the bad value instead. Co-authored-by: Isaac <no-reply@databricks.com>
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at 63b26a4. Each finding was verified against the code; simonfaltum may follow up.
No blocking issues found, approving. 17 non-blocking comment(s) and 2 question(s) below.
This PR adds ./task aitools-quick and ./task aitools-full as fast and pre-PR verification tiers for aitools, plus a benchmark and an agent rule. The tiers are a useful idea and nothing blocking turned up. Some gaps undercut their promises, though: package-level test failures (timeouts, init panics) are dropped from the summary, moving HOME for the acceptance run hides uv's Python installs from the harness, and the importer and golden lists miss acceptance/internal/fakeagent and air/image-push. The only red CI job failed on a JFrog 503, which this PR didn't cause.
Questions
PACKAGESandACCEPTANCEintools/aitools_quick.pyduplicate the--packages/-runlists oftest-exp-aitoolsand add more goldens (help,cmd/version,quickstart,air/*) and the importer tests. testmask sends an aitools-only PR totest-exp-aitoolsalone, so CI never runs those extras on such PRs. Should they also go intotest-exp-aitools? Or should the two tasks share one list, so the local tier and CI don't drift? (non-blocking)- The description explains why the bench exists (re-measuring the quick/full trade-off). Does it need to land with the two tiers, or could
tools/aitools_quick_bench.pyand theaitools-quick-benchtask go in a follow-up PR, given AGENTS.md's one-change-per-PR rule? The tiers and the agent rule work without it. (non-blocking)
Agreeing with existing threads
- [nit] OMNIGENT missing from AGENT_ENV: #7825 (comment)
- [nit] --rounds 0 crashes in percentile([]): #7825 (comment)
Co-authored-by: Isaac <no-reply@databricks.com>
aitools-quick: - Read the agent-detection variables from acceptance/acceptance_test.go instead of copying them, so the two lists can't drift; also remove CLAUDE_CODE_CHILD_SESSION and COPILOT_HOME. - Report packages that fail without a failing test (timeout, init panic, TestMain exit) with their output. - Run the acceptance step with the host environment; the harness isolates each test and finds uv's cache through the real HOME. - Build acceptance/internal/fakeagent and run all of TestAccept/air. - Default --ref to origin/main when it exists. - Share changed_go_dirs with tools/lintdiff.py, which now passes --no-renames and keeps only directories that still hold a Go file. - Pass the Taskfile's acceptance timeout to the full tier. - Prefix test-log findings with the package directory. aitools-quick-bench: - Reject --json and --budget with --cold. - Require both tiers to pass on the unmodified tree; run the full tier only on mutants the quick tier misses. - Walk directories in a fixed order and skip import blocks. - Edit one file in every aitools package. - Pin the diff format used to copy uncommitted changes. Also drop the timing claim from the agent rule and move the tasks next to lint-q, away from the testmask comment. Co-authored-by: Isaac <no-reply@databricks.com>
Changes
Adds two verification tasks for the aitools command group (
cmd/aitools,libs/aitools,experimental/aitools), plus a benchmark for the faster one:./task aitools-quickruns three steps in parallel: golangci-lint and the unit tests of the aitools packages, andgo buildof the packages that import aitools (the root package,cmd,cmd/root,cmd/appsand others). Packages with Go changes since the merge base withmain(--refpicks another ref) are also linted and tested, so editing a shared package such aslibs/telemetryruns its tests too. Every step gets a one-line summary, and failed steps also print their output. With-- --json <path>it also writes a machine-readable report withfile:linefindings../task aitools-fullruns the same lint, then the unit tests of the importers, the acceptance tests that exercise aitools (cmd/aitools,experimental/aitools,apps, the help and version goldens,quickstart,airhelp) andtools/check_deadcode.py.DATABRICKS_*variables removed, and withHOMEpointing at a fixed directory under the user cache dir. This way the agent a developer happens to run in can't change aitools behavior, and Go's test cache stays valid between runs. The Go cache, module cache andgo envfile are pinned so movingHOMEdoesn't force a cold build../task aitools-quick-benchmeasures the quick tier.timemakes a small edit in one file per aitools package and reports p50/p95 wall time, failing if p50 exceeds--budget.detectapplies source mutants one at a time and reports what share of the mutants caught by the full tier the quick tier also catches. Both modes work in a temporary git worktree and never write to the checkout..agents/rulesfile (with its.cursor/rulessymlink), scoped to the aitools paths, tells coding agents to runaitools-quickafter every edit there andaitools-fullbefore opening a PR.Why
To check an aitools edit today you run either
test-exp-aitoolsor the whole suite. Both take long enough that you don't run them after each edit. The quick tier covers the common failures (lint, unit tests, and broken importers such ascmd/root, which calls aitools from the root command's pre-run) in a time short enough to run after every edit. The full tier catches the output and wiring changes the quick tier misses. The bench script lets anyone re-measure that trade-off when the package lists change.Tests
./task aitools-quickpasses on this branch. Its three steps ran in parallel and the run took about 25s, with the test step the slowest.tools/aitools_quick.pyandtools/aitools_quick_bench.pypass (python3 -m doctest).