Skip to content

aitools: stop polling tests from waiting on the real poll interval - #7824

Merged
anton-107 merged 4 commits into
mainfrom
aitools-fast-poll-tests
Nov 20, 2026
Merged

anton-107 merged 4 commits into
mainfrom
aitools-fast-poll-tests

Conversation

@anton-107

Copy link
Copy Markdown
Contributor

Changes

This only changes unit tests in experimental/aitools/cmd. Production code is untouched.

  • query_test.go: TestExecuteAndPollWithPolling and TestExecuteAndPollFailsDuringPolling now build their client with sqlexec.WithPollInterval(time.Millisecond, time.Millisecond). They still go through the poll loop, but each poll waits 1ms.
  • batch_test.go, statement_test.go: TestExecuteBatchPreservesInputOrder and TestGetStatementResultPolls now call t.Parallel(). These two tests can't take a poll interval: executeBatch and getStatementResult create their own sqlexec.Client from the API interface. Running them in parallel means their 1s waits overlap.

Why

The sqlexec client polls with an additive backoff that starts at 1s and grows by 1s per poll. With mocked responses, every test that reached the poll loop slept for real, one test after another:

Test Polls Wait before Wait after
TestExecuteAndPollWithPolling 2 1s + 2s ~2ms
TestExecuteAndPollFailsDuringPolling 1 1s ~1ms
TestExecuteBatchPreservesInputOrder 1 1s 1s (in parallel)
TestGetStatementResultPolls 1 1s 1s (in parallel)

That's about 6s of sleeping before (my estimate from the backoff schedule) and about 1s after. The tests check the same things as before.

Tests

go test ./experimental/aitools/cmd -count=1 -run 'TestExecuteAndPoll|TestExecuteBatch|TestGetStatementResult' -v
--- PASS: TestExecuteAndPollWithPolling (0.00s)
--- PASS: TestExecuteAndPollFailsDuringPolling (0.00s)
--- PASS: TestGetStatementResultPolls (1.00s)
--- PASS: TestExecuteBatchPreservesInputOrder (1.00s)
ok  github.com/databricks/cli/experimental/aitools/cmd  1.015s

This is a test-only change, so there's no changelog fragment.

## Changes

This only changes unit tests in `experimental/aitools/cmd`. Production code is untouched.

- `query_test.go`: `TestExecuteAndPollWithPolling` and `TestExecuteAndPollFailsDuringPolling` now build their client with `sqlexec.WithPollInterval(time.Millisecond, time.Millisecond)`. They still go through the poll loop, but each poll waits 1ms.
- `batch_test.go`, `statement_test.go`: `TestExecuteBatchPreservesInputOrder` and `TestGetStatementResultPolls` now call `t.Parallel()`. These two tests can't take a poll interval: `executeBatch` and `getStatementResult` create their own `sqlexec.Client` from the API interface. Running them in parallel means their 1s waits overlap.

## Why

The `sqlexec` client polls with an additive backoff that starts at 1s and grows by 1s per poll. With mocked responses, every test that reached the poll loop slept for real, one test after another:

| Test | Polls | Wait before | Wait after |
| --- | --- | --- | --- |
| `TestExecuteAndPollWithPolling` | 2 | 1s + 2s | ~2ms |
| `TestExecuteAndPollFailsDuringPolling` | 1 | 1s | ~1ms |
| `TestExecuteBatchPreservesInputOrder` | 1 | 1s | 1s (in parallel) |
| `TestGetStatementResultPolls` | 1 | 1s | 1s (in parallel) |

That's about 6s of sleeping before (my estimate from the backoff schedule) and about 1s after. The tests check the same things as before.

## Tests

```
go test ./experimental/aitools/cmd -count=1 -run 'TestExecuteAndPoll|TestExecuteBatch|TestGetStatementResult' -v
--- PASS: TestExecuteAndPollWithPolling (0.00s)
--- PASS: TestExecuteAndPollFailsDuringPolling (0.00s)
--- PASS: TestGetStatementResultPolls (1.00s)
--- PASS: TestExecuteBatchPreservesInputOrder (1.00s)
ok  github.com/databricks/cli/experimental/aitools/cmd  1.015s
```

This is a test-only change, so there's no changelog fragment.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 requested review from a team and lennartkats-db as code owners November 19, 2026 19:34
@github-actions github-actions Bot added aitools AI tools (databricks aitools) related changes test-only-fix PRs that update failing of flaky tests without changing CLI code. labels Nov 19, 2026

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 4f887ea. Each finding was verified against the code.

No blocking issues found. 1 non-blocking comment(s) below.

This test-only PR reduces polling-test wall-clock time from ~6 s to ~1 s by injecting a fastPoll option into the two tests that build their own sqlexec.Client, and by marking the other two (whose callers hard-code sqlexec.New without option forwarding) as t.Parallel(). The approach is correct and an improvement, though the parallel workaround still sleeps 1 s per test when run in isolation.

Comment thread experimental/aitools/cmd/batch_test.go Resolved
The polling tests for executeBatch and getStatementResult can now use
the 1ms poll interval too, so they no longer wait on the real 1s
interval or depend on running in parallel.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot removed the test-only-fix PRs that update failing of flaky tests without changing CLI code. label Nov 19, 2026

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review from simonfaltum's review agent (pr-review skill) at 2a44125. Each finding was verified against the code; simonfaltum may follow up.

No blocking issues found, approving. 4 non-blocking comment(s) and 1 question(s) below.

This PR makes the aitools polling tests use a 1ms poll interval instead of the real 1s+ backoff. To do that, it adds a variadic sqlexec.Option parameter to executeBatch and getStatementResult. The code is correct and production behaviour is unchanged: the package's tests pass, including under -race, and take ~0s instead of ~6s. The main ask is to update the PR description, which still describes the first commit's t.Parallel() approach and will become the squash commit message. The rest are small comment and test nits, plus one optional question about passing a *sqlexec.Client instead of options.

Questions

  1. executeBatch and getStatementResult now take opts ...sqlexec.Option, but only tests pass any. Both production callers (query.go:243, statement_get.go:33) pass none, and both functions use api/warehouseID only to build the client. In the same package, executeAndPoll and cancelInFlight already take a *sqlexec.Client, which is how TestExecuteAndPollWithPolling injects fastPoll. Would you consider having these two functions take a *sqlexec.Client that their callers build? That would remove the test-only parameter and match the existing seam. Keeping the options is also reasonable if you prefer them. (non-blocking)

Accepted answers

  • Forward poll options so the last two polling tests skip the real 1s wait: #7824 (comment)

Comments outside the changed lines

  • [nice to have] experimental/aitools/cmd/batch.go: The PR description still describes the first commit, and this repo squash-merges with the PR body as the commit message. It says "Production code is untouched" and that TestExecuteBatchPreservesInputOrder / TestGetStatementResultPolls call t.Parallel() because they "can't take a poll interval". At head, executeBatch (batch.go:63) and getStatementResult (statement_get.go:63) take opts ...sqlexec.Option, both tests pass fastPoll, and there is no t.Parallel(). These are also stale: the table's "1s (in parallel)" rows, the "about 1s after" estimate, and the pasted test output (1.00s each, 1.015s total). All four tests now run in ~0s. Please update Changes/Why/Tests to match the head commit. Skipping the changelog is still right (experimental/, no user-visible change), but "test-only change" no longer holds. While you're there: the commits are co-authored by Isaac, and .agents/skills/pr-checklist asks for a one-line italic agent disclosure at the end of the body.

Comment thread experimental/aitools/cmd/batch.go Resolved
Comment thread experimental/aitools/cmd/batch_test.go Resolved
Comment thread experimental/aitools/cmd/query_test.go Resolved
anton-107 and others added 2 commits November 19, 2026 23:05
Co-authored-by: Isaac <no-reply@databricks.com>
- Say why executeBatch and getStatementResult take poll options.
- Move fastPoll to the top of query_test.go and fix its comment.
- Make TestExecuteBatchPreservesInputOrder deterministic: with two slots,
  the slow statement's poll blocks until the 'fast2' submission starts,
  so it always completes last.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 31e9487 Nov 20, 2026
23 checks passed
@anton-107
anton-107 deleted the aitools-fast-poll-tests branch November 20, 2026 00:21
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

❌ Integration test report

Commit: 31e9487

Run: 44645292564

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 27 1619 1419 178:19
🔄​ azure linux-2core-8gb 2 25 1599 1427 166:40
🔄​ gcp linux-2core-8gb 3 26 1604 1425 187:12
29 interesting tests: 24 RECOVERED, 5 flaky
Test Name aws linux-2core-8gb azure linux-2core-8gb gcp linux-2core-8gb
🔄​ TestAccept 💚​R 💚​R 🔄​f
💚​ TestAccept/bundle/deployment/bind/alert 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/deployment/bind/alert/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/deployment/bind/alert/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/generate/alert 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/generate/alert/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/generate/alert/DMS=true 💚​R 💚​R 💚​R
🔄​ TestAccept/bundle/generate/pipeline_and_deploy ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/generate/pipeline_and_deploy/DMS= ✅​p ✅​p 🔄​f
💚​ TestAccept/bundle/resources/alerts/basic 💚​R 💚​R 💚​R
🔄​ TestAccept/bundle/resources/alerts/basic/DMS= 💚​R 🔄​f 💚​R
🔄​ TestAccept/bundle/resources/alerts/basic/DMS=true 💚​R 🔄​f 💚​R
💚​ TestAccept/bundle/resources/alerts/with_file 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/with_file/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/alerts/with_file/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started/DMS=true 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/basic 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/basic/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS= 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS=true 💚​R 💚​R 💚​R
Top 50 slowest tests (at least 2 minutes):
duration env testname
10:55 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:30 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:27 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:19 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:02 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:47 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:33 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:24 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
7:23 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=
7:21 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=
7:02 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
6:51 gcp linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
6:46 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
6:35 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:24 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
6:23 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:19 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:09 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
5:47 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
5:40 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:34 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
5:29 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
5:04 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=true
5:00 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
4:47 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
4:35 azure linux-2core-8gb TestAccept/pipelines/run/refresh-flags
4:24 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
4:23 gcp linux-2core-8gb TestAccept/pipelines/run/refresh-flags
4:16 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
4:08 aws linux-2core-8gb TestAccept/pipelines/run/refresh-flags
4:06 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=
3:57 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
3:56 gcp linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
3:54 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=
3:50 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true
3:43 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
3:41 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
3:38 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
3:38 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=
3:37 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
3:35 aws linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
3:35 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
3:30 gcp linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
3:19 azure linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
3:18 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=
3:17 gcp linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
3:16 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
3:15 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
3:11 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=
3:11 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

aitools AI tools (databricks aitools) related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants