Skip to content

air: Say when --idempotency-key returns an existing run - #7529

Open
abdulkareem-db wants to merge 7 commits into
mainfrom
abdulkareem/air-idempotency-existing-run
Open

abdulkareem-db wants to merge 7 commits into
mainfrom
abdulkareem/air-idempotency-existing-run

Conversation

@abdulkareem-db

@abdulkareem-db abdulkareem-db commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Changes

When air run is given an idempotency key that was already used (via --idempotency-key or the YAML idempotency_token) and the API returns the existing run, the CLI now says so:

Returned existing workload with Job Run ID: 555 (idempotency key "my-key" was already used)
View job run at: <host>/jobs/runs/555

Fresh submits print the same output as before. The -o json submit result now has an existing field:

  • true when the key returned an earlier run.
  • false when the run is new, including every submit with a generated token, since that token can't repeat.
  • Omitted when the CLI can't tell. It isn't reported as a new run in that case. Dry-run JSON also omits it, because nothing is submitted.

The --watch JSONL events are unchanged. --watch on an existing run prints the line above and then streams that run's logs as before.

Why

The --idempotency-key help says "Return the existing run if this key was already used", but a repeated submit printed "Submitted workload with Job Run ID: …" and identical JSON. Users couldn't tell a deduplicated submit from a new run.

runs/submit returns only run_id, so the CLI checks the run afterwards. Each submit uploads its command.sh to a fresh launch directory named <run>_<16 hex> (cliLaunchDir). runs/get returns the run's ai_runtime_task.deployments[].command_path, which air get already reads. The CLI compares only the launch directory name from that path with this submit's. If they differ, the run came from an earlier submit. The rest of the path isn't compared, because its prefix can differ between the submitted path and the one runs/get returns (/Users/... vs /Workspace/Users/...). Timestamps weren't used, because clock skew makes them unreliable for the quick retries idempotency keys are meant for.

  • The extra runs/get is made only when the key is user-supplied (flag or YAML) and the result is shown: text output, with or without --watch, and non-watch -o json. It's skipped under --watch -o json, whose events don't carry existing. Generated tokens skip it too, so most submits make no extra call.
  • The lookup times out after 5 seconds, so a slow backend can't hold up the success line.
  • If the CLI can't tell, it logs a warning when runs/get failed, prints the usual "Submitted workload" line, and leaves existing out of the JSON. This covers four cases: runs/get failed or timed out, the run has no deployment, its command path isn't a recognized launch directory, or the workload uses containers. The lookup therefore can't fail a submit that succeeded.
  • submitWorkload's signature is unchanged. air run opts in with withExistingRunReport(ctx), following the context-pointer pattern of armDryRunValidationAttemptBudget, because the command path is built inside submitWorkload and the caller can't recompute it.
  • Precedence between the flag and the YAML token is unchanged.

Known limitation: for workloads that use containers the CLI can't tell whether the run existed, because their deployment spec carries no command_path. They print the usual line and omit existing.

libs/testserver didn't model runs/submit idempotency. It now returns the earlier run for a repeated token, following the existing run-now pattern, and keeps the submitted ai_runtime_task on the run.

Tests

  • Unit tests in cmd/air/run_test.go:
    • reuse via the flag, reuse via the YAML token, and the JSON existing field. Fresh-submit text output is unchanged.
    • --watch text mode with a reused key prints the existing-run line before the streamed logs, and logs still stream.
    • the lookup is made for text (with or without --watch) and non-watch -o json, and skipped under --watch -o json.
    • the launch directory matches when the submitted and returned path prefixes differ (/Users vs /Workspace/Users).
    • when runs/get fails or times out, there's no deployment, or the command path is empty or unrecognized, existing is omitted and the text line is unchanged.
    • containers workloads make no lookup.
  • Every added executable line in cmd/air/run.go and cmd/air/runsubmit.go is covered by go test ./cmd/air/... -coverprofile.
  • TestJobsSubmit_IdempotencyTokenReturnsExistingRun for the test server.
  • New acceptance test acceptance/air/run-idempotency: fresh submit, reuse via the flag (text and JSON), a different key, and reuse via YAML. No existing golden outputs changed.
  • go test ./cmd/air/... ./libs/testserver/... and go test ./acceptance -run TestAccept/air pass. Lint reports 0 issues.

This pull request and its description were written by Isaac.

abdulkareem-db and others added 3 commits November 11, 2026 22:48
Resubmitting `air run` with an already-used idempotency key (flag or YAML
idempotency_token) printed "Submitted workload with Job Run ID" exactly
like a fresh submit, so users could not tell a deduplicated submit from a
new one (CNXT-2520).

runs/submit returns only a run id, so when the token is caller-supplied the
CLI now fetches the run and compares its command_path with the one this
submit uploaded. Every submit uses a fresh launch directory, so a run
pointing at a different command path was started by an earlier submit.
Generated tokens skip the lookup.

- Text output says "Returned existing workload with Job Run ID: <id>
  (idempotency key "<key>" was already used)".
- JSON results (PENDING envelope, --watch SUBMITTED event and closing
  envelope) carry an `existing` boolean; dry runs omit it.
- The fake workspace's runs/submit now honors idempotency tokens and keeps
  the submitted ai_runtime_task on the run.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Keep submitWorkload's signature unchanged: run opts in through the
context, and submitWorkload records whether a caller-supplied
idempotency token returned an existing run. Drop `existing` from the
--watch JSONL events; the text line and the non-watch -o json result
still report it.

Co-authored-by: Isaac <no-reply@databricks.com>
@abdulkareem-db
abdulkareem-db requested review from a team as code owners November 12, 2026 00:06
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 872308f07 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (8f35f97) latest (v1.26.0)
TestAccept/air/run-idempotency ✅ ❌ ❌
TestAccept/air/run-idempotency: branch ✅ | main (8f35f97) ❌ | latest (v1.26.0) ❌

main (8f35f97):

acceptance_test.go:1380: Diff:
        --- air/run-idempotency/output.txt
        +++ /tmp/TestAcceptairrun-idempotency3077355146/002/output.txt
        @@ -14,7 +14,7 @@
         
         >>> [CLI] air run -f run.yaml --idempotency-key flag-key
         Submitting experiment: idempotency-smoke
        -Returned existing workload with Job Run ID: [FLAG_KEY_RUN_ID] (idempotency key "flag-key" was already used)
        +Submitted workload with Job Run ID: [FLAG_KEY_RUN_ID]
         View job run at: [DATABRICKS_URL]/jobs/runs/[FLAG_KEY_RUN_ID]?o=[WORKSPACE_ID]
         
         Tip: use --watch when submitting a run to stream logs to your terminal.
        @@ -30,21 +30,20 @@
           "data": {
             "status": "PENDING",
             "run_id": "[FLAG_KEY_RUN_ID]",
        -    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/[FLAG_KEY_RUN_ID]?o=[WORKSPACE_ID]",
        -    "existing": true
        +    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/[FLAG_KEY_RUN_ID]?o=[WORKSPACE_ID]"
           }
         }
         
         === a different key starts a new run
         
         >>> [CLI] air run -f run.yaml --idempotency-key other-key -o json
        -false
        +null
         
         === a YAML idempotency_token is deduplicated the same way
         
         >>> [CLI] air run -f run-token.yaml
         Submitting experiment: idempotency-smoke
        -Returned existing workload with Job Run ID: [YAML_TOKEN_RUN_ID] (idempotency key "yaml-key" was already used)
        +Submitted workload with Job Run ID: [YAML_TOKEN_RUN_ID]
         View job run at: [DATABRICKS_URL]/jobs/runs/[YAML_TOKEN_RUN_ID]?o=[WORKSPACE_ID]
         
         Tip: use --watch when submitting a run to stream logs to your terminal.
        
FAIL

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- air/run-idempotency/output.txt
        +++ /tmp/TestAcceptairrun-idempotency3415640600/002/output.txt
        @@ -4,7 +4,7 @@
         >>> [CLI] air run -f run.yaml --idempotency-key flag-key
         Submitting experiment: idempotency-smoke
         Submitted workload with Job Run ID: [FLAG_KEY_RUN_ID]
        -View job run at: [DATABRICKS_URL]/jobs/runs/[FLAG_KEY_RUN_ID]?o=[WORKSPACE_ID]
        +View job run at: [DATABRICKS_URL]/jobs/runs/[FLAG_KEY_RUN_ID]
         
         Tip: use --watch when submitting a run to stream logs to your terminal.
         Stream logs after submission using:
        @@ -14,8 +14,8 @@
         
         >>> [CLI] air run -f run.yaml --idempotency-key flag-key
         Submitting experiment: idempotency-smoke
        -Returned existing workload with Job Run ID: [FLAG_KEY_RUN_ID] (idempotency key "flag-key" was already used)
        -View job run at: [DATABRICKS_URL]/jobs/runs/[FLAG_KEY_RUN_ID]?o=[WORKSPACE_ID]
        +Submitted workload with Job Run ID: [FLAG_KEY_RUN_ID]
        +View job run at: [DATABRICKS_URL]/jobs/runs/[FLAG_KEY_RUN_ID]
         
         Tip: use --watch when submitting a run to stream logs to your terminal.
         Stream logs after submission using:
        @@ -30,22 +30,21 @@
           "data": {
             "status": "PENDING",
             "run_id": "[FLAG_KEY_RUN_ID]",
        -    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/[FLAG_KEY_RUN_ID]?o=[WORKSPACE_ID]",
        -    "existing": true
        +    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/[FLAG_KEY_RUN_ID]"
           }
         }
         
         === a different key starts a new run
         
         >>> [CLI] air run -f run.yaml --idempotency-key other-key -o json
        -false
        +null
         
         === a YAML idempotency_token is deduplicated the same way
         
         >>> [CLI] air run -f run-token.yaml
         Submitting experiment: idempotency-smoke
        -Returned existing workload with Job Run ID: [YAML_TOKEN_RUN_ID] (idempotency key "yaml-key" was already used)
        -View job run at: [DATABRICKS_URL]/jobs/runs/[YAML_TOKEN_RUN_ID]?o=[WORKSPACE_ID]
        +Submitted workload with Job Run ID: [YAML_TOKEN_RUN_ID]
        +View job run at: [DATABRICKS_URL]/jobs/runs/[YAML_TOKEN_RUN_ID]
         
         Tip: use --watch when submitting a run to stream logs to your terminal.
         Stream logs after submission using:
        
FAIL

abdulkareem-db and others added 2 commits November 12, 2026 00:24
…o json

The --watch JSON events never carry `existing`, so the runs/get that
detects a reused idempotency key was wasted there. Keep it for text
output (with or without --watch) and the non-watch JSON result.

Add tests for --watch text output printing the existing-run line before
the streamed logs, the per-mode runs/get lookup count, the runs/get
failure and missing command path fallbacks, and the containers skip.

Co-authored-by: Isaac <no-reply@databricks.com>
The existing-run report is now tri-state: JSON omits `existing` when the
CLI could not tell (runs/get failed or timed out, no deployment, an
unrecognized command path, or a container run) instead of claiming false.
Generated-token submits still report false, since they can never repeat.

The lookup compares only the unique <run>_<hex> launch directory, so a
different workspace path prefix in the runs/get echo cannot turn a fresh
run into an existing one, and is bounded by a 5s timeout.

Co-authored-by: Isaac <no-reply@databricks.com>
@abdulkareem-db

Copy link
Copy Markdown
Contributor Author

End-to-end check against a real workspace, using a CLI built from 6e31540 and a minimal 1×A10 workload (command: echo ..., no code_source):

# Command Result
1 air run --idempotency-key K (first use) Submitted workload with Job Run ID: <A>
2 same key again Returned existing workload with Job Run ID: <A> (idempotency key "K" was already used)
3 same key, -o json "run_id": "<A>", "existing": true
4 different key, -o json new run <B>, "existing": false
5 same key, --watch existing-run line printed before the logs, then run <A>'s logs streamed to completion, exit 0

So the backend echoes the deployment command_path the CLI submitted, and the launch-directory comparison tells a deduplicated submit apart from a fresh one. Steps 2, 3 and 5 started no new runs.

This comment was written by Isaac.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 872308f

Run: 43936041495

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 16 9:13
💚​ aws-windows-latest-4core-16gb 1 272 14 7:38
💚​ azure linux-2core-8gb 1 269 16 10:09
💚​ azure-windows-latest-4core-16gb 1 271 14 7:46
💚​ gcp linux-2core-8gb 1 270 16 11:23
💚​ gcp-windows-latest-4core-16gb 1 272 14 7:55
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 13 slowest tests (at least 2 minutes):
duration env testname
4:06 gcp-windows-latest-4core-16gb TestAccept
3:51 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsRead
3:45 azure-windows-latest-4core-16gb TestAccept
3:20 aws-windows-latest-4core-16gb TestAccept
3:16 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:55 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
2:48 azure-windows-latest-4core-16gb TestSyncIncrementalFileSync
2:34 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:32 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files
2:32 aws-windows-latest-4core-16gb TestSyncIncrementalFileSync
2:06 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
2:04 gcp linux-2core-8gb TestImportDir
2:03 aws-windows-latest-4core-16gb TestImportDirDoesNotOverwrite

@pardis-beikzadeh-db pardis-beikzadeh-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description is out of date since 6e31540. Please update it to match the code:

  • When the CLI can't tell whether the run already existed, existing is left out of the JSON. It isn't reported as new.
  • Only the launch directory name is compared.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants