Skip to content

serving-endpoints: explain a failed create or update-config wait - #7669

Open
anshul0713 wants to merge 2 commits into
anshul0713/serving-endpoints-wait-progressfrom
anshul0713/serving-endpoints-wait-failure-details
Open

anshul0713 wants to merge 2 commits into
anshul0713/serving-endpoints-wait-progressfrom
anshul0713/serving-endpoints-wait-failure-details

Conversation

@anshul0713

Copy link
Copy Markdown
Contributor

Stacked on #7659 (wait progress). Review that one first; this PR's base will move to main once it merges.

Changes

When serving-endpoints create or update-config fails or times out while waiting, the error now adds each served entity's last deployment state and the commands that show its logs. Real output from a model with an uninstallable pip requirement:

Error: failed to reach NOT_UPDATING, got UPDATE_FAILED: current status: UPDATE_FAILED
bad-again: Container build failed. Cause: user_pip_resolution. See Build Logs for details.
  build logs:  databricks serving-endpoints build-logs ag-cli-audit-fail bad-again
  server logs: databricks serving-endpoints logs ag-cli-audit-fail bad-again

The suggested build-logs command then shows ERROR: Could not find a version that satisfies the requirement this-package-does-not-exist==0.0.1.

Why

The generated wait error never says why the update failed, but the served entity's deployment_state_message already does. In a terminal, the spinner that showed that state is cleared before the error prints. Both agent trials of a "debug a failed deployment" journey had to run get and then build-logs to find a cause the CLI already had.

Backward compatibility

  • The original error message comes first and is still wrapped (errors.Is keeps working). The exit code is unchanged.
  • Successful waits, --no-wait, stdout and requests are unchanged.
  • If the last poll has no served entity state, the error is unchanged.

Tests

  • Unit tests: a failed entity (shape taken from a real UPDATE_FAILED endpoint), no poll result, and entities without state.
  • Existing serving-endpoint acceptance tests (including serving-endpoints: show served entity deployment state while waiting #7659's) and bundle tests pass. The fake server never fails an update, so the failure path is covered by unit tests and the manual run.
  • Ran against a staging workspace: the output above comes from update-config onto a model version whose requirements can't be installed.
  • ./task lint passes.

This pull request and its description were written by Isaac.

When the endpoint update failed, the wait only returned "failed to reach
NOT_UPDATING, got UPDATE_FAILED: current status: UPDATE_FAILED", even
though the served entity's state already named the cause (for example
"Container build failed. Cause: user_pip_resolution. See Build Logs for
details."). In a terminal, the spinner that showed that state is gone by
then. Add each served entity's last state to the error, with the
build-logs and logs commands for it. The original error stays first and
is still wrapped, and the exit code is unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Anshul Gupta <anshul.gupta@databricks.com>
@anshul0713
anshul0713 requested a review from a team as a code owner November 16, 2026 08:00
@anshul0713
anshul0713 requested a review from anton-107 November 16, 2026 08:00
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Anshul Gupta <anshul.gupta@databricks.com>
@github-actions

Copy link
Copy Markdown
Contributor

Commit: cfa2897cd | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (620ea05) latest (v1.26.0)
TestAccept/cmd/workspace/serving-endpoints ✅ ❌ ❌
TestAccept/cmd/workspace/serving-endpoints: branch ✅ | main (620ea05) ❌ | latest (v1.26.0) ❌

main (620ea05):

acceptance_test.go:1374: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1374
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1115
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n=== Create waits and reports progress on stderr; stdout is the endpoint JSON\n\n>>> [CLI] serving-endpoints create --json @input.json\n{\n  \"name\": \"iris-clf\",\n  \"state\": {\n    \"config_update\": \"NOT_UPDATING\",\n    \"ready\": \"NOT_READY\"\n  }\n}\n\n=== Create with --no-wait prints the create response and no progress\n\n>>> [CLI] serving-endpoints create --no-wait --json {\"name\": \"iris-clf-2\"}\n{\n  \"name\": \"iris-clf-2\"\n}\n\n=== Update config waits and reports progress\n\n>>> [CLI] serving-endpoints update-config iris-clf --json @update.json\ncurrent status: IN_PROGRESS ([ELAPSED])\n{\n  \"name\": \"iris-clf\",\n  \"state\": {\n    \"config_update\": \"NOT_UPDATING\",\n    \"ready\": \"NOT_READY\"\n  }\n}\n\n=== Help mentions the wait and --no-wait\n\n>>> [CLI] serving-endpoints create --help\n"
        	            	actual  : "\n=== Create waits and reports progress on stderr; stdout is the endpoint JSON\n\n>>> [CLI] serving-endpoints create --json @input.json\n{\n  \"name\": \"iris-clf\",\n  \"state\": {\n    \"config_update\": \"NOT_UPDATING\",\n    \"ready\": \"NOT_READY\"\n  }\n}\n\n=== Create with --no-wait prints the create response and no progress\n\n>>> [CLI] serving-endpoints create --no-wait --json {\"name\": \"iris-clf-2\"}\n{\n  \"name\": \"iris-clf-2\"\n}\n\n=== Update config waits and reports progress\n\n>>> [CLI] serving-endpoints update-config iris-clf --json @update.json\n{\n  \"name\": \"iris-clf\",\n  \"state\": {\n    \"config_update\": \"NOT_UPDATING\",\n    \"ready\": \"NOT_READY\"\n  }\n}\n\n=== Help mentions the wait and --no-wait\n\n>>> [CLI] serving-endpoints create --help\ncontains error: 'Use --no-wait to return immediately' not found in the output.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -22,3 +22,2 @@
        	            	 >>> [CLI] serving-endpoints update-config iris-clf --json @update.json
        	            	-current status: IN_PROGRESS ([ELAPSED])
        	            	 {
        	            	@@ -34,2 +33,5 @@
        	            	 >>> [CLI] serving-endpoints create --help
        	            	+contains error: 'Use --no-wait to return immediately' not found in the output.
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/cmd/workspace/serving-endpoints
        	Messages:   	cmd/workspace/serving-endpoints/output.txt vs /tmp/TestAcceptcmdworkspaceserving-endpoints3748123939/002/output.txt
FAIL

latest (v1.26.0): same output as main (620ea05)

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: cfa2897

Run: 43827996367

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 16 3:48
💚​ aws-windows-latest-4core-16gb 1 272 14 3:33
💚​ azure linux-2core-8gb 1 269 16 3:43
💚​ azure-windows-latest-4core-16gb 1 271 14 3:24
💚​ gcp linux-2core-8gb 1 270 16 3:54
💚​ gcp-windows-latest-4core-16gb 1 272 14 3:24
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:28 aws-windows-latest-4core-16gb TestAccept
3:18 azure-windows-latest-4core-16gb TestAccept
3:18 gcp-windows-latest-4core-16gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants