Skip to content

serving-endpoints: show served entity deployment state while waiting - #7659

Open
anshul0713 wants to merge 2 commits into
mainfrom
anshul0713/serving-endpoints-wait-progress
Open

anshul0713 wants to merge 2 commits into
mainfrom
anshul0713/serving-endpoints-wait-progress

Conversation

@anshul0713

@anshul0713 anshul0713 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Changes

databricks serving-endpoints create and update-config now show each served entity's deployment state and the time waited while they wait for the endpoint. Before, they showed only current status: IN_PROGRESS, which stays the same for the whole deployment.

When stderr isn't a terminal, each state change is printed as one line on stderr. Before, nothing was printed. Interactive terminals still get the spinner, now with the new text. Stdout is unchanged in both cases.

Real run on a Small CPU endpoint with stderr redirected:

anshul_obs_demo_model-1: Creating resources for served entity. (0s)
anshul_obs_demo_model-1: Provisioning resources for served entities. (1m9s)
anshul_obs_demo_model-1: Deploying served entities. (2m2s)

The --help for both commands now also says that they wait by default (up to --timeout), and points to --no-wait plus serving-endpoints get.

This is a hand-written override (cmd/workspace/serving-endpoints/overrides.go), because the generated wait only reads state.config_update. It reuses the generated flags and --json handling, so the request sent to the API doesn't change.

Why

In agent trials of a train → register → serve → query journey, every agent's serving-endpoints create call blocked for 7-9 minutes with no output and hit its tool timeout. Each agent then wrote its own serving-endpoints get polling loop. The per-entity deployment_state_message already says which phase the endpoint is in; the CLI just never showed it.

Backward compatibility

  • Stdout, exit codes, flags and the request body are unchanged.
  • If the endpoint has already finished updating at the first poll, nothing new is printed, so output for commands that finish immediately is the same as before. The existing bundle acceptance tests that call serving-endpoints create pass without regenerating their outputs.

Tests

  • New unit tests for the progress message (pending config, multiple entities, fallback to the active config, fallback to the old current status text) and for non-TTY deduplication.
  • New acceptance test acceptance/cmd/workspace/serving-endpoints for create, create --no-wait, update-config and --help.
  • go test ./acceptance -run 'TestAccept/bundle/deployment/bind/model-serving-endpoint|TestAccept/bundle/resources/model_serving_endpoints' passes unchanged.
  • ./task lint passes.
  • Ran manually against a real e2-dogfood endpoint (output above).

This pull request and its description were written by Isaac.

`serving-endpoints create` and `update-config` only showed
`current status: IN_PROGRESS` in a spinner for the whole deployment, and
printed nothing when stderr is not a terminal. Show each served entity's
deployment_state_message with the time waited, and print one stderr line
per change when there is no TTY. Stdout is unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Anshul Gupta <anshul.gupta@databricks.com>
@anshul0713
anshul0713 requested a review from a team as a code owner November 16, 2026 06:56
@anshul0713
anshul0713 requested a review from anton-107 November 16, 2026 06:56
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Anshul Gupta <anshul.gupta@databricks.com>
@github-actions

Copy link
Copy Markdown
Contributor

Commit: e4f05bf8b | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (620ea05) latest (v1.26.0)
TestAccept/cmd/workspace/serving-endpoints ✅ ❌ ❌
TestAccept/cmd/workspace/serving-endpoints: branch ✅ | main (620ea05) ❌ | latest (v1.26.0) ❌

main (620ea05):

acceptance_test.go:1374: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1374
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1115
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n=== Create waits and reports progress on stderr; stdout is the endpoint JSON\n\n>>> [CLI] serving-endpoints create --json @input.json\n{\n  \"name\": \"iris-clf\",\n  \"state\": {\n    \"config_update\": \"NOT_UPDATING\",\n    \"ready\": \"NOT_READY\"\n  }\n}\n\n=== Create with --no-wait prints the create response and no progress\n\n>>> [CLI] serving-endpoints create --no-wait --json {\"name\": \"iris-clf-2\"}\n{\n  \"name\": \"iris-clf-2\"\n}\n\n=== Update config waits and reports progress\n\n>>> [CLI] serving-endpoints update-config iris-clf --json @update.json\ncurrent status: IN_PROGRESS ([ELAPSED])\n{\n  \"name\": \"iris-clf\",\n  \"state\": {\n    \"config_update\": \"NOT_UPDATING\",\n    \"ready\": \"NOT_READY\"\n  }\n}\n\n=== Help mentions the wait and --no-wait\n\n>>> [CLI] serving-endpoints create --help\n"
        	            	actual  : "\n=== Create waits and reports progress on stderr; stdout is the endpoint JSON\n\n>>> [CLI] serving-endpoints create --json @input.json\n{\n  \"name\": \"iris-clf\",\n  \"state\": {\n    \"config_update\": \"NOT_UPDATING\",\n    \"ready\": \"NOT_READY\"\n  }\n}\n\n=== Create with --no-wait prints the create response and no progress\n\n>>> [CLI] serving-endpoints create --no-wait --json {\"name\": \"iris-clf-2\"}\n{\n  \"name\": \"iris-clf-2\"\n}\n\n=== Update config waits and reports progress\n\n>>> [CLI] serving-endpoints update-config iris-clf --json @update.json\n{\n  \"name\": \"iris-clf\",\n  \"state\": {\n    \"config_update\": \"NOT_UPDATING\",\n    \"ready\": \"NOT_READY\"\n  }\n}\n\n=== Help mentions the wait and --no-wait\n\n>>> [CLI] serving-endpoints create --help\ncontains error: 'Use --no-wait to return immediately' not found in the output.\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -22,3 +22,2 @@
        	            	 >>> [CLI] serving-endpoints update-config iris-clf --json @update.json
        	            	-current status: IN_PROGRESS ([ELAPSED])
        	            	 {
        	            	@@ -34,2 +33,5 @@
        	            	 >>> [CLI] serving-endpoints create --help
        	            	+contains error: 'Use --no-wait to return immediately' not found in the output.
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/cmd/workspace/serving-endpoints
        	Messages:   	cmd/workspace/serving-endpoints/output.txt vs /tmp/TestAcceptcmdworkspaceserving-endpoints2025432887/002/output.txt
FAIL

latest (v1.26.0): same output as main (620ea05)

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: e4f05bf

Run: 43820526264

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 16 3:44
💚​ aws-windows-latest-4core-16gb 1 272 14 2:55
💚​ azure linux-2core-8gb 1 269 16 3:44
💚​ azure-windows-latest-4core-16gb 1 271 14 3:19
💚​ gcp linux-2core-8gb 1 270 16 3:41
💚​ gcp-windows-latest-4core-16gb 1 272 14 3:32
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:27 gcp-windows-latest-4core-16gb TestAccept
3:13 azure-windows-latest-4core-16gb TestAccept
2:51 aws-windows-latest-4core-16gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants