Repository navigation
serving-endpoints: explain a failed create or update-config wait - #7669
Open
anshul0713 wants to merge 2 commits into
Open
anshul0713 wants to merge 2 commits into
anshul0713 wants to merge 2 commits into
Conversation
When the endpoint update failed, the wait only returned "failed to reach NOT_UPDATING, got UPDATE_FAILED: current status: UPDATE_FAILED", even though the served entity's state already named the cause (for example "Container build failed. Cause: user_pip_resolution. See Build Logs for details."). In a terminal, the spinner that showed that state is gone by then. Add each served entity's last state to the error, with the build-logs and logs commands for it. The original error stays first and is still wrapped, and the exit code is unchanged. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Anshul Gupta <anshul.gupta@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Anshul Gupta <anshul.gupta@databricks.com>
Contributor
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 1 tests: regression=1 unreleased=0 coverage=0 failing=0
TestAccept/cmd/workspace/serving-endpoints: branch ✅ | main (620ea05) ❌ | latest (v1.26.0) ❌main (620ea05): latest (v1.26.0): same output as main (620ea05) |
Collaborator
✅ Integration test reportCommit: cfa2897
Top 3 slowest tests (at least 2 minutes):
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #7659 (wait progress). Review that one first; this PR's base will move to
mainonce it merges.Changes
When
serving-endpoints createorupdate-configfails or times out while waiting, the error now adds each served entity's last deployment state and the commands that show its logs. Real output from a model with an uninstallable pip requirement:The suggested
build-logscommand then showsERROR: Could not find a version that satisfies the requirement this-package-does-not-exist==0.0.1.Why
The generated wait error never says why the update failed, but the served entity's
deployment_state_messagealready does. In a terminal, the spinner that showed that state is cleared before the error prints. Both agent trials of a "debug a failed deployment" journey had to rungetand thenbuild-logsto find a cause the CLI already had.Backward compatibility
errors.Iskeeps working). The exit code is unchanged.--no-wait, stdout and requests are unchanged.Tests
UPDATE_FAILEDendpoint), no poll result, and entities without state.update-configonto a model version whose requirements can't be installed../task lintpasses.This pull request and its description were written by Isaac.