Repository navigation
serving-endpoints: warn when query output drops response fields - #7664
Open
anshul0713 wants to merge 2 commits into
Open
anshul0713 wants to merge 2 commits into
anshul0713 wants to merge 2 commits into
Conversation
The SDK's QueryEndpointResponse only has some of the fields serving endpoints return. Everything else is dropped silently, for example choices[].finish_reason (the struct tag says finishReason), tool calls, service_tier and agents' custom outputs, and content-part arrays turn into strings. Users and agents read the output as the endpoint's response. Send the query with the same path, headers and body as the SDK, keep printing the typed response so the output doesn't change, and print a warning on stderr that names the dropped fields and points to `databricks api post` for the full response. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Anshul Gupta <anshul.gupta@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Anshul Gupta <anshul.gupta@databricks.com>
Contributor
|
Commit: Regression test reportSelected 2 changed tests (limit=20, 0 not selected) 2 tests: regression=2 unreleased=0 coverage=0 failing=0
TestAccept/cmd/workspace/serving-endpoints/query-dropped-fields: branch ✅ | main (620ea05) ❌ | latest (v1.26.0) ❌main (620ea05): latest (v1.26.0): same output as main (620ea05) TestAccept/cmd/workspace/serving-endpoints/query-stream: branch ✅ | main (620ea05) ❌ | latest (v1.26.0) ❌main (620ea05): latest (v1.26.0): same output as main (620ea05) |
Collaborator
✅ Integration test reportCommit: 6d5dbdb
Top 3 slowest tests (at least 2 minutes):
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #7660 (
query --stream). Review that one first; this PR's base will move tomainonce it merges.Changes
databricks serving-endpoints querynow prints a warning on stderr when its output leaves out fields of the endpoint's response, and says how to get the full response:Stdout is still the same typed response as before.
To do this, the override now sends the non-streaming query itself, with the same path, headers (including the workspace routing header via
auth.WorkspaceIDHeaders) and body as the SDK'sQuery. It decodes the body into the sameserving.QueryEndpointResponse, and compares that response, re-marshaled, with the raw body.Why
The SDK's
QueryEndpointResponseonly models some of what serving endpoints return, and the rest disappears without a trace. Users and agents read the CLI output as the endpoint's response, so they draw the wrong conclusions. What I saw against real endpoints:choices[].finish_reasonis dropped from every chat completion. The SDK field is taggedfinishReason, while the API sendsfinish_reason.[{"type":"text","text":...}], returned by some models) is turned into a JSON-encoded string.service_tier,tool_calls, agents'custom_outputsand the Responses APIoutputare not in the struct, so they're dropped.The root fix belongs in the API spec. Until then, users should at least know the output is incomplete.
Backward compatibility
Tests
omitempty), agent outputs, and invalid JSON.acceptance/cmd/workspace/serving-endpoints/query-dropped-fieldswith stubbed chat and custom-model responses. Thequery-streamacceptance test still passes../task lintpasses.This pull request and its description were written by Isaac.