Skip to content

serving-endpoints: warn when query output drops response fields - #7664

Open
anshul0713 wants to merge 2 commits into
anshul0713/serving-endpoints-query-streamfrom
anshul0713/serving-endpoints-query-dropped-fields
Open

anshul0713 wants to merge 2 commits into
anshul0713/serving-endpoints-query-streamfrom
anshul0713/serving-endpoints-query-dropped-fields

Conversation

@anshul0713

Copy link
Copy Markdown
Contributor

Stacked on #7660 (query --stream). Review that one first; this PR's base will move to main once it merges.

Changes

databricks serving-endpoints query now prints a warning on stderr when its output leaves out fields of the endpoint's response, and says how to get the full response:

Warning: the endpoint's response has fields that this output doesn't show: choices[].finish_reason, service_tier

To see the full response, send the same request with:
  databricks api post /serving-endpoints/NAME/invocations --json ...

Stdout is still the same typed response as before.

To do this, the override now sends the non-streaming query itself, with the same path, headers (including the workspace routing header via auth.WorkspaceIDHeaders) and body as the SDK's Query. It decodes the body into the same serving.QueryEndpointResponse, and compares that response, re-marshaled, with the raw body.

Why

The SDK's QueryEndpointResponse only models some of what serving endpoints return, and the rest disappears without a trace. Users and agents read the CLI output as the endpoint's response, so they draw the wrong conclusions. What I saw against real endpoints:

  • choices[].finish_reason is dropped from every chat completion. The SDK field is tagged finishReason, while the API sends finish_reason.
  • A content-parts array ([{"type":"text","text":...}], returned by some models) is turned into a JSON-encoded string.
  • service_tier, tool_calls, agents' custom_outputs and the Responses API output are not in the struct, so they're dropped.

The root fix belongs in the API spec. Until then, users should at least know the output is incomplete.

Backward compatibility

  • Stdout is unchanged. I compared the previous build and this one against real chat and custom-model endpoints: the custom model's output is byte-identical, and the chat output matches apart from the generated text.
  • Exit codes, error messages (missing endpoint, wrong payload shape, invalid JSON) and the request are unchanged. These were also compared against a real workspace.
  • The warning is new stderr output. It appears only when the response contains a non-null, non-zero field that the output omits or changes the JSON type of.

Tests

  • Unit tests for the comparison: nothing dropped, chat fields, content parts, null and zero values (ignored, because of omitempty), agent outputs, and invalid JSON.
  • New acceptance test acceptance/cmd/workspace/serving-endpoints/query-dropped-fields with stubbed chat and custom-model responses. The query-stream acceptance test still passes.
  • ./task lint passes.

This pull request and its description were written by Isaac.

The SDK's QueryEndpointResponse only has some of the fields serving
endpoints return. Everything else is dropped silently, for example
choices[].finish_reason (the struct tag says finishReason), tool calls,
service_tier and agents' custom outputs, and content-part arrays turn
into strings. Users and agents read the output as the endpoint's
response.

Send the query with the same path, headers and body as the SDK, keep
printing the typed response so the output doesn't change, and print a
warning on stderr that names the dropped fields and points to
`databricks api post` for the full response.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Anshul Gupta <anshul.gupta@databricks.com>
@anshul0713
anshul0713 requested a review from a team as a code owner November 16, 2026 07:23
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Anshul Gupta <anshul.gupta@databricks.com>
@github-actions

Copy link
Copy Markdown
Contributor

Commit: 6d5dbdb97 | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0
test branch main (620ea05) latest (v1.26.0)
TestAccept/cmd/workspace/serving-endpoints/query-dropped-fields ✅ ❌ ❌
TestAccept/cmd/workspace/serving-endpoints/query-stream ✅ ❌ ❌
TestAccept/cmd/workspace/serving-endpoints/query-dropped-fields: branch ✅ | main (620ea05) ❌ | latest (v1.26.0) ❌

main (620ea05):

acceptance_test.go:1374: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1374
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1115
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n=== Fields the output doesn't show are named on stderr; stdout is unchanged\n\n>>> [CLI] serving-endpoints query chat-endpoint --json {\"messages\": [{\"role\": \"user\", \"content\": \"hi\"}]}\nWarning: the endpoint's response has fields that this output doesn't show: choices[].finish_reason, service_tier\n\nTo see the full response, send the same request with:\n  databricks api post /serving-endpoints/chat-endpoint/invocations --json ...\n\n{\n  \"choices\": [\n    {\n      \"index\": 0,\n      \"message\": {\n        \"content\": \"Hello\",\n        \"role\": \"assistant\"\n      }\n    }\n  ],\n  \"id\": \"chatcmpl-1\",\n  \"object\": \"chat.completion\"\n}\n\n=== No warning when nothing is dropped\n\n>>> [CLI] serving-endpoints query custom-endpoint --json {\"dataframe_records\": [{\"text\": \"I love this\"}]}\n{\n  \"predictions\": [\n    {\n      \"confidence\": 0.9,\n      \"label\": \"positive\"\n    }\n  ]\n}\n"
        	            	actual  : "\n=== Fields the output doesn't show are named on stderr; stdout is unchanged\n\n>>> [CLI] serving-endpoints query chat-endpoint --json {\"messages\": [{\"role\": \"user\", \"content\": \"hi\"}]}\n{\n  \"choices\": [\n    {\n      \"index\": 0,\n      \"message\": {\n        \"content\": \"Hello\",\n        \"role\": \"assistant\"\n      }\n    }\n  ],\n  \"id\": \"chatcmpl-1\",\n  \"object\": \"chat.completion\"\n}\n\n=== No warning when nothing is dropped\n\n>>> [CLI] serving-endpoints query custom-endpoint --json {\"dataframe_records\": [{\"text\": \"I love this\"}]}\n{\n  \"predictions\": [\n    {\n      \"confidence\": 0.9,\n      \"label\": \"positive\"\n    }\n  ]\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,7 +4,2 @@
        	            	 >>> [CLI] serving-endpoints query chat-endpoint --json {"messages": [{"role": "user", "content": "hi"}]}
        	            	-Warning: the endpoint's response has fields that this output doesn't show: choices[].finish_reason, service_tier
        	            	-
        	            	-To see the full response, send the same request with:
        	            	-  databricks api post /serving-endpoints/chat-endpoint/invocations --json ...
        	            	-
        	            	 {
        	Test:       	TestAccept/cmd/workspace/serving-endpoints/query-dropped-fields
        	Messages:   	cmd/workspace/serving-endpoints/query-dropped-fields/output.txt vs /tmp/TestAcceptcmdworkspaceserving-endpointsquery-dropped-fields749417941/002/output.txt
FAIL

latest (v1.26.0): same output as main (620ea05)

TestAccept/cmd/workspace/serving-endpoints/query-stream: branch ✅ | main (620ea05) ❌ | latest (v1.26.0) ❌

main (620ea05):

acceptance_test.go:1374: Diff:
        --- cmd/workspace/serving-endpoints/query-stream/output.txt
        +++ /tmp/TestAcceptcmdworkspaceserving-endpointsquery-stream3304231542/002/output.txt
        @@ -2,38 +2,36 @@
         === --stream prints one JSON chunk per line
         
         >>> [CLI] serving-endpoints query chat-endpoint --stream --json {"messages": [{"role": "user", "content": "hi"}]}
        -{"choices":[{"delta":{"content":"Hel"},"index":0}]}
        -{"choices":[{"delta":{"content":"lo"},"finish_reason":"stop","index":0}]}
        +Error: failed to unmarshal response body: invalid character 'd' looking for beginning of value. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:
        +```
        +POST /serving-endpoints/chat-endpoint/invocations
        +> * Host: 
        +> * Accept: application/json
        +> * Authorization: REDACTED
        +> * Content-Type: application/json
        +> * Traceparent: (redacted)
        +> * User-Agent: cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/linux cmd/serving-endpoints_query cmd-exec-id/[UUID] interactive/none auth/pat cicd/github
        +> * X-Databricks-Workspace-Id: [WORKSPACE_ID]
        +> {
        +>   "messages": [
        +>     {
        +>       "content": "**REDACTED**",
        +>       "role": "user"
        +>     }
        +>   ],
        +>   "stream": true
        +> }
        +< HTTP/1.1 200 OK
        +< * Content-Length: 154
        +< * Content-Type: text/event-stream
        +< * Date: (redacted)
        +< data: {"choices":[{"delta":{"content":"Hel"},"index":0}]}
        +< 
        +< data: {"choices":[{"delta":{"content":"lo"},"finish_reason":"stop","index":0}]}
        +< 
        +< data: [DONE]
        +< 
        +< 
        +```
         
        -=== "stream": true in --json also streams
        -
        ->>> [CLI] serving-endpoints query chat-endpoint --json {"messages": [{"role": "user", "content": "hi"}], "stream": true}
        -Hello
        -
        ->>> print_requests.py //serving-endpoints
        -{
        -  "method": "POST",
        -  "path": "/serving-endpoints/chat-endpoint/invocations",
        -  "body": {
        -    "messages": [
        -      {
        -        "content": "hi",
        -        "role": "user"
        -      }
        -    ],
        -    "stream": true
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/serving-endpoints/chat-endpoint/invocations",
        -  "body": {
        -    "messages": [
        -      {
        -        "content": "hi",
        -        "role": "user"
        -      }
        -    ],
        -    "stream": true
        -  }
        -}
        +Exit code: 1

latest (v1.26.0): same output as main (620ea05)

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 6d5dbdb

Run: 43823691010

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 271 17 3:46
✅​ aws-windows-latest-4core-16gb 273 15 4:06
✅​ azure linux-2core-8gb 270 17 3:29
✅​ azure-windows-latest-4core-16gb 272 15 3:42
✅​ gcp linux-2core-8gb 271 17 4:08
✅​ gcp-windows-latest-4core-16gb 273 15 3:33
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:01 aws-windows-latest-4core-16gb TestAccept
3:38 azure-windows-latest-4core-16gb TestAccept
3:29 gcp-windows-latest-4core-16gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants