Skip to content

serving-endpoints: make query --stream work - #7660

Open
anshul0713 wants to merge 3 commits into
mainfrom
anshul0713/serving-endpoints-query-stream
Open

anshul0713 wants to merge 3 commits into
mainfrom
anshul0713/serving-endpoints-query-stream

Conversation

@anshul0713

Copy link
Copy Markdown
Contributor

Changes

databricks serving-endpoints query --stream, or "stream": true in --json, now prints the streamed response as it arrives, one JSON chunk per line, without the server-sent events framing (data: prefixes, blank lines, the [DONE] sentinel). For a chat endpoint, the reply text is then jq -rj '.choices[0].delta.content' away.

Non-streaming queries are unchanged: they still run the generated code.

Why

The generated command decodes the whole response as a single JSON object. A streamed response is an event stream, so --stream always failed:

Error: failed to unmarshal response body: invalid character 'd' looking for beginning of value. This is likely a bug in the Databricks SDK for Go ...

The --stream flag exists and is documented in --help, but there was no way to use it. In agent trials of shell-only LLM usage, agents tried it, hit this error, and fell back to curl with a hand-fetched token.

Backward compatibility

  • Non-streaming queries go through the generated code exactly as before, so their output and request are unchanged.
  • Streaming requests send the same body as before (stream: true). Only the response handling is new, and that path previously always errored.
  • Lines in the response that aren't SSE framing (for example a plain JSON body from a model that ignores stream) are printed as is.

Tests

  • Unit tests for SSE parsing (data lines, [DONE], comments and other fields, plain JSON bodies, 1 MiB lines) and for detecting stream mode from the flag or from --json.
  • New acceptance test acceptance/cmd/workspace/serving-endpoints/query-stream. It stubs the invocations route, which the test server doesn't model.
  • Checked against a real databricks-gpt-5-4 pay-per-token endpoint on a staging workspace: chunks arrive one per line and end at [DONE].
  • ./task lint passes.

This pull request and its description were written by Isaac.

The generated query command decodes the response as one JSON object, so a
streamed (server-sent events) response always failed with "failed to
unmarshal response body". When --stream or "stream": true is set, read the
event stream and print each chunk's JSON on its own line. Non-streaming
queries still run the generated code.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Anshul Gupta <anshul.gupta@databricks.com>
@anshul0713
anshul0713 requested a review from a team as a code owner November 16, 2026 07:08
@anshul0713
anshul0713 requested a review from rugpanov November 16, 2026 07:08
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Anshul Gupta <anshul.gupta@databricks.com>
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 1fd8e8286 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (620ea05) latest (v1.26.0)
TestAccept/cmd/workspace/serving-endpoints/query-stream ✅ ❌ ❌
TestAccept/cmd/workspace/serving-endpoints/query-stream: branch ✅ | main (620ea05) ❌ | latest (v1.26.0) ❌

main (620ea05):

acceptance_test.go:1374: Diff:
        --- cmd/workspace/serving-endpoints/query-stream/output.txt
        +++ /tmp/TestAcceptcmdworkspaceserving-endpointsquery-stream3765597906/002/output.txt
        @@ -2,38 +2,36 @@
         === --stream prints one JSON chunk per line
         
         >>> [CLI] serving-endpoints query chat-endpoint --stream --json {"messages": [{"role": "user", "content": "hi"}]}
        -{"choices":[{"delta":{"content":"Hel"},"index":0}]}
        -{"choices":[{"delta":{"content":"lo"},"finish_reason":"stop","index":0}]}
        +Error: failed to unmarshal response body: invalid character 'd' looking for beginning of value. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:
        +```
        +POST /serving-endpoints/chat-endpoint/invocations
        +> * Host: 
        +> * Accept: application/json
        +> * Authorization: REDACTED
        +> * Content-Type: application/json
        +> * Traceparent: (redacted)
        +> * User-Agent: cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/linux cmd/serving-endpoints_query cmd-exec-id/[UUID] interactive/none auth/pat cicd/github
        +> * X-Databricks-Workspace-Id: [WORKSPACE_ID]
        +> {
        +>   "messages": [
        +>     {
        +>       "content": "**REDACTED**",
        +>       "role": "user"
        +>     }
        +>   ],
        +>   "stream": true
        +> }
        +< HTTP/1.1 200 OK
        +< * Content-Length: 154
        +< * Content-Type: text/event-stream
        +< * Date: (redacted)
        +< data: {"choices":[{"delta":{"content":"Hel"},"index":0}]}
        +< 
        +< data: {"choices":[{"delta":{"content":"lo"},"finish_reason":"stop","index":0}]}
        +< 
        +< data: [DONE]
        +< 
        +< 
        +```
         
        -=== "stream": true in --json also streams
        -
        ->>> [CLI] serving-endpoints query chat-endpoint --json {"messages": [{"role": "user", "content": "hi"}], "stream": true}
        -Hello
        -
        ->>> print_requests.py //serving-endpoints
        -{
        -  "method": "POST",
        -  "path": "/serving-endpoints/chat-endpoint/invocations",
        -  "body": {
        -    "messages": [
        -      {
        -        "content": "hi",
        -        "role": "user"
        -      }
        -    ],
        -    "stream": true
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/serving-endpoints/chat-endpoint/invocations",
        -  "body": {
        -    "messages": [
        -      {
        -        "content": "hi",
        -        "role": "user"
        -      }
        -    ],
        -    "stream": true
        -  }
        -}
        +Exit code: 1
        
FAIL

latest (v1.26.0): same output as main (620ea05)

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Anshul Gupta <anshul.gupta@databricks.com>
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 1fd8e82

Run: 43823228473

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 16 3:42
💚​ aws-windows-latest-4core-16gb 1 272 14 3:28
💚​ azure linux-2core-8gb 1 269 16 3:39
💚​ azure-windows-latest-4core-16gb 1 271 14 3:22
💚​ gcp linux-2core-8gb 1 270 16 3:47
💚​ gcp-windows-latest-4core-16gb 1 272 14 3:35
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:30 gcp-windows-latest-4core-16gb TestAccept
3:24 aws-windows-latest-4core-16gb TestAccept
3:17 azure-windows-latest-4core-16gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants