🐛 Bug Report: llama streams: close() before reading never ends the span; aclose() leaves the HTTP response (connection) open #4608

Activity

1. Suddha-Abhitej commented on Oct 12, 2026

   

   Suddha-Abhitej

   on Oct 12, 2026

   More actions

   I'd like to take this on.

   Approach, following the suggestion in the issue:

   1. Sync: return a small wrapper class (__iter__ / __next__ / close()) instead of the bare generator. close() ends the span if iteration never started, and __del__ covers a stream that is dropped without being read. The generator's finally also closes the wrapped Ollama stream.
   2. Async: same shape with __aiter__ / __anext__ / aclose(). The finally awaits response.aclose(), so the HTTP response is released immediately instead of at GC.
   3. Regression tests using an in-process httpx.MockTransport (no cassettes): close/aclose before reading, dropped before reading, close/aclose after a partial read, idempotent close, and full iteration unchanged.

   Two notes:

   • With close() before the first read, no HTTP request is actually sent, since Ollama's stream is lazy. The span is still ended, with status left unset. Happy to set a different status if you prefer.
   • The return value is an iterator rather than a real generator, so send() / throw() are no longer available. Ollama's type hints only promise Iterator / AsyncIterator.
2. armanunix commented on Oct 13, 2026

   

   armanunix

   on Oct 13, 2026

   Author

   More actions

   A related case in the same wrapper: an app that stops reading at Ollama's final done record (for chunk in stream: ... if chunk.done: break, a common pattern) has the complete answer, but the span records no output.

   _accumulate_streaming_response sets the accumulated response after its for loop and ends the span in finally. When the app breaks out, the generator receives GeneratorExit at its yield, the code after the loop never runs, and finally ends the span without gen_ai.completion.*. Recording the response when the done record arrives (or in the GeneratorExit path) would fix it.

   import gc, json, httpx, ollama
   from importlib.metadata import version
   from opentelemetry.sdk.trace import SpanProcessor, TracerProvider
   from opentelemetry.instrumentation.ollama import OllamaInstrumentor
   
   class Spans(SpanProcessor):
       def __init__(self): self.ended = []
       def on_end(self, span): self.ended.append(span)
   
   spans = Spans()
   provider = TracerProvider(); provider.add_span_processor(spans)
   OllamaInstrumentor().instrument(tracer_provider=provider)
   
   def respond(request):              # local stand-in for the Ollama server: "Hel", "lo", then the done record
       lines = [{"model": "m", "created_at": "2026-01-01T00:00:00Z", "done": False,
                 "message": {"role": "assistant", "content": w}} for w in ("Hel", "lo")]
       lines.append({"model": "m", "created_at": "2026-01-01T00:00:00Z", "done": True, "done_reason": "stop",
                     "message": {"role": "assistant", "content": ""}, "prompt_eval_count": 3, "eval_count": 2})
       return httpx.Response(200, headers={"content-type": "application/x-ndjson"},
                             content="".join(json.dumps(x) + "\n" for x in lines).encode())
   
   client = ollama.Client(host="http://mock", transport=httpx.MockTransport(respond))
   print(f"opentelemetry-instrumentation-ollama {version('opentelemetry-instrumentation-ollama')} | ollama {version('ollama')}")
   for label, stop_at_done in (("loop to the end", False), ("break at the done record", True)):
       spans.ended.clear(); text = ""
       for chunk in client.chat(model="m", messages=[{"role": "user", "content": "hi"}], stream=True):
           text += chunk.message.content
           if stop_at_done and chunk.done:
               break                  # the app has the complete answer
       gc.collect()
       out = [k for s in spans.ended for k, v in (s.attributes or {}).items() if "Hello" in str(v)]
       print(f"{label:26s} app got {text!r} | spans {len(spans.ended)} | attributes with the output: {out or 'none'}")

   opentelemetry-instrumentation-ollama 0.62.4 | ollama 0.6.3
   loop to the end            app got 'Hello' | spans 1 | attributes with the output: ['gen_ai.completion.0.content']
   break at the done record   app got 'Hello' | spans 1 | attributes with the output: none
   
3. Suddha-Abhitej commented on Oct 13, 2026

   

   Suddha-Abhitej

   on Oct 13, 2026

   More actions

   Fixed, along with two more cases the same investigation turned up. PR: #4617

   Your case had the same root as case 1, from the other side: GeneratorExit
   subclasses BaseException, so breaking out at the done record misses the
   except Exception arm, the else block never runs, and finally ends the span
   bare. Fixed by accumulating before the yield and adding an except GeneratorExit: arm that records what was received and re-raises. The span is
   now identical to a full iteration, token usage included.

   Two things that fell out of it:

   • Cancellation. A cancelled async stream — request timeout, client
     disconnect — also ended the span empty. It now records what the app received,
     with error.type=CancelledError. The CancelledError itself is re-raised
     untouched, and there's a test asserting it reaches the awaiting caller.
   • llm.streaming.time_to_generate. Not emitted when the app breaks at
     done, even though generation had completed. Now emitted for that case; a
     mid-stream break still emits nothing.

   22 regression tests, 18 of which fail on current main. Verified against a real
   Ollama server and checked in Jaeger: on main the close()-before-reading case
   exports only the parent span, since ollama.chat never ends — on the branch
   both are there.
4. Suddha-Abhitej commented on Oct 13, 2026

   

   Suddha-Abhitej

   on Oct 13, 2026

   More actions