Skip to content

Make retrieval timeout terminal SSE events request-correlated and measurable #137

New issue
New issue
Closed
Closed
Make retrieval timeout terminal SSE events request-correlated and measurable#137
Labels
bugSomething isn't working

Description

@WilliamAGH
WilliamAGH
opened on Jul 28, 2026

Bug

A normal chat request on dev.javachat.ai intermittently returned the designed HTTP-200 SSE terminal error code=retrieval.timeout, stage=retrieval, retryable=true. The immediate retry succeeded.

The underlying Qdrant 10-second query deadline is logged per collection, but the chat controller returns the retryable terminal SSE event before its generic pipeline error logger. The terminal classification therefore has no request-correlated application log and /actuator/prometheus has no dedicated counter for it.

Evidence

  • Reproduced while dogfooding deployed revision cb7e486c24bf755847ac3c13394ab48ae2b96ad9.
  • Qdrant timeout logs occurred at 2026-07-28T08:24:02.425Z and 2026-07-28T08:24:05.712Z.
  • The response contract remained HTTP 200 SSE and was retryable; this issue does not change timeout or retry behavior.

Acceptance criteria

  • Every emitted retrieval.timeout terminal event increments one stable Prometheus counter.
  • Normal chat writes one safe structured WARN containing its request token, bounded session ID, stable code/stage, and exception type before returning the SSE event.
  • Guided Learning writes equivalent bounded request context for the same shared timeout terminal path.
  • Regression tests prove the SSE contract is unchanged, the metric increments exactly once per terminal event, safe correlation fields are logged, and upstream exception messages are not exposed.
  • The deployed revision is verified on dev.javachat.ai; the metric is present in /actuator/prometheus and live chat behavior remains healthy.

Non-goals

  • Do not increase or otherwise change Qdrant, reranker, response-preparation, or provider timeout behavior.
  • Do not add fallback retrieval paths or change the retryable SSE contract.

Activity

  1. WilliamAGH
    added
    bugSomething isn't working
    on Jul 28, 2026

  2. WilliamAGH commented on Jul 28, 2026

    @WilliamAGH
    WilliamAGH
    on Jul 28, 2026
    OwnerAuthor

    Resolved and verified on the deployed dev revision 3f1964691a1461d6fce49ad6ab291bbc8d3c6a57.

    Live acceptance evidence:

    • /actuator/info, local HEAD, and origin/dev all reported the same exact revision.
    • /actuator/health, liveness, readiness, and dependency health were UP.
    • /actuator/prometheus exposed javachat_sse_terminal_retrieval_timeouts_total.
    • During the isolated acceptance observation, the counter value matched seven safe terminal WARNs.
    • Normal and guided failures both logged a unique request token, bounded session context, stable code="retrieval.timeout", stage="retrieval", and exceptionType="HybridSearchPartialFailureException".
    • The guided WARN included the bounded lesson slug.
    • WARN lines did not expose upstream exception messages or throwables.
    • The HTTP-200 retryable SSE contract remained unchanged, and successful normal and guided responses still streamed content and citations.

    Dogfood also confirmed the underlying Qdrant query latency defect is real and separate from this observability scope. It is tracked in #138.

  3. WilliamAGH
    closed this as completedon Jul 28, 2026

  4. WilliamAGH commented on Jul 28, 2026

    @WilliamAGH
    WilliamAGH
    on Jul 28, 2026
    OwnerAuthor

    Final deployed acceptance evidence for this closed follow-up:

    • authoritative remote dev and live build/deployment identity match abf2aaa;
    • GitHub Actions run 30397134096 passed build, frontend, and Docker/Qdrant smoke for that exact SHA;
    • health, readiness, and dependency health are UP;
    • three normal streams, three guided Records streams, and a concurrent normal/guided pair returned text plus citations without terminal errors;
    • desktop and 390x844 mobile browser passes completed without browser or console errors;
    • javachat_sse_terminal_retrieval_timeouts_total remained 0.0;
    • the bounded Loki window from 2026-07-28T20:37:00Z through 20:49:10Z had proven stream coverage and zero retrieval.timeout, Qdrant search-timeout, query-failure, or preparation-timeout matches;
    • active Java Chat Grafana alerts were zero.

    This supplements the issue-specific implementation and test evidence already recorded here. Exact-SHA CI: https://github.com/WilliamAGH/java-chat/actions/runs/30397134096

  5. WilliamAGH commented on Nov 2, 2026

    @WilliamAGH
    WilliamAGH
    on Nov 2, 2026
    OwnerAuthor

    HTTP rejections, terminal SSE errors, and handled browser failures do not consistently retain correlation and upstream rate-limit metadata. The fix adds request IDs, bounded client-reported observations, and provider status/retry diagnostics through the existing application logs. Regression coverage checks HTTP 429, HTTP-200 SSE errors, missing retry timing, CSRF protection, and no automatic search replay.

  6. WilliamAGH
    reopened this on Nov 2, 2026
  7. WilliamAGH
    added
    parked:elsewhereOwned outside the incident dispatcher
    agent:needs-attentionAn active owner is completing the recorded follow-up
    on Nov 2, 2026

  8. WilliamAGH commented on Nov 2, 2026

    @WilliamAGH
    WilliamAGH
    on Nov 2, 2026
    OwnerAuthor

    incident-agent · investigator · claim: finish-java-chat, haiku-7, ses_e651faff3ffeDdTophrwg3Q2V7; completing the approved diagnostic repair's production delivery and correlated live acceptance before closure.

  9. WilliamAGH commented on Nov 2, 2026

    @WilliamAGH
    WilliamAGH
    on Nov 2, 2026
    OwnerAuthor

    incident-agent · investigator, fix-engineer · diagnostic repair and served acceptance.

    Closure reason: fixed; the correlated diagnostic repair and its review corrections are included in the deployed development and production revisions.

    Development serves bee5ced46d558fd4049d81e7bb45f30fc5de3fb0; production and www serve 9f693d7e8688a1e10aeb541f9251741d4f4b21be, which contains the repair. All three health checks are UP. Development CI, production CI, and promotion #272 completed successfully.

    • HTTP completion records retain the final outcome exactly once across normal, asynchronous and error dispatches.
    • Terminal SSE diagnostics preserve request and upstream-failure correlation; bounded browser observations remain explicitly client-reported.
    • Normal and guided searches streamed text and citations without terminal errors. CSRF rejection, diagnostic submission and missing-lesson responses retained the correct final statuses in native and central logs.
    • Focused regressions, full push gates and browser failure-path checks passed. Signed-out browser acceptance passed on the served application.
    • The existing retrieval-timeout counter remains exported. Timeouts, quotas and search replay behavior were not loosened.

    The controlled 429 observation verifies diagnostic retention; it is not an actual upstream refusal or a guarantee against legitimate future overload responses. The separately clarified book-search repair is not attributed to JavaChat.

  10. WilliamAGH
    closed this as completedon Nov 2, 2026
  11. WilliamAGH
    removed
    agent:needs-attentionAn active owner is completing the recorded follow-up
    parked:elsewhereOwned outside the incident dispatcher
    on Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions