Skip to content

fix(sdk): save ask_agent usage with the conversation state - #6845

Merged
enyst merged 1 commit into
mainfrom
claude/fix-ask-agent-stats-persisted
Nov 14, 2026
Merged

enyst merged 1 commit into
mainfrom
claude/fix-ask-agent-stats-persisted

Conversation

@enyst

@enyst enyst commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

HUMAN:
Engel Nyst (@enyst) set up a daily automation that keeps the verify-agent-server and verify-sdk feature maps current. This PR fixes a bug its run for main@3c66af1 found.


AGENT:
I'm an AI agent helping Engel Nyst (@enyst) with project work.

Why

A side question (ask_agent, /btw in Agent Canvas) records its usage in state.stats under ask-agent-llm and changes nothing else. ConversationState saves base_state.json when a field is assigned, so that usage stayed in memory until the next unrelated state change.

The agent server answers GET /api/conversations/{id} from base_state.json, and since #6690 it pushes a stats event on the socket for the ask. So the socket reported the usage, REST did not, and a server restarted before the next run lost it. In Canvas the cost meter rose after /btw, fell back on reload, and was short after a restart. The SDK alone has the same gap: after LocalConversation.ask_agent the usage is not in base_state.json, and a resumed conversation does not count it.

The missing save predates #6690 (same REST, disk and restart values at fb75ae9). #6690 made it visible by sending the socket event.

Summary

  • ConversationState.save_soon(): saves the state for a change made inside a field. If the state lock is free it saves now, under the lock. If another thread holds the lock it marks the state as having a pending save, and that holder's with state: exit writes it. It never waits.
  • LocalConversation.ask_agent calls it after the LLM call. An ask during a run still returns at once.
  • ConversationState.__exit__ clears the pending-save flag before it serializes instead of after (and sets it again if the save fails), so a mark made by another thread during the write is kept for the next flush.

Issue Number

Fixes #6844

How to Test

uv run pytest tests/sdk/conversation/test_ask_agent.py -q
uv run pytest tests/cross/test_remote_conversation_live_server.py -q -k ask_agent_usage

New tests, each fails without the SDK change:

  • test_ask_agent_usage_is_saved_with_the_conversation: the usage is in base_state.json when ask_agent returns, a resumed conversation counts it, and a second ask adds to it.
  • test_ask_agent_does_not_wait_for_the_state_lock_to_save_usage: with another thread holding the state lock, ask_agent answers while it is held, and the usage is on disk once the lock is released.
  • test_ask_agent_usage_is_in_conversation_info_at_once (live server): GET /api/conversations/{id} has ask-agent-llm right after RemoteConversation.ask_agent.

Also run: tests/sdk/conversation (1035 passed), and tests/agent_server/test_event_service.py, tests/agent_server/test_conversation_router.py, tests/sdk/llm/test_llm_session.py, tests/sdk/llm/test_llm_registry.py, tests/cross/test_conversation_parity.py, tests/cross/test_remote_conversation_live_server.py (357 passed, 1 skipped, 1 xfailed). pre-commit passes on the four changed files.

Real server, launched from this branch (52a132e) with control-agent-server and a scripted llm-stub provider (no model key). The script asks three side questions (stub usage 1111/11, 2222/22, 3333/33 prompt/completion tokens) around two restarts and one run, then one during a run blocked on its model:

S=<scratch dir> bash repro.sh --checkout <this worktree>    # https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/fixes/fix-ask-agent-stats-persisted/repro.sh

Video/Screenshots

With the fix (ˋ```````` https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/fixes/fix-ask-agent-stats-persisted/fix-run1.txtˋ), ```````` REST and disk equal the socket after every ask, and the totals survive both restarts:

ask 1 -> SIDE1   (stub usage 1111/11)
A2 after ask 1                     socket full_state on connect: default=10/5
A2 after ask 1                     socket stats events: 1; last: ask-agent-llm=1111/11 default=10/5
A2 after ask 1                     REST GET stats:    ask-agent-llm=1111/11 default=10/5
A2 after ask 1                     base_state.json:   ask-agent-llm=1111/11 default=10/5
-- server restarted (graceful) --
A3 after restart                   REST GET stats:    ask-agent-llm=1111/11 default=10/5
A3 after restart                   base_state.json:   ask-agent-llm=1111/11 default=10/5
ask 2 -> SIDE2   (stub usage 2222/22)
A4 after restart + ask 2           socket full_state on connect: ask-agent-llm=1111/11 default=10/5
A4 after restart + ask 2           socket stats events: 1; last: ask-agent-llm=3333/33 default=10/5
A4 after restart + ask 2           REST GET stats:    ask-agent-llm=3333/33 default=10/5
A4 after restart + ask 2           base_state.json:   ask-agent-llm=3333/33 default=10/5
ask 3 -> SIDE3   (stub usage 3333/33)
A6 after ask 3                     socket full_state on connect: ask-agent-llm=3333/33 default=20/10
A6 after ask 3                     socket stats events: 1; last: ask-agent-llm=6666/66 default=20/10
A6 after ask 3                     REST GET stats:    ask-agent-llm=6666/66 default=20/10
A6 after ask 3                     base_state.json:   ask-agent-llm=6666/66 default=20/10
-- server restarted (graceful) --
A7 after restart                   REST GET stats:    ask-agent-llm=6666/66 default=20/10
A7 after restart                   base_state.json:   ask-agent-llm=6666/66 default=20/10
ask B -> SIDEB in 415 ms   (stub usage 4444/44); status: running
B1 after ask, still running        socket full_state on connect: default=0/0
B1 after ask, still running        socket stats events: 1; last: ask-agent-llm=4444/44 default=0/0
B1 after ask, still running        REST GET stats:    ask-agent-llm=4444/44 default=0/0
B1 after ask, still running        base_state.json:   ask-agent-llm=4444/44 default=0/0

On main at 3c66af1 (ˋ````````` https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/fixes/fix-ask-agent-stats-persisted/target-run1.txtˋ) ````````` the same lines read:

A2 after ask 1                     socket stats events: 1; last: ask-agent-llm=1111/11 default=10/5
A2 after ask 1                     REST GET stats:    default=10/5
A2 after ask 1                     base_state.json:   default=10/5
A3 after restart                   REST GET stats:    default=10/5
A6 after ask 3                     socket stats events: 1; last: ask-agent-llm=5555/55 default=20/10
A6 after ask 3                     REST GET stats:    ask-agent-llm=2222/22 default=20/10
A7 after restart                   REST GET stats:    ask-agent-llm=2222/22 default=20/10
B1 after ask, still running        REST GET stats:    default=0/0

The SDK program (ˋ`````````` https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/fixes/fix-ask-agent-stats-persisted/sdk_twin.pyˋ) `````````` prints ask usage is durable on this branch (ˋ``````````` https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/fixes/fix-ask-agent-stats-persisted/sdk-twin-fix.txtˋ) ``````````` and NOT in base_state.json / lost on reload on main (ˋ```````````` https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/fixes/fix-ask-agent-stats-persisted/sdk-twin-target.txtˋ). ````````````

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

  • save_soon() is a new public method on ConversationState. It adds no lock and no second writer: the write is _save_base_state under the state's own lock, either taken without blocking or already held by the thread that flushes on exit.
  • When the lock is held, the pending mark is written without the lock. If it lands in the few instructions between a holder's flush check and its release, the save happens at the next with state: exit instead of this one (on the server, the stats event for the same ask takes the lock right after). The usage is never dropped from memory.
  • Each ask now writes base_state.json once. The size of that write is the subject of perf(sdk): base_state.json rewrites the full LLM metrics history on every event (O(N^2) over a run) #6432.
  • Not covered: the ACP path. ACPAgent.ask_agent returns before this code, and it does not record fork usage today ([Bug]: ACP ask_agent loses the fork's cost and stalls 2 s waiting for a UsageUpdate it dropped #6646, PR fix(acp): bill fork usage updates in ask_agent #6666). When it does, it needs the same call.
  • Not checked: other utility calls that record usage without a later state change (for example title generation with the agent's LLM) may have the same lag.
  • The verify-agent-server map (F10.ask-agent, F10.ask-agent-running) does not assert stats after an ask; a bullet for it can follow this fix.

Generated by Claude Code


🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:52a132e-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-52a132e-python \
  ghcr.io/openhands/agent-server:52a132e-python

All tags pushed for this build

ghcr.io/openhands/agent-server:52a132e-golang-amd64
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-golang-amd64
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-golang-amd64
ghcr.io/openhands/agent-server:52a132e-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:52a132e-golang-arm64
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-golang-arm64
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-golang-arm64
ghcr.io/openhands/agent-server:52a132e-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:52a132e-java-amd64
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-java-amd64
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-java-amd64
ghcr.io/openhands/agent-server:52a132e-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:52a132e-java-arm64
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-java-arm64
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-java-arm64
ghcr.io/openhands/agent-server:52a132e-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:52a132e-python-amd64
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-python-amd64
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-python-amd64
ghcr.io/openhands/agent-server:52a132e-python-node-runtime-amd64
ghcr.io/openhands/agent-server:52a132e-python-arm64
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-python-arm64
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-python-arm64
ghcr.io/openhands/agent-server:52a132e-python-node-runtime-arm64
ghcr.io/openhands/agent-server:52a132e-python-minimal-amd64
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-python-minimal-amd64
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-python-minimal-amd64
ghcr.io/openhands/agent-server:52a132e-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:52a132e-python-minimal-arm64
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-python-minimal-arm64
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-python-minimal-arm64
ghcr.io/openhands/agent-server:52a132e-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:52a132e-python-slim-amd64
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-python-slim-amd64
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-python-slim-amd64
ghcr.io/openhands/agent-server:52a132e-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:52a132e-python-slim-arm64
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-python-slim-arm64
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-python-slim-arm64
ghcr.io/openhands/agent-server:52a132e-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:52a132e-golang
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-golang
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-golang
ghcr.io/openhands/agent-server:52a132e-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:52a132e-java
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-java
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-java
ghcr.io/openhands/agent-server:52a132e-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:52a132e-python-minimal
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-python-minimal
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-python-minimal
ghcr.io/openhands/agent-server:52a132e-python-node-runtime-minimal
ghcr.io/openhands/agent-server:52a132e-python-slim
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-python-slim
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-python-slim
ghcr.io/openhands/agent-server:52a132e-python-node-runtime-slim
ghcr.io/openhands/agent-server:52a132e-python
ghcr.io/openhands/agent-server:52a132eba938f030e06f378c741232af9f634334-python
ghcr.io/openhands/agent-server:claude-fix-ask-agent-stats-persisted-python
ghcr.io/openhands/agent-server:52a132e-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., 52a132e-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 52a132e-python-amd64) are also available if needed

ask_agent records its usage inside state.stats (usage id ask-agent-llm)
and changes nothing else, so base_state.json was not rewritten until the
next unrelated state change. The agent server answers
GET /api/conversations/{id} from that file: the socket's stats event
already carried the usage while REST and a restarted server did not.

ConversationState.save_soon() saves at once when the state lock is free
and otherwise leaves the save to the lock holder's exit, so ask_agent
still never waits for a running step.

Co-authored-by: openhands <openhands@all-hands.dev>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@enyst

enyst commented Nov 14, 2026

Copy link
Copy Markdown
Member Author

I'm an AI agent helping Engel Nyst (@enyst) with project work.

"Validate PR description" is red because the linked issue #6844 was filed minutes before this PR and has no ready-for-dev label yet. Nothing in the diff causes it; the check should pass on a re-run once triage labels #6844.


Generated by Claude Code

@github-actions

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-sdk/openhands/sdk/conversation
   state.py337997%339, 481–482, 491–493, 923–925
openhands-sdk/openhands/sdk/conversation/impl
   local_conversation.py12756995%186, 423, 490, 526–529, 531, 786–787, 911, 1357, 1359, 1430, 1554, 1586, 1720, 1725–1730, 1783, 1804–1806, 1855, 2262–2264, 2267–2268, 2276–2277, 2281–2283, 2287–2288, 2581, 2584–2585, 2803, 2840, 2844, 2848, 2869, 2921, 2925, 2959, 2966–2967, 2975, 2979–2980, 2984, 2991, 3299–3300, 3318–3319, 3354, 3569, 3573, 3594, 3645, 3652–3653
TOTAL55965875684% 

@enyst enyst mentioned this pull request Nov 14, 2026
1 of 5 tasks
@enyst
enyst requested a review from all-hands-bot November 14, 2026 10:53

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Reviewed at head 52a132e against main (3c66af1), with the repo review guide and the SDK/agent-server AGENTS.md applied.

Scope. The change stays inside this repository's ownership (SDK conversation state persistence plus the LocalConversation.ask_agent path) and implements the bounded scope and acceptance criteria of the linked issue #6844, which is open, labeled bug/priority:medium/ready-for-dev, and matches this diff. No repository move or maintainer decision is needed.

The fix works. save_soon() reuses ConversationState's single save path (_save_base_state under the state's own FIFOLock) with no second writer of base_state.json. When the lock is held it only marks _dirty (a non-persisted PrivateAttr) and returns, so an ask during a run still answers immediately, matching the issue's non-blocking requirement. _flush_deferred_save() clearing _dirty before serializing (and re-marking on save failure) closes the lost-mark window between the flush check and the write; a mark from another thread that lands mid-write survives to the next flush. When the lock is free, the immediate flush covers the common idle case, so REST and a restart see the usage as soon as ask_agent returns. The stale docstring ("does not modify, persist, or become part of the conversation state") is corrected to state that only the usage is kept.

Concurrency. With the lock held by the run thread, a post-fix mark set inside the holder's with state: block is still flushed at the holder's exit (the flag was cleared before the write, and __exit__ calls _flush_deferred_save() again). An asker thread entering blocking with state: joins the FIFO queue and flushes exactly once on exit, the same semantics a pre-fix asker got when it eventually acquired the lock, so the dirty flag does not reorder saves for that path. The agent-server flushes the same ask's stats event right after, which normally covers the narrow mark-between-check-and-release window the PR notes; the usage is never dropped from memory in any case.

Compatibility. Additive public method, no removals or persisted-shape changes (_dirty is a PrivateAttr; base_state.json payload is unchanged); OpenAPI, persisted-settings, public-type-budget, and deprecation checks are green on this exact head.

Verification.

  • tests/sdk/conversation/test_ask_agent.py (17 passed) and tests/sdk/conversation (1035 passed) in this workspace; the two new SDK tests fail on a copy of this head with only save_soon() removed and pass with it, so they genuinely guard the fix. The lock-held test exercises the real FIFOLock with a real second thread, no mocks on the path under test.
  • tests/agent_server/test_event_service.py, test_conversation_router.py, tests/sdk/llm/test_llm_session.py, test_llm_registry.py (280 passed). tests/cross/test_conversation_parity.py: the 2 failures (update_secrets_before_run, secret_value_in_tool_output) reproduce identically on an origin/main worktree, so they are unrelated to this diff. tests/cross/test_remote_conversation_live_server.py: 50 passed incl. the new live ask-usage test; the one failure (secret_masked_timestamp_cursor) is also unrelated to state saving and passes in CI on this head.
  • GitHub Actions on 52a132e: all checks pass, including the earlier "Validate PR description" red explained in the PR comments (linked issue lacked a label when filed).
  • The PR's real agent-server before/after transcripts (restarts, ask during a blocked run) provide the live evidence the repo review guide requires for a persisted-state recovery fix.

No material correctness, security, compatibility, or acceptance-criterion defects found at this head.

✅ APPROVED

@enyst
enyst merged commit 44d4264 into main Nov 14, 2026
53 of 54 checks passed
@enyst
enyst deleted the claude/fix-ask-agent-stats-persisted branch November 14, 2026 22:25
@enyst enyst mentioned this pull request Nov 15, 2026
1 of 5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: ask_agent usage is not saved: REST and a restarted server miss what the stats socket event already reported

2 participants