Repository navigation
fix(sdk): save ask_agent usage with the conversation state - #6845
Conversation
ask_agent records its usage inside state.stats (usage id ask-agent-llm)
and changes nothing else, so base_state.json was not rewritten until the
next unrelated state change. The agent server answers
GET /api/conversations/{id} from that file: the socket's stats event
already carried the usage while REST and a restarted server did not.
ConversationState.save_soon() saves at once when the state lock is free
and otherwise leaves the save to the lock holder's exit, so ask_agent
still never waits for a running step.
Co-authored-by: openhands <openhands@all-hands.dev>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
I'm an AI agent helping Engel Nyst (@enyst) with project work. "Validate PR description" is red because the linked issue #6844 was filed minutes before this PR and has no Generated by Claude Code |
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Coverage Report •
|
||||||||||||||||||||||||||||||
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Reviewed at head 52a132e against main (3c66af1), with the repo review guide and the SDK/agent-server AGENTS.md applied.
Scope. The change stays inside this repository's ownership (SDK conversation state persistence plus the LocalConversation.ask_agent path) and implements the bounded scope and acceptance criteria of the linked issue #6844, which is open, labeled bug/priority:medium/ready-for-dev, and matches this diff. No repository move or maintainer decision is needed.
The fix works. save_soon() reuses ConversationState's single save path (_save_base_state under the state's own FIFOLock) with no second writer of base_state.json. When the lock is held it only marks _dirty (a non-persisted PrivateAttr) and returns, so an ask during a run still answers immediately, matching the issue's non-blocking requirement. _flush_deferred_save() clearing _dirty before serializing (and re-marking on save failure) closes the lost-mark window between the flush check and the write; a mark from another thread that lands mid-write survives to the next flush. When the lock is free, the immediate flush covers the common idle case, so REST and a restart see the usage as soon as ask_agent returns. The stale docstring ("does not modify, persist, or become part of the conversation state") is corrected to state that only the usage is kept.
Concurrency. With the lock held by the run thread, a post-fix mark set inside the holder's with state: block is still flushed at the holder's exit (the flag was cleared before the write, and __exit__ calls _flush_deferred_save() again). An asker thread entering blocking with state: joins the FIFO queue and flushes exactly once on exit, the same semantics a pre-fix asker got when it eventually acquired the lock, so the dirty flag does not reorder saves for that path. The agent-server flushes the same ask's stats event right after, which normally covers the narrow mark-between-check-and-release window the PR notes; the usage is never dropped from memory in any case.
Compatibility. Additive public method, no removals or persisted-shape changes (_dirty is a PrivateAttr; base_state.json payload is unchanged); OpenAPI, persisted-settings, public-type-budget, and deprecation checks are green on this exact head.
Verification.
tests/sdk/conversation/test_ask_agent.py(17 passed) andtests/sdk/conversation(1035 passed) in this workspace; the two new SDK tests fail on a copy of this head with onlysave_soon()removed and pass with it, so they genuinely guard the fix. The lock-held test exercises the real FIFOLock with a real second thread, no mocks on the path under test.tests/agent_server/test_event_service.py,test_conversation_router.py,tests/sdk/llm/test_llm_session.py,test_llm_registry.py(280 passed).tests/cross/test_conversation_parity.py: the 2 failures (update_secrets_before_run,secret_value_in_tool_output) reproduce identically on anorigin/mainworktree, so they are unrelated to this diff.tests/cross/test_remote_conversation_live_server.py: 50 passed incl. the new live ask-usage test; the one failure (secret_masked_timestamp_cursor) is also unrelated to state saving and passes in CI on this head.- GitHub Actions on 52a132e: all checks pass, including the earlier "Validate PR description" red explained in the PR comments (linked issue lacked a label when filed).
- The PR's real agent-server before/after transcripts (restarts, ask during a blocked run) provide the live evidence the repo review guide requires for a persisted-state recovery fix.
No material correctness, security, compatibility, or acceptance-criterion defects found at this head.
✅ APPROVED
HUMAN:
Engel Nyst (@enyst) set up a daily automation that keeps the verify-agent-server and verify-sdk feature maps current. This PR fixes a bug its run for main@3c66af1 found.
AGENT:
I'm an AI agent helping Engel Nyst (@enyst) with project work.
Why
A side question (
ask_agent,/btwin Agent Canvas) records its usage instate.statsunderask-agent-llmand changes nothing else.ConversationStatesavesbase_state.jsonwhen a field is assigned, so that usage stayed in memory until the next unrelated state change.The agent server answers
GET /api/conversations/{id}frombase_state.json, and since #6690 it pushes astatsevent on the socket for the ask. So the socket reported the usage, REST did not, and a server restarted before the next run lost it. In Canvas the cost meter rose after/btw, fell back on reload, and was short after a restart. The SDK alone has the same gap: afterLocalConversation.ask_agentthe usage is not inbase_state.json, and a resumed conversation does not count it.The missing save predates #6690 (same REST, disk and restart values at fb75ae9). #6690 made it visible by sending the socket event.
Summary
ConversationState.save_soon(): saves the state for a change made inside a field. If the state lock is free it saves now, under the lock. If another thread holds the lock it marks the state as having a pending save, and that holder'swith state:exit writes it. It never waits.LocalConversation.ask_agentcalls it after the LLM call. An ask during a run still returns at once.ConversationState.__exit__clears the pending-save flag before it serializes instead of after (and sets it again if the save fails), so a mark made by another thread during the write is kept for the next flush.Issue Number
Fixes #6844
How to Test
New tests, each fails without the SDK change:
test_ask_agent_usage_is_saved_with_the_conversation: the usage is inbase_state.jsonwhenask_agentreturns, a resumed conversation counts it, and a second ask adds to it.test_ask_agent_does_not_wait_for_the_state_lock_to_save_usage: with another thread holding the state lock,ask_agentanswers while it is held, and the usage is on disk once the lock is released.test_ask_agent_usage_is_in_conversation_info_at_once(live server):GET /api/conversations/{id}hasask-agent-llmright afterRemoteConversation.ask_agent.Also run:
tests/sdk/conversation(1035 passed), andtests/agent_server/test_event_service.py,tests/agent_server/test_conversation_router.py,tests/sdk/llm/test_llm_session.py,tests/sdk/llm/test_llm_registry.py,tests/cross/test_conversation_parity.py,tests/cross/test_remote_conversation_live_server.py(357 passed, 1 skipped, 1 xfailed).pre-commitpasses on the four changed files.Real server, launched from this branch (52a132e) with
control-agent-serverand a scriptedllm-stubprovider (no model key). The script asks three side questions (stub usage 1111/11, 2222/22, 3333/33 prompt/completion tokens) around two restarts and one run, then one during a run blocked on its model:Video/Screenshots
With the fix (
ˋ```````` https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/fixes/fix-ask-agent-stats-persisted/fix-run1.txtˋ), ````````REST and disk equal the socket after every ask, and the totals survive both restarts:On
mainat 3c66af1 (ˋ````````` https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/fixes/fix-ask-agent-stats-persisted/target-run1.txtˋ) `````````the same lines read:The SDK program (
ˋ`````````` https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/fixes/fix-ask-agent-stats-persisted/sdk_twin.pyˋ) ``````````printsask usage is durableon this branch (ˋ``````````` https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/fixes/fix-ask-agent-stats-persisted/sdk-twin-fix.txtˋ) ```````````andNOT in base_state.json / lost on reloadonmain(ˋ```````````` https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/fixes/fix-ask-agent-stats-persisted/sdk-twin-target.txtˋ). ````````````Type
Notes
save_soon()is a new public method onConversationState. It adds no lock and no second writer: the write is_save_base_stateunder the state's own lock, either taken without blocking or already held by the thread that flushes on exit.with state:exit instead of this one (on the server, thestatsevent for the same ask takes the lock right after). The usage is never dropped from memory.base_state.jsononce. The size of that write is the subject of perf(sdk): base_state.json rewrites the full LLM metrics history on every event (O(N^2) over a run) #6432.ACPAgent.ask_agentreturns before this code, and it does not record fork usage today ([Bug]: ACP ask_agent loses the fork's cost and stalls 2 s waiting for a UsageUpdate it dropped #6646, PR fix(acp): bill fork usage updates in ask_agent #6666). When it does, it needs the same call.F10.ask-agent,F10.ask-agent-running) does not assert stats after an ask; a bullet for it can follow this fix.Generated by Claude Code
🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:52a132e-pythonRun
All tags pushed for this build
About Multi-Architecture Support
52a132e-python) is a multi-arch manifest supporting both amd64 and arm6452a132e-python-amd64) are also available if needed