Skip to content

fix(agent-server): one owner for confirmation_policy, security_analyzer and secrets (base_state.json) - #6764

Draft
enyst wants to merge 50 commits into
claude/openhands-sdk-dual-source-truth-qp1x6r-2-fork-keeps-statefrom
claude/openhands-sdk-dual-source-truth-qp1x6r-3-single-owner
Draft

enyst wants to merge 50 commits into
claude/openhands-sdk-dual-source-truth-qp1x6r-2-fork-keeps-statefrom
claude/openhands-sdk-dual-source-truth-qp1x6r-3-single-owner

Conversation

@enyst

@enyst enyst commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

HUMAN:
Engel Nyst asked me to find the fork PR that removes the two sources of truth for a conversation's settings (meta.json in the agent-server and base_state.json in the SDK), which is #4813 by @BSmick6; to put its commits on branches in this repo, split so that each concern can be reviewed on its own; and to open new PRs that credit the author and say that we do the final review and tweaks. Then to review each PR, address what the review finds, and prove that the changes work by running the SDK and the agent-server the way a program or an agent does, with the repo's verify-sdk and verify-agent-server skills. The principle behind the request: the agent-server is a REST interface to the SDK. It should not be a different thing, and it should not repeat what the SDK already does, unless a particular case gives a special reason.

This is part 3 of 3: the agent-server stops keeping its own copy of confirmation_policy, security_analyzer and secrets. base_state.json is their only home.


AGENT:

I'm Claude (Fable 5.2), an AI agent working for Engel Nyst. I brought #4813 up to date with main, split it into a stack of three PRs, reviewed each, and verified them against real servers and through the Agent Canvas UI. The design and the core refactor in this PR (commit 6c14b62d) are @BSmick6's, from #4813, with one fix from Engel Nyst's follow-ups on that PR (c3d1a02d) and one test (11703e03) split out of a commit an AI agent authored on that PR. The rest was added in the final review and is listed under "Final review".

The stack: 1. #6762 (secrets are saved the moment they change) → 2. #6763 (a fork keeps the state) → 3. this PR. It is based on part 2, so its diff shows only its own change. #4813's own commits, merged with main at 56d559f6, are on the branch claude/openhands-sdk-dual-source-truth-qp1x6r.

Why

A conversation's confirmation policy, security analyzer and secrets were stored twice: by the SDK in base_state.json, and by the agent-server in meta.json. The runtime routes (POST .../confirmation_policy, .../security_analyzer, .../secrets) never wrote the second; a secret reached even the first only at the next save of the state, which #6762 fixes. Every load of the conversation, after a restart or after the default 20-minute idle eviction, re-applied the second, the copy from creation time. On main:

  • a conversation switched to AlwaysConfirm goes back to NeverConfirm and runs actions unasked (or the other way round);
  • the analyzer goes back to the one it was created with;
  • a secret replaced at runtime goes back to its old value;
  • forks and child conversations start from the creation-time copy.

A safety setting that silently reverts is the serious part. The agent-server is meant to be a REST interface to the SDK; for these three values it kept a second copy and let it win.

Summary

  • StoredConversation (meta.json) can no longer carry confirmation_policy, security_analyzer, secrets or secrets_encrypted. Its base class is a new StoredConversationConfig, which has no field for them, the same way fix(agent-server): base_state.json as single source of truth for the agent (end meta.json duplication) #4440 kept agent out of the stored record. ConversationConfig and StartConversationRequest keep their released shape, so the REST body and the OpenAPI document are unchanged.
  • EventService receives the policy and the analyzer as start fields that are applied when a conversation is created; a resume keeps what base_state.json holds. Secrets reach the conversation through the SDK's own update_secrets.
  • The create path decrypts the request's secrets and analyzer when secrets_encrypted is set, since they no longer pass through the stored payload that was validated with the cipher.
  • The CODEX fallback credential goes to the EventService as an argument instead of a temporary patch of stored.secrets.
  • A child conversation inherits the three from its parent's conversation state.
  • Telemetry's conversation_created event reads the policy from the conversation state, since the stored record no longer has it. As before, that is the policy the conversation starts with; for a fork or a child it is now its source's current policy, not the source's creation-time one.

Final review: what was found and changed on top of #4813

  1. Child conversations hung. The one open review thread on fix(agent-server): own confirmation_policy, security_analyzer, secrets in ConversationState #4813 was right: reading the parent's values took the parent's state lock, which the parent's own step holds while its tool waits for this server to answer. The request stalled for the 300-second timeout, was reported as failed, and the child appeared afterwards anyway. The read is now lock-free, like the agent and the budget next to it. The regression test holds the lock both ways a running step does and times out without the change.
  2. An upgrade could lose secrets. A conversation created by an older server with start-time secrets, and not saved since, has them only in meta.json. fix(agent-server): own confirmation_policy, security_analyzer, secrets in ConversationState #4813 ignored that key and then rewrote the file without it. The first load now takes over the names base_state.json lacks (base_state.json wins for the ones it has), through the SDK's own update_secrets. One case needs more. With a credential binding, the credential scrub rewrites meta.json before the conversation is loaded. The scrub already rewrites base_state.json as well, so it now writes those secrets there in the same step, and a load that fails right after does not lose them. Two edges. The take-over is decided per load: when this server's key cannot decrypt what the two files hold (another key, or no key at all), nothing is taken over, so the take-over adds no save of the conversation state on a load that could not read it; the legacy secrets are then dropped by the next write of meta.json, and the log says so. That is all it guarantees: a conversation with a credential binding still has both files rewritten by the credential scrub on such a load, as on main, and the regression test covers conversations without a binding. And each legacy entry is read on its own, so one that cannot be parsed (a secret kind this server does not know) is reported in the log and costs only itself. Shown live below with the latest release.
  3. Released servers on a newer SDK. openhands-agent-server does not pin the SDK version, and the released StoredConversation extends ConversationConfig and reads the three fields from it. Taking them off ConversationConfig, as fix(agent-server): own confirmation_policy, security_analyzer, secrets in ConversationState #4813 did, made a released server fail every conversation start with AttributeError once it ran on the new SDK. Hence the slim base under an unchanged ConversationConfig: the agent-server code of the 1.62.0 release starts conversations on this SDK (result below).
  4. A retried create lost its safety settings. The start fields were applied only when base_state.json did not exist. A create that fails after that file is written (a full disk, a kill) leaves it behind, and the retry with the same id then ran with NeverConfirm and no analyzer; main applied the requested values. The start fields are now applied whenever the caller passes an agent, which only the create path does; whether base_state.json exists no longer decides it.
  5. A resume installed cipher text as the Codex credential. With secrets_encrypted set, the create path decrypted the request's secrets and the resume path did not. It behaves the same on main; the Docker runtime forwards every start that way. The resume path now reads the secrets the way create does.
  6. Two conversations could share one analyzer object when the same request object started both, because the analyzer no longer made a trip through the stored record. Each conversation gets its own copy, as forks and children do.
  7. A server with no cipher at all (neither OH_SECRET_KEY nor a session key) writes no secret values to disk, so the in-memory copy of meta.json used to be the only thing that brought start-time secrets back after an idle eviction or a fork. An ACP agent's provider credential is such a secret. Without that second copy, the loaded conversation is the only place that has the values, so such a server does not evict an idle conversation that holds secrets or an analyzer with a key of its own, and a fork receives its source's secrets from memory.
  8. Less duplication. apply_resume_secrets calls LocalConversation.update_secrets instead of copying the registry by hand; a child's analyzer and secrets are copied with the helper fork() uses; the four fields are left out of the stored payload through the existing exclude set; the start fields are named like main's initial_max_budget_per_run; bound secret names are left out at the one place the start secrets are used.
  9. Tests. The mock-based test class for the start fields is replaced by tests on a real EventService. Added, each failing without the code it guards: idle eviction keeps the three, a legacy meta.json is taken over (also when the first load fails, with a bound credential next to it, and with an unreadable entry next to it), a retried create, a child on a real service inherits the parent's current values and does not wait for its lock, an encrypted create and an encrypted resume are decrypted, one request gives two analyzers, telemetry reports the conversation's policy, a load under another key or none, and the three no-cipher cases.
  10. Feature maps. New recipes for restart, on-disk layout, fork, child, sandbox pause and upgrade; three credential-binding recipes that read meta.json's secrets now read the registry in base_state.json.

Issue Number

Fixes #4810.

The issue's literal acceptance criteria ask for the runtime routes to write meta.json as well. They are replaced by the decision recorded in the issue thread, that state the conversation owns is written only to base_state.json and read from there: #4810 (comment)

How to Test

uv run pytest tests/cross/test_agent_server_restore_mutations.py tests/agent_server/test_event_service.py \
  tests/agent_server/test_conversation_service.py tests/agent_server/test_parent_child_conversations.py \
  tests/agent_server/test_conversation_eviction.py tests/agent_server/test_credential_binding.py \
  tests/agent_server/test_models.py tests/agent_server/telemetry/test_telemetry_subscriber.py -q

Regression tests, each run without its change:

Test without the change with it
test_eviction_keeps_runtime_policy_analyzer_and_secrets (on main) fails: policy is NeverConfirm after eviction passes
test_mutations_survive_a_server_restart_and_a_later_fork (on main) fails passes
test_resume_keeps_the_saved_policy_and_analyzer_over_the_start_fields (resume guard removed) fails: NeverConfirm passes
test_start_child_conversation_does_not_wait_for_parent_state_lock (lock-taking read) fails: TimeoutError passes
test_secrets_an_older_server_left_in_meta_json_are_kept (no legacy read) fails: KeyError: 'ONLY_IN_META' passes
test_secrets_from_an_older_meta_json_survive_a_failed_first_load (the scrub does not carry them) fails: AssertionError: assert 'ONLY_IN_META' in {} (base_state.json after the failed load) passes
test_retry_of_a_create_that_failed_halfway_gets_the_requested_settings (seeding by file absence) fails: NeverConfirm passes
test_child_inherits_what_its_parent_has_now (seeding removed) fails: NeverConfirm passes
test_resume_decrypts_an_encrypted_fallback, live and cold (also on main) fails: the credential is gAAAAA... passes
test_conversations_started_from_one_request_have_their_own_analyzer fails: the same object passes
test_conversation_whose_secrets_cannot_be_reloaded_is_not_evicted fails: evicted passes
test_fork_keeps_secret_values_on_a_server_without_a_cipher fails: None == 'start-value' passes
test_conversation_whose_analyzer_key_cannot_be_reloaded_is_not_evicted fails: evicted passes
test_legacy_secrets_this_key_cannot_decrypt_do_not_cost_the_agent_its_key, three cases: a static secret under another key, a lookup secret under another key, a static secret with no key the last two fail with the per-entry check: base_state.json is rewritten without the agent's key passes
test_server_without_a_key_does_not_take_legacy_cipher_text_for_a_value fails: the cipher text is registered as the secret's value passes
test_unreadable_legacy_meta_secret_costs_only_itself (the key read as one mapping) fails: the readable legacy secret is not taken over passes

Live, with control-agent-server

Every attempt is its own freshly launched agent-server; "restart" is a real process restart on the same state directory. Here and below, main is main at 3c66af1d, the commit this stack is merged with.

The bug and the fix, with the repo's "verify an existing fix" loop. The recipes carry a known-bug marker in a copy of main's map and were replayed against a server launched from main, then from this branch (--checkout), two attempts each. xfail means the bullet failed with exactly the documented failure; xpass means that failure no longer reproduces.

Bullet main this branch
F08.state-on-disk xfail, xfail xpass, xpass
F08.policy-restart xfail, xfail xpass, xpass
F08.analyzer-restart xfail, xfail xpass, xpass
F08.analyzer-clear-restart xfail, xfail xpass, xpass
F08.secrets-override-restart xfail, xfail xpass, xpass
F08.fork-after-restart xfail, xfail xpass, xpass
F08.child-inherits-current xfail, xfail xpass, xpass
F08.reload-after-pause xfail, xfail xpass, xpass

What main showed in those attempts: meta.json held confirmation_policy, security_analyzer, secrets and secrets_encrypted; a conversation switched to another policy at runtime answered with its creation-time policy after the restart; a command got the secret's old value; a fork, a child and a reload after a sandbox pause all started from the creation-time copy.

Whole families on this branch's final code (cac2960d), to look for regressions next to the fix. In this branch's own map the bullets above carry no marker and are part of F08. The last column is the same family replayed on main with main's own map:

Family this branch main (3c66af1d), its own map
F08 confirmation, security, secrets 34 of 34 pass 22 of 24 pass, 2 xfail
F10 fork, navigate, condense, ask 25 of 27 pass; 2 fail that also fail on main (F10.ask-agent, F10.ask-agent-running) 22 of 26 pass, 2 fail, 2 xfail
F04 conversation lifecycle 36 of 37 pass; 1 xfail that main has too (F04.create-initial-message-run-false) 35 of 37 pass, 2 xfail
F12 runtime and credentials 29 of 32 pass; 1 xfail that main has too (F12.sandbox-pause-websocket); F12.docker-runtime blocked (needs a Docker daemon); F12.docker-startup-timeout blocked (needs a Docker daemon) 29 of 32 pass, 1 xfail, 2 blocked
F31 persistence and limits 22 of 24 pass; 1 xfail that main has too (F31.takeover-keeps-metadata); F31.storage-retention blocked (needs a Docker daemon) 22 of 24 pass, 1 xfail, 1 blocked
F33 Python RemoteConversation: run, fork, confirmation, secrets 5 of 5 pass 5 of 5 pass
F34 TypeScript client: create, run, fork/navigate/ask, confirm 4 of 5 pass; 1 fail that also fails on main (F34.conversation-branching) 4 of 5 pass, 1 fail
S04 local conversation (SDK map) 26 of 27 pass; S04.factory labelled blocked in the map 25 of 26 pass, 1 blocked
S13 security and confirmation (SDK map) 7 of 9 pass; the live-model bullet (S13.llm-analyzer, S13.confirm-wait, S13.confirm-approve-reject) failed in this run and was blocked on main: see the note below; S13.confirm-approve-reject, S13.llm-analyzer labelled blocked in the map 7 of 9 pass, 2 blocked

Every bullet that does not pass on this branch does not pass on main either (compared bullet by bullet). main has five more that do not pass: the bullets its map marks as known bugs, which this stack fixes (F04.create-secrets-get, F08.secrets-visible, F08.secrets-unsaved-restart, F10.fork-keeps-credentials, F10.fork-run-memory). The three failures the two share are recipes about a side question: after ask_agent the event count is one higher than F10.ask-agent, F10.ask-agent-running and F34.conversation-branching expect, on main as here. I left those recipes alone; they are not about this change.

The S13 bullet that did not pass is the live-model one (S13.llm-analyzer): it needs DeepSeek to rate rm -rf scratch as HIGH, and the model often says MEDIUM. Repeated alone four times on each side, it passed once, failed twice and was blocked once on this stack's top branch, and exactly the same on main. The stack does not touch how a model rates a command.

F08.upgrade-from-release is part of the F08 run: conversations are created by the latest released server (1.62.0), which leaves a start-time secret only in meta.json, and the same state directory is then restarted on this code. The secret, the policy and the analyzer are all there. With the legacy take-over removed, that bullet fails twice (the command gets an empty secret).

export PATH="$PWD/.agents/skills/verify-agent-server/scripts:$PWD/.agents/skills/verify-sdk/scripts:$PATH"
control-agent-server map run --file F08 --fresh        # also F10, F04, F12, F31
control-agent-server map run --file F08 --only F08.state-on-disk,F08.policy-restart,F08.secrets-override-restart \
  --fresh --repeat 2 --checkout ../main                # the same bullets against a main checkout: fail, fail

Through the UI

The full OpenHands UI (Agent Canvas, built from the OpenHands/OpenHands repository) was run with that repository's verify-openhands skill (control-openhands launch --sdk-path <checkout>), once on main's SDK and agent-server (91e8b5b6) and once on this branch's (ae2e835c), with a real model (DeepSeek flash). That is one merge of main before this branch's current code commit; the source of the four packages is the same in both (git diff ae2e835c..cac2960d is empty for them). The Canvas has no control that changes confirmation mode on an existing conversation, and it only ever sends lookup secrets, so two steps were made through the REST API: that switch, and the replacement of the secret's value. Everything else was typed and clicked in the UI, and each stack was restarted for real.

Scenario, observed in the UI main this branch
Conversation switched to AlwaysConfirm, then a backend restart, then a new command ran without asking, in 2 of 2 attempts waited for approval, in 2 of 2 attempts; Continue ran it, Cancel rejected the next one
Secret replaced at runtime (18 bytes → 25), then a restart; length read by the agent 18: the old value is back 25
New conversation started in confirmation mode: reject, approve, restart, next command waits (creation-time value) waits
meta.json read back at every step had confirmation_policy, security_analyzer, secrets in 29 of 29 read-backs had none of them in 31 of 31

policy after a restart

replaced secret after a restart

On main the revert is also written back: once the conversation is loaded after the restart, base_state.json holds the creation-time policy again. While the Add-secret form is being filled, the typed value is in that masked field; otherwise no secret value appears in the page (text and DOM), in the API events or in the conversation's state directory, on either side, and the screenshots show only lengths. An independent pass recounted every number in this table from the saved read-backs and step logs.

Checks that CI runs only for PRs into main

This PR is stacked, so those workflows do not run on it yet. Run locally on the code commit cac2960d (the head adds only the verification record under .pr/):

  • The exported OpenAPI document (export_agent_server_openapi.py) is byte-identical to main's.
  • REST API breakage check (check_agent_server_rest_api_breakage.py, with oasdiff 1.19.1): passes. What it lists against the 1.62.0 release are main's own additive changes (the reaction events of feat(agent-server): Conversation reactions, the first experimental feature #6609); with an identical OpenAPI document, this branch adds nothing to them.
  • Persisted settings compatibility (check_persisted_settings_compat.py): passes, the payloads of the 1.62.0 release included.
  • SDK API breakage check (check_sdk_api_breakage.py): no breaking changes in the three packages.
  • make test-server-schema: passes.
  • The agent-server code of the 1.62.0 release (tag v1.62.0) starts a conversation on this branch's SDK.

The last line is the mixed-version check. With the four fields taken off ConversationConfig, as in #4813, the same script ends in AttributeError: 'StoredConversation' object has no attribute 'secrets'.

GitHub CI on this branch's head (5c298f9e: the code commit cac2960d plus the verification record under .pr/): all 14 checks that ran are green.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

  • API surface. The exported OpenAPI document is byte-identical to main's. ConversationConfig and StartConversationRequest have the same fields as on main, with the same types, defaults and descriptions. The four fields now come after prompt instead of mid-list, so the server's own /openapi.json lists StartConversationRequest's properties in a different order; the key-sorted export does not show that. The only name this PR adds to the SDK is StoredConversationConfig (in openhands.sdk.conversation.request, re-exported from openhands.agent_server.models). In the agent-server, launch.request_security_analyzer and three EventService fields (initial_confirmation_policy, initial_security_analyzer, secrets) are new.
  • Older servers on the same files: this is a one-way change, and it needs a maintainer's decision. Code from before this PR re-applies meta.json's policy and analyzer on every load. On a meta.json written by this code it finds none, applies its defaults (NeverConfirm, no analyzer) and saves them over base_state.json; the reset then stays after rolling forward. The secrets in base_state.json are left as they are. Two ways to get there:
    • A rollback to an older server, or two server versions on one conversation store.
    • Docker runtime mode with a host and a container of different versions. The container owns the conversation and writes both files into the host's directory, but a host that starts while a conversation is still marked running (the container died mid-run and its lease expired) loads it with its own code, provided it can create the conversation's stored workspace path: always for a worktree conversation, and for the plain /workspace path only on a host that can create that directory (otherwise the load fails and writes nothing). With an older host and a container on this code, that load resets the policy. With a host on this code and an older container, the host's load drops the fields from meta.json and the container's next load resets it. The default container image is latest-python, which follows main and not a release, so a released host gets a newer container with its next pull. I reproduced both sequences in-process, with two checkouts on one conversation directory, the kill and the expired lease simulated by editing base_state.json and removing the lease file, and one shared key; no Docker daemon was available here, so the mount, the proxy routing and the per-conversation key are from reading the code.
    • The alternative is a transition period in which meta.json still gets the live policy and analyzer as a write-only mirror that this code never reads. It protects older readers and keeps a second copy on disk for a while, which is the thing this PR removes. I did not add it. A narrower step that helps only hosts on this code is to have the host's start-up recovery skip Docker-backed conversations, which also removes a second writer that main has today.
  • No cipher at all. Across restarts nothing changes: without a cipher both files only ever held the ********** placeholder. Inside a running process, see point 7 above. Two things remain different from main there: a sandbox pause loses the values as a restart does, and a fork does not carry the key inside a security analyzer (it never carried the agent's own model key). The README says so.
  • Saves during a step. A policy, analyzer or secret posted while a run waits on its model is saved at once, like any other change: fix(sdk): persist state changes made while a step waits on the model #5776, which closed that window, is in main and merged into this stack.
  • Other open PRs. This replaces fix(agent-server): own confirmation_policy, security_analyzer, secrets in ConversationState #4813. It also covers the agent-server half of fix(agent-server): persist secret, policy and analyzer changes in base_state #5788, which kept the meta.json copies as seeds. fix(agent-server): re-read meta.json when loading a persisted conversation #5777 changes the same load and reuse path for another reason. Both PRs replace the in-place patch of the catalog record with a parameter passed down to EventService (secrets here, resume_secrets there), and fix(agent-server): re-read meta.json when loading a persisted conversation #5777 then writes it into stored.secrets, which this PR removes. A trial merge conflicts in conversation_service.py and event_service.py; whichever lands second has to redo that plumbing on top of the other.
  • Not in this PR. StoredConversation still duplicates other state: hook_config, max_iterations, stuck_detection, tags and workspace. They are follow-up material; hook_config is the one that can drift through the public API today.
  • Merging the stack. Each stacked PR needs to be retargeted to main and rebased onto it after the one below merges; that push is what starts the main-only workflows on it.

🤖 Generated with Claude Code

https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY

BSmick6 and others added 11 commits November 11, 2026 18:49
…nd secrets out of meta.json

set_confirmation_policy, set_security_analyzer and update_secrets wrote
only to ConversationState (base_state.json), while StoredConversation
(meta.json) kept its own creation-time copy of all three. On every
resume, after an idle eviction or a server restart, EventService.start()
re-applied the meta.json copy and silently reverted the changes.

Follow the pattern of #4440, which moved `agent` off StoredConversation:
the three fields (and `secrets_encrypted`) move from ConversationConfig
down to StartConversationRequest, so StoredConversation cannot carry
them and there is no second copy to drift. base_state.json is their only
owner after creation.

- StartConversationRequest: now declares the four fields itself. The
  REST request body is unchanged.
- EventService: takes `confirmation_policy`, `security_analyzer` and
  `secrets` as init-only fields, like `agent`. start() seeds the policy
  and analyzer only when no base_state.json exists yet; a resume keeps
  what the state holds.
- ConversationService: passes the three to the EventService on create
  and no longer splats them into StoredConversation. The CODEX fallback
  secret goes to the EventService as an argument instead of a temporary
  patch of `stored.secrets`. apply_resume_secrets stops mirroring into
  meta.json.
- Child conversations inherit the three from the parent's
  ConversationState, since the parent's StoredConversation no longer has
  them.
- Telemetry reports the conversation's current policy, not the one it
  was created with.

Old meta.json files that still have the keys load fine: they are
ignored.

Fixes #4810.

This is #4813 brought up to date with main: the squash of its
agent-server commits (30f79f0, 44eecbc, aa8047d, d800fc2, c402636,
d948733, b0c0093, c6d100f, 6494ba1, cca43ac, a4e7a74, 8f3443d, 3ad137e)
plus the merge resolutions. The SDK parts of #4813 are in the two PRs
this one is stacked on.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Engel Nyst <engel.nyst@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…ed is set

The analyzer no longer goes through the StoredConversation payload that
is validated with the cipher, so with secrets_encrypted=True its nested
credentials (e.g. the guardrail LLM api_key) would stay ciphertext for
the first session. Validate it with the cipher, like the agent and the
secrets (`request_security_analyzer`, next to `request_secrets`).

Split out of #4813 (commit 5466de5, as adapted in its merge 77499fb).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…start

Drives a real ConversationService through a restart: a confirmation
policy, a security analyzer and a secret set after start are still
there, none of them is in meta.json, and a fork made after the restart
carries all three.

Split out of #4813 (commit aedf746).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…te lock

A parent starts a child from inside its own step. The step holds the
parent's state lock until the tool call returns, and the tool call
returns only when this server answers the start request. Reading the
inherited policy, analyzer and secrets under that lock therefore never
finished: the launch hung until the tool's timeout.

Read the three without the lock, as the agent and the budget already
are. Each is replaced on change, never mutated in place (update_secrets
now assigns a new registry), so a plain read sees a whole value. The
child also gets its own copies of the secret sources, as a fork does.

Raised in review on #4813
(#4813 (comment)).
The new test holds the parent's lock the way a running step does; it
times out without this change.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…pdate_secrets

apply_resume_secrets copied the secret registry, updated the copy and
assigned it back by hand. That is what LocalConversation.update_secrets
does now, so call it. The state lock is still held around the update and
the ACP credential restart, as before.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…d payload in one place

_create_conversation dumped the request, then popped confirmation_policy,
security_analyzer, secrets and secrets_encrypted from the result. The
dump already has an exclude set for what StoredConversation does not
carry (the agent sources, max_budget_per_run); name the four there.

Telemetry reads the policy through the conversation's public `state`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
Before this stack, update_secrets mutated the registry in place, so a
conversation created with start-time secrets and not saved since (no
message, no other state change) had them in meta.json and nowhere else.
With `secrets` gone from StoredConversation, the first resume by the new
server would ignore that key and then rewrite meta.json without it: the
only copy of the secret, lost on upgrade.

On resume, read the legacy `secrets` key from meta.json and add the
names base_state.json does not have. base_state.json wins for the names
it has, so a saved runtime override is not reverted. Names bound to a
runtime credential binding stay scrubbed. The next save_meta drops the
key, so this runs once per conversation.

The policy and the analyzer need nothing: base_state.json always had
them, and its value is the newer one.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…sume

The comment said all of them are unused on resume. `secrets` is not: it
is added on every start, which is how the CODEX fallback credential
reaches a resumed conversation.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…udget's

main's budget change keeps `max_budget_per_run` off meta.json through a
keyword-only `initial_max_budget_per_run` on EventService. The policy
and the analyzer follow the same rule (they seed a new conversation and
are ignored on a resume), so they take the same shape:
`initial_confirmation_policy` and `initial_security_analyzer`,
keyword-only. `secrets` keeps its name because it is not initial-only:
it is added on every start.

The mock-based test class for these fields is replaced by two tests on
a real EventService: a new conversation has its start settings in
base_state.json and none in meta.json, and a resume keeps the saved
policy and analyzer whatever the EventService was constructed with.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…and secrets

An idle conversation is evicted after 20 minutes by default and loaded
again on its next use. On main that reload re-applied the creation-time
values: a conversation switched to AlwaysConfirm went back to
NeverConfirm, and a replaced secret went back to its old value.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
… and secrets

F08 gains five sub-features: where the three live on disk (in
base_state.json, never in meta.json), a runtime policy and analyzer
after a restart, a replaced start-time secret after a restart, and an
upgrade from the latest release. F10 gains a fork that keeps settings
changed after the source was created.

The three credential-binding recipes in F12 that read
`meta.json`'s `secrets` now read the secret registry in
base_state.json, and F31's layout line names the new homes.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-agent-server/openhands/agent_server
   conversation_service.py131710492%222–223, 305, 326, 535–536, 709, 716–717, 817, 868–869, 873, 878, 928, 980–981, 988, 1021–1022, 1038, 1070, 1074, 1087, 1107, 1128–1129, 1138, 1208, 1218, 1245, 1251–1252, 1256–1257, 1265, 1292, 1298, 1392, 1398, 1403, 1409, 1417–1418, 1427–1430, 1439, 1452, 1470, 1507, 1511–1515, 1538, 1573, 1593, 1603, 1896, 1930–1933, 1987, 2115, 2118, 2125–2127, 2130–2131, 2137, 2147, 2151, 2180, 2189, 2260–2262, 2294, 2434, 2437, 2464, 2474, 2479–2482, 2496, 2507, 2519–2520, 2552, 2655, 2725, 2783, 2808–2809, 3216, 3269, 3272
   event_service.py120311890%203, 209–210, 381–382, 404–407, 553, 557, 562, 588, 595, 618, 621–622, 626–627, 638, 644, 715–716, 722, 778–779, 900, 919, 948, 958, 981–982, 986, 1046, 1081–1083, 1086, 1108–1109, 1120, 1127, 1133, 1346, 1379, 1382, 1466, 1484, 1498, 1691–1692, 1748, 1782, 1936–1938, 1987, 1991, 1998, 2012, 2028, 2036, 2060, 2107–2108, 2199, 2221, 2224–2225, 2252, 2259, 2275, 2277, 2284, 2292, 2294, 2311, 2329, 2376, 2387–2390, 2399, 2404, 2406, 2411, 2420–2421, 2427, 2439, 2449, 2487–2489, 2574–2575, 2577–2584, 2586–2587, 2597, 2607, 2623, 2634, 2658, 2664, 2675, 2684–2685, 2707, 2717
   launch.py131398%167, 170–171
openhands-sdk/openhands/sdk/conversation
   request.py126894%222, 510, 524, 533–534, 546, 552, 560
TOTAL57861932584% 

A full replay of F04 on this branch failed two recipes that asserted
the old one: `F04.create-settings` expected `secrets_encrypted` in
meta.json, and `F04.create-secrets` expected the secret there. They now
check that meta.json has neither and that base_state.json holds the
secret encrypted.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
Brings in the review fixes made on parts 1 and 2: the masked string form
of state update events, one copy idiom for what a fork takes over, hook
verdicts carried into a fork, and the GraySwan key serializer.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY

# Conflicts:
#	.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md
… fork's

A deep copy of a secret source fails on one that holds a lock or a
client. Use the same dump/validate round trip as for the analyzer, which
is what LocalConversation.fork does for both.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…holds the lock

arun() holds the parent's state lock on the event-loop thread while it
awaits the tool; run() holds it on a worker thread. The lock is
reentrant per thread, so the single-case test would still have passed
with a lock taken inline on the loop thread. Both cases now block a
thread on the launch the way the tool's HTTP request does.

The inherited secret sources are read from a snapshot of the registry's
mapping.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…d base

The agent-server package depends on openhands-sdk without a version
bound, so a released agent-server can run on a newer SDK. The released
StoredConversation extends ConversationConfig and reads
confirmation_policy, security_analyzer and secrets from it. Taking those
fields off ConversationConfig made such a server fail every conversation
start with AttributeError.

ConversationConfig keeps the four fields. The fields a server record may
hold move to a new base under it, StoredConversationConfig, and the
agent-server's StoredConversation extends that one. The record still
cannot carry the four fields, and StartConversationRequest is the same
model as on main: the exported OpenAPI document is byte-identical.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…ts are used

EventService.secrets is read once, in start(). Bound names were removed
from it by the credential scrub and again by a late binding, the second
time on a dict nothing reads any more. start() now leaves bound names out
when it hands the secrets to the conversation, and both removals go.

The tests that looked at that input dict look at the secret registry
instead. test_failed_resume_does_not_retain_plaintext_fallback is removed:
it guarded the temporary patch of stored.secrets, which no longer exists,
and its remaining assertion could not fail.

Two comments are corrected: a budget given on a resume does replace the
saved one, and secret updates no longer replace the stored record.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…efore the scrub

The secrets an older server left in meta.json were read into memory, and
written to base_state.json only after the conversation was built. In
between, the credential scrub rewrites meta.json without them. A load
that failed after the scrub (for example an unusable workspace path) lost
them for good.

They are now written to base_state.json first, for the names it lacks and
that have no credential binding, so the take-over no longer depends on the
rest of the load. A meta.json whose secrets cannot be read is reported in
the log instead of skipped silently.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
A start request for an existing conversation can carry a Codex fallback
credential. With secrets_encrypted set, the create path decrypted the
request's secrets and the resume path did not, so the cipher text was
installed as the credential, on a loaded and on an unloaded conversation.
The Docker runtime forwards every start with secrets_encrypted set.

The resume path reads the secrets through request_secrets, as create does.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…'s analyzer

The request's security analyzer went to the conversation as the same
object unless the request was encrypted. Two conversations started from
one request object then ran on one analyzer, and an analyzer can keep
per-conversation history. Before this stack the analyzer was copied by
its trip through the stored record.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
… without a cipher

Without OH_SECRET_KEY and without a session API key, base_state.json
holds no secret values. The loaded conversation is their only copy, where
the in-memory stored record used to be a second one for start-time
secrets. An ACP agent's provider credential is such a secret.

- A conversation that holds secrets is not unloaded when idle.
- A fork receives its source's secrets from memory.

A sandbox pause still loses them, as a restart does; the README says so.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
The stored record has not carried the agent for a while, and a child gets
its parent's current secrets, not the ones from the parent's start.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…eps-state' into claude/openhands-sdk-dual-source-truth-qp1x6r-3-single-owner

# Conflicts:
#	.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md
… conversation

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…eps-state' into claude/openhands-sdk-dual-source-truth-qp1x6r-3-single-owner
… again

The take-over of secrets an older server left in meta.json had become a
second place where the server parses and rewrites base_state.json by
hand. It goes back to LocalConversation.update_secrets after the
conversation is built.

The one case that needed the early write is kept, where it arises: the
credential scrub rewrites meta.json before the load, and only when the
conversation has a credential binding. The scrub already rewrites
base_state.json at that point, so it now carries along the legacy names
that file lacks. Without a binding nothing touches meta.json until the
load has succeeded, and a failed load can simply be retried.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…analyzer

EventService seeded the request's confirmation policy and analyzer only
when base_state.json did not exist. A create that fails after that file
is written (a full disk while the start secrets are saved, a kill) leaves
it behind, and the retry with the same id then ran with NeverConfirm and
no analyzer. main applied the requested values.

The seeding now depends on who is calling: only a creating caller passes
its agent to the EventService; a resume and a fork pass none.

Two tests on a real ConversationService: the retried create, and a child
that inherits its parent's current policy, analyzer and secrets. The
existing child test runs against a mocked EventService and passed with
the seeding removed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…under the single owner

F08 gains five bullets and richer versions of four: the start-time values
after a restart, a cleared analyzer, a fork made right after a restart, a
child that inherits the parent's current values, and a reload after a
sandbox pause. The policy bullet now also drives the conversation and
checks that the action waits. The on-disk bullet tells an absent key from
a null one with jq.

F10's fork bullet checks that the fork's secret is cipher text.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
Each of these was green with part of the change removed:

- telemetry reports the conversation's own policy at creation;
- a create with secrets_encrypted set decrypts the conversation's secrets,
  and refuses the request on a server that has no cipher;
- a start secret named like a bound credential stays out of the registry;
- a legacy meta.json that holds a bound credential next to other secrets:
  the others are taken over, the bound one is not;
- a legacy secrets key this server cannot parse does not block the resume;
- a second start request for an existing id leaves its saved policy,
  analyzer and secrets alone;
- a child's inherited secret is its own object.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…eps-state' into claude/openhands-sdk-dual-source-truth-qp1x6r-3-single-owner
…loaded too

The guard that keeps a conversation loaded when a reload would lose its
secrets looked at the secret registry only. A security analyzer can carry
a key of its own (a guardrail LLM, GraySwan). Without a cipher that key
is not saved either, and an analyzer without its key rates every action
UNKNOWN, which ConfirmRisky(confirm_unknown=False) lets through. main
re-applied the analyzer from memory after an idle unload.

The guard now asks of the registry and of the analyzer whether a dump
without credentials differs from one with them, so a lookup secret with
no secret header and an analyzer with no key stay evictable.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…en over

A secret an older server left in meta.json reads as unset when the server
runs under a key that cannot decrypt it. Taking it over called
update_secrets, which saves the state, and that save wrote out everything
else the load could not decrypt as unset: after the right key was
restored the agent's LLM key was gone. On main such a load does not write
base_state.json.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…eft behind

- Two tests asserted the secrets kwarg of a mocked EventService; that
  the secrets reach the conversation is tested on a real service.
- The telemetry context test no longer claims to check where the policy
  is read from: it is an argument now. It keeps what it still shows.
- The trajectory zip test scanned for secrets that meta.json cannot hold.
- The restart test asserted that two objects loaded from two files
  differ.
- Two tests read the conversation's private state; they use get_state().

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…eps-state' into claude/openhands-sdk-dual-source-truth-qp1x6r-3-single-owner
…thers

The secrets an older server left in meta.json were validated as one
mapping, so a single entry this server cannot parse (a secret kind from a
plugin that is gone) made the whole key unreadable, and the next write of
meta.json dropped all of them. Each entry is now read on its own: the
unreadable one is reported by name in the log and left behind, the rest
are taken over.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
The live before/after results on main and on the three branches, the
family replays, the UI run with two screenshots, and the local runs of
the checks CI keeps for PRs into main. Temporary, under .pr/.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…eps-state' into claude/openhands-sdk-dual-source-truth-qp1x6r-3-single-owner

# Conflicts:
#	.agents/skills/verify-agent-server/references/feature-map/F08-confirmation-security-secrets.md
#	.agents/skills/verify-agent-server/references/feature-map/README.md
ConversationService takes one ServerStores since #6702; the two tests
that restart a service on an older meta.json still passed secrets_store.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…eps-state' into claude/openhands-sdk-dual-source-truth-qp1x6r-3-single-owner
Taking a secret over saves the conversation state. The check was per
entry and only skipped a StaticSecret whose value read as unset, so two
loads that cannot decrypt the stored state still saved it: another key
with a legacy LookupSecret, and no key at all with a legacy
StaticSecret, whose cipher text was then also used as the value. Both
wrote the agent's key and the other secrets as missing.

The legacy mapping and base_state.json are now searched for a Fernet
token this server cannot decrypt, and if there is one nothing is taken
over. The warning says that the next write of meta.json drops those
secrets.

The failed-first-load test now also asserts what the credential scrub
did during that load, so it cannot pass without a resolved binding.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…eps-state' into claude/openhands-sdk-dual-source-truth-qp1x6r-3-single-owner
The runs are repeated on main at 91e8b5b and on the final commits of
the three parts: the before/after pairs, the whole families next to the
same families on main, the Agent Canvas UI, and the checks CI runs only
for PRs into main.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…eps-state' into claude/openhands-sdk-dual-source-truth-qp1x6r-3-single-owner

# Conflicts:
#	.agents/skills/verify-agent-server/references/feature-map/F10-fork-navigate-condense-ask.md
#	.agents/skills/verify-agent-server/references/feature-map/README.md
The proof is repeated on the stack merged with main at 3c66af1 and on
that main: the before/after pairs, the whole families on both sides
compared bullet by bullet, the SDK maps, and the checks CI runs only
for PRs into main. The UI section stays on the commits it was run on.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
Conflicts, all between the single-owner change and work that landed on
main since the last merge:

- conversation_service.py: main added `initial_sender` to the launch
  path (#6975); the launch helper now takes it next to the policy, the
  analyzer and the secrets this branch passes to the event service.
- event_service.py: main moved state-lock work onto the state executor
  (#6554). `apply_resume_secrets` runs its update through
  `run_state_op` as on main and still writes no secrets to meta.json.
- tests: both sides' imports and tests are kept.
- feature maps: main's new bullets and counts, plus this branch's
  single-owner bullets; the bullet that said a load re-applies the
  creation-time copy from meta.json stays removed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
main replaced the hand-written `{"cipher": ...}` and
`{"expose_secrets": True}` dicts with `cipher_context()` and
`client_context()` (#6779). The three places this branch adds now do the
same; behaviour is unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…sation

main added three tests that build a `ConversationState` with
`confirmation_policy=stored.confirmation_policy`. `StoredConversation`
no longer has that field on this branch, and none of the three is about
the policy, so they now take the state's default, like their siblings.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Agent-server conversation start benchmark

Base claude/openhands-sdk-dual-source-truth-qp1x6r-2-fork-keeps-s vs this PR merge b3f33aa8d6, measured in the same job on the same runner, interleaved: 5 server starts per side and scenario. Medians with 95% bootstrap CIs. The LLM is a mock that answers instantly, so this is agent-server overhead only.

base this PR change
Create conversation (warm) 0.112 s [0.104, 0.126] 0.122 s [0.110, 0.132] +9% ≈ same
↳ first agent event 0.159 s [0.152, 0.179] 0.164 s [0.159, 0.182] +3% ≈ same
First create right after server start 0.335 s [0.275, 0.459] 0.230 s [0.101, 0.341] -31% ≈ same
↳ first agent event 0.652 s [0.476, 0.793] 0.463 s [0.277, 0.628] -29% ≈ same
First create 3 s after server start 0.149 s [0.113, 0.182] 0.138 s [0.103, 0.170] -8% ≈ same
↳ first agent event 0.370 s [0.321, 0.434] 0.342 s [0.280, 0.387] -8% ≈ same
Create in a new git worktree (warm, 0.5 s git fetch) 0.652 s [0.630, 0.673] 0.660 s [0.652, 0.676] +1% ≈ same
↳ first agent event 0.695 s [0.673, 0.720] 0.713 s [0.700, 0.728] +3% ≈ same
Worktree creates that stalled the server > 20 ms 0/20 (0%) 0/20 (0%) +0 pts ≈ same
Create with one stdio MCP server (warm, 3 s startup) 1.112 s [1.098, 1.122] 1.106 s [1.101, 1.118] -1% ≈ same
↳ first agent event 3.750 s [3.713, 3.861] 3.709 s [3.667, 3.763] -1% ≈ same
Creates that stalled the server > 20 ms 0/35 (0%) 1/35 (3%) +3 pts ≈ same
Server startup (/alive) 4.071 s [3.891, 4.149] 4.092 s [3.837, 4.329] +1% ≈ same
How this is measured

Each create is a Canvas-shaped POST /api/conversations (agent profile, worktree: true, first message included, so the request waits for agent startup). A side task polls /alive every 20 ms to catch event-loop stalls. Scenarios, each on its own server starts:

  • default: a fresh directory per create, 8 creates per start.
  • pause: the first create waits 3 s after startup, as a user opening the UI would.
  • worktree: every create branches a new worktree from a local repository whose origin answers fetches after 0.5 s.
  • mcp: the agent profile enables one stdio MCP server that starts after 3 s, like an npx/uvx launch.

A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest /alive took over 20 ms; stall rows are flagged when the share of stalled creates moves by at least 20 points with a Fisher exact p below 0.01. The first-create rows have one sample per server start, so they are flagged only on a clear change. A "↳ first agent event" row times the create above it until the agent's first event arrives on a socket opened after the create returns, as Agent Canvas does, so it includes the connect and replay. Runner: blacksmith-2vcpu-ubuntu-2404-oo3mkfzxu6najd46, 178 environment variables. Run it locally with python scripts/agent_server_benchmarks/conversation_start.py --help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants