Repository navigation
fix(sdk): lock and atomically write installation metadata - #5690
vnktadithya wants to merge 23 commits into
Conversation
|
The TypeScript client integration-test failure is in "Set up Docker Buildx", before dependencies are installed or the branch image is built. Docker Hub timed out while pulling the BuildKit image: |
|
The |
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Scope and ownership
This is a Python SDK / Agent Server change (openhands-sdk/openhands/sdk/extensions/installation/ and openhands-agent-server/.../canvas_extensions/installed.py), which is exactly what this repository owns per AGENTS.md cross-repository boundaries. It needs no maintainer decision and no move.
What I verified
- Current state: head is still
06394f14and matches the reviewed SHA; PR is open, non-draft, no labels; the only existing review is a Copilot quota-error comment (no review threads). Linked issue #5496 is open with thebuglabel only. All current-head checks are green, includingsdk-tests,agent-server-tests,windows-tests,pre-commit, and the OpenAPI/type-budget gates. The two failure notes in the PR comments (TypeScript Docker Buildx timeout, OpenRouter catalog tests) are infrastructure /main-wide and do not reflect this head. - Focused tests:
uv run pytest tests/sdk/extensions/installation -q-> 67 passed, including the newtest_installation_concurrency.py.
Assessment
No material correctness, security, or compatibility defect found on the current head.
- Torn read fixed:
save_to_dirnow writes throughatomic_write_text(temp file +os.replace, explicitmode=0o644, matching the prior umask-derived permissions). Readers can no longer observe a truncated.installed.json, which was the root cause of thesource="local"rediscovery. - Lost update fixed:
InstallationMetadata.open()acquires aFileLock(installed_dir/.metadata.lock, 30s timeout) before loading and holds it throughMetadataSession.__exit__, so load-modify-save cycles serialize across threads and processes. The lock is released in afinally, including whensave_to_dirraises or the body errors..metadata.lockstarts with., sodiscover_untracked/validate_trackedskip it. - Windows
os.replaceinteraction handled:get()and canvas_tracked_namesswitched to the new lock-holdingread(), so no unlocked reader holds the metadata file open while a save replaces it. - Caller compatibility: every production caller already uses
open()in awithblock (manager.pyinstall/uninstall/_set_enabled/list_installed, andinstalled.py:462), so holding the lock for the session lifetime does not leak it. Thelockargument is optional and additive;read()is new. Nested sessions on the same directory are explicitly disallowed and I found none:_tracked_namesandinstall/disableacquire sequentially, never nested. - Residual race is correctly scoped as follow-up:
install()still rmtree+copytree outside the lock, so a concurrent list landing in that window can prune the entry and a force-reinstall can loseenabledstate. This is documented in the PR Notes, is a much smaller window than the fixed torn-read path, and does not affect the reported symptom (source is no longer rewritten aslocal). It does not block this fix. - Live evidence: the PR provides authentic before/after runs through the real Agent Server HTTP endpoints (
/api/plugins/install,PATCH /api/plugins/installed/{name},/refresh) with concurrentGET /api/plugins/installed, plus an explicit statement of the environment difference (Windows/local-path plugin vs the reported Linux/catalog path). This meets the repository's production-facing bug-fix evidence bar.
All four acceptance criteria on #5496 are met by this head. Note for the maintainer: this is a concurrency/timing fix, so a lightweight eval is not required, but the residual install()-window race noted above is worth tracking as a follow-up.
✅ APPROVED
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Scope and ownership
Python SDK / Agent Server change (openhands-sdk/openhands/sdk/extensions/installation/ and openhands-agent-server/.../canvas_extensions/installed.py), which this repository owns per the AGENTS.md cross-repository boundaries. No product/architecture decision or repository move is required.
Current state re-verified
- Head is
3d65e49fand matches the reviewed SHA; PR open, non-draft, no labels. - The head moved since the prior review (
06394f14). The only code delta issave_to_dir's mode0o644->0o600(plus a regression test).InstallationInfo.sourceis persisted verbatim and can carry URL credentials for private sources, so owner-only is the correct mode; it matchesatomic_write_text's default and the LLM/agent profile stores, and it is the fix for Copilot's one open security finding (thread now resolved/outdated). - No review from this account exists on
3d65e49f, so this is a fresh review of the moved head. - Linked issues: #5496 is open with
bugonly. The author moved the residual directory-swap race to #5985, which is open withbug,priority:low,ready-for-dev. - Focused tests:
uv run pytest tests/sdk/extensions/installation -q-> 68 passed (includes the new concurrency suite and the new owner-only-mode test).
Assessment
No material correctness, security, or compatibility defect on this head.
- Torn read / lost update closed:
save_to_dirwrites throughatomic_write_text(temp file +os.replace), andopen()holds aFileLock(installed_dir/.metadata.lock, 30s timeout) from before load untilMetadataSession.__exit__. The lock usesfcntl.flock/msvcrton a persistent lock file, so it is genuinely effective across processes, and the lock covers the load-modify-save cycle.read()gives read-only callers the same protection and avoids the Windowsos.replacehandle conflict..metadata.lockstarts with., so discovery/validation skip it. - Lock release is deterministic: released in a
finally;open()releases and re-raises if construction fails. Every production caller already usesopen()in awithblock, and no caller nests sessions on the same directory (_tracked_namesand install/disable acquire sequentially). - Security: metadata mode tightened to
0o600, with a regression test that pre-chmods the file to0o644. No credentials are added to logs. - Compatibility: additive (
lockkwarg, newread()); no public signature or persisted-shape break. - Remaining race is correctly out of scope:
install()still rmtree+copytree outside the lock; a list in that window can prune the entry and a force-reinstall can loseenabledstate. This is pre-existinginstall()behavior, not introduced here, does not cause the reportedsource="local"regression, and is now tracked in #5985 (ready-for-dev). It does not block this fix. - Live evidence: authentic before/after runs through the real Agent Server HTTP endpoints with concurrent
GET /api/plugins/installed, plus an explicit environment-difference statement, meet the repository's production-facing bug-fix evidence bar. The real-workflow run predates the mode-only commit, which does not touch the locking or read paths.
Non-blocking CI notes for the maintainer (process gates, not code defects)
Two required checks currently fail on this head and are owned by the deterministic automation / maintainer:
Validate PR description(.github/scripts/check_pr_description.py): linked issue #5496 carries neitherready-for-devnor a pre-rollout creation date (opened 2026-10-05, after the 2026-08-13 rollout), so the gate blocks. This is an issue-readiness label question, not a defect in the diff.unresolved-review-threads: one thread remains open (Copilot's directory-swap race note). The author replied scoping it to #5985; resolving the thread or adding areview-thread-waiver:line to the PR body clears the gate.
All code-level checks pass on this head (sdk-tests, agent-server-tests, windows-tests, pre-commit, OpenAPI/type-budget gates, etc.).
✅ APPROVED
|
A short status summary for whoever picks this up: the repo's review agent has approved the current head, and the code checks pass. The one remaining gate is the PR description check, which needs |
|
Thanks for the write-up and the before/after runs, the torn read is real and the atomic replace is the right first move. Before merging I ran the branch locally (head 3d65e49, macOS, Python 3.13) to see what the lock actually covers, and I think it needs another pass. The script and its output are at the bottom; everything below comes from that run or from the linked code. The folder swap in
The lock timeout surfaces as a bare Two smaller things. On the tests: What I would do: move the folder swap under the lock (or stop pruning entries whose folder is missing while an install is in flight), drop the read lock on POSIX and handle the Windows replace failure with a bounded retry inside repro_5690.py and its output on 3d65e49import json, os, tempfile, threading
from pathlib import Path
from pydantic import BaseModel
from openhands.sdk.extensions.installation import (
InstallationInterface, InstallationManager, InstallationMetadata,
)
class Ext(BaseModel):
name: str
version: str = "0.0.1"
description: str = "x"
class Iface(InstallationInterface):
@staticmethod
def load_from_dir(d: Path):
return Ext.model_validate_json((d / "extension.json").read_text())
def manager():
tmp = Path(tempfile.mkdtemp())
src = tmp / "src" / "magic-test"
(src / "assets").mkdir(parents=True)
(src / "extension.json").write_text(Ext(name="magic-test").model_dump_json())
for i in range(600): # big enough that copytree takes a few ms
(src / "assets" / f"f{i}").write_bytes(b"x" * 8192)
return src, InstallationManager(installation_dir=tmp / "installed", installation_interface=Iface())
# 1. update() racing list_installed() re-enables a disabled extension
src, m = manager()
m.install(str(src))
flips = 0
for _ in range(60):
m.disable("magic-test")
stop = threading.Event()
def reader():
while not stop.is_set():
try:
m.list_installed()
except Exception:
pass
readers = [threading.Thread(target=reader) for _ in range(3)]
for t in readers:
t.start()
m.update("magic-test")
stop.set()
for t in readers:
t.join()
flips += m.get("magic-test").enabled
print(f"1. disabled extension came back enabled after update(): {flips}/60")
# 2. get() on a read-only install dir
src, m = manager()
m.install(str(src))
(m.installation_dir / ".metadata.lock").unlink(missing_ok=True)
os.chmod(m.installation_dir, 0o555)
try:
print("2. get() on a 0o555 install dir:", m.get("magic-test").enabled)
except Exception as e:
print("2. get() on a 0o555 install dir raised:", type(e).__name__)
finally:
os.chmod(m.installation_dir, 0o755)
# 3. an unreadable .installed.json is replaced by rediscovered local/enabled entries
src, m = manager()
m.install(str(src))
m.disable("magic-test")
p = InstallationMetadata.get_metadata_path(m.installation_dir)
os.chmod(p, 0)
try:
m.list_installed()
except Exception as e:
print("3. list_installed() raised:", type(e).__name__)
finally:
os.chmod(p, 0o644)
e = json.loads(p.read_text())["extensions"]["magic-test"]
print(f"3. after list_installed() with an unreadable file: source={e['source']!r} enabled={e['enabled']}") |
|
Thanks for running the branch and writing the repro, @VascoSch92, and sorry for the slow reply. You’re right on all of it. The description’s table couldn’t catch these cases: with a one-file plugin, a list almost never lands between On #5985: the folder swap is on the Update path, so it can’t wait for a follow-up. I’ll fix it in this PR and close #5985 with it. What I’m changing:
Also in this push:
CI doesn’t run Windows, so I’ll rerun the suite and the Agent Server workflow on Windows with a large bundle, and replace the table in the description with those results. On the folder swap, I’ll go with what I described on #5985:
The swap is under the lock and the copy isn’t. That keeps lock time independent of bundle size, and the live path never holds a partially copied folder. On permissions, saves will keep the existing file’s mode, and a new file is created |
The enable/disable, uninstall and revoke-secret-grant Canvas endpoints are async def and called disable/enable/uninstall_canvas_extension directly. Those take the installation metadata FileLock, which waits with a blocking sleep for up to 30 s, so a held lock stalled the whole event loop. Run them through asyncio.to_thread, as the install endpoint already does. Adds a regression test that holds the metadata lock and checks /alive still answers while each endpoint waits for it.
The metadata lock timeout raised a bare TimeoutError, an OSError: plugin and skill routes answered 500 and the Canvas install route reported it as an invalid extension (422). Raise a dedicated InstallationMetadataLockTimeoutError instead and map it to 503 in one app-level handler, next to the existing domain-error handlers, so every plugin, skill and Canvas route returns a retryable 503. InstallationMetadataLoadError still answers 500: retrying cannot fix a corrupt metadata file.
Saves forced 0o600, narrowing a metadata file other local users could read. An existing file now keeps its mode, and a new file gets the umask's mode as a plain write gives it, as before saves became atomic. atomic_write_text gains mode=None for that; its 0o600 default and its other callers are unchanged. metadata_session becomes a method: it takes the metadata lock, which an attribute access should not do.
…data A root process sharing an installation directory with its user (an agent-server container on a bind mount) left a root-owned .metadata.lock the user could no longer open, and replaced the user's metadata file with a root-owned one. When running as root, give the lock file to the directory's owner and keep the metadata file's previous owner, as the in-place write did before saves became atomic. Best effort and POSIX only. The lock file is changed through a descriptor opened without following symlinks, and only when it has a single link.
…r the listing's lock A list racing a force reinstall could take the installed extension for a new one and disable it. Also drops an unused test helper.
Adapts the branch to two upstream changes in the same area: - OpenHands#6780 maps domain errors to HTTP in one place. The metadata lock timeout is now a LockTimeout, so ERROR_STATUS answers 503; the branch's own exception handler in api.py is gone, and routes declare the 503 with error_responses(). Canvas install re-raises LockTimeout before its OSError clause, since a LockTimeout is also an OSError. - OpenHands#5756 adds strict loading. strict stays on list_installed, load_installed, sync and load_from_dir. InstallationMetadata.open() drops it: a session already refuses metadata it cannot load, so strict had no effect there. OpenHands#6702 removed reset_stores(); the responsiveness test no longer needs it, as create_app() builds its own stores.
|
@VascoSch92, I've pushed a reworked version and rewritten the description for it. Point by point: 1–3. Folder swap outside the lock (SDK and Canvas)
I went with "copy outside, swap inside" instead of putting the whole delete-and-copy under the lock, so the lock time doesn't grow with the bundle size. Canvas: Tests:
Your 4. Read lock Reads no longer take the lock, on any platform. Tests: 5. Unreadable metadata A write session now raises Tests: 6. Lock timeout There's a new Test: 7. Event loop Enable, disable and uninstall now run the lock-taking calls through Test: 8. It's a method now. The description notes the compatibility change. 9. Mode and owner An existing regular file keeps its mode. A new file's mode comes from the umask ( Tests: I haven't opened the owner-only metadata issue I offered. With the umask deciding, 10. Threaded test
Also in this push
Your call on these
Also, "Validate PR description" check needs the |
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Fresh review of head cf1a3506 (merge of upstream main 7435f4be9 plus the rework addressing @VascoSch92's review of 3d65e49f). I re-verified current PR state, linked issues, CI, and ran the tests locally; no material findings.
Design — verified against the concerns raised in the previous round
- The folder swap now happens inside the metadata session:
install(force=True)/update()copy into a dot-prefixed.installing-staging dir outside the lock, then read the prior entry, carry overenabled, swap with two renames (with rollback), and write the entry in one session. A list can no longer prune the entry mid-copy or mid-swap, and cannot run between the renames. Canvasinstall_with_outcomedecideswas_installedinside the install's own session instead of from a stale pre-snapshot, andapply_canvas_extension_updatedoes its renames inside the session, with a per-apply backup root so a failed apply's rollback can't clear another apply's backup. - Reads take no lock and load via
load_from_dir; writes are serialized per installation directory and saved withatomic_write_text. I checkedfilelock3.20.3 semantics empirically: two sessions opened from different threads on the same dir block each other (freshFileLockinstances don't share the thread-local refcount), so the documented "do not nest sessions" rule holds and no code path here nests one. - A write session over an existing-but-unreadable/unparseable
.installed.jsonnow raisesInstallationMetadataLoadErrorand leaves the file unchanged, instead of saving rediscoveredlocal/enabled=Trueentries over it; a missing file still loads as empty. - Lock timeout maps to 503 via
LockTimeout→BusyinERROR_STATUS, declared on the 17 routes that can hit the lock; the Canvas install endpoint re-raisesLockTimeoutbefore itsOSError→ 422 mapping. Enable/disable/uninstall/revoke endpoints run the lock-taking calls throughasyncio.to_thread, and the responsiveness test verifies/alivekeeps answering while a lock is held.
Tests (this workspace, Linux, Python 3.13)
- The PR's suites (
test_installation_concurrency.py,test_installation_manager.py,test_installation_metadata.py,test_files.py,test_installation_lock_timeout.py,test_canvas_extensions_lock_responsiveness.py, plugins/canvas/skill router tests): all pass, including the event-forced interleavings for the swap windows and the real reader-thread test. - Full
tests/sdksuite: 8284 passed, 7 skipped; the 3 failures (XDG credentials-dir and subagent-loader tests) fail identically on a cleanmaincheckout — environment-dependent, not introduced here. - Full
tests/agent_serversuite (minus stress and the backend process tests): 3180 passed; the 14 credential-binding/launch-parity failures are byte-identical to a cleanmaincheckout. check_sdk_api_breakage.pypasses; themetadata_sessionproperty→method change is properly contained inopenhands.sdk.extensions.installation.
CI
All code checks pass on this head. The only failing check, "Validate PR description", is the label gate: issue #5496 currently has only bug and still lacks ready-for-dev (#5985 has it). As the author noted on Nov 15, that label addition is needed for the check to pass; it's a maintainer-side action, not a code defect in this PR.
The two open design questions the author left (brief 404 on GET .../installed/{name} between swap renames, and non-strict SDK load paths skipping installed extensions on lock timeout with a logged warning) match main's behavior or degrade gracefully, and are reasonable to settle in follow-ups.
✅ APPROVED

HUMAN:
Overlapping plugin requests (a list call landing while an install or enable/disable was saving .installed.json, or while an Update was replacing the plugin’s folder) rewrote the plugin’s real source as
local, made Update fail, and could silently re-enable a disabled plugin. I ran the same Agent Server workflow on Windows before and after the fix; the results are in the table below.AGENT:
Why
.installed.jsonrecords each installed plugin, skill and Canvas extension, with its source and enabled state. Requests read and write it without coordination, and Agent Canvas sends two or threeGET /api/plugins/installedright after an install (#5496). When a list overlaps another request:save_to_dirwrites in place (Path.write_texttruncates first). A list that reads the half-written file rediscovers the extension assource="local",enabled=Trueand saves that over the real entry. Update then fails, and a disabled plugin comes back enabled.install(force=True)andupdate()delete the installed folder and copy the new one before opening the metadata session. A list in that window prunes the entry, so the reinstall finds no previous state and writesenabled=True..plugin/plugin.jsonwhile a force install deletes the folder, so install and disable answer HTTP 500[WinError 32].Summary
FileLocksession (30 s timeout) and saves withatomic_write_text(temp file, thenos.replace). A session that can't read or parse an existing file raisesInstallationMetadataLoadErrorinstead of saving rediscovered entries over it. On Windows, the replace and the unlocked reads retryPermissionErrorfor up to 1 s (10 ms × 100).install(force=True)andupdate()copy the new version into a dot-prefixed.<name>.installing-*folder outside the lock, then swap it in with two renames and write the entry inside one locked session, keeping the previousenabledstate. Canvas install (install_with_outcome), apply and listing decide inside their own locked sessions.InstallationMetadataLockTimeoutError, aLockTimeout, so the agent server answers 503 as it does for other busy stores; the routes that can hit the lock declare it witherror_responses(). The async Canvas endpoints that take the lock call it throughasyncio.to_thread.Issue Number
Fixes #5496
Fixes #5985
How to Test
Unit tests (run in CI on Linux):
test_installation_concurrency.pyforces each interleaving with events instead of relying on timing, including a list during an Update's copy (test_list_while_update_copies_keeps_the_entry) and a list between the swap renames (test_list_cannot_run_between_the_swap_renames).test_reader_thread_finds_the_extension_during_savesruns a real reader thread against repeated saves.Real workflow on Windows. A running Agent Server, driven over HTTP by
api_repro_5496.py(below). Each cycle: force install, then disable, each raced by 3 threads callingGET /api/plugins/installed; read the plugin; click Update (POST .../refresh) alone; then click Update again, raced by the same lists. The raced Update starts from a disabled plugin with its real source: if the earlier steps lost either, a plain install and disable with nothing racing them restore it first. Fresh server and persistence dir per run,LOG_JSON=1, 30 cycles.--assets 600adds 600 files of 8 KiB, so the copy takes long enough for a list to land in the swap window.mainis7435f4be9, the upstream commit this branch merges.main, 1 filemain, 601 fileslocalThe script prints one combined "re-enabled" count; the split comes from its per-cycle
after raced Updatelines. With the source kept, the cause is the swap window (#5985). With the source rewritten tolocal, it's the torn read (#5496). The large bundle makes the swap window far more likely: 5 cases with 601 files against 1 with one file.On
main, every failed Update returned 400Failed to fetch the source of plugin 'magic-test'. The installed copy was kept., and every 500, from install, disable or a raced Update, was[WinError 32]on...\plugins\installed\magic-test\.plugin\plugin.json. With 601 files, three raced Updates answered 404Plugin 'magic-test' is not installedfor an installed plugin, because a list pruned it mid-Update.Suites on this branch:
main: five tests create symlinks, which need a privilege Windows withholds by default (WinError 1314), andtest_apply_rejects_absolute_resolved_reffails because_staged_path(feat(agent-server): Canvas Extensions staged refresh (check/apply) [4/4] #4374) accepts an absoluteC:\...path.test_reader_thread_finds_the_extension_during_saves20/20 passed.test_root_session_keeps_the_directory_users_ownership,test_root_session_does_not_give_away_a_hard_linked_lock_file) skip on CI's non-root runners. They pass when run as root on Linux.uv run pre-commit run --files <changed files>passes.CI's Windows job runs only
tests/tools, so none of these tests run on Windows in CI. The Windows results above are local.api_repro_5496.pyVideo/Screenshots
Not a visual change. API results and logs are in How to Test.
Type
Notes
Compatibility
InstallationManager.metadata_sessionis now a method. It is exported fromopenhands.sdk.extensions.installation, not fromopenhands.sdk, socheck_sdk_api_breakage.pypasses. Code using the property (with manager.metadata_session as s:) needsmetadata_session().openhands.sdk.extensions.installation:InstallationMetadataLoadError,InstallationMetadataLockTimeoutError(aLockTimeout, so also aTimeoutError) andInstallationOutcome, plusInstallationManager.install_with_outcome().InstallationMetadata.open()no longer takesstrict(added onmainin fix(sdk): drop a since-disabled ambient plugin's skills and MCP servers on resume #5756, not yet released): a session always raises on metadata it cannot load.strictstays onlist_installed(),load_installed(),MetadataSession.sync()andload_from_dir().atomic_write_textgainsmode=None(use the umask) andowner; the default stays0o600. New helperread_text_retrying. Its other callers (acp_file_credentials.py,io/local.py,event_service.py) get the same Windows retry, which applies only on Windows and only to aPermissionErrorthat would otherwise have been raised..installed.jsonkeeps an existing regular file's mode. A new file's mode comes from the umask (0o644under022,0o600under077). When the agent server runs as root on POSIX, the lock file goes to the install directory's owner and an existing metadata file keeps its owner (best effort, guarded byO_NOFOLLOWand a single-link check).create_app()gains 17503responses and is otherwise identical tomain's. The client is told to retry; the local path is only logged. Unreadable or unparseable metadata reaches the routers as a 500 (InstallationMetadataLoadError) and the file is left unchanged.