Skip to content

fix(sdk): lock and atomically write installation metadata - #5690

Open
vnktadithya wants to merge 23 commits into
OpenHands:mainfrom
vnktadithya:fix/installed-json-race-5496
Open

vnktadithya wants to merge 23 commits into
OpenHands:mainfrom
vnktadithya:fix/installed-json-race-5496

Conversation

@vnktadithya

@vnktadithya vnktadithya commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

HUMAN:

Overlapping plugin requests (a list call landing while an install or enable/disable was saving .installed.json, or while an Update was replacing the plugin’s folder) rewrote the plugin’s real source as local, made Update fail, and could silently re-enable a disabled plugin. I ran the same Agent Server workflow on Windows before and after the fix; the results are in the table below.


AGENT:

Why

.installed.json records each installed plugin, skill and Canvas extension, with its source and enabled state. Requests read and write it without coordination, and Agent Canvas sends two or three GET /api/plugins/installed right after an install (#5496). When a list overlaps another request:

Summary

  • Writes are serialized and atomic; reads take no lock. Every load-modify-save runs in one FileLock session (30 s timeout) and saves with atomic_write_text (temp file, then os.replace). A session that can't read or parse an existing file raises InstallationMetadataLoadError instead of saving rediscovered entries over it. On Windows, the replace and the unlocked reads retry PermissionError for up to 1 s (10 ms × 100).
  • Install and Update swap folders under the lock. install(force=True) and update() copy the new version into a dot-prefixed .<name>.installing-* folder outside the lock, then swap it in with two renames and write the entry inside one locked session, keeping the previous enabled state. Canvas install (install_with_outcome), apply and listing decide inside their own locked sessions.
  • The server answers 503 when busy and keeps its event loop free. A lock timeout raises InstallationMetadataLockTimeoutError, a LockTimeout, so the agent server answers 503 as it does for other busy stores; the routes that can hit the lock declare it with error_responses(). The async Canvas endpoints that take the lock call it through asyncio.to_thread.

Issue Number

Fixes #5496
Fixes #5985

How to Test

Unit tests (run in CI on Linux):

uv run pytest tests/sdk/extensions tests/sdk/utils/test_files.py tests/agent_server/canvas_extensions tests/agent_server/test_plugins_router.py tests/agent_server/test_installation_lock_timeout.py tests/agent_server/test_canvas_extensions_lock_responsiveness.py -q

test_installation_concurrency.py forces each interleaving with events instead of relying on timing, including a list during an Update's copy (test_list_while_update_copies_keeps_the_entry) and a list between the swap renames (test_list_cannot_run_between_the_swap_renames). test_reader_thread_finds_the_extension_during_saves runs a real reader thread against repeated saves.

Real workflow on Windows. A running Agent Server, driven over HTTP by api_repro_5496.py (below). Each cycle: force install, then disable, each raced by 3 threads calling GET /api/plugins/installed; read the plugin; click Update (POST .../refresh) alone; then click Update again, raced by the same lists. The raced Update starts from a disabled plugin with its real source: if the earlier steps lost either, a plain install and disable with nothing racing them restore it first. Fresh server and persistence dir per run, LOG_JSON=1, 30 cycles. --assets 600 adds 600 files of 8 KiB, so the copy takes long enough for a list to land in the swap window. main is 7435f4be9, the upstream commit this branch merges.

Windows 11 Home Single Language, Python 3.13.14 main, 1 file main, 601 files this PR, 1 file this PR, 601 files
Source lost 19/30 14/30 0/30 0/30
Disable lost 12/30 16/30 0/30 0/30
Update failed (400) 19/30 14/30 0/30 0/30
HTTP 500 on install/disable 8 7 0 0
Raced Update ran (state restored first) 30 (19) 30 (17) 30 (0) 30 (0)
Raced Update re-enabled the plugin, source kept 1/30 5/30 0/30 0/30
Raced Update re-enabled the plugin, source became local 12/30 1/30 0/30 0/30
Raced Update HTTP errors 1 4 0 0
"Discovered untracked extension" log entries 49 55 0 0
"Failed to load installed extension metadata" log entries 46 52 0 0

The script prints one combined "re-enabled" count; the split comes from its per-cycle after raced Update lines. With the source kept, the cause is the swap window (#5985). With the source rewritten to local, it's the torn read (#5496). The large bundle makes the swap window far more likely: 5 cases with 601 files against 1 with one file.

On main, every failed Update returned 400 Failed to fetch the source of plugin 'magic-test'. The installed copy was kept., and every 500, from install, disable or a raced Update, was [WinError 32] on ...\plugins\installed\magic-test\.plugin\plugin.json. With 601 files, three raced Updates answered 404 Plugin 'magic-test' is not installed for an installed plugin, because a list pruned it mid-Update.

Suites on this branch:

  • Windows (the PR's tests plus the related Canvas, router, domain-error and plugin-loading tests): 462 passed, 14 skipped, 6 failed. The 6 failures also fail on main: five tests create symlinks, which need a privilege Windows withholds by default (WinError 1314), and test_apply_rejects_absolute_resolved_ref fails because _staged_path (feat(agent-server): Canvas Extensions staged refresh (check/apply) [4/4] #4374) accepts an absolute C:\... path.
  • Windows: test_reader_thread_finds_the_extension_during_saves 20/20 passed.
  • WSL (Ubuntu 24.04, non-root), adding the plugin, skill, file-router and API tests: 1665 passed, 3 skipped, 0 failed.
  • The two root-only tests (test_root_session_keeps_the_directory_users_ownership, test_root_session_does_not_give_away_a_hard_linked_lock_file) skip on CI's non-root runners. They pass when run as root on Linux.
  • uv run pre-commit run --files <changed files> passes.
    CI's Windows job runs only tests/tools, so none of these tests run on Windows in CI. The Windows results above are local.
api_repro_5496.py
"""Real-workflow repro for #5496 against a running Agent Server.
 
Mirrors what Agent Canvas does: each install and each enable/disable is
immediately followed by several concurrent GET /api/plugins/installed calls.
Each cycle then clicks Update (POST .../refresh), the action the issue reports
failing with "Unable to parse extension source: local".
 
It then clicks Update a second time, raced by the same concurrent lists. That
Update swaps the plugin's directory (#5985); the cycle checks the plugin is
still disabled afterwards. It starts from a disabled plugin with its real
source: if the steps above lost either, a plain install and disable, with
nothing racing them, restore it first. --assets N adds N 8 KiB files so the copy takes long enough for a
list to land in the swap window.
 
Usage:
    1. Start the server with an isolated persistence dir, e.g. (PowerShell):
         $env:OH_PERSISTENCE_DIR = "$env:TEMP\\oh-5496"
         uv run python -m openhands.agent_server --host 127.0.0.1 --port 8765
    2. In another terminal:
         uv run python api_repro_5496.py --url http://127.0.0.1:8765 --cycles 30
         (add --assets 600 for the large bundle)
"""
 
import argparse
import json
import tempfile
import threading
from pathlib import Path
 
import httpx
 
NAME = "magic-test"
 
 
def make_plugin(assets: int) -> Path:
    plugin_dir = Path(tempfile.mkdtemp()) / NAME
    (plugin_dir / ".plugin").mkdir(parents=True)
    (plugin_dir / ".plugin" / "plugin.json").write_text(
        json.dumps({"name": NAME, "version": "1.0.0", "description": NAME})
    )
    if assets:
        (plugin_dir / "assets").mkdir()
        for i in range(assets):
            (plugin_dir / "assets" / f"f{i}").write_bytes(b"x" * 8192)
    return plugin_dir
 
 
def with_concurrent_lists(client: httpx.Client, request, readers: int = 3):
    """Run *request* while *readers* GET /installed calls race it, like Canvas."""
    start = threading.Barrier(readers + 1)
    result: dict[str, httpx.Response] = {}
 
    def reader() -> None:
        start.wait()
        for _ in range(3):
            client.get("/api/plugins/installed")
 
    def writer() -> None:
        start.wait()
        result["response"] = request()
 
    threads = [threading.Thread(target=reader) for _ in range(readers)]
    threads.append(threading.Thread(target=writer))
    for thread in threads:
        thread.start()
    for thread in threads:
        thread.join()
    return result["response"]
 
 
def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--url", default="http://127.0.0.1:8765")
    parser.add_argument("--cycles", type=int, default=30)
    parser.add_argument("--assets", type=int, default=0)
    args = parser.parse_args()
 
    source = str(make_plugin(args.assets))
    bad_source = bad_enabled = errors = update_failed = 0
    restored = raced_runs = raced_failed = raced_reenabled = raced_source_lost = 0
 
    # No keep-alive: the server may close a connection after a 500, and a
    # reused dead connection would surface as a client-side reset.
    limits = httpx.Limits(max_keepalive_connections=0)
    with httpx.Client(base_url=args.url, timeout=60, limits=limits) as client:
        try:
            initial = client.get("/api/plugins/installed").json()["plugins"]
        except httpx.ConnectError:
            raise SystemExit(f"Server not reachable at {args.url}; start it first.")
        if initial:
            raise SystemExit(
                "Persistence dir is not fresh (plugins already installed)."
            )
        for cycle in range(1, args.cycles + 1):
            install = with_concurrent_lists(
                client,
                lambda: client.post(
                    "/api/plugins/install", json={"source": source, "force": True}
                ),
            )
            disable = with_concurrent_lists(
                client,
                lambda: client.patch(
                    f"/api/plugins/installed/{NAME}", json={"enabled": False}
                ),
            )
            for response in (install, disable):
                if response.status_code >= 400:
                    errors += 1
                    try:
                        detail = response.json().get("exception", response.text)
                    except ValueError:
                        detail = response.text
                    print(f"cycle {cycle}: HTTP {response.status_code} {detail}")
 
            # Same as Canvas reloading the page, then reading the plugin.
            client.get("/api/plugins/installed")
            info = client.get(f"/api/plugins/installed/{NAME}").json()
            if info.get("source") != source:
                bad_source += 1
                print(f"cycle {cycle}: source became {info.get('source')!r}")
            if info.get("enabled") is not False:
                bad_enabled += 1
                print(
                    f"cycle {cycle}: disabled plugin is enabled={info.get('enabled')!r}"
                )
 
            # Same as clicking Update in Canvas.
            update = client.post(f"/api/plugins/installed/{NAME}/refresh")
            if update.status_code >= 400:
                update_failed += 1
                try:
                    detail = update.json().get("exception", update.text)
                except ValueError:
                    detail = update.text
                print(f"cycle {cycle}: Update HTTP {update.status_code} {detail}")
 
            # Update again, raced by lists, from a correct state so an earlier
            # loss isn't counted twice. Restore it if the steps above lost it.
            before = client.get(f"/api/plugins/installed/{NAME}").json()
            if before.get("source") != source or before.get("enabled") is not False:
                restored += 1
                client.post(
                    "/api/plugins/install", json={"source": source, "force": True}
                )
                client.patch(f"/api/plugins/installed/{NAME}", json={"enabled": False})
                before = client.get(f"/api/plugins/installed/{NAME}").json()
            if before.get("source") == source and before.get("enabled") is False:
                raced_runs += 1
                raced = with_concurrent_lists(
                    client,
                    lambda: client.post(f"/api/plugins/installed/{NAME}/refresh"),
                )
                if raced.status_code >= 400:
                    raced_failed += 1
                    try:
                        detail = raced.json().get("exception", raced.text)
                    except ValueError:
                        detail = raced.text
                    print(
                        f"cycle {cycle}: raced Update HTTP {raced.status_code} {detail}"
                    )
                after = client.get(f"/api/plugins/installed/{NAME}").json()
                if after.get("enabled") is not False:
                    raced_reenabled += 1
                    print(
                        f"cycle {cycle}: after raced Update, "
                        f"enabled={after.get('enabled')!r}"
                    )
                if after.get("source") != source:
                    raced_source_lost += 1
                    print(
                        f"cycle {cycle}: after raced Update, "
                        f"source={after.get('source')!r}"
                    )
 
    print(
        f"\n{args.cycles} cycles: source lost {bad_source}x, "
        f"disable lost {bad_enabled}x, HTTP errors {errors}x, "
        f"Update failed {update_failed}x"
    )
    print(
        f"raced Update ran in {raced_runs}/{args.cycles} cycles "
        f"(state restored first in {restored}): "
        f"re-enabled {raced_reenabled}x, source lost {raced_source_lost}x, "
        f"HTTP errors {raced_failed}x"
    )
 
 
if __name__ == "__main__":
    main()

Video/Screenshots

Not a visual change. API results and logs are in How to Test.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

Compatibility

  • InstallationManager.metadata_session is now a method. It is exported from openhands.sdk.extensions.installation, not from openhands.sdk, so check_sdk_api_breakage.py passes. Code using the property (with manager.metadata_session as s:) needs metadata_session().
  • New exports from openhands.sdk.extensions.installation: InstallationMetadataLoadError, InstallationMetadataLockTimeoutError (a LockTimeout, so also a TimeoutError) and InstallationOutcome, plus InstallationManager.install_with_outcome().
  • InstallationMetadata.open() no longer takes strict (added on main in fix(sdk): drop a since-disabled ambient plugin's skills and MCP servers on resume #5756, not yet released): a session always raises on metadata it cannot load. strict stays on list_installed(), load_installed(), MetadataSession.sync() and load_from_dir().
  • atomic_write_text gains mode=None (use the umask) and owner; the default stays 0o600. New helper read_text_retrying. Its other callers (acp_file_credentials.py, io/local.py, event_service.py) get the same Windows retry, which applies only on Windows and only to a PermissionError that would otherwise have been raised.
  • .installed.json keeps an existing regular file's mode. A new file's mode comes from the umask (0o644 under 022, 0o600 under 077). When the agent server runs as root on POSIX, the lock file goes to the install directory's owner and an existing metadata file keeps its owner (best effort, guarded by O_NOFOLLOW and a single-link check).
  • REST API: the OpenAPI spec generated from create_app() gains 17 503 responses and is otherwise identical to main's. The client is told to retry; the local path is only logged. Unreadable or unparseable metadata reaches the routers as a 500 (InstallationMetadataLoadError) and the file is left unchanged.
  • No TypeScript client change here: CI checks the client against the pinned release, so the 503s arrive with the version bump after release.

@vnktadithya

Copy link
Copy Markdown
Contributor Author

The TypeScript client integration-test failure is in "Set up Docker Buildx", before dependencies are installed or the branch image is built. Docker Hub timed out while pulling the BuildKit image: Get "https://auth.docker.io/token?...": net/http: request canceled (Client.Timeout exceeded while awaiting headers). The same Docker Hub errors (429 / 500 from docker.io) are failing the other open PRs right now. Could someone re-run it once Docker Hub recovers?

@vnktadithya
vnktadithya marked this pull request as ready for review October 9, 2026 21:44
Copilot AI balanced review requested due to automatic review settings October 9, 2026 21:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vnktadithya

Copy link
Copy Markdown
Contributor Author

The sdk-tests / agent-server-tests failures are OpenRouter model-catalog tests that are failing on main and the other open PRs, not just this one. They read LiteLLM's live catalog (model_prices_and_context_window.json on BerriAI/litellm main), which currently has no openrouter/* entries. test_openrouter_entries_are_real_catalog_ids fails the same way on unmodified main(checked locally).

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Scope and ownership

This is a Python SDK / Agent Server change (openhands-sdk/openhands/sdk/extensions/installation/ and openhands-agent-server/.../canvas_extensions/installed.py), which is exactly what this repository owns per AGENTS.md cross-repository boundaries. It needs no maintainer decision and no move.

What I verified

  • Current state: head is still 06394f14 and matches the reviewed SHA; PR is open, non-draft, no labels; the only existing review is a Copilot quota-error comment (no review threads). Linked issue #5496 is open with the bug label only. All current-head checks are green, including sdk-tests, agent-server-tests, windows-tests, pre-commit, and the OpenAPI/type-budget gates. The two failure notes in the PR comments (TypeScript Docker Buildx timeout, OpenRouter catalog tests) are infrastructure / main-wide and do not reflect this head.
  • Focused tests: uv run pytest tests/sdk/extensions/installation -q -> 67 passed, including the new test_installation_concurrency.py.

Assessment

No material correctness, security, or compatibility defect found on the current head.

  • Torn read fixed: save_to_dir now writes through atomic_write_text (temp file + os.replace, explicit mode=0o644, matching the prior umask-derived permissions). Readers can no longer observe a truncated .installed.json, which was the root cause of the source="local" rediscovery.
  • Lost update fixed: InstallationMetadata.open() acquires a FileLock (installed_dir/.metadata.lock, 30s timeout) before loading and holds it through MetadataSession.__exit__, so load-modify-save cycles serialize across threads and processes. The lock is released in a finally, including when save_to_dir raises or the body errors. .metadata.lock starts with ., so discover_untracked / validate_tracked skip it.
  • Windows os.replace interaction handled: get() and canvas _tracked_names switched to the new lock-holding read(), so no unlocked reader holds the metadata file open while a save replaces it.
  • Caller compatibility: every production caller already uses open() in a with block (manager.py install/uninstall/_set_enabled/list_installed, and installed.py:462), so holding the lock for the session lifetime does not leak it. The lock argument is optional and additive; read() is new. Nested sessions on the same directory are explicitly disallowed and I found none: _tracked_names and install/disable acquire sequentially, never nested.
  • Residual race is correctly scoped as follow-up: install() still rmtree+copytree outside the lock, so a concurrent list landing in that window can prune the entry and a force-reinstall can lose enabled state. This is documented in the PR Notes, is a much smaller window than the fixed torn-read path, and does not affect the reported symptom (source is no longer rewritten as local). It does not block this fix.
  • Live evidence: the PR provides authentic before/after runs through the real Agent Server HTTP endpoints (/api/plugins/install, PATCH /api/plugins/installed/{name}, /refresh) with concurrent GET /api/plugins/installed, plus an explicit statement of the environment difference (Windows/local-path plugin vs the reported Linux/catalog path). This meets the repository's production-facing bug-fix evidence bar.

All four acceptance criteria on #5496 are met by this head. Note for the maintainer: this is a concurrency/timing fix, so a lightweight eval is not required, but the residual install()-window race noted above is worth tracking as a follow-up.

✅ APPROVED

@vnktadithya
vnktadithya requested a balanced review from Copilot October 13, 2026 13:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical race and file-permission issues, plus synchronous lock blocking in async endpoints.

2 open findings

🧠 Review effort: Lite

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Scope and ownership

Python SDK / Agent Server change (openhands-sdk/openhands/sdk/extensions/installation/ and openhands-agent-server/.../canvas_extensions/installed.py), which this repository owns per the AGENTS.md cross-repository boundaries. No product/architecture decision or repository move is required.

Current state re-verified

  • Head is 3d65e49f and matches the reviewed SHA; PR open, non-draft, no labels.
  • The head moved since the prior review (06394f14). The only code delta is save_to_dir's mode 0o644 -> 0o600 (plus a regression test). InstallationInfo.source is persisted verbatim and can carry URL credentials for private sources, so owner-only is the correct mode; it matches atomic_write_text's default and the LLM/agent profile stores, and it is the fix for Copilot's one open security finding (thread now resolved/outdated).
  • No review from this account exists on 3d65e49f, so this is a fresh review of the moved head.
  • Linked issues: #5496 is open with bug only. The author moved the residual directory-swap race to #5985, which is open with bug, priority:low, ready-for-dev.
  • Focused tests: uv run pytest tests/sdk/extensions/installation -q -> 68 passed (includes the new concurrency suite and the new owner-only-mode test).

Assessment

No material correctness, security, or compatibility defect on this head.

  • Torn read / lost update closed: save_to_dir writes through atomic_write_text (temp file + os.replace), and open() holds a FileLock (installed_dir/.metadata.lock, 30s timeout) from before load until MetadataSession.__exit__. The lock uses fcntl.flock/msvcrt on a persistent lock file, so it is genuinely effective across processes, and the lock covers the load-modify-save cycle. read() gives read-only callers the same protection and avoids the Windows os.replace handle conflict. .metadata.lock starts with ., so discovery/validation skip it.
  • Lock release is deterministic: released in a finally; open() releases and re-raises if construction fails. Every production caller already uses open() in a with block, and no caller nests sessions on the same directory (_tracked_names and install/disable acquire sequentially).
  • Security: metadata mode tightened to 0o600, with a regression test that pre-chmods the file to 0o644. No credentials are added to logs.
  • Compatibility: additive (lock kwarg, new read()); no public signature or persisted-shape break.
  • Remaining race is correctly out of scope: install() still rmtree+copytree outside the lock; a list in that window can prune the entry and a force-reinstall can lose enabled state. This is pre-existing install() behavior, not introduced here, does not cause the reported source="local" regression, and is now tracked in #5985 (ready-for-dev). It does not block this fix.
  • Live evidence: authentic before/after runs through the real Agent Server HTTP endpoints with concurrent GET /api/plugins/installed, plus an explicit environment-difference statement, meet the repository's production-facing bug-fix evidence bar. The real-workflow run predates the mode-only commit, which does not touch the locking or read paths.

Non-blocking CI notes for the maintainer (process gates, not code defects)

Two required checks currently fail on this head and are owned by the deterministic automation / maintainer:

  • Validate PR description (.github/scripts/check_pr_description.py): linked issue #5496 carries neither ready-for-dev nor a pre-rollout creation date (opened 2026-10-05, after the 2026-08-13 rollout), so the gate blocks. This is an issue-readiness label question, not a defect in the diff.
  • unresolved-review-threads: one thread remains open (Copilot's directory-swap race note). The author replied scoping it to #5985; resolving the thread or adding a review-thread-waiver: line to the PR body clears the gate.

All code-level checks pass on this head (sdk-tests, agent-server-tests, windows-tests, pre-commit, OpenAPI/type-budget gates, etc.).

✅ APPROVED

@vnktadithya

Copy link
Copy Markdown
Contributor Author

A short status summary for whoever picks this up: the repo's review agent has approved the current head, and the code checks pass. The one remaining gate is the PR description check, which needs ready-for-dev on #5496, since that issue was opened after the readiness rollout, and only maintainers can add that label. If anything in the approach should change first, I'm happy to update it. The leftover directory-swap race is tracked separately in #5985.

@VascoSch92

Copy link
Copy Markdown
Member

Thanks for the write-up and the before/after runs, the torn read is real and the atomic replace is the right first move. Before merging I ran the branch locally (head 3d65e49, macOS, Python 3.13) to see what the lock actually covers, and I think it needs another pass. The script and its output are at the bottom; everything below comes from that run or from the linked code.

The folder swap in install(force=True) is still outside the lock: rmtree and copytree run first and the session is only opened afterwards. A list that lands in that window prunes the entry, so the reinstall finds no previous enabled and resets it to true. I know that is the #5985 note, but it is exactly the Update button path, and it does not need a hook to hit: with a 600-file bundle and three threads calling list_installed(), update() re-enabled a disabled extension in 20 of 60 cycles. With a one-file extension I got 0 of 40, so the table in the description (install and disable raced against lists, with a tiny plugin) does not cover it. The Canvas side has the same shape in install_canvas_extension, where the name is already in pre_existing so _force_disable_new skips it, and in apply_canvas_extension_update, which swaps the folders and only then opens the session.

read() now takes the file lock, and filelock opens the lock file read-write, creating it when missing. So get() on an install dir the process cannot write to raises PermissionError where the merge-base returns the info. That reaches the plugin and skill GET routes, the Canvas detail and bundle routes and the backend revision check. I would not lock reads on POSIX at all: once the save is an atomic replace, an unlocked read sees either the old file or the new one, which is the point of the replace.

load_from_dir still turns any read error into empty metadata, and a write session then saves the rediscovered local/enabled entries over the real file. On the merge-base the same situation fails loudly because the in-place write hits the same permission error; with the atomic replace it succeeds silently. In my run, one list_installed() while .installed.json was unreadable rewrote the source to local and enabled to true, which is the #5496 symptom through a different door. A write session should raise when it cannot load the file instead of saving the fallback.

The lock timeout surfaces as a bare TimeoutError and nothing in plugins_router catches it, so while one request holds the lock, GET /plugins/installed and GET /plugins/installed/{name} return 500 once the timeout passes (checked with a TestClient and the timeout set to 1 s). On the merge-base those reads never failed this way. Related: the Canvas enable/disable and uninstall endpoints are async def and call the sync, lock-taking functions directly, while install already goes through asyncio.to_thread. filelock waits by polling with time.sleep, so a request that holds the lock for a couple of seconds (an uninstall rmtree, discovery on a slow disk) stalls the whole event loop for that long. A 2 s hold gave me a 1.9 s gap on a 50 ms heartbeat task.

Two smaller things. metadata_session is a property that now acquires the lock, so manager.metadata_session.extensions outside a with holds it until the object is collected; the next get() timed out for me. A method would make that harder to get wrong. And the save now writes 0o600 through a new inode, which the new test asserts. That changes owner and mode on every save, so a root agent-server on a bind-mounted ~/.openhands leaves a file the host user cannot read, and the fallback above takes over from there.

On the tests: test_unlocked_reader_never_sees_partial_metadata_during_save patches os.replace for the whole process and runs its reader inside the writer's thread, so there is no concurrent reader in it, and CI does not run the suite on Windows, where the sharing-violation path lives. All 68 tests in tests/sdk/extensions/installation pass on the branch for me; none of them hit the cases above.

What I would do: move the folder swap under the lock (or stop pruning entries whose folder is missing while an install is in flight), drop the read lock on POSIX and handle the Windows replace failure with a bounded retry inside atomic_write_text, raise instead of saving when a write session cannot load the file, run the sync calls in the Canvas endpoints through to_thread like install does, and map the lock timeout to a 503 in the routers.

repro_5690.py and its output on 3d65e49
$ LOG_LEVEL=ERROR uv run python repro_5690.py
1. disabled extension came back enabled after update(): 20/60
2. get() on a 0o555 install dir raised: PermissionError
3. after list_installed() with an unreadable file: source='local' enabled=True
import json, os, tempfile, threading
from pathlib import Path
from pydantic import BaseModel
from openhands.sdk.extensions.installation import (
    InstallationInterface, InstallationManager, InstallationMetadata,
)

class Ext(BaseModel):
    name: str
    version: str = "0.0.1"
    description: str = "x"

class Iface(InstallationInterface):
    @staticmethod
    def load_from_dir(d: Path):
        return Ext.model_validate_json((d / "extension.json").read_text())

def manager():
    tmp = Path(tempfile.mkdtemp())
    src = tmp / "src" / "magic-test"
    (src / "assets").mkdir(parents=True)
    (src / "extension.json").write_text(Ext(name="magic-test").model_dump_json())
    for i in range(600):  # big enough that copytree takes a few ms
        (src / "assets" / f"f{i}").write_bytes(b"x" * 8192)
    return src, InstallationManager(installation_dir=tmp / "installed", installation_interface=Iface())

# 1. update() racing list_installed() re-enables a disabled extension
src, m = manager()
m.install(str(src))
flips = 0
for _ in range(60):
    m.disable("magic-test")
    stop = threading.Event()
    def reader():
        while not stop.is_set():
            try:
                m.list_installed()
            except Exception:
                pass
    readers = [threading.Thread(target=reader) for _ in range(3)]
    for t in readers:
        t.start()
    m.update("magic-test")
    stop.set()
    for t in readers:
        t.join()
    flips += m.get("magic-test").enabled
print(f"1. disabled extension came back enabled after update(): {flips}/60")

# 2. get() on a read-only install dir
src, m = manager()
m.install(str(src))
(m.installation_dir / ".metadata.lock").unlink(missing_ok=True)
os.chmod(m.installation_dir, 0o555)
try:
    print("2. get() on a 0o555 install dir:", m.get("magic-test").enabled)
except Exception as e:
    print("2. get() on a 0o555 install dir raised:", type(e).__name__)
finally:
    os.chmod(m.installation_dir, 0o755)

# 3. an unreadable .installed.json is replaced by rediscovered local/enabled entries
src, m = manager()
m.install(str(src))
m.disable("magic-test")
p = InstallationMetadata.get_metadata_path(m.installation_dir)
os.chmod(p, 0)
try:
    m.list_installed()
except Exception as e:
    print("3. list_installed() raised:", type(e).__name__)
finally:
    os.chmod(p, 0o644)
e = json.loads(p.read_text())["extensions"]["magic-test"]
print(f"3. after list_installed() with an unreadable file: source={e['source']!r} enabled={e['enabled']}")

@vnktadithya

Copy link
Copy Markdown
Contributor Author

Thanks for running the branch and writing the repro, @VascoSch92, and sorry for the slow reply. You’re right on all of it. The description’s table couldn’t catch these cases: with a one-file plugin, a list almost never lands between rmtree and copytree.

On #5985: the folder swap is on the Update path, so it can’t wait for a follow-up. I’ll fix it in this PR and close #5985 with it.

What I’m changing:

  1. Folder swap under the lock in install(force=True) and update(). On the Canvas side: install_canvas_extension decides whether the extension is new inside the install’s own metadata session, instead of from a snapshot taken before it. apply_canvas_extension_update does its renames inside the session.
  2. No read lock. read() and get() go back to an unlocked load. atomic_write_text gets a bounded retry when os.replace fails on Windows because a reader has the file open. That util is shared with other stores, so they get the retry too.
  3. Write sessions raise on an unloadable file. If .installed.json exists but can’t be read or parsed, a write session raises instead of saving the rediscovered local/enabled entries. A missing file still loads as empty.
  4. Canvas endpoints off the event loop. The enable/disable and uninstall endpoints call the sync functions through asyncio.to_thread, like install already does.
  5. Lock timeout returns 503. It becomes a dedicated exception, mapped to 503 in the plugin, skill, and Canvas routers.

Also in this push:

  • metadata_session becomes a method.
  • test_unlocked_reader_never_sees_partial_metadata_during_save is replaced by a test that reads from a separate thread.
  • Your three cases become regression tests: a test that forces a list call into the swap window during update(), get() on a 0o555 dir, and a list over an unreadable file.
  • The branch is updated with main, since manager.py changed in fix(sdk): skip auto-loaded plugins and installed extensions that fail to load #6439.

CI doesn’t run Windows, so I’ll rerun the suite and the Agent Server workflow on Windows with a large bundle, and replace the table in the description with those results.

On the folder swap, I’ll go with what I described on #5985:

  • install(force=True) copies the new version into a dot-prefixed staging dir. Discovery already skips dot-dirs, and I’ll use a name separate from Canvas’s .staging/<name>.
  • Inside the metadata session, it reads the previous enabled, swaps the folders with two renames and rollback as in apply_canvas_extension_update, and writes the entry.

The swap is under the lock and the copy isn’t. That keeps lock time independent of bundle size, and the live path never holds a partially copied folder.

On permissions, saves will keep the existing file’s mode, and a new file is created 0o644, matching main under the default 022 umask. 0o600 for new files would still leave a root-created file unreadable to the host user on a fresh setup. source can carry URL credentials, but main already stores them at 0o644, so this isn’t a regression. If you want owner-only metadata, I can open a separate issue for it. I’ll update the test that asserts 0o600.

@vnktadithya
vnktadithya marked this pull request as draft November 10, 2026 03:20
The enable/disable, uninstall and revoke-secret-grant Canvas endpoints are
async def and called disable/enable/uninstall_canvas_extension directly.
Those take the installation metadata FileLock, which waits with a blocking
sleep for up to 30 s, so a held lock stalled the whole event loop. Run them
through asyncio.to_thread, as the install endpoint already does.

Adds a regression test that holds the metadata lock and checks /alive
still answers while each endpoint waits for it.
The metadata lock timeout raised a bare TimeoutError, an OSError: plugin and skill routes answered 500 and the Canvas install route reported it as an invalid extension (422). Raise a dedicated InstallationMetadataLockTimeoutError instead and map it to 503 in one app-level handler, next to the existing domain-error handlers, so every plugin, skill and Canvas route returns a retryable 503.

InstallationMetadataLoadError still answers 500: retrying cannot fix a corrupt metadata file.
Saves forced 0o600, narrowing a metadata file other local users could read. An existing file now keeps its mode, and a new file gets the umask's mode as a plain write gives it, as before saves became atomic. atomic_write_text gains mode=None for that; its 0o600 default and its other callers are unchanged.

metadata_session becomes a method: it takes the metadata lock, which an attribute access should not do.
…data

A root process sharing an installation directory with its user (an agent-server container on a bind mount) left a root-owned .metadata.lock the user could no longer open, and replaced the user's metadata file with a root-owned one. When running as root, give the lock file to the directory's owner and keep the metadata file's previous owner, as the in-place write did before saves became atomic.

Best effort and POSIX only. The lock file is changed through a descriptor opened without following symlinks, and only when it has a single link.
…r the listing's lock

A list racing a force reinstall could take the installed extension for a new one and disable it. Also drops an unused test helper.
Adapts the branch to two upstream changes in the same area:

- OpenHands#6780 maps domain errors to HTTP in one place. The metadata lock timeout
  is now a LockTimeout, so ERROR_STATUS answers 503; the branch's own
  exception handler in api.py is gone, and routes declare the 503 with
  error_responses(). Canvas install re-raises LockTimeout before its
  OSError clause, since a LockTimeout is also an OSError.
- OpenHands#5756 adds strict loading. strict stays on list_installed, load_installed,
  sync and load_from_dir. InstallationMetadata.open() drops it: a session
  already refuses metadata it cannot load, so strict had no effect there.

OpenHands#6702 removed reset_stores(); the responsiveness test no longer needs it,
as create_app() builds its own stores.
@vnktadithya
vnktadithya marked this pull request as ready for review November 15, 2026 09:44
@vnktadithya

Copy link
Copy Markdown
Contributor Author

@VascoSch92, I've pushed a reworked version and rewritten the description for it. Point by point:

1–3. Folder swap outside the lock (SDK and Canvas)

install(force=True) and update() now copy the new version into a dot-prefixed .<name>.installing-* directory outside the lock (discovery skips it). Inside one metadata_session() they read the previous entry, carry over its enabled, swap the folders with two renames (_swap_directory, which rolls back on failure) and write the entry. The old version is deleted after the lock is released.

I went with "copy outside, swap inside" instead of putting the whole delete-and-copy under the lock, so the lock time doesn't grow with the bundle size.

Canvas: install_canvas_extension decides whether the extension is new inside the install's own session (install_with_outcome), and apply_canvas_extension_update does its renames inside the session.

Tests:

  • SDK: test_list_while_update_copies_keeps_the_entry (your case), test_list_cannot_run_between_the_swap_renames, test_failed_swap_keeps_the_installed_version, test_previous_version_is_deleted_after_the_lock_is_released, test_install_does_not_replace_a_directory_that_appeared_meanwhile
  • Canvas: test_force_reinstall_with_a_concurrent_list_keeps_disabled_state, test_force_reinstall_after_a_concurrent_uninstall_lands_disabled, test_list_cannot_run_between_the_apply_renames, test_apply_keeps_the_previous_bundle_when_the_rollback_fails
  • Router: test_refresh_with_a_concurrent_list_keeps_a_disabled_plugin_disabled

Your repro.py case 1 now gives 0/60 (Linux). The description's Windows table covers the same case over HTTP: with a 601-file bundle, a raced Update re-enabled a disabled plugin with its source kept in 5/30 cycles on main, and 0/30 with this PR.

4. Read lock

Reads no longer take the lock, on any platform. get() reads through load_from_dir, and InstallationMetadata.read() is gone. On Windows, both the replace and the unlocked read retry PermissionError (10 ms × 100, about 1 s), because there a reader can also fail to open the file while it's being replaced.

Tests: test_get_reads_a_read_only_installation_dir (your case 2), test_get_does_not_wait_for_an_open_session, and the retry tests in test_files.py.

5. Unreadable metadata

A write session now raises InstallationMetadataLoadError when .installed.json exists but can't be read or parsed. A missing file still loads as empty.

Tests: test_open_refuses_unreadable_metadata, test_open_refuses_metadata_it_cannot_load, test_open_without_metadata_file_starts_empty, test_list_installed_keeps_unreadable_metadata (your case 3).

6. Lock timeout

There's a new InstallationMetadataLockTimeoutError. It subclasses LockTimeout from #6780, so ERROR_STATUS answers 503, and the 17 routes that can hit the lock declare it with error_responses(), including the plugin and skill marketplace routes. Since a LockTimeout is also an OSError, Canvas install re-raises it before its except OSError clause, which would otherwise report a busy lock as 422 "Invalid canvas extension". The client is told to retry; the local path is only logged.

Test: test_lock_timeout_returns_503, parametrized over the 17 routes.

7. Event loop

Enable, disable and uninstall now run the lock-taking calls through asyncio.to_thread. So does revoke-secret-grant, which makes the same call but wasn't in either of our lists.

Test: test_alive_answers_while_canvas_request_waits_for_lock holds the lock from another FileLock and checks that /alive answers within 2 s, for each of the four endpoints.

8. metadata_session

It's a method now. The description notes the compatibility change.

9. Mode and owner

An existing regular file keeps its mode. A new file's mode comes from the umask (0o644 under 022, 0o600 under 077), not a fixed 0o644 as I wrote earlier. When the server runs as root on POSIX, the lock file goes to the install directory's owner and an existing metadata file keeps its owner.

Tests: test_save_to_dir_keeps_existing_file_mode, test_save_to_dir_creates_new_file_with_umask_mode, and two root-only tests that skip on CI's non-root runners; I ran those as root.

I haven't opened the owner-only metadata issue I offered. With the umask deciding, 077 already gives 0o600 for new files. I can still open it if you want.

10. Threaded test

test_unlocked_reader_never_sees_partial_metadata_during_save is replaced by test_reader_thread_finds_the_extension_during_saves, which reads from its own thread during repeated saves. CI's Windows job only runs tests/tools, so the Windows results in the description are local runs.

Also in this push

  • list_installed_canvas_extensions now decides which entries are new inside its own session. I found this while testing: a listing that ran between the swap renames saw an enabled extension as new and disabled it. On main the same snapshot is also taken outside any session, so the race isn't new. Test: test_list_during_a_reinstall_keeps_an_enabled_extension_enabled.
  • The other callers of atomic_write_text get the same retry. It only applies on Windows, and only to a PermissionError that would otherwise have been raised.
  • Merged main up to 7435f4be9 and adapted to two changes there: the lock timeout is a LockTimeout (refactor(agent-server): map domain errors to HTTP once #6780, see point 6), and fix(sdk): drop a since-disabled ambient plugin's skills and MCP servers on resume #5756's strict stays everywhere except InstallationMetadata.open(), where a session already refuses metadata it can't load.

Your call on these

  1. GET .../installed/{name} reads without the lock, so between the two renames it can briefly answer 404. main has the same gap for its whole delete-and-copy. OK to leave?
  2. load_user_skills() and the default load_user_plugins() catch every error, so a lock timeout or unreadable metadata there skips the installed extensions for that one load and logs a warning. With strict=True (conversation resume, fix(sdk): drop a since-disabled ambient plugin's skills and MCP servers on resume #5756) the error propagates instead. Leave the default path as it is, or should it surface the error too?
  3. On Windows, the Canvas backend's status() and prepare() call revision() on the event loop, so under contention the read retry can sleep there for up to about 1 s. Move it to a thread here, or in a follow-up?
  4. The two error-branch disable() calls in the Canvas enable endpoint have no test; deleting either one still passes. They predate this PR, which only moved them to a thread. Do you want tests for them here?
  5. filelock opens the lock file with O_TRUNC, so a root session would truncate a root-owned file hard-linked into a user's install directory before the ownership check runs. fs.protected_hardlinks=1, the Linux default, blocks that link. I've treated it as out of scope.
  6. No unit test pins the 10 ms × 100 retry schedule. It's based on Windows runs, where the previous 7-step schedule let the writer give up in 2 of 10 runs of the threaded test.
  7. I didn't add feature-map entries or a live-server test under tests/cross/. The 503 and responsiveness tests run in process through create_app. Tell me if you want either.

Also, "Validate PR description" check needs the ready-for-dev label on #5496. Could you please add it?

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Fresh review of head cf1a3506 (merge of upstream main 7435f4be9 plus the rework addressing @VascoSch92's review of 3d65e49f). I re-verified current PR state, linked issues, CI, and ran the tests locally; no material findings.

Design — verified against the concerns raised in the previous round

  • The folder swap now happens inside the metadata session: install(force=True)/update() copy into a dot-prefixed .installing- staging dir outside the lock, then read the prior entry, carry over enabled, swap with two renames (with rollback), and write the entry in one session. A list can no longer prune the entry mid-copy or mid-swap, and cannot run between the renames. Canvas install_with_outcome decides was_installed inside the install's own session instead of from a stale pre-snapshot, and apply_canvas_extension_update does its renames inside the session, with a per-apply backup root so a failed apply's rollback can't clear another apply's backup.
  • Reads take no lock and load via load_from_dir; writes are serialized per installation directory and saved with atomic_write_text. I checked filelock 3.20.3 semantics empirically: two sessions opened from different threads on the same dir block each other (fresh FileLock instances don't share the thread-local refcount), so the documented "do not nest sessions" rule holds and no code path here nests one.
  • A write session over an existing-but-unreadable/unparseable .installed.json now raises InstallationMetadataLoadError and leaves the file unchanged, instead of saving rediscovered local/enabled=True entries over it; a missing file still loads as empty.
  • Lock timeout maps to 503 via LockTimeout → Busy in ERROR_STATUS, declared on the 17 routes that can hit the lock; the Canvas install endpoint re-raises LockTimeout before its OSError → 422 mapping. Enable/disable/uninstall/revoke endpoints run the lock-taking calls through asyncio.to_thread, and the responsiveness test verifies /alive keeps answering while a lock is held.

Tests (this workspace, Linux, Python 3.13)

  • The PR's suites (test_installation_concurrency.py, test_installation_manager.py, test_installation_metadata.py, test_files.py, test_installation_lock_timeout.py, test_canvas_extensions_lock_responsiveness.py, plugins/canvas/skill router tests): all pass, including the event-forced interleavings for the swap windows and the real reader-thread test.
  • Full tests/sdk suite: 8284 passed, 7 skipped; the 3 failures (XDG credentials-dir and subagent-loader tests) fail identically on a clean main checkout — environment-dependent, not introduced here.
  • Full tests/agent_server suite (minus stress and the backend process tests): 3180 passed; the 14 credential-binding/launch-parity failures are byte-identical to a clean main checkout.
  • check_sdk_api_breakage.py passes; the metadata_session property→method change is properly contained in openhands.sdk.extensions.installation.

CI

All code checks pass on this head. The only failing check, "Validate PR description", is the label gate: issue #5496 currently has only bug and still lacks ready-for-dev (#5985 has it). As the author noted on Nov 15, that label addition is needed for the check to pass; it's a maintainer-side action, not a code defect in this PR.

The two open design questions the author left (brief 404 on GET .../installed/{name} between swap renames, and non-strict SDK load paths skipping installed extensions on lock timeout with a logged warning) match main's behavior or degrade gracefully, and are reasonable to settle in follow-ups.

✅ APPROVED

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

4 participants