[WSLC] (2/2) Admit concurrent execs and refuse a busy container

[Soham Das (SohamDas2021)]

📖 Description

A caller could not run two sandboxes at once. The daemon admitted a single exec stream, so a second exec — even against a different container — was refused while the first ran, and the concurrency the previous change unlocked was unreachable.

The daemon now admits up to eight exec streams, one per container. A second exec naming a container that already has one in flight is refused immediately as busy instead of waiting behind a workload of unknown duration. Lifecycle commands on that container still wait for the run, because deleting the container would free a handle the run is using.

The bound of eight comes from memory. A streaming exec's output lives only in its bounded live-output queue, so the daemon stays near 128 MB of live output against clients that never drain. A streaming run no longer also fills the capped capture buffers it never reads, which is what keeps that figure true.

This is 2 of 2 for the work item, and includes the test work originally scoped as a third change.

🔗 References

• ADO work item 63911810
• Stacked on [WSLC] (1/2) Run the state-aware daemon's exec off the single worker thread #1407
• docs/wsl/wslc-state-aware.md updated: exec admission, the memory bound, a table of the two conditions that surface as busy, the cancellation lane, and the limitation entry

🔍 Validation

• cargo test -p mxc-sdk --features wslc — 4406 passed, 0 failed
• cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all -- --check — clean
• WSL2 host (not reachable from CI): five #[ignore] daemon tests, four daemon_ipc pipe tests, and the full E2E suite — one-shot 35/35, state-aware 93/93
• New coverage, over the real control pipe: two clients executing concurrently on their own sandboxes with no cross-talk and measured in-container overlap, and enough simultaneous clients to prove the cap refuses the excess before admission. At E2E level, a full provision/start/exec/stop lifecycle on one sandbox completes while a long exec runs on another
• Each new test was checked against a reverted implementation and fails for the intended reason: zeroing the cancellation lane starves cancellation, dropping the slot frees capacity early, restoring unconditional capture doubles per-exec memory, and raising the cap above the client count admits every client

✅ Checklist

• Signed the Contributor License Agreement
• Linked to an issue
• Updated documentation (if applicable)
• Updated Copilot instructions (if build, architecture, or conventions changed)
• If this PR changes Cargo.lock, the dependency-feed-check check passes (see docs/pull-requests.md)

📋 Issue Type

• Bug fix
• Feature
• Task

Microsoft Reviewers: Open in CodeFlow

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Copilot review overview

🟡 Changes recommended

Exec permits can be released while output remains buffered, exceeding the stated memory bound.

Review effort: Balanced
Findings: 1 · 2

Open (3)

• Share permit until both worker and handler finish · New
• Add worker barrier to prevent premature exec dispatch · New
• Record actual process exit time before disposal · New

What changed in this PR

Enables concurrent WSLc execution across containers, building on the off-worker execution support from #1407.

Changes:

• Raises exec capacity to eight and rejects same-container concurrent runs as busy.
• Adds reserved cancellation capacity and removes duplicate streaming-output capture.
• Expands concurrency tests and documents admission behavior.

File	Description
tests/​scripts/​run_wslc_state_aware_tests.ps1	Adds asynchronous phase helpers and concurrency scenarios.
src/​mxc-sdk/​tests/​wslc_daemon_ipc.rs	Tests concurrent pipe clients and capacity refusal.
src/​mxc-sdk/​src/​bin/​wslc_daemon/​session_manager.rs	Adds busy rejection, slot ownership, and worker tests.
src/​mxc-sdk/​src/​bin/​wslc_daemon/​control_server.rs	Raises admission limits and reserves cancellation capacity.
src/​mxc-sdk/​src/​backends/​wslc/​common/​container_steps.rs	Bypasses capture buffers when streaming output.
docs/​backends/​wslc/​wslc-state-aware.md	Documents concurrency, limits, and busy responses.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Soham Das (SohamDas2021)]

Leaving this open - I addressed the guarantee but not the way you suggested, so it's not fully closed.

You're right that provisioning establishes no barrier and the active_execs check only proves registration. Rather than add a test-only barrier to the worker, I injected the run starter (RunStarter on Worker) and wrote a deterministic unit test, a_cancelled_exec_never_reaches_the_runner, that sets the cancellation flag, dispatches, and asserts the runner was never invoked at all. No SDK, no race. It fails if you remove the pre-run cancellation check.

That covers the guarantee this live test exists to protect, and covers it more strongly - the old assertion could only observe that done reported Cancelled, not that the process was never created.

What I did not do is remove the race in the #[ignore] live test itself, so it can still flake on a slow host. It passed 5/5 across four runs here, and it's host-gated so it never runs in CI, but the window you describe is still there.

[Gudge (MGudgin)]

Stack-only review summary

This comment-only review compares #1428 at e2a5c84 to its actual PR base 7ce1c8e (#1407's head). The merge-base equals that base, and the local six-file diff matches gh pr diff after newline normalization. Findings on #1407 alone are not charged to this PR. The new run-bound permit addresses #1407's handler-disconnect over-admission at this tip.

Verified clean with receipts: The worker now checks its per-container in-flight map before deciding whether to return pre-admission Busy or park lifecycle work. Streaming callbacks choose the sink instead of the capture buffers, and the new real-pipe overlap test compares stamped in-container run intervals. Test markers in control_server.rs increased 20 to 23, session_manager.rs 42 to 47, and wslc_daemon_ipc.rs 2 to 4. I inspected source and test assertions; I did not rerun WSL2 E2E tests in this filing.

Previously raised on this same head (confirmed, not duplicated inline)

High — Output-queue lifetime exceeds the exec-slot lifetime. introduced_by_change: moving the permit into InFlightExec releases it on run completion even when a connected non-reading client's handler still holds a full 16 MiB output queue. Up to 16 general handlers can retain about 256 MiB of queued payload, rather than the advertised eight-queue/~128 MiB ceiling. Keep an output-memory claim until the handler drains or drops the queue while retaining a run claim until process completion. This is already anchored in the existing Copilot review (ID 5437849230).

Medium — E2E lifecycle-overlap assertion timestamps A too late. introduced_by_change: J4 records $cEnded before calling Wait-StateAware for A, whose EndedAt is then set to collection time. The comparison therefore succeeds even if A finished before C began. Compare against A's actual Process.ExitTime or an asynchronously recorded completion. Already anchored in review 5437849230.

Low — Queued-cancellation test assumes task poll order. introduced_by_change: the new test spawns provisioning and exec independently and waits for client-side exec registration, not for provisioning to occupy the worker. A worker admission barrier would make the intended queued-before-cancelled case deterministic. Already anchored in review 5437849230.

Verified pre-existing — not attributed to #1428

• The no-timeout exec continuing after its client disconnects was already possible in #1407. Counting that run against the new capacity is an intentional safety tradeoff, not a new orphaning defect here.
• The global client limiter and its saturation under enough parked lifecycle requests existed at #1407. Its separate cancellation reserve is new here, and the distinct capacity issue in the inline comments is attributed to that new lane.
• daemon_protocol.rs is byte-identical base to head. Its Busy variant's older doc was already incomplete for global capacity; the new third Busy case is instead retargeted at this PR's added two-case documentation table.

The remaining distinct findings are anchored below. The verdict is COMMENT, not a request for changes.

[Gudge (MGudgin)]

Medium (reliability, performance, correctness) — A cancellation slot can be held before the request is classified.

Attribution: introduced_by_change — the new cancellation fallback acquires a reserved permit before this cancel_only check runs.

With the 16 general slots occupied, eight same-owner connections can take all eight cancellation permits and stall while sending a first frame. The existing read_frame accepts a length up to 16 MiB and allocates the whole body before reading it; the first-frame deadline is 30 seconds. Thus these connections can hold ~128 MiB of input buffers and block a subsequent cancellation for that period. An exhausted lane drops the next connection without handling its request. This is an owner-only local liveness risk, not remote access.

Fix: Give lane requests a much smaller pre-allocation frame limit and shorter first-frame timeout, or identify cancellation via a separate compact endpoint before reserving its slots. Test cancellation under stalled partial-frame connections.

[Soham Das (SohamDas2021)]

Fixed with your first option: cancel-lane connections now get LANE_MAX_FRAME_BYTES = 1024 (sized off the 304 bytes a worst-case cancel_exec encodes to) and LANE_FIRST_FRAME_TIMEOUT = 2s, so your scenario drops from 8 × 16 MiB held 30s to 8 × 1 KiB held 2s. Covered by a_cancellation_survives_a_lane_saturated_by_stalled_connections, which fills every lane slot with partial-frame stalls and lands a real cancellation in 2.00s

[Copilot]

🟡 Changes recommended

Worker unwind cleanup can release the session and SDK while an image pull still uses them.

2 open findings

• Outstanding image pulls can outlive released session resources · New
• Add worker barrier to prevent premature exec dispatch

2 resolved since last review

• Share permit until both worker and handler finish
• Record actual process exit time before disposal

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[Copilot]

🔵 Needs a closer look

Concurrent native-handle ownership and Windows-only failure-containment paths warrant final human review.

4 open findings

• Outstanding image pulls can outlive released session resources
• Add worker barrier to prevent premature exec dispatch
• Document quarantine cleanup after successful container deletion · New
• Retry only after the exec permit is released · New

2 resolved since last review

• Share permit until both worker and handler finish
• Record actual process exit time before disposal

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[Copilot]

🟡 Changes recommended

The cancellation lane rejects valid escaped identifiers when general client capacity is exhausted.

5 open findings

• Cancellation rejects valid escaped identifiers over the body-size limit · New
• Outstanding image pulls can outlive released session resources
• Add worker barrier to prevent premature exec dispatch
• Retry only after the exec permit is released
• Document quarantine cleanup after successful container deletion

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[Copilot]

🔵 Needs a closer look

Native SDK concurrency and handle ownership require final human review of the Windows/WSL2 validation.

4 open findings

• Add worker barrier to prevent premature exec dispatch
• Document cancellation frame body size limit · New
• Retry only after the exec permit is released
• Document quarantine cleanup after successful container deletion

2 resolved since last review

• Cancellation rejects valid escaped identifiers over the body-size limit
• Outstanding image pulls can outlive released session resources

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.