Skip to content

Close backend validation coverage gaps - #1431

Merged
Elliot (theelliotm) merged 5 commits into
mainfrom
user/emichlin/close-test-coverage-gaps
Oct 7, 2026
Merged

Elliot (theelliotm) merged 5 commits into
mainfrom
user/emichlin/close-test-coverage-gaps

Conversation

@theelliotm

@theelliotm Elliot (theelliotm) commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

📖 Description

Closes gaps in the backend validation suites where a behaviour was claimed but never actually exercised. No product code changes — this is tests, fixtures, and one shared helper.

Seatbelt (macOS)

  • deniedPaths is now probed for what it actually denies: writes, directory listing, delete, rename, and metadata reads.
  • A denied subtree nested under a readwritePaths grant is checked for AF_UNIX bind() and connect(). A socket under a broad read-write root is a control plane (Docker, ssh-agent, gpg-agent), so reaching one would be an escape. The listener runs on the host because the sandbox is the client here.
  • An aliased deniedPaths entry (deny written as /tmp/…, grants written as /private/tmp/…) is checked to still outrank both a read-only and a read-write grant — the deny only outranks if alias resolution ran first.

WSLc

  • The isolated/bridged egress pair is now a real oracle. Both fixtures run the same raw-IP TCP connect, so "reached" vs "blocked" reflects the posture rather than DNS or a missing interpreter. The previous isolated fixture used wget against a hostname, which reports the same thing whether egress is blocked or the image simply lacks wget.
  • Per-destination egress rules (allow and deny) are now asserted to be rejected. WSLc networking is all-or-nothing — no CAP_NET_ADMIN for in-container rules — so a rule must be refused, not quietly widened to the posture default.
  • A proxy URL carrying credentials is asserted to be rejected: process.env without inheritDefaultEnv launches through env -i NAME=VALUE, which puts the URL in argv and therefore in /proc/<pid>/cmdline.

IsolationSession

  • New "Lifecycle G" group in the state-aware suite covering what a live session does rather than just its request/response shape: that the mandated all-allow network posture really carries traffic, that exec output reaches the caller mid-run, that process.timeout ends a command and leaves the session usable, that a caller killed mid-exec doesn't take the sandbox with it, that repeating a lifecycle call doesn't corrupt the sandbox, and that the agent account named in provision metadata is created and removed.
  • New tests/scripts/lib/LoopbackAnchor.ps1 gives the network assertions a positive oracle — a host loopback listener an isolated session can reach — so they cover the session's posture instead of the runner's outbound internet access.

Known failures

The new coverage found two real bugs, so this cannot go green as-is. Both are filed and neither is caused by this PR:

🔗 References

Surfaced by the new coverage in this PR (not fixed by it):

🔍 Validation

Scheduled Validation Tests, nightly plan: 36 of 39 jobs passed. Every failure is accounted for above.

✅ Checklist

📋 Issue Type

  • Bug fix
  • Feature
  • Task
Microsoft Reviewers: Open in CodeFlow

Copilot AI balanced review requested due to automatic review settings October 6, 2026 22:04
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The loopback oracle can inherit proxy settings and test the runner proxy instead of direct host loopback.

Review effort: Balanced
Findings: 3 Medium severity

Open (3)
What changed in this PR

Expands backend validation coverage for Seatbelt, WSLc, and IsolationSession, surfacing two tracked product defects.

Changes:

  • Adds filesystem, socket, egress, proxy, streaming, timeout, and lifecycle assertions.
  • Introduces a reusable host-loopback network oracle.
  • Replaces ambiguous WSLc probes with controlled raw-IP checks.
File Description
tests/​scripts/​run_wslc_all_tests.ps1 Adds WSLc network assertions.
tests/​scripts/​run_seatbelt_unix_socket_test.sh Tests denied AF_UNIX operations.
tests/​scripts/​run_seatbelt_path_resolution_test.sh Tests aliased deny precedence.
tests/​scripts/​run_seatbelt_filesystem_test.sh Expands denied-path coverage.
tests/​scripts/​run_isolation_session_tests.ps1 Adds network and streaming tests.
tests/​scripts/​run_isolation_session_state_aware_tests.ps1 Adds Lifecycle G scenarios.
tests/​scripts/​README.md Documents the loopback helper.
tests/​scripts/​lib/​LoopbackAnchor.ps1 Implements the loopback HTTP anchor.
tests/​configs/​wslc_network_proxy_credentials_rejected.json Tests credential rejection.
tests/​configs/​wslc_network_isolated.json Uses a raw-IP denied-egress probe.
tests/​configs/​wslc_network_egress_rules_rejected.json Tests allow-rule rejection.
tests/​configs/​wslc_network_egress_deny_rules_rejected.json Tests deny-rule rejection.
tests/​configs/​wslc_network_bridged.json Adds the allowed-egress control.
tests/​configs/​seatbelt_path_alias_denied.json Defines aliased deny coverage.
tests/​configs/​seatbelt_fs_denied_write.json Defines denied-write coverage.
tests/​configs/​seatbelt_fs_denied_unix_socket.json Defines denied-socket probes.
tests/​configs/​seatbelt_fs_denied_metadata.json Defines metadata and mutation probes.
tests/​configs/​isolation_session_streaming_smoke.json Lengthens streaming intervals.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/scripts/lib/LoopbackAnchor.ps1 Outdated
Comment thread tests/scripts/run_isolation_session_state_aware_tests.ps1 Outdated
Comment thread tests/scripts/run_isolation_session_tests.ps1 Outdated
A second deprovision on the same sandboxId is documented to report
stale_id, and start/stop/exec all do. deprovision does not: RemoveUser
reports success for an agent user that is already gone, so the runner
never sees the ERROR_NOT_FOUND it would promote.

MXC's side is already correct -- deprovision_agent_user classifies with
StalePromotion::Eligible -- and synthesizing the error here would
manufacture the provenance that classify_api_failure explicitly forbids.
So record the gap rather than asserting it, and let #1429 track the fix.

Assert-KnownGap never fails the suite, and reports when the expectation
starts holding so the waiver leaves with the fix.

Refs #1429

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 7, 2026 06:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The streaming oracles can false-pass during process-exit races, and the known-gap helper makes a documented contract violation pass.

Review effort: Balanced
Findings: 4 Medium severity

Open (4)
Previously missed (2)

In code that hasn't changed since last review

Medium severity Recheck process exit after reading the streaming marker

tests/​scripts/​run_isolation_session_state_aware_tests.ps1:376

The process is checked only before reading the file. If buffered output is flushed while the process exits between that check and Read-LiveFile, this returns true even though the marker was never observed during execution; that can falsely pass the streaming test and make the caller-kill test target an already-exited process. Recheck HasExited after observing the marker.

Medium severity Prevent post-exit output from satisfying incremental delivery

tests/​scripts/​run_isolation_session_tests.ps1:762

This can mark $sawEarly after the process has exited: output may flush between the loop's HasExited check and the file read. In that race, an implementation that buffers everything until exit passes the incremental-delivery oracle. Recheck the process after reading the marker.

Comment thread tests/scripts/run_isolation_session_state_aware_tests.ps1
Updated curl commands to include --noproxy option for better handling of requests.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 7, 2026 06:20
@theelliotm
Elliot (theelliotm) marked this pull request as ready for review October 7, 2026 06:21
@theelliotm
Elliot (theelliotm) requested a review from a team as a code owner October 7, 2026 06:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The Seatbelt assertion guarantees red validation, and the timeout timing window does not verify the configured deadline.

Review effort: Balanced
Findings: 1 High severity

Open (1)
Resolved since last review (4)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Timing test does not verify the configured process timeout

tests/​scripts/​run_isolation_session_state_aware_tests.ps1:1964

The only timing check is < 45s for a configured 3s timeout, so an implementation that ignores this value and applies a 30–40s timeout still passes; without a lower bound, an immediate unrelated termination after the first echo can also pass. Bound the observation around the requested deadline with reasonable CI tolerance so this test actually verifies process.timeout, rather than merely proving the 59s command did not finish naturally.

Comment thread tests/scripts/run_seatbelt_filesystem_test.sh
Copilot AI balanced review requested due to automatic review settings October 7, 2026 06:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The new loopback helper is omitted from both packaged IsolationSession test-bundle staging paths, causing those suites to fail at startup.

Review effort: Balanced
Findings: 2 High severity

Open (2)
Resolved since last review (1)

Comment thread tests/scripts/run_isolation_session_state_aware_tests.ps1
Comment thread tests/scripts/run_isolation_session_tests.ps1
@theelliotm
Elliot (theelliotm) merged commit 2d225ad into main Oct 7, 2026
31 checks passed
@theelliotm
Elliot (theelliotm) deleted the user/emichlin/close-test-coverage-gaps branch October 7, 2026 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants