feat(sec-254): vsock CID allocation and end-to-end tests#6
Draft
jasonhernandez wants to merge 16 commits into
Draft
feat(sec-254): vsock CID allocation and end-to-end tests#6jasonhernandez wants to merge 16 commits into
jasonhernandez wants to merge 16 commits into
Conversation
Add vsock device support across both Firecracker (Linux) and AVF (macOS) backends, enabling structured host↔guest communication over a Unix domain socket instead of SSH polling. CLI: `ember vm create myvm --image base --vsock` YAML config: `vsock: true` UDS created at: `<state_dir>/vms/<name>/vsock.sock` Linux (Firecracker): - New `PUT /vsock` API call with guest CID and UDS path - Firecracker natively creates the UDS and bridges to guest AF_VSOCK macOS (AVF): - VZVirtioSocketDeviceConfiguration added to VM config - ember-vz implements a UDS bridge: accepts host connections on the UDS and proxies them to guest vsock port 1024, and accepts guest-initiated connections on port 1024 and bridges them back to the UDS Both platforms expose the same UDS interface — Thermite's code path is identical regardless of the underlying hypervisor. Co-Authored-By: Claude <noreply@anthropic.com>
ember vm stop --all # stop all running VMs ember vm stop --all --force # SIGKILL all running VMs ember vm delete --all --force # stop + delete every VM Useful for cleanup and for ending all VMs (including non-pool control agent VMs that pool destroy doesn't touch). Co-Authored-By: Claude <noreply@anthropic.com>
Lightweight Rust daemon that runs inside Ember VMs and serves the JSON-lines protocol expected by Thermite's EmberdClient. Listens on vsock port 1024 (Linux) or a Unix domain socket (--uds, for testing). Operations: ping, exec, read_file, write_file, agent_status. - New `emberd/` workspace member with minimal dependencies - 15 unit + integration tests (all via UDS on any platform) - Makefile targets: `make emberd`, `make emberd-release` - Workspace fmt/check/clippy/test updated to include emberd Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Protocol reference, build instructions, architecture diagram, and image integration guide. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add emberd binary and systemd service to both ubuntu-dev and ubuntu-slim Dockerfiles. The binary is pre-built on the host with `make emberd-image` and staged at images/emberd for COPY. - images/emberd.service: systemd unit (Type=simple, Restart=always) - Dockerfile.ubuntu-dev: COPY emberd + enable service - Dockerfile.ubuntu-slim: COPY emberd + enable service - Makefile: `make emberd-image` target (native on Linux, cross-compile on macOS) - .gitignore: exclude staged binary Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Show build, pull, and list commands instead of only suggesting pull. Most custom images (ubuntu-dev, ubuntu-slim) need to be built from a Dockerfile, not pulled from a registry. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix Backlog type in listen_vsock (nix 0.29 on Linux requires Backlog::new() instead of raw integer) - Makefile emberd-image: use Docker (rust:latest) for Linux builds on macOS instead of requiring cross-compilation toolchain - Dockerfiles: clean up emberd COPY comments Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
If `ember vm create` succeeds but the subsequent start fails (e.g., ember-vz crash, missing binary), delete the created VM instead of leaving orphaned state behind. Previously, the start rollback only cleaned up network/process but left the VM metadata and disk. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two bugs in the ember-vz vsock UDS bridge: 1. VZVirtioSocketDevice.connect(toPort:) was called from a background queue, but AVF requires it on the main queue. The completion handler never fired, so host→guest connections silently failed. 2. VZVirtioSocketConnection was not retained during bridgeConnection(), so ARC could deallocate it and close the fd mid-transfer. Fixes: - Dispatch connect(toPort:) to DispatchQueue.main - Hold strong ref to VZVirtioSocketConnection via DispatchGroup - Log ember-vz stderr to <vm_dir>/ember-vz.log for debugging - Add diagnostic logging throughout the bridge Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
`ember vm create` now waits up to 90s (configurable via --wait) for SSH to become reachable before reporting success. This means `ember exec` works immediately after create — no manual polling needed. Also add --wait flag to `ember exec` for configuring the SSH connect timeout (default: 30s, can be increased for heavy images). If the wait times out, the VM is still running — just SSH is slow. A hint is printed suggesting `ember exec --wait`. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When `ember exec vm -- "echo hi | tee /tmp/out"` has one argument after `--`, pass it directly to the SSH channel without quoting. The remote shell interprets pipes and redirects correctly. Previously, shell_escape_join would single-quote arguments containing `|` or `>`, preventing the remote shell from interpreting them. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ember exec now tries vsock (emberd) first, falling back to SSH: - Connects to the VM's vsock UDS and sends JSON-lines exec request - No SSH dependency — works immediately after boot (emberd starts fast) - Falls back to SSH automatically if vsock fails - --ssh flag to force SSH path ember vm list now shows IP address and vsock status: NAME STATUS IP VSOCK CPUS MEM DISK val-smoke running 192.168.64.2 ✓ 1 16 GiB 8 GiB Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- `ember vm create --format json` returns VM metadata as JSON on stdout - All progress messages (Cloning, Growing, Injecting, Starting, Waiting) now go to stderr so stdout is clean for JSON piping - `ember exec` also reformatted by cargo fmt This makes ember scriptable: `ember vm create foo --image bar --format json | jq .` outputs clean JSON while progress is visible on stderr. 201 tests pass (186 ember + 15 emberd), clippy clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…tests Replace hardcoded guest_cid=3 with a proper CID allocator that assigns unique CIDs (starting at 3) per VM, persisted in vsock/cids.json. CIDs are freed on VM delete and reused lowest-first, following the same pattern as IP allocation in network/ip.rs. - state/vsock.rs: allocate()/release() with flock-based locking (6 tests) - cli/vm.rs: create and fork use CID allocator; delete releases CIDs - cli/vm.rs: validate_uds_path() rejects paths >= 104 bytes (macOS sun_path limit) with actionable error message (3 tests) - error.rs: Error::Vsock variant for CID allocation failures - state/store.rs: vsock_allocations_path(), vsock/ dir in init() - tests/vsock.rs: 6 integration tests (CID uniqueness, reuse after delete, inspect JSON/table output, vm list checkmark, end-to-end UDS connectivity on macOS and Linux) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jasonhernandez
force-pushed
the
feat/sec-254-vsock-cid-allocation
branch
from
276757c to
4afb65f
Compare
…OU) (#7) * network: migrate IP allocator to SQLite, fixes parallel-allocate TOCTOU The prior `network/allocations.json` + flock store had per-call (not per-transaction) locking: read_optional acquired a shared lock, read, dropped it; write acquired an exclusive lock, wrote, dropped it. Between read and write, another process could run its own read+modify+write and both would think they had the lowest free slot. Observed manifestation: after a host crash, six parallel `ember vm start` invocations each saw an empty allocations file, each picked slot 0, and configured their NICs with the same IP. The persisted state captured only the last writer; on the bridge, ARP resolved each duplicate IP to one MAC, so SSH to three different VM names landed on one VM. A subsequent thermite dispatch silently consolidated three agents onto one host. (See SEC-459.) The structural fix: relational state moves to a SQLite database at `<state>/state.db` with a `(subnet, block_index) PRIMARY KEY` plus `vm_name UNIQUE` constraint. Double-allocation is impossible at the data layer — the second INSERT for the same slot fails with SQLITE_CONSTRAINT_PRIMARYKEY, so even if the read-modify-write logic regresses, the schema catches it. Allocators run under `BEGIN IMMEDIATE`, which acquires the write lock at transaction start (not lazily on first write). This eliminates the SELECT/INSERT race window. Connections are opened per call; SQLite's WAL mode + busy_timeout handles concurrent readers and serializes writers correctly without per-process pooling. Changes: * `crates/ember-core/src/state/db.rs` — new module that opens `state.db`, runs the schema bootstrap (idempotent CREATE TABLE IF NOT EXISTS), enables WAL mode, and sets a 5s busy timeout. Schema uses STRICT tables for type enforcement. * `crates/ember-core/src/network/ip.rs` — `IpAllocations` JSON struct deleted; `allocate` and `release` rewritten against the SQLite schema. New `check_invariants()` helper used by `vm list`. * `crates/ember-core/src/state/store.rs` — `network_allocations_path` and the `network/` directory init are removed. State store no longer touches the JSON allocator file. * `src/cli/vm.rs` — `vm list` flags VMs whose state is corrupted (two VMs sharing a guest_ip; allocator anomalies from check_invariants) with `[CORRUPTED]` and prints a recovery hint. Belt-and-suspenders for the hypothetical case where state was migrated from the old store or hand-edited around the schema. * New rusqlite dep (bundled feature — single .a file, no system libsqlite3 runtime dep, consistent with ember's "no fragile bindings" philosophy). No backwards compatibility. Existing `network/allocations.json` files are stale (this fix lands on R&D pool VMs that are recreated regularly, and the state file is corrupt today anyway). The serde types and the read/write paths are deleted, not deprecated. Tests: * Schema-level: PRIMARY KEY and UNIQUE constraints reject duplicate inserts (covers the structural-impossibility property) * Behavior preserved: all existing allocate/release tests pass against the new backend * Stress: 50 OS threads each calling `allocate()` against a shared store with distinct VM names — assert all 50 returned slots are unique and the table has 50 rows. This is the regression test for the original TOCTOU bug. Passes in both debug and release builds. * `check_invariants` returns empty list on a clean store cargo build, cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings all clean. Closes SEC-459. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * network: address SEC-459 review feedback Three non-blocking polish items from the PR review: 1. Stress test now uses std::sync::Barrier so all 50 threads enter BEGIN IMMEDIATE near-simultaneously rather than serializing on scheduler luck. The reviewer ran a 200-thread barrier-synchronized variant independently to validate the property; this brings the in-tree test up to the same level of confidence. 2. db::open now sets busy_timeout BEFORE the WAL journal_mode pragma. The first-time WAL conversion takes a write lock; if two processes race that conversion, one would otherwise see SQLITE_BUSY without retry. After the first conversion, the pragma is a no-op. (This was actually a real race — the in-tree test flaked once on the pre-reorder build until the reorder took effect.) 3. Removed the stale "vsock/cids.json" reference from db.rs's module docstring. That allocator hasn't been implemented yet; future work should add its own table to this same database. The remaining review item (refactor check_invariants to return structured Anomaly { vm_names } and stop swallowing DB errors via unwrap_or_default) is more substantive and is being filed as a follow-up issue rather than bundled here. cargo test --workspace + cargo clippy --workspace --all-targets clean. parallel_allocate stress test passes 10/10 with the barrier in place. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Jason Hernandez <7144515+jasonhernandez@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
# Conflicts: # crates/ember-core/src/state/store.rs # src/cli/vm.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #5.
guest_cid=3with a proper CID allocator (state/vsock.rs) that assigns unique CIDs per VM, persisted invsock/cids.json, freed on delete, reused lowest-firstsun_pathlimit (104 bytes) with actionable error before allocating resourcesvm create --format jsonwith progress to stderrFiles changed (key additions over #5)
crates/ember-core/src/state/vsock.rsvsock/cids.jsoncrates/ember-core/src/state/store.rsvsock_allocations_path()methodemberd/src/cli/exec.rssrc/cli/vm.rsember-vz/Sources/EmberVZ/Start.swifttests/vsock.rsimages/Rebased onto main after the ember-core/ember-linux/ember-macos workspace restructuring.
Test plan
cargo test --workspace) — 9 CID allocator + 3 UDS validationcargo buildclean,cargo clippy --workspaceclean,cargo fmtcleancargo test --test vsock -- --ignoredon macOS with ember-vz builtcargo test --test vsock -- --ignoredon Linux with Firecracker + KVM🤖 Generated with Claude Code