Docker VMM on Windows 4.93.0: bbolt corruption, stale vsock socket, and container filesystem errors after restart

Bug report

Summary

Docker Desktop 4.93.0 on Windows 11 Pro with Docker VMM/libkrun became unable to start its engine. The incident exposed two distinct startup blockers: a containerd bbolt multiple-reference panic, and failure to remove an AF_UNIX vsock-1999 listener. The engine now responds using a separately reconstructed working VHDX, but container recovery is NOT complete: an overlay mount reports structure needs cleaning, PostgreSQL volume reads return Bad message, and engine summary counts disagree with per-container state.

This is an incident report, not a recommendation to reset data or blindly apply custom metadata reconstruction. The root cause and any causal connection between the socket failure and disk corruption are not established.

Related: #554 and #531 for stale AF_UNIX startup failures; #713 describes a separate VMM incident with bbolt corruption. We have not established that the shutdown/flush mechanism suggested in that other report occurred here.

Environment and storage layout

Windows 11 Pro, 10.0.26300 / build 26300.
Docker Desktop 4.93.0; Engine 29.8.1; containerd v2.3.5; runc 1.5.1.
Guest kernel 7.0.14-linuxkit, linux/amd64.
Docker VMM/libkrun: UseLibkrun=true, WslEngineEnabled=false; containerd overlayfs image store.
VM disk is on D:. The existing %LOCALAPPDATA%/Docker directory is a junction to a Docker directory on D:. This non-default layout is disclosed as a possible contributing factor, not a proven cause.
Dynamic VHDX host-file size 65,281,196,032 bytes (approximately 60.8 GiB); virtual capacity 1 TiB; ext4 partition.
No switch to WSL2/Hyper-V during recovery.
Data preservation was mandatory; no new recovery storage on C:.

Original errors

starting engine: resetting socket forwarder: preparing vsock listener:
listening on AF_UNIX vsock 1999:
remove C:\Users\<USER>\AppData\Local\Docker\vm-data\00000002.000007cf:
The file cannot be accessed by the system.

service containerd failed: panic detected in containerd:
panic: freepages: failed to get all reachable pages
(page 1817: multiple references (stack: [1817]))

goroutine 74 [running]:
go.etcd.io/bbolt.(*DB).freepages.func2()
    /go/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:1248 +0x8d
created by go.etcd.io/bbolt.(*DB).freepages in goroutine 67
    /go/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:1246 +0x1ef

Repeated occurrences had different goroutine numbers but the same page-1817 error.

Everything tried, including unsuccessful attempts

Normal Docker launches/restarts did not resolve the problem. The user restarted Windows and tried an Administrator PowerShell command followed by another launch; Docker remained stuck at Starting the Docker Engine. The exact earlier administrator command is not in the retained record and is not invented here.
Read-only backend/settings/path checks identified VMM/libkrun, the D: VHDX, and the existing LocalAppData-to-D: junction. No backend switch, WSL unregister, or data reset was attempted.
Docker was stopped with its normal Desktop CLI. One earlier stop timed out; a later normal stop completed. A complete VHDX backup was made while the original was held read-only and exclusively locked. Full source and backup SHA-256 values matched.
The user's elevated vssadmin list shadows /for=D: check returned No items found that satisfy the query. There was no D: VSS recovery point.
Read-only extraction from the verified backup reproduced the exact page-1817 panic. Both main metadata committed transaction states (9902/9903) and both overlay states (1666/1667) failed integrity validation. Rollback/meta-page trials were limited to disposable extracted database copies and failed. Historical root-page candidates yielded no intact database. None of those unsuccessful trials was installed in the original disk or backup.
Corruption was identified in three databases:
- /desktop-containerd/daemon/io.containerd.metadata.v1.bolt/meta.db
- /desktop-containerd/daemon/io.containerd.snapshotter.v1.overlayfs/metadata.db
- /docker/buildkit/containerd-overlayfs/metadata_v2.db
  The separate volume metadata, network metadata, and checked BuildKit cache/history databases passed structural checks; that did NOT prove referenced data files intact.
A bulk-read boundary bug in an inspection library was found and corrected by splitting VHDX reads at payload-block boundaries. Authoritative extractions were repeated; the main database still reproduced the panic. Earlier incorrect diagnostic extracts were not used for repair.
With explicit approval, a separate full working VHDX copy on D: was verified identical to the backup before edits. Populated main, overlay and BuildKit metadata were reconstructed on this working copy ONLY, from surviving records and existing snapshots. Valid values were preserved, but some missing/aliased snapshot/cache records required inferred fields. Exact historical recovery is not claimed. No empty-database reset was used.
Snapshot coverage was checked against all 501 physical directories, and parent references/cycles were checked. All 28 valid image records and five containerd runtime-container records were retained; these are separate from 24 Docker container directories. Non-expiring preservation leases were added to inhibit containerd GC during recovery.
The reconstructed databases passed official bbolt integrity checks and independent logical comparison against reconstruction manifests. Same-size replacements used existing initialized payload space; sparse tails remained zero. Required VHDX headers received valid checksums. Full comparison of all 65,281,196,032 host-file bytes, masking only the intended database payloads, two temporary restart-policy holds and required VHDX header fields, verified every other byte unchanged. No inode, extent, allocation, journal, BAT or region-table repair was performed. The same 32 volume, 24 Docker container and 501 snapshot directories remained.
All 665 physical content blobs were hashed: 662 matched; three pre-existing mismatches were in build history, not the main image-content namespace. Those files were preserved, not deleted or relabeled. Build-history integrity remains limited.
With explicit permission, Docker's existing small settings file was backed up to D: and ONLY SailorDataFolder changed to the repaired working-copy folder on D:. Other settings were verified unchanged. Recovery tools, temp/cache files, extracts and reports were placed on D:; earlier scratch files were moved off C:.
Original and backup VHDX files were held read-only and exclusively locked during startup validation. A normal docker desktop start --detach --timeout 30 failed at the AF_UNIX vsock-1999 removal stage; the backend exited. A subsequent normal stop said Desktop was not running.
The physical stale listener on D: was zero bytes with AF_UNIX reparse tag 0x80000023, NOT a VM disk. Moving only this socket recoverably into a recovery folder did NOT fix startup. A second normal launch failed again and left a zero-byte .stale socket.
With Docker processes absent, the exact runtime parent was verified to contain only that zero-byte socket, no VHDX or subdirectories. Its D: parent was renamed to a preserved timestamped sibling, and a fresh directory created at the original runtime path. Nothing was deleted. This was the socket runtime directory, NOT the separately located VHDX data folder.
A normal Desktop CLI start then succeeded. At 2026-10-04 08:29:41 UTC the API returned Engine 29.8.1, 24 containers and 28 images, and Desktop showed Engine running. The repaired working VHDX was in use while the original and backup were simultaneously held exclusively, confirming which disk booted.
Read-only disk protection was explicitly released after validation; original/backup sizes and last-write times were unchanged. Both temporary unless-stopped -> no restart-policy holds were restored to unless-stopped and verified; neither restoration started its stopped container.
All 24 container records were then audited read-only: state/errors, image registration, mount registration, Windows bind-source existence and bounded logs. No mass start, migration rerun, pull, prune, factory reset, metadata deletion, backend switch or filesystem repair was performed.

What finally started Docker Desktop's engine

Both stages mattered in this successful recovery sequence: use the populated, repaired WORKING VHDX on D:, then preserve/rename the stale socket's runtime PARENT DIRECTORY and create a fresh one. Rebooting/restarting, the earlier administrator attempt, and moving the individual socket alone did not resolve the full incident.

This is one successful startup, not proof the stale-socket workaround is durable across future stop/start cycles. Engine recovery is not full application recovery.

Container problems remaining after startup

At approximately 2026-10-04 08:39 UTC:

All 24 individual container inspections report State.Running=false and Status=exited. docker ps -a and the Desktop UI agree.
docker info nevertheless reports Containers=24, ContainersRunning=4, ContainersStopped=20 across repeated checks.
Recorded exit codes: 16 containers with 0; four with 255; one with 1; two with 143; one with 137. Historical exits are not treated as proof every container is broken.
A PostgreSQL proxy's recorded overlay mount error ends with:

failed to mount /var/lib/docker/rootfs/overlayfs/<CONTAINER_ID>
... fstype: overlay ...
upperdir=.../snapshots/562/fs
lowerdir=.../snapshots/561/fs:.../snapshots/1/fs
err: structure needs cleaning

A PostgreSQL database container has a fresh start attempt followed by exit 1. Its logs contain many filesystem reads such as:

2026-10-04T08:38:23Z find: /var/lib/postgresql/data/base/<DB_OID>/<FILE>: Bad message

These are inside the existing data volume, not a missing Windows bind source. Filesystem/application integrity is not established.

All referenced named volumes are registered and all checked Windows bind-source paths exist. Existence does not prove readability or successful guest mounting.
One previously successful stopped migration container references an image ID docker image inspect cannot find; the other 23 containers' image IDs are registered. Missing-reference timing/cause is unknown. The migration was not rerun.
Four application containers have exit 255 around engine initialization but empty State.Error strings; no common cause for those four is established.
Most restart policies are no. One-shot migration/permission jobs intentionally remain stopped; we did not start everything indiscriminately.

The reconstruction changed no snapshot or volume payload bytes before startup, but it is a custom recovery and must be disclosed when assessing subsequent errors. No claim is made that all errors share a cause, the disk is fully repaired, or VMM is conclusively responsible for the corruption.

Expected behavior and investigation requested

Safely recover an inaccessible stale AF_UNIX runtime listener instead of aborting the whole backend.
Investigate VMM persistence/shutdown behavior in relation to multi-database corruption and filesystem-level errors.
Provide a supported DATA-PRESERVING recovery procedure; reset or deleting volumes/metadata is not acceptable here.
Investigate the running-container count inconsistency.

Only redacted evidence is posted. Full disk images, environment variables, application specs/logs and private paths are not attached publicly. No diagnostic bundle has been uploaded.

Platform

Windows

Version information

Windows 11 Pro 10.0.26300 / build 26300
Docker Desktop 4.93.0
Docker Engine 29.8.1, API 1.56
containerd v2.3.5
runc 1.5.1
Guest kernel 7.0.14-linuxkit, linux/amd64
Docker VMM/libkrun: UseLibkrun=true, WslEngineEnabled=false
containerd overlayfs image store
VHDX on D:; existing LocalAppData Docker junction to D:
24 container records, 28 image records after recovery

Diagnostics ID

None — no diagnostics bundle uploaded; redacted local evidence only.

Update: data-preserving recovery completed (4.94.0 / Engine 29.8.2), root causes and suggestions

Follow-up to this report. The working VHDX described above later failed again, and this time we found the underlying causes and completed a recovery with no data loss (23 containers, 30 images, 37 volumes, 11 networks all present and readable afterwards). Details below are redacted in the same way as the original report.

What failed after the original recovery

Environment now: Windows 11 Pro 10.0.26300, Docker Desktop 4.94.0, Engine 29.8.2, Docker VMM (UseLibkrun=true, WslEngineEnabled=false), containerd image store, VHDX on a second drive.

service containerd failed: panic detected in containerd: panic: freepages: failed to get all reachable pages (page 757: multiple references (stack: [757])) (same class of error as in this report).
starting engine: resetting socket forwarder: preparing vsock listener: listening on AF_UNIX vsock 1999: rename …\vm-data\00000002.000007cf …\vm-data\00000002.000007cf.stale: The file cannot be accessed by the system. (4.94.0 now tries to rename the stale socket to .stale; the rename fails when a .stale from the previous failed start already exists.)
After 1 and 2 were fixed: service command exited with code 2: the Docker network database was corrupt and has been reset: panic detected in command: panic: invalid freelist page: 16, page type is branch.
After 1 to 3: failed to retrieve container list: rw layer snapshot not found for container <id> for most containers.

Root causes we established

containerd meta.db. Both bbolt meta pages pointed at the same root and the freelist is not persisted (freelist = MaxUint64), so Open() rebuilds it by walking the tree. One bucket was reachable from two parents (v1/<ns>/<bucket> and v1/<bucket>, same root page), which makes Open() panic in freepages; that also breaks bbolt check/compact and containerd. The tree the meta pointed to was additionally an older/reduced generation whose moby namespace was empty (no images, containers, snapshots). Other, complete generations of the tree were still present in the file. (Note: this machine had previously gone through the custom reconstruction described above, so we cannot say how much of this state is Docker's and how much is ours.)
Network DB local-kv.db. The newest meta's freelist pointed at a page that is a branch page of the previous transaction (torn commit), and 29 of 72 keys were unreachable by key lookup in the newest tree although all 72 were intact in the previous transaction. Docker "resets" the DB in this case; on a test copy the file was gone from its directory afterwards, which would silently drop all user-defined networks.
Stale AF_UNIX socket handling (see also Startup crash: cannot remove stale unix socket files after unclean shutdown #554). Windows returns error 1920 ("The file cannot be accessed by the system") for any normal open of an AF_UNIX socket file, so this looks like a lock but is not. Renaming the parent directory aside (not the socket) cleared the failure. A rename of the socket onto an existing .stale fails.
containerd GC on a state that is valid but unprotected. With the newest root that passed bbolt check (leases already cleaned up), containerd GC removed 331 of 673 snapshot records within minutes; 19 of 23 containers then reported rw layer snapshot not found. With the previous consistent root (leases still present) only 22 records were removed, all transient -view snapshots, none belonging to a container or a surviving parent. "Newest valid root" is therefore not a safe criterion on its own.

How it was fixed (all steps first proven on copies of a hash-verified backup)

Stop Docker; full copy of the VHDX; SHA-256 verified on source and copy.
Read the three bbolt files out of the VHDX read-only (VHDX + ext4 parser, no VM): containerd meta.db, overlayfs metadata.db, local-kv.db.
Find candidate roots by scanning every page for a leaf containing the known top-level bucket (v1, libnetwork).
For each candidate build a working copy whose newest meta page points to it (snippet below) and run bbolt check. Candidates with shared pages panic; consistent ones print OK.
Choose among consistent candidates by content, not only recency: the namespace must contain containers/images/snapshots; every container directory ID under /docker/containers must have a snapshot key; every snapshot record's backend name must exist in the snapshotter's own metadata.db; leases intact (point 4 above). Also verify keys with real branch-routing lookups, because bbolt check does not catch every routing problem.
Network DB: rebuilt a fresh bbolt file from the previous consistent transaction (72/72 keys), bbolt check OK.
Boot the engine on a copy with the repaired files; check docker info, docker system df, per-container docker ps -a -s, networks, volumes; re-count snapshot records after the engine has run for a few minutes.
Patch the live disk with a data-only write (overwrite the existing data blocks of those files; no allocation, inode, journal or VHDX header change), re-read both files, and compare the whole disk with the pre-patch backup outside the replaced ranges.

For meta.db the final change was one 4 KiB page (the meta page). Every old data page was left untouched.

Meta-page writer (bbolt v1.x layout), run on a copy only

import struct, sys
src, root, dst = sys.argv[1], int(sys.argv[2]), sys.argv[3]
raw = bytearray(open(src, 'rb').read())
ps = struct.unpack_from('<I', raw, 16 + 8)[0]

def fnv64a(b):                         # bbolt meta checksum
    h = 14695981039346656037
    for x in b:
        h = ((h ^ x) * 1099511628211) & 0xFFFFFFFFFFFFFFFF
    return h

metas = []
for i in (0, 1):
    off = i * ps
    magic, ver, psz, flags, r, rseq, fl, pgid, txid, chk = struct.unpack_from('<IIIIQQQQQQ', raw, off + 16)
    assert chk == fnv64a(raw[off + 16: off + 16 + 56])
    metas.append((i, txid, r, rseq, fl, pgid))

cur = max(metas, key=lambda m: m[1])
newtx = cur[1] + 1
slot = newtx % 2
body = struct.pack('<IIIIQQQQQ', 0xED0CDAED, 2, ps, 0, root, 0, cur[4], cur[5], newtx)
page = bytearray(ps)
struct.pack_into('<QHHI', page, 0, slot, 0x04, 0, 0)
page[16:16 + len(body)] = body
struct.pack_into('<Q', page, 16 + len(body), fnv64a(body))
raw[slot * ps:(slot + 1) * ps] = page
open(dst, 'wb').write(raw)

Suggestions for Docker VMM

Do not discard the network DB on corruption. Keep a .corrupt copy and try the previous bbolt transaction first.
On bbolt Open() failure for containerd/network DBs, fall back to the previous valid meta/generation instead of failing the engine.
Persist the bbolt freelist, or at least avoid a hard panic when reachable-page reconstruction fails.
Gate containerd GC at engine start until metadata is checked against on-disk containers, images and leases.
Make socket start-up idempotent: remove or uniquely rename stale AF_UNIX sockets and .stale leftovers with a reparse-aware API, or rename the parent directory, and show a clear error instead of "The file cannot be accessed by the system".
Offer a repair or "restore previous metadata" option before "Reset to factory defaults". Here Quit and Reset were the only actions while all data was recoverable.
Snapshot the three bbolt files on clean shutdown so a known-good previous generation always exists.

Caveats

We could not establish what produced the first bad meta page (unclean shutdown on a machine with an earlier custom reconstruction).
The procedure is expert-level and was only safe because each step was done on verified copies first. The tooling was ad hoc, not a supported repair tool.
We have not tested whether the stale-socket workaround survives repeated stop/start cycles beyond this recovery.

Navigation Menu

Repository navigation

Docker VMM on Windows 4.93.0: bbolt corruption, stale vsock socket, and container filesystem errors after restart #726

Bug report

Summary

Environment and storage layout

Original errors

Everything tried, including unsuccessful attempts

What finally started Docker Desktop's engine

Container problems remaining after startup

Expected behavior and investigation requested

Platform

Version information

Diagnostics ID

Update: data-preserving recovery completed (4.94.0 / Engine 29.8.2), root causes and suggestions

What failed after the original recovery

Root causes we established

How it was fixed (all steps first proven on copies of a hash-verified backup)

Suggestions for Docker VMM

Caveats

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Docker VMM on Windows 4.93.0: bbolt corruption, stale vsock socket, and container filesystem errors after restart #726

Description

Bug report

Summary

Environment and storage layout

Original errors

Everything tried, including unsuccessful attempts

What finally started Docker Desktop's engine

Container problems remaining after startup

Expected behavior and investigation requested

Platform

Version information

Diagnostics ID

Activity

nicefellow1234 commented on Oct 8, 2026

Update: data-preserving recovery completed (4.94.0 / Engine 29.8.2), root causes and suggestions

What failed after the original recovery

Root causes we established

How it was fixed (all steps first proven on copies of a hash-verified backup)

Suggestions for Docker VMM

Caveats

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions