Make jemalloc the default Rust allocator on Linux

[kriszyp]

⊙ Problem

On Linux fan-out nodes, symphony keeps its peak memory after connections drop, and the peak creeps up day by day. The cause is glibc's malloc: memory freed across its per-thread arenas is not returned to the OS.

🤔 Is a default allocator change the right size for this?

Yes. Every Linux deployment has the problem, and an opt-in build would not reach them. malloc_trim was ruled out because each call stalls allocation for up to 689 ms.

💡 Solution

On Linux, the addon's Rust allocations now go to jemalloc. Node keeps its own malloc. Measured on current main, 100k ↔ 10k connections under churn, 5 cycles, 3 repeats per allocator:

• glibc keeps ~100% of peak at every trough, and at zero connections. Its peak grows ~100 MiB over 5 cycles.
• jemalloc keeps 28% at every trough and stays flat.
• Cost: +1.9% bytes per parked connection. CPU is the same for 256 B messages; a 4 KiB result is still being re-measured (below).

--no-default-features builds on the system allocator, for bisecting.

⚖️ Alternatives

• LD_PRELOAD jemalloc: same memory result, but every deployment must ship the library, Node uses it too, and apps embedding the addon can't use it.
• MALLOC_ARENA_MAX=2: still kept 100% of peak in #60.
• Periodic malloc_trim: brings the trough to 15%, but each call holds arena locks 55 ms p50, 689 ms max.

🤔 64 KiB jemalloc pages on arm64?

So one arm64 binary starts on 4, 16 and 64 KiB kernels; a 4 KiB build aborts on 64 KiB ones. Bigger slabs may keep more at troughs; being measured on x86 with the same geometry.

🤔 Purge threads for idle processes?

Kept jemalloc's defaults. With live traffic, background_thread changed nothing at the trough (742 vs 750 MiB). It only helps a process with zero connections and no traffic, where defaults keep ~25% of peak (717 → 196 MiB). If that matters, it can be switched on at runtime later.

🔧 Changes

• Allocator — Cargo.toml makes the existing jemalloc feature a default whose dependency builds only on Linux; src/lib.rs installs it as the global allocator on Linux only, so macOS is unchanged. Symbols stay prefixed, so Node's malloc is untouched. rust_allocations_come_from_jemalloc asks jemalloc whether a live Box is its own. That proves the allocator is selected, not just linked.
• arm64 page size — jemalloc aborts on a kernel with larger pages than it was built for, and a cross build picks up the builder's 4 KiB. .cargo/config.toml builds both aarch64 targets for 64 KiB, which also runs on 4 and 16 KiB kernels. Look hardest here: its memory cost was measured on x86 only (below).
• CI — a new job runs npm test on the arm64-gnu, x64-musl and arm64-musl artifacts through docker run, and fails if an artifact lacks jemalloc. Before this, only x64 glibc was ever loaded in CI. The release build checks for jemalloc too.
• Rust↔Node boundary — no Rust allocation is freed by Node or by C free(), and no JS allocation by Rust. JS Buffer cert arguments are borrowed: napi-rs only unrefs them on drop, and pem_bytes copies them. Values returned to JS are copied by napi_create_*. The class wrapper and threadsafe-function payloads are freed by Rust code in napi-rs's finalizers. Recorded with the invariant in CLAUDE.md.
• Benchmark (__test__/bench-copy-memory-oop.ts):
  • a soak phase: N churned peak/drop cycles, measured against a fresh process at the trough population, then a traffic-free trough and a zero-connection hold;
  • a cpu phase: symphony CPU seconds per GB and per 1M delivered messages, with delivery latency from send timestamps the client reads back;
  • --symphony-cpus pinning, --fixed-order and --arm-env.
  • Its glibc arms now run a --no-default-features build; the repo's own build is the rust-jemalloc arm.
• Docs — README build note and opt-out.

✅ Verification

Bench host: i7-12700H at 2 GHz, Linux 7.0, glibc 2.43, Node 26.2. Arms interleaved; symphony in its own cgroup, swap off. The measured tree is merged main's.

Gate, set before collecting	Result
Parked 100k: ≤ +10% bytes/connection	+1.9% total, +1.8% marginal ✅
Soak: jemalloc trough ≤ 35% of peak, every cycle	27.8–28.1% ✅ (glibc 99.6–100.1%)
Soak: jemalloc trough growth c1→c5 ≤ 5% of peak	+13 MiB, 0.5% ✅ (glibc +86 MiB)
CPU per GB and per 1M messages ≤ +5%	256 B: −0.5% ✅; 4 KiB: +24% on 3 noisy runs ⏳ re-measuring
p99 delivery latency ≤ +10% or within spread	256 B: within spread ✅; 4 KiB ⏳

• cargo test: 159 pass on the default build, including the allocator check and the 4 KiB future-size guard; 158 with --no-default-features. cargo clippy --all-targets -D warnings is clean on both.
• npm test: 158/158 on the default (jemalloc) build.
• The --no-default-features build has no jemalloc in it.
• Not verified here: musl and arm64 builds and tests (CI runs them), and a 64 KiB-page arm64 kernel.

Parked, 100k connections (abc phase), 3 runs per arm

arm	runs	parked total	parked marginal	parked over idle	active total	active Δ over parked	active RSS slope KiB/s	delivered % of target	symphony CPU % (active)
glibc	3	27,758 (27,754–27,760)	27,131 (27,126–27,133)	27,179 (27,175–27,182)	27,759 (27,756–27,761)	1 (1–2)	-17.0 (-17.2–-16.8)	99.2 (99.2–99.2)	381 (323–420)
rust-jemalloc	3	28,287 (28,285–28,289)	27,621 (27,619–27,624)	27,697 (27,695–27,699)	28,288 (28,286–28,290)	1 (1–1)	-17.2 (-17.2–-17.0)	99.2 (99.2–99.2)	438 (408–472)

Soak, 5 cycles, 100k ↔ 10k under churn

Soak, 5 cycles of ramp to 100000 under churn → hold 120 s → drop to 10% → hold 390 s. Footprint = VmRSS + VmSwap, MiB:

arm	runs	peak/conn c1 (B)	peak c1	peak c2	peak c3	peak c4	peak c5	trough c1	trough c2	trough c3	trough c4	trough c5	trough c1 % of peak	trough c2 % of peak	trough c3 % of peak	trough c4 % of peak	trough c5 % of peak	trough c1 +10 s % of peak	peak growth c1→last	trough growth c1→last
glibc	3	30,961 (30,937–30,994)	2,953 (2,950–2,956)	3,018 (3,005–3,025)	3,034 (3,013–3,045)	3,051 (3,039–3,066)	3,052 (3,038–3,067)	2,954 (2,950–2,957)	3,012 (3,001–3,018)	3,030 (3,006–3,042)	3,041 (3,031–3,053)	3,040 (3,028–3,061)	100.1 (100.0–100.1)	99.8 (99.7–99.9)	99.9 (99.8–100.0)	99.7 (99.6–99.8)	99.6 (99.3–99.8)	100.1 (100.1–100.1)	100 (88–111)	86 (77–103)
rust-jemalloc	2	28,558 (28,557–28,559)	2,723 (2,723–2,724)	2,739 (2,738–2,741)	2,738 (2,738–2,739)	2,744 (2,744–2,744)	2,747 (2,746–2,747)	757 (755–759)	766 (761–770)	766 (765–767)	770 (769–770)	770 (767–774)	27.8 (27.7–27.9)	28.0 (27.8–28.1)	28.0 (27.9–28.0)	28.1 (28.0–28.1)	28.0 (27.9–28.2)	35.7 (34.8–36.7)	23 (23–24)	13 (8–18)

Soak floors, MiB: fresh-low = a process that never saw the peak, at the trough population, before cycle 1; quiet = trough population with traffic and churn stopped; zero = every connection closed (start = +settle, end = +trough hold):

arm	runs	idle	fresh-low	trough c1 over fresh-low	trough c2 over fresh-low	trough c3 over fresh-low	trough c4 over fresh-low	trough c5 over fresh-low	quiet start	quiet end	zero start	zero end
glibc	3	55 (55–56)	354 (351–357)	2,600 (2,599–2,602)	2,658 (2,650–2,664)	2,676 (2,655–2,689)	2,688 (2,680–2,696)	2,686 (2,677–2,703)	3,040 (3,028–3,061)	3,040 (3,028–3,060)	2,998 (2,983–3,014)	2,998 (2,984–3,015)
rust-jemalloc	2	56 (56–56)	333 (333–333)	424 (423–426)	433 (429–438)	433 (432–435)	437 (437–437)	438 (434–441)	770 (767–774)	766 (763–770)	723 (719–728)	723 (719–728)

Fan-out CPU and latency, 256 B messages, symphony pinned to CPUs 0-7

Fan-out CPU, 20000 connections, 256 B at 10/s/conn, 3 × 60 s, symphony pinned to CPUs 0-7. GB = payload bytes through symphony, both directions; latency = upstream write → client read:

arm	runs	msgs/s	delivered %	symphony cores	CPU s per GB	CPU s per 1M msgs	minor faults/s	p50 ms	p99 ms	p99.9 ms
glibc	3	200,005 (199,982–200,031)	100.0 (100.0–100.0)	3.466 (3.313–3.586)	67.70 (64.69–70.05)	17.332 (16.562–17.932)	–	1.40 (0.86–1.88)	13.46 (11.59–15.02)	41.92 (23.17–60.10)
rust-jemalloc	3	200,007 (199,993–200,015)	100.0 (100.0–100.0)	3.448 (3.319–3.536)	67.35 (64.82–69.05)	17.241 (16.594–17.677)	–	1.22 (1.12–1.33)	13.88 (11.59–15.02)	30.41 (19.48–38.97)

Fan-out CPU and latency, 4096 B messages, symphony pinned to CPUs 0-7

Fan-out CPU, 20000 connections, 4096 B at 2/s/conn, 3 × 60 s, symphony pinned to CPUs 0-7. GB = payload bytes through symphony, both directions; latency = upstream write → client read:

arm	runs	msgs/s	delivered %	symphony cores	CPU s per GB	CPU s per 1M msgs	minor faults/s	p50 ms	p99 ms	p99.9 ms
glibc	3	40,000 (39,989–40,010)	100.0 (100.0–100.0)	1.258 (1.119–1.451)	7.68 (6.83–8.85)	31.448 (27.989–36.263)	–	0.51 (0.36–0.72)	6.06 (4.87–7.51)	12.12 (9.74–15.02)
rust-jemalloc	3	40,001 (39,998–40,006)	100.0 (100.0–100.0)	1.560 (1.305–1.731)	9.52 (7.96–10.56)	39.005 (32.618–43.272)	–	0.93 (0.56–1.22)	7.89 (5.79–8.93)	15.99 (10.62–19.48)

Signed: Claude Opus 5.5

🤖 Generated with Claude Code

https://claude.ai/code/session_016hppFxKQ8eH3zUG7dRrjyw

Related PRs: #60 overlaps

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex; adjudicated=domain; blocked=claude(fallback)(out-of-budget); declined=cursor-grok,cursor-kimi,cursor-muse,muse; rounds=4; full=1 @ b0ece2f}

_{Review-Attention: study ~15m (sensitive: CI.yml; decisions: jemalloc-default-on-linux, arm64-single-64k-page-build, idle-reclaim-off, variant-legs-run-as-root, do-less-alternative) @ b0ece2f}

[gemini-code-assist]

Code Review

This pull request configures jemalloc as the default global allocator for Linux builds to optimize memory reclamation after connections close, preventing glibc's resident peak footprint issue. It updates the build configuration, documentation, and dependencies to support this change, including cross-compilation fixes for arm64 page sizes. Additionally, the out-of-process memory benchmark tool (bench-copy-memory-oop.ts) is significantly enhanced with new soak and cpu benchmarking phases, latency tracking, minor page fault monitoring, and support for testing with the system allocator. A unit test is also added to verify that Rust allocations are correctly routed to jemalloc. There are no review comments, so no further feedback is provided.