Add an out-of-process memory benchmark for active, stalled and churning connections by kriszyp · Pull Request #60 · HarperFast/symphony

kriszyp · 2026-10-13T22:56:39Z

⊙ Problem

On fan-out nodes, symphony 1.3.x holds 3–4× the ~19–25 KB per connection that an earlier parked-connection test reported, and its RSS stays at the peak after the connection count falls. That test's harness was never committed, so nothing could say which cause dominates.

🤔 Is a committed benchmark with default-off hooks the right size for this?

As specified, yes. A production node cannot try an allocator swap or a rustls queue cap, and the in-process benchmarks measure their own client. The hooks do nothing unless an env var or cargo feature is set; removing them later is one small revert.

💡 Solution

npm run benchmark:memory-oop samples only symphony-server, in its own cgroup, while separate processes drive parked, active, stalled-reader and churn-then-drop load. At 100k connections (repeats within 0.1%), before #61:

The gap is parked state, not traffic. An idle 1.3.x connection costs 67 KB: a 42.8 KB task future plus resident copy buffers.
Traffic adds ~0. A stalled reader adds 74 KB RSS plus 90 KB kernel send queue, then stops growing.
RSS stays at the peak because glibc keeps freed memory. malloc_trim or jemalloc bring it to 15–16% of peak after a 90% drop.

#61's body has post-#61 numbers: parked 66.8 → 27.2 KB per connection at 30k.

🤔 Which follow-up first?

With the future-size fix (−34.6 KB per connection, merged as #61 Cut parked per-connection memory 59% by shrinking each connection's task) in, 1024/4096 copy buffers on the MQTT listener bring a connection to 21.5 KB (−68%). Then jemalloc as the Rust allocator: it frees memory after drops, for 6–7% more RSS.

⚖️ Alternatives

Extend benchmark.ts — it hosts the proxy in its load generator; its RSS is mostly the client's.
Node-only stalled readers — Node cannot set SO_RCVBUF, so the client's own kernel absorbs the backlog; a python3 client sets it and a production-like MSS.
Only process-level knobs (LD_PRELOAD, MALLOC_ARENA_MAX, glibc tunables) — none returns free pages from the middle of an arena, and none caps rustls' queue.
jemalloc by LD_PRELOAD only — it cannot show whether the addon alone gets the benefit; rust-jemalloc shows it does.

🤔 Should the two hooks ship in the published addon?

SYMPHONY_TLS_BUFFER_LIMIT and SYMPHONY_MALLOC_TRIM_INTERVAL_MS are read in every build, so a field node can try a lever without a rebuild. A malformed value fails proxy construction. The alternative is a cargo feature like jemalloc: no unsupported surface, but experiments need a special build.

🔧 Changes

__test__/bench-copy-memory-oop.ts (new, run with npm run benchmark:memory-oop): an orchestrator plus three child roles, all in one file.
- Isolation: when systemd-run --user works (else a plain spawn, with a warning), symphony-server runs from dist/server.js in its own systemd-run --user --scope with MemorySwapMax=0. Before loading anything, the harness checks the pid against /health, the addon and preload library against /proc/<pid>/maps, and the cgroup and memory.swap.max.
- Load generators: Node TLS clients use one loopback source address per connection. UDS upstreams strip PROXY v1, echo, and push the fan-out. A python3 stalled reader sets SO_RCVBUF/TCP_MAXSEG.
- Sampling: /proc/<pid>/status and smaps_rollup, the scope's memory.stat, ss skmem on both socket ends, and /proc/net/sockstat. It also reports delivered bytes and symphony's CPU time per arm, so a reader can check the arms did equal work.
- Run validity: a byte-count check fails a parked/active/slow run unless byte totals match in both directions and no connection was lost; a churn run fails on any lost connection. Guards stop at low host or cgroup memory and near tcp_mem pressure.
- Arms: glibc, arena2, trim, jemalloc-preload, rust-jemalloc, buflimit, buf-small, and any --addon-arm NAME=PATH build. --summarize pools separate invocations and refuses to mix builds under one arm name.
src/bench_hooks.rs (new): reads SYMPHONY_TLS_BUFFER_LIMIT (≥ 1024; 0 would spin tokio-rustls) and SYMPHONY_MALLOC_TRIM_INTERVAL_MS (≥ 100, glibc only). A malformed value is a construction error, and an active hook says so on stderr. trim_periodically runs malloc_trim(0) on a blocking thread and logs each call's duration.
src/proxy_conn.rs: tls_accept applies set_buffer_limit through accept_with only when the limit is set. Unset, it is acceptor.accept, the code this replaced. That is one branch per handshake and nothing per byte.
src/proxy.rs: reads both hooks at construction. The trim loop starts in start() and ends on the shutdown broadcast, like the eviction tasks.
src/lib.rs, Cargo.toml: an optional jemalloc feature installs tikv-jemallocator (prefixed symbols, disable_initial_exec_tls for dlopen) as the global allocator. It is off by default, CI never enables it, and npm run build:bench-jemalloc builds it.
package.json: adds the two scripts and puts bench-hooks.spec.js in npm test. README.md: points the sizing caveat at the new benchmark. .gitignore: ignores bench-results/.

✅ Verification

cargo test: 157 pass. the_tls_buffer_limit_bounds_what_a_stalled_reader_can_queue fails if tls_accept drops the limit: with a 4 KiB cap a stalled client must pin under 16 KB, and over 48 KB without one. Clippy is clean on the default build and with --features jemalloc.
npm test: 156 pass. bench-hooks.spec.ts covers malformed values rejected, a 512 KiB patterned echo intact under a 1 KiB rustls cap, and the trim loop on two proxies in turn.
Benchmark: 100k connections, 8 arms, 3 repeats under bench-run (CPU capped at 2 GHz); no lost or failed connections. Also v1.2.0 vs current at 30k (2 repeats), and the combined levers at 100k (2 repeats).

Which hypothesis accounts for the gap

Per connection, 1.3.x with host-manager's config (8 KiB copy buffers, IP affinity, PROXY v1), 100k connections, glibc:

Hypothesis	Accounts for	Evidence	Confidence
Not on the list: the per-connection task future	42.8 KB of 67 KB; 17.9 KB of that is new in 1.3.0	`-Zprint-type-sizes`. Shrinking it by 34,616 B (`futfix`) lowered parked RSS by 34,586 B. v1.2.0's 17,936 B smaller future lowered it by 17,990 B	high
Active-traffic state: copy buffers	16 KB, resident even when parked	`buf-small` (1024/4096) lowered parked RSS by 11,253 B; the buffers shrank by 11,264 B	high
Active-traffic state: rustls queues, deframer growth	~0 at 1 msg/s of 256 B	active Δ +1 B/conn, flat slope, 99% delivered	high for small messages
Slow-reader queues	+74 KB RSS and +90 KB kernel per stalled connection, then flat	slow phase; `buflimit` 16 KiB → +18.9 KB	high for the bound; how many stall in production is unknown
Allocator retention after a drop	all of it: glibc keeps 100% of peak RSS	`trim` → 15% of peak, but each `malloc_trim(0)` holds arena locks for 55 ms (p50) to 689 ms (max); jemalloc → 16% with no pauses	high
Kernel socket buffers	~0 for a healthy connection	`ss` skmem and cgroup `sock` ≈ 1 B/conn at peak	high on loopback; real RTT adds in-flight bytes

The other ~8.6 KB is rustls session state, its 4 KiB deframer buffer, tokio I/O registrations and IP-affinity entries; it is not broken down further.

The 18.7 KB marginal of the earlier test does not reproduce. v1.2.0 under that test's config measures 37.5 KB marginal at 30k. 1.3.x adds the future's 18 KB (55.5 KB), and the default 8 KiB buffers add 11 KB (66.7 KB). That is inside the 56–75 KB/conn seen in production, with no traffic effect needed. Confidence that this transfers to production is medium: loopback, a glibc build, no RTT.

Results, 100k connections, 3 repeats

N=100000, 8 client procs, 4 UDS upstreams (ipAffinity true), msg 256 B at 1/s/conn, ping 8 B every 30 s, slow fraction 0.1, churn turnover 300 s, drop to 10% held 390 s. Cells: mean (min–max) over runs; bytes per connection unless noted. RSS = VmRSS; kernel = symphony-side TCP skmem r+w+f from ss.

Parked → active → slow:

arm	runs	parked total	parked marginal	parked over idle	active total	active Δ over parked	active RSS slope KiB/s	delivered % of target	symphony CPU % (active)	slow Δ per slow conn	slow 2nd-half Δ per slow conn	kernel/conn active	kernel per slow conn	cgroup sock per slow conn	global TCP mem in slow MiB
futfix	3	32,807 (32,804–32,811)	32,167 (32,162–32,173)	32,226 (32,221–32,233)	32,807 (32,805–32,811)	1 (-0–1)	-17.2 (-17.5–-16.9)	99.2 (99.1–99.2)	389 (298–458)	74,451 (74,410–74,509)	-28 (-34–-23)	0 (0–0)	90,116 (90,113–90,119)	90,078 (90,069–90,091)	963 (959–971)
arena2	3	67,390 (67,384–67,393)	66,741 (66,728–66,748)	66,811 (66,805–66,815)	67,391 (67,385–67,395)	2 (1–3)	-14.7 (-18.3–-9.6)	99.1 (99.1–99.1)	440 (421–467)	74,588 (74,451–74,659)	12 (-17–67)	0 (0–0)	90,116 (90,112–90,120)	90,122 (90,104–90,145)	965 (959–970)
glibc	3	67,393 (67,384–67,404)	66,742 (66,729–66,753)	66,813 (66,803–66,826)	67,394 (67,384–67,405)	1 (1–2)	-15.9 (-17.4–-13.6)	98.9 (98.6–99.1)	397 (279–457)	74,481 (74,448–74,517)	33 (-21–61)	0 (0–0)	90,113 (90,112–90,113)	90,109 (90,107–90,112)	967 (961–970)
buf-small	3	56,140 (56,131–56,151)	55,494 (55,475–55,511)	55,561 (55,550–55,574)	56,142 (56,132–56,153)	2 (1–2)	-15.6 (-17.3–-14.7)	99.1 (99.0–99.2)	384 (302–468)	72,377 (68,304–74,542)	34 (-21–63)	0 (0–1)	90,114 (90,105–90,123)	90,105 (90,096–90,109)	951 (940–971)
trim	3	67,388 (67,378–67,408)	66,729 (66,714–66,758)	66,807 (66,797–66,829)	67,388 (67,376–67,411)	-0 (-2–3)	-9.9 (-12.1–-7.1)	99.1 (99.1–99.2)	396 (326–462)	74,384 (74,299–74,461)	-47 (-98–-21)	0 (0–0)	90,113 (90,112–90,113)	90,115 (90,094–90,134)	961 (958–965)
buflimit	3	67,405 (67,405–67,406)	66,758 (66,750–66,762)	66,825 (66,824–66,825)	67,407 (67,406–67,408)	2 (1–3)	-14.4 (-16.6–-12.1)	99.1 (99.1–99.1)	427 (353–472)	18,866 (18,851–18,881)	35 (-20–65)	0 (0–0)	90,116 (90,112–90,118)	90,116 (90,094–90,141)	960 (957–964)
jemalloc-preload	3	72,151 (72,142–72,162)	71,422 (71,413–71,436)	71,503 (71,494–71,514)	72,155 (72,146–72,166)	4 (3–4)	-11.3 (-12.7–-10.0)	99.1 (99.1–99.1)	417 (406–439)	82,820 (82,757–82,879)	52 (-15–87)	0 (0–0)	90,114 (90,113–90,115)	90,129 (90,112–90,162)	963 (960–965)
rust-jemalloc	3	72,090 (72,075–72,119)	71,397 (71,382–71,421)	71,498 (71,483–71,527)	72,092 (72,077–72,120)	1 (1–2)	-17.1 (-17.4–-16.8)	99.1 (99.1–99.1)	448 (412–476)	82,760 (82,597–82,919)	-27 (-120–60)	0 (0–0)	90,117 (90,112–90,121)	90,125 (90,112–90,135)	965 (962–972)

Churn → peak → drop → second peak → quiesced:

arm	runs	peak total	drop-start RSS % of peak	drop-end RSS % of peak	drop-end conns % of peak	retained over idle %	drop-end per conn	second peak total	second-peak growth per peak conn	quiesced Δ	kernel/conn at peak	cgroup sock/conn at peak	malloc_trim p50 ms	malloc_trim max ms
futfix	3	33,993 (33,961–34,023)	100.1 (100.1–100.1)	100.0 (100.0–100.0)	10.0 (10.0–10.0)	100.0 (100.0–100.0)	340,005 (339,766–340,221)	37,049 (36,891–37,245)	3,056 (2,896–3,223)	5 (4–8)	1 (1–1)	2 (0–4)	–	–
arena2	3	67,785 (67,685–67,875)	100.0 (100.0–100.0)	100.0 (100.0–100.0)	10.0 (10.0–10.0)	100.0 (100.0–100.0)	677,788 (676,878–678,499)	68,661 (67,911–69,388)	876 (225–1,512)	2 (2–2)	1 (1–2)	3 (2–3)	–	–
glibc	3	68,631 (68,605–68,660)	100.0 (100.0–100.0)	99.9 (99.7–100.0)	10.0 (10.0–10.0)	99.9 (99.7–100.0)	685,534 (684,829–686,037)	73,983 (73,333–74,458)	5,351 (4,728–5,830)	8 (2–13)	1 (0–1)	1 (1–2)	–	–
buf-small	3	57,149 (57,095–57,255)	100.0 (100.0–100.0)	99.9 (99.9–99.9)	10.0 (10.0–10.0)	99.9 (99.9–99.9)	570,808 (570,235–571,831)	61,906 (61,458–62,251)	4,757 (4,363–4,995)	7 (2–15)	1 (1–2)	2 (1–5)	–	–
trim	3	68,501 (68,422–68,609)	92.1 (91.9–92.2)	15.0 (14.9–15.0)	10.0 (10.0–10.0)	14.2 (14.2–14.3)	102,537 (102,234–102,949)	68,313 (68,268–68,343)	-186 (-282–-127)	49 (39–58)	1 (1–2)	1 (1–1)	55.1 (53.4–56.1)	688.9 (678.5–697.8)
jemalloc-preload	3	72,892 (72,765–73,091)	21.7 (21.1–22.5)	16.0 (16.0–16.1)	10.0 (10.0–10.0)	15.3 (15.2–15.3)	116,975 (116,296–117,454)	73,486 (73,462–73,511)	594 (420–698)	8 (-52–51)	1 (0–1)	1 (1–2)	–	–
rust-jemalloc	3	72,819 (72,702–72,921)	22.2 (21.6–22.6)	16.0 (15.8–16.2)	10.0 (10.0–10.0)	15.3 (15.1–15.5)	116,577 (115,320–117,934)	73,428 (73,326–73,605)	610 (404–772)	-18 (-57–42)	1 (1–1)	1 (1–1)	–	–

Repeat 1's jemalloc-preload and rust-jemalloc runs first failed to start, because their admin socket path passed the 108-byte limit; the harness now puts sockets under $XDG_RUNTIME_DIR. Those two arms were re-run for repeat 1 with identical binaries. A stalled reader here has SO_RCVBUF 4096 and MSS 1448; symphony's send buffer for it autotunes to 85 KB. Symphony's CPU varies 280–480% across arms with no per-arm pattern, so it is host load, not an arm effect.

v1.2.0 vs current, the earlier test's config, 30k connections

1024/4096 copy buffers, no IP affinity, no PROXY header. v120 is v1.2.0's native addon run with --addon-arm.

N=30000, 3 client procs, 4 UDS upstreams (ipAffinity false), msg 256 B at 1/s/conn, ping 8 B every 30 s, slow fraction 0, churn turnover 300 s, drop to 10% held 390 s. Cells: mean (min–max) over runs; bytes per connection unless noted. RSS = VmRSS; kernel = symphony-side TCP skmem r+w+f from ss.

Parked → active → slow:

arm	runs	parked total	parked marginal	parked over idle	active total	active Δ over parked	active RSS slope KiB/s	delivered % of target	symphony CPU % (active)	slow Δ per slow conn	slow 2nd-half Δ per slow conn	kernel/conn parked	kernel/conn active	kernel per slow conn	cgroup sock per slow conn	global TCP mem in slow MiB	max VmSwap MiB
glibc	2	57,611 (57,605–57,617)	55,494 (55,482–55,505)	55,679 (55,674–55,684)	57,593 (57,587–57,600)	-18 (-19–-18)	-18.3 (-19.4–-17.2)	99.1 (99.1–99.2)	120 (116–123)	–	–	0 (0–0)	2 (1–2)	–	–	–	0 (0–0)
v120	2	39,627 (39,625–39,628)	37,504 (37,500–37,507)	37,687 (37,685–37,689)	39,607 (39,603–39,611)	-20 (-22–-17)	-20.9 (-23.4–-18.4)	99.1 (99.1–99.2)	126 (112–139)	–	–	0 (0–0)	2 (2–3)	–	–	–	0 (0–0)

Combined: the future-size patch plus 1024/4096 buffers, 100k connections

futfix is the patched build and glibc today's build, both with 1024/4096 copy buffers.

N=100000, 8 client procs, 4 UDS upstreams (ipAffinity true), msg 256 B at 1/s/conn, ping 8 B every 30 s, slow fraction 0.1, churn turnover 300 s, drop to 10% held 390 s. Cells: mean (min–max) over runs; bytes per connection unless noted. RSS = VmRSS; kernel = symphony-side TCP skmem r+w+f from ss.

Parked → active → slow:

arm	runs	parked total	parked marginal	parked over idle	active total	active Δ over parked	active RSS slope KiB/s	delivered % of target	symphony CPU % (active)	slow Δ per slow conn	slow 2nd-half Δ per slow conn	kernel/conn parked	kernel/conn active	kernel per slow conn	cgroup sock per slow conn	global TCP mem in slow MiB	max VmSwap MiB
glibc	2	56,140 (56,140–56,140)	55,496 (55,494–55,497)	55,558 (55,557–55,560)	56,142 (56,142–56,142)	2 (2–2)	-14.6 (-14.8–-14.3)	99.2 (99.1–99.2)	435 (430–439)	74,513 (74,506–74,519)	62 (61–63)	0 (0–0)	0 (0–0)	90,117 (90,116–90,119)	90,111 (90,103–90,119)	961 (959–964)	0 (0–0)
futfix	2	21,540 (21,539–21,541)	20,898 (20,893–20,904)	20,959 (20,958–20,961)	21,541 (21,540–21,542)	1 (1–1)	-16.9 (-17.4–-16.4)	99.2 (99.2–99.2)	466 (463–469)	74,373 (74,349–74,397)	-23 (-23–-22)	0 (0–0)	0 (0–0)	90,114 (90,114–90,114)	90,098 (90,085–90,112)	965 (963–968)	0 (0–0)

The future-size patch (the futfix arm, measured before #61 merged it)

handle's future goes from 42,808 to 8,192 bytes. Boxing costs one allocation, only on routes that rewrite HTTP headers.

diff --git a/src/proxy_conn.rs b/src/proxy_conn.rs
index ceef334..6572f6b 100644
--- a/src/proxy_conn.rs
+++ b/src/proxy_conn.rs
@@ -419,7 +419,7 @@ where
 	// of a per-connection cache line per read chunk; do not "optimise" it away without also
 	// deciding to lose that metric.
 	let watch = HalfCloseWatch::new(&ctx.listener_metrics);
-	let copy = async {
+	let copy = std::pin::pin!(async {
 		write_connection_prefix(upstream, sf).await?;
 		let mut client = Watched::client(client, &watch);
 		let mut upstream = Watched::upstream(upstream, &watch);
@@ -433,17 +433,21 @@ where
 			)
 			.await
 		} else {
-			crate::http_proxy::proxy_http1_rewriting(&mut client, &mut upstream, &rewrites).await
+			Box::pin(crate::http_proxy::proxy_http1_rewriting(
+				&mut client,
+				&mut upstream,
+				&rewrites,
+			))
+			.await
 		}
-	};
+	});
 
 	let body = async {
 		if ctx.half_close_timeout.is_zero() {
 			copy.await.map_err(classify_copy_failure)
 		} else {
-			tokio::pin!(copy);
 			tokio::select! {
-				result = &mut copy => result.map_err(classify_copy_failure),
+				result = copy => result.map_err(classify_copy_failure),
 				_ = watch.expired(ctx.half_close_timeout) => Err(ErrorKind::HalfClosed),
 			}
 		}

Method limits

Loopback only. A stalled reader holds about 165 KB before the upstream sees backpressure (its window, symphony's 85 KB send buffer, the 64 KiB rustls queue and the copy buffer). At one 256 B message per minute that takes about 11 hours; at the benchmark's 1/s, about 11 minutes.
glibc host. If production runs the musl build, the allocator results do not apply as measured. The harness can take a musl .node via --addon-arm on a musl host.
Run records: ~/dev/tmp/bench-runs/symphony-memory-{r1,r1b,r2,r3,v120,combo}-*/meta.txt on the bench host (CPU capped at 2 GHz).

_{🤖 Generated by Claude (Opus 5.5); posted via @kriszyp.}

🤖 Generated with Claude Code

https://claude.ai/code/session_01CpGsSRp9pXECi7rMgHedg5

Related PRs: #61 overlaps (same TLS-accept block in proxy_conn.rs, merged first; resolved in the rebase), #57 independent
Complexity: medium

_{Dispatch: task symphony-active-conn-memory-bench · queued by unknown · ran by claude/opus/xhigh · worker kzyp-xps-1}

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex; adjudicated=domain; declined=cursor-grok,cursor-kimi,cursor-muse,muse; rounds=9; full=2 @ 77005d8}

_{Review-Attention: study ~14m (decisions: do-less-alternative, failed-connection-policy, swap-tolerance, bench-hooks-in-npm-test) @ 77005d8}

gemini-code-assist

Code Review

This pull request introduces an out-of-process memory benchmark (bench-copy-memory-oop.ts) and associated environment hooks in Rust to measure and control memory usage under various load conditions, including TLS buffer limits and periodic malloc_trim intervals. The review feedback highlights several robustness improvements in the benchmark script: explicitly checking if child.send is a function before calling it, handling cases where the spawned process PID is undefined, and safely checking for the existence of the memory.swap.max cgroup file to prevent unhandled ENOENT exceptions.

gemini-code-assist on #60: without swap accounting the scope has no memory.swap.max, and every scoped run failed with ENOENT. VmSwap is still sampled and reported, so the run proceeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpGsSRp9pXECi7rMgHedg5 Dispatch-Task: symphony-active-conn-memory-bench

…connections `npm run benchmark:memory-oop` runs symphony-server in its own process (its own systemd scope with swap disabled, when available), drives it from separate Node TLS clients, python3 slow readers and UDS upstreams, and samples only the server: /proc status + smaps_rollup, its cgroup's memory.stat, and ss skmem of its sockets. Phases: parked, active fan-out, stalled readers, and churn → peak → drop → second peak. Arms: glibc, MALLOC_ARENA_MAX=2, periodic malloc_trim, jemalloc via LD_PRELOAD, jemalloc as the addon's Rust allocator, a rustls buffer cap, and the README's small copy buffers. Two env hooks and a cargo feature make those arms possible; all are inert unless set: - SYMPHONY_TLS_BUFFER_LIMIT caps rustls' outgoing queues (set_buffer_limit). - SYMPHONY_MALLOC_TRIM_INTERVAL_MS runs malloc_trim(0) on a blocking thread while the proxy is started. - `--features jemalloc` (npm run build:bench-jemalloc) installs tikv-jemallocator as the global allocator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpGsSRp9pXECi7rMgHedg5 Dispatch-Task: symphony-active-conn-memory-bench

…rors Review round 1 (gemini, cursor-composer): a non-UTF-8 hook value was read as unset, and an active hook now says so on stderr; an unhandled spawn or log stream 'error' could crash the orchestrator before cleanup, and a closed IPC channel made stopAll throw. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpGsSRp9pXECi7rMgHedg5 Dispatch-Task: symphony-active-conn-memory-bench

… of git Review round 1 (adjudicated): compare /proc maps against real paths, keep an 'error' listener on every child, fail a run whose last sample raised a fault, name lost connections in a payload mismatch, ignore bench-results/, and drop machine-specific wording from comments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpGsSRp9pXECi7rMgHedg5 Dispatch-Task: symphony-active-conn-memory-bench

…vocations `--addon-arm NAME=PATH` runs another .node build as its own arm, so a code change can be priced against the same workload. `--summarize DIR...` pools separate invocations, so repeats can each take a shared benchmark lock for two hours instead of one run holding it for six. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpGsSRp9pXECi7rMgHedg5 Dispatch-Task: symphony-active-conn-memory-bench

… shutdown Review round 2 (codex, gemini, cursor-muse): - payload accounting counts a write when its callback succeeds; - holds, deadlines and sample times use performance.now(); - a signal during start-up kills anything spawned after the stop snapshot; - the validity barrier checks child exits directly, not just the 2 s poll; - the python reader rechecks pause per selected event; - kernel page size is read, not assumed 4 KiB; - over-long Unix socket paths fail fast; - --clients cannot overlap the slow readers' source nets, and the rust-jemalloc arm must actually contain jemalloc; - the TLS buffer limit rejects values that do not fit a usize, and the hook spec restores pre-existing environment values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpGsSRp9pXECi7rMgHedg5 Dispatch-Task: symphony-active-conn-memory-bench

…per tick Review round 2 (adjudicated): Node documents its IPC channel as unsupported with a non-Node child, so the slow reader now speaks the same JSON-lines protocol on its stdin/stdout. The churn budget is capped at the live population, and a timeline sample answered after the run ended is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpGsSRp9pXECi7rMgHedg5 Dispatch-Task: symphony-active-conn-memory-bench

Review round 3 (gemini, domain; codex's partial output): - the Unix socket path check runs once before the sweep, with the real `.<pid>` margin, instead of aborting it part-way; - --summarize refuses to pool one arm name across different addon builds, and needs at least one directory; - the sweep stops at the next run after a signal; - a failed page-size query is not cached; - --addon-arm names cannot shadow built-in arms or break paths; - a lost connection invalidates a parked/active/slow run; - a truncated JSON line from the python reader fails that child instead of throwing out of the reader. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpGsSRp9pXECi7rMgHedg5 Dispatch-Task: symphony-active-conn-memory-bench

…se losses Review round 4 (codex, gemini, domain): - Unix sockets go in a private directory under $XDG_RUNTIME_DIR (else TMPDIR), so a deep checkout's default results path no longer overflows the 108-byte socket address; - meta.json records the harness file's hash, and --summarize warns when it pools runs from different harness versions; - a connection closed without the client closing it invalidates a churn-then-drop run too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpGsSRp9pXECi7rMgHedg5 Dispatch-Task: symphony-active-conn-memory-bench

Repeats run from different frozen copies of the same build carry different addon and preload paths; the per-arm addon hash already identifies the build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpGsSRp9pXECi7rMgHedg5 Dispatch-Task: symphony-active-conn-memory-bench

Review round 5 (nits): Child and Run declare their fields explicitly, so the file stays type-strippable; a child that fails rejects its pending stats requests at once instead of after the 15 s IPC timeout; socket directories are removed on a signal and when setup fails; --summarize ignores --phases, --no-cgroup and --min-available-mb when comparing workloads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpGsSRp9pXECi7rMgHedg5 Dispatch-Task: symphony-active-conn-memory-bench

Review round 6: --summarize no longer compares preload library paths, so each run now records the preloaded library's hash and pooling requires it to match. A run without its own cgroup no longer pools with isolated runs, and --upstreams is bounded to 1–64. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpGsSRp9pXECi7rMgHedg5 Dispatch-Task: symphony-active-conn-memory-bench

gemini-code-assist on #60: without swap accounting the scope has no memory.swap.max, and every scoped run failed with ENOENT. VmSwap is still sampled and reported, so the run proceeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpGsSRp9pXECi7rMgHedg5 Dispatch-Task: symphony-active-conn-memory-bench

gemini-code-assist Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread __test__/bench-copy-memory-oop.ts

Comment thread __test__/bench-copy-memory-oop.ts

Comment thread __test__/bench-copy-memory-oop.ts Outdated

kriszyp mentioned this pull request Oct 14, 2026

Cut parked per-connection memory 59% by shrinking each connection's task #61

Merged

kriszyp marked this pull request as ready for review October 14, 2026 17:07

kriszyp and others added 12 commits October 14, 2026 11:14

kriszyp force-pushed the bench/active-connection-memory branch from 2bc999b to 77005d8 Compare October 14, 2026 17:47

kriszyp merged commit 7d2fbdf into main Oct 14, 2026
11 checks passed

kriszyp mentioned this pull request Oct 15, 2026

Make jemalloc the default Rust allocator on Linux #66

Draft

Navigation Menu

Repository navigation

Add an out-of-process memory benchmark for active, stalled and churning connections - #60

Add an out-of-process memory benchmark for active, stalled and churning connections#60
kriszyp merged 12 commits into
mainfrom
bench/active-connection-memory

kriszyp commented Oct 13, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

kriszyp commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⊙ Problem

🤔 Is a committed benchmark with default-off hooks the right size for this?

💡 Solution

🤔 Which follow-up first?

⚖️ Alternatives

🤔 Should the two hooks ship in the published addon?

🔧 Changes

✅ Verification

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

kriszyp commented Oct 13, 2026 •

edited

Loading