Make jemalloc the default Rust allocator on Linux

.cargo/config.toml

@@ -0,0 +1,5 @@
+[env]
+# jemalloc fixes its page size when it is built and aborts on a kernel with larger pages. arm64
+# kernels run 4, 16 or 64 KiB pages and a cross build would detect the builder's, so build for 64 KiB.
+AARCH64_UNKNOWN_LINUX_GNU_JEMALLOC_SYS_WITH_LG_PAGE = "16"
+AARCH64_UNKNOWN_LINUX_MUSL_JEMALLOC_SYS_WITH_LG_PAGE = "16"

.github/workflows/CI.yml

@@ -102,6 +102,15 @@ jobs:
               export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-musl-gcc
               export CC_aarch64_unknown_linux_musl=aarch64-linux-musl-gcc
             fi
+            # jemalloc's configure links a test program, which the image's clang cannot do for these
+            # targets on its own. aarch64 keeps clang (ring's assembly needs it) and borrows the cross
+            # GCC toolchain's linker and runtime.
+            if [ "${{ matrix.target }}" = "x86_64-unknown-linux-musl" ]; then
+              export CC_x86_64_unknown_linux_musl=gcc
+            fi
+            if [ "${{ matrix.target }}" = "aarch64-unknown-linux-gnu" ]; then
+              export CFLAGS_aarch64_unknown_linux_gnu="${CFLAGS:-} --gcc-toolchain=/usr/aarch64-unknown-linux-gnu"
+            fi
             npm run build -- --target ${{ matrix.target }}
             strip --strip-unneeded ${{ matrix.artifact }} 2>/dev/null || true
 
@@ -196,12 +205,56 @@ jobs:
         with:
           name: ${{ matrix.artifact }}
 
+      - name: Check the artifact links jemalloc
+        if: runner.os == 'Linux'
+        run: grep -q '<jemalloc>' ${{ matrix.artifact }}
+
       - name: Install dependencies
         run: npm ci
 
       - name: Run tests
         run: npm test
 
+  test-linux-variants:
+    # Each Linux artifact links its own jemalloc (libc and page size differ per target).
+    name: Test – ${{ matrix.artifact }}
+    runs-on: ${{ matrix.os }}
+    needs: [build-linux]
+    timeout-minutes: 15
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - os: ubuntu-24.04-arm
+            image: node:22-bookworm
+            artifact: symphony.linux-arm64-gnu.node
+          - os: ubuntu-latest
+            image: node:22-alpine
+            artifact: symphony.linux-x64-musl.node
+          - os: ubuntu-24.04-arm
+            image: node:22-alpine
+            artifact: symphony.linux-arm64-musl.node
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Download artifact
+        uses: actions/download-artifact@v4
+        with:
+          name: ${{ matrix.artifact }}
+
+      # `docker run`, not a job container: the runner cannot run JavaScript actions inside Alpine on
+      # arm64. The local-build loader looks only for the flat `<platform>-<arch>[-gnu].node` names.
+      - name: Run tests
+        run: |
+          grep -q '<jemalloc>' ${{ matrix.artifact }}
+          docker run --rm -v "$PWD:/w" -w /w ${{ matrix.image }} sh -ec '
+            arch=$(node -p process.arch)
+            [ -f "symphony.linux-$arch-gnu.node" ] || mv ${{ matrix.artifact }} "symphony.linux-$arch.node"
+            if command -v apk >/dev/null; then apk add --no-cache openssl; fi
+            npm ci
+            npm test'
+
   integration-test:
     name: Integration Test – Harper UDS proxy
     runs-on: ubuntu-latest

.github/workflows/release.yml

@@ -59,8 +59,18 @@ jobs:
               export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-musl-gcc
               export CC_aarch64_unknown_linux_musl=aarch64-linux-musl-gcc
             fi
+            # jemalloc's configure links a test program, which the image's clang cannot do for these
+            # targets on its own. aarch64 keeps clang (ring's assembly needs it) and borrows the cross
+            # GCC toolchain's linker and runtime.
+            if [ "${{ matrix.target }}" = "x86_64-unknown-linux-musl" ]; then
+              export CC_x86_64_unknown_linux_musl=gcc
+            fi
+            if [ "${{ matrix.target }}" = "aarch64-unknown-linux-gnu" ]; then
+              export CFLAGS_aarch64_unknown_linux_gnu="${CFLAGS:-} --gcc-toolchain=/usr/aarch64-unknown-linux-gnu"
+            fi
             npm run build -- --target ${{ matrix.target }}
             strip --strip-unneeded ${{ matrix.artifact }} 2>/dev/null || true
+            grep -q '<jemalloc>' ${{ matrix.artifact }}
 
       - name: Upload artifact
         uses: actions/upload-artifact@v4

CLAUDE.md

@@ -48,7 +48,7 @@ TCP accept (SO_REUSEPORT per worker thread)
 
 | File | Responsibility |
 |---|---|
-| `src/lib.rs` | Crate root; `mod` declarations; `#[macro_use] napi_derive` |
+| `src/lib.rs` | Crate root; `mod` declarations; `#[macro_use] napi_derive`; jemalloc as the global allocator on Linux |
 | `src/proxy.rs` | All `#[napi]`-exposed types and methods; config parsing helpers |
 | `src/listener.rs` | TCP accept loop for TLS listeners; SO_REUSEPORT per worker; RLIMIT_NOFILE |
 | `src/liveness.rs` | Dead-peer detection: TCP keepalive on the listening socket; the half-close bound |
@@ -151,6 +151,15 @@ Two further things are deliberately *not* solved: rustls' ring `Ticketer` rotate
 ### Send + Sync in napi classes
 napi `Buffer` contains raw pointers (`*mut napi_env__`, `*mut napi_ref__`) that are not `Sync`. The `SymphonyProxyWrap` struct must be `Send + Sync`. Solution: napi types (`Buffer`, `JsCertConfig`, etc.) are only used as constructor/method *parameters*, immediately converted to plain Rust (`Vec<u8>`, `ListenerTlsSpec`, etc.), and never stored in the struct. The struct only holds types that are provably `Send + Sync`.
 
+### jemalloc as the Rust allocator on Linux (`src/lib.rs`, default `jemalloc` feature)
+glibc keeps a fan-out node's peak footprint after its connections close: freed chunks are scattered across per-thread arenas, and glibc returns only the top of a heap. On a 100k ↔ 10k churned soak glibc held ~100% of peak at every trough and at zero connections, and its peaks crept up cycle over cycle; jemalloc fell to ~28% of peak at each trough and stayed flat. `--cargo-flags=--no-default-features` builds on the system allocator, for bisecting.
+
+- **Prefixed symbols are what make it safe in a Node process.** tikv-jemallocator exports `_rjem_*`, so only Rust's `#[global_allocator]` reaches it and Node/V8 stay on glibc. The invariant that follows: memory is freed by the allocator that allocated it. It holds because nothing crosses the boundary by ownership — JS `Buffer` arguments are borrowed (napi-rs unrefs them on drop, never frees the bytes) and `pem_bytes` copies them; values returned to JS are copied by `napi_create_*`; the class wrapper and threadsafe-function payloads are freed by Rust code in napi-rs's finalizers. Returning a Rust `Vec` to JS as an external buffer would still be fine (its finalizer is Rust), but a C library that `free()`s memory Rust allocated, or vice versa, would crash.
+- **`disable_initial_exec_tls`** is required: Node `dlopen()`s the addon, and jemalloc's default TLS model cannot be dlopened.
+- **arm64 page size is fixed at build time** (`.cargo/config.toml`): jemalloc refuses to start on a kernel whose pages are larger than the ones it was built for, and a cross build would detect the builder's 4 KiB. The aarch64 targets are built for 64 KiB, which also runs on 4 and 16 KiB kernels.
+- **Defaults are deliberate.** jemalloc purges freed pages only when that arena sees allocation activity, so a process with zero connections and no traffic keeps them (~25% of peak in the soak); `background_thread:true` returned them (717 → 196 MiB) but made no difference at a trough with live traffic, where the residue is pages pinned by surviving connections. If idle reclamation matters, enable it with `mallctl("background_thread")` after initialization — jemalloc documents that turning it on through `malloc_conf` at startup can deadlock.
+- `allocator_tests::rust_allocations_come_from_jemalloc` asks jemalloc (`arenas.lookup`) whether a live `Box` is its own, which proves selection, not just linkage.
+
 ---
 
 ## Conventions

Cargo.toml

@@ -27,12 +27,14 @@ ring        = "0.17"
 md-5        = "0.10"
 sha2        = "0.10"
 zeroize     = "1"
+
+[target.'cfg(target_os = "linux")'.dependencies]
 # `disable_initial_exec_tls`: jemalloc's default TLS model cannot be dlopen()ed, which is how Node
 # loads this cdylib.
 tikv-jemallocator = { version = "0.6", optional = true, features = ["disable_initial_exec_tls"] }
 
 [features]
-# Memory-benchmark arm only (see src/lib.rs); never enabled in published builds.
+default = ["jemalloc"]
 jemalloc = ["dep:tikv-jemallocator"]
 
 [dev-dependencies]

README.md

@@ -821,6 +821,11 @@ npm run build:debug    # builds a dev .node file
 npm run build          # release build (LTO, stripped)
 ```
 
+On Linux the addon's Rust allocations go to a statically linked jemalloc, which returns memory to the
+OS after connections close; glibc's malloc keeps the peak resident. Its symbols are prefixed, so Node's
+own allocations stay on the system allocator. To build on the system allocator instead, for example
+to bisect a memory problem: `npm run build -- --cargo-flags=--no-default-features`.
+
 ### Cross-compilation
 
 Use the napi-rs Docker images (same ones used in CI):