Skip to content

perf(gateway): evaluate jemalloc so freed memory returns to the OS #5404

Open

Description

@FrostGod

User Story

I run shared OpenShell gateways where agents create short-lived sandboxes all day (create, run a few commands, delete), and the gateways are expected to stay up for weeks. In HA runs, each gateway held 0.7–1.3 GiB with 300 sandboxes on a 5-node Kubernetes cluster, and 2.7 GiB with 1,080 sandboxes on a larger HA run: about 2–3.6 MiB per sandbox on every gateway. After sandboxes are deleted, gateway memory doesn't go back down, so we can't tell freed memory from leaked memory or size gateways from measurements.

Description

The gateway uses the default glibc allocator (distroless/cc-debian13 base image; no #[global_allocator] in the gateway binary). Freed blocks, such as the ~1.7 MiB per-sandbox buffers, often stay mapped in the process, so RSS does not drop after sandboxes are deleted. Evaluate jemalloc, or tuning such as MALLOC_ARENA_MAX, so freed memory returns to the OS.

Context

This is not the leak itself; the leaks are tracked in #5399, #5400, #5401 and #5402. Once those land, there is less memory to return, so measure before switching.

Definition of Done

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:gatewayGateway server and control-plane work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions