Skip to content

chore: serve static file benchmarks from memory - #2189

Closed
dinwwwh wants to merge 1 commit into
mainfrom
claude/static-file-bench-determinism
Closed

dinwwwh wants to merge 1 commit into
mainfrom
claude/static-file-bench-determinism

Conversation

@dinwwwh

@dinwwwh dinwwwh commented Oct 6, 2026

Copy link
Copy Markdown
Member

The static file handler benchmarks read real files, so CodSpeed's single measured call depended on thread pool scheduling and on whatever else the event loop picked up, rather than on plugin code. They now serve the same files from an in-memory fs that still runs the whole plugin path: path resolution, containment checks, content type, ETag, conditional and range handling, and body streaming. Every measured call is now the same microtask chain, so these benches should stop flagging false regressions on unrelated PRs. The new numbers reset the baseline for all five benches.

Evidence

  • False flags. Across the CodSpeed comments on fix(rpc): reject non-Blob values in serialized blobs #2060–fix(openapi): fix regex pattern for numeric path validation #2186, "serve file" was flagged 19 times, almost always at −11.3% to −12.9% (3.5 ms ↔ 4.0 ms). "range request" was flagged 4 times, "serve deeply nested encoded path" 3 times and "not modified (304)" twice. None of those PRs touched the plugin, and 25 of the 28 comments warned about different runtime environments.
  • Unrelated code moves the numbers. chore: remove unreliable octet stream benchmarks #2188 only removes the octet stream benches and their helpers from __shared__/payloads.ts, yet CodSpeed reports range request +60%, nested path +51% and serve file +48%. This bench imported that module just for drainBody.
  • Most of the measured call isn't the plugin. CodSpeed records 17–183 syscalls inside the single measured call of these benches (serve file: 161–183 across the last 43 runs), against 0 for almost every other bench. Under the CodSpeed analysis runner with strace, each serve call makes 6 fs requests through the libuv thread pool, with ~10 event-loop turns and ~18 futex handoffs. The per-call syscall count also varies from call to call (53–149) with identical code.
  • Unrelated callbacks run inside calls. While a call waits on fs, the worker handles vitest's IPC traffic, such as the reply to the previous bench's [CodSpeed] … done log. Locally that lands in a warm-up call, but which call it lands in depends only on timing.

Results

Measured locally with the CodSpeed analysis runner (CODSPEED_ENV=1 CODSPEED_RUNNER_MODE=simulation) under strace. Valgrind isn't available here, so this is not a CodSpeed run.

  • Syscalls inside the measured call drop from 14–149 (fs, futex, epoll) to none for four benches. Serve file is left with only V8's JIT code-map writes.
  • Three runs, one with chore: remove unreliable octet stream benchmarks #2188's payloads.ts change applied, give identical syscall counts for all 40 calls. Before this change they differed between runs.
  • All five scenarios still return the expected responses: 200, 200, 206 bytes 0-1023/10240, 304, and fall through.
  • "serve file" still costs more than the others. As the first bench in the file, its measured call includes V8's baseline compilation of the shared request path, which is fixed for a given code state.

The static file handler benches read real files, so CodSpeed's single
measured call hopped through the libuv thread pool and event loop and
picked up unrelated callbacks, flipping results by 12% or more on PRs
that never touched the plugin. Back the plugin with an in-memory fs and
stop importing the shared payloads, so every measured call is the same
microtask chain.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
orpc 08e13f5 Commit Preview URL

Branch Preview URL
Oct 06 2026, 04:09 AM

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Serves bench files from memory instead of disk — benches/static-file-handler.bench.ts now injects an in-memory fs via vi.mock('node:fs') / vi.mock('node:fs/promises') (createReadStream, stat, realpath) so the single call CodSpeed measures no longer waits on the libuv thread pool or absorbs unrelated callbacks, which was the source of the false regressions across unrelated PRs.
  • Drops the ./__shared__/payloads import — the file only used it for drainBody; a local copy is inlined so building those payloads no longer shifts V8 compilation into the first measured call.
  • Deterministic representation metadata — fixed mtime and fixed 10KB buffers give a stable ETag, so all five scenarios still exercise the full plugin path (path resolution, containment, content type, ETag, conditional, range, streaming) and return 200, 200, 206 bytes 0-1023/10240, 304, and fall through.

I verified the bench runs to completion, and the file passes eslint and the root tsc. The mock replaces only the fs surface the plugin actually reads, and the hoisted function declarations are safe to reference from the lazy vi.mock factories.

Pullfrog  | View workflow run | Using deepseek-v4.1-flash (free via Pullfrog for OSS) | 𝕏

@pkg-pr-new

pkg-pr-new Bot commented Oct 6, 2026

Copy link
Copy Markdown
More templates

@orpc/ai-sdk

npm i https://pkg.pr.new/@orpc/ai-sdk@2189

@orpc/arktype

npm i https://pkg.pr.new/@orpc/arktype@2189

@orpc/bun

npm i https://pkg.pr.new/@orpc/bun@2189

@orpc/client

npm i https://pkg.pr.new/@orpc/client@2189

@orpc/cloudflare

npm i https://pkg.pr.new/@orpc/cloudflare@2189

@orpc/contract

npm i https://pkg.pr.new/@orpc/contract@2189

@orpc/experimental-effect

npm i https://pkg.pr.new/@orpc/experimental-effect@2189

@orpc/evlog

npm i https://pkg.pr.new/@orpc/evlog@2189

@orpc/hibernation

npm i https://pkg.pr.new/@orpc/hibernation@2189

@orpc/json-schema

npm i https://pkg.pr.new/@orpc/json-schema@2189

@orpc/experimental-lock

npm i https://pkg.pr.new/@orpc/experimental-lock@2189

@orpc/experimental-msw

npm i https://pkg.pr.new/@orpc/experimental-msw@2189

@orpc/nest

npm i https://pkg.pr.new/@orpc/nest@2189

@orpc/next

npm i https://pkg.pr.new/@orpc/next@2189

@orpc/node

npm i https://pkg.pr.new/@orpc/node@2189

@orpc/openapi

npm i https://pkg.pr.new/@orpc/openapi@2189

@orpc/opentelemetry

npm i https://pkg.pr.new/@orpc/opentelemetry@2189

@orpc/pinia-colada

npm i https://pkg.pr.new/@orpc/pinia-colada@2189

@orpc/pino

npm i https://pkg.pr.new/@orpc/pino@2189

@orpc/publisher

npm i https://pkg.pr.new/@orpc/publisher@2189

@orpc/ratelimit

npm i https://pkg.pr.new/@orpc/ratelimit@2189

@orpc/server

npm i https://pkg.pr.new/@orpc/server@2189

@orpc/shared

npm i https://pkg.pr.new/@orpc/shared@2189

@orpc/swr

npm i https://pkg.pr.new/@orpc/swr@2189

@orpc/tanstack-query

npm i https://pkg.pr.new/@orpc/tanstack-query@2189

@orpc/trpc

npm i https://pkg.pr.new/@orpc/trpc@2189

@orpc/valibot

npm i https://pkg.pr.new/@orpc/valibot@2189

@orpc/zod

npm i https://pkg.pr.new/@orpc/zod@2189

commit: 08e13f5

@codecov

codecov Bot commented Oct 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@codspeed

codspeed Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will improve performance by 52.74%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
✅ 26 untouched benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ not modified (304) 361.2 µs 212.4 µs +70.05%
⚡ range request 1,016.6 µs 667.8 µs +52.24%
⚡ serve deeply nested encoded path 1,020.7 µs 672.8 µs +51.72%
⚡ not found fall through 329.9 µs 238.1 µs +38.58%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/static-file-bench-determinism (08e13f5) with main (86c31c3)

Open in CodSpeed

@dinwwwh dinwwwh closed this Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant