Benchmark Track: Stable dart:io (AOT, fletch-style) vs Experimental dart-zig by kartikey321 · Pull Request #692 · MDA2AV/HttpArena

kartikey321 · 2026-05-06T03:33:02Z

Purpose

This PR is a benchmark track PR only and is not intended to merge into main.

Goal:

Compare stable Dart dart:io against experimental dart-zig under HttpArena using the same benchmark profiles.

What This PR Changes

How `dart-zig` Differs from `dart:io`

Replaces the dart:io socket/event pipeline with direct kernel backends (io_uring on Linux, kqueue on macOS).
Reworks the native boundary through custom runtime embedding + native resolver calls (@pragma('vm:external-name') path).
Reimplements TCP and HTTP server hot paths in Zig (accept/read/write, parser/routing, completion batching/pooling).
Uses an explicit multi-worker runtime model (SO_REUSEPORT semantics, worker topology experiments).
Splits runtime execution paths clearly for JIT/AOT (dart-zig + .dill vs dart-zig-aot + .so).
Current scope is server/runtime performance substrate, not full production-complete dart:io parity across all APIs.

Benchmark Scope (current)

Profiles in this track:

baseline
pipelined
limited-conn
json

(Advanced profiles such as async-db/json-tls/json-comp are intentionally out of scope for this PR.)

Validation

bash scripts/validate.sh dart-io ✅
bash scripts/validate.sh dart-zig ✅ (on this branch setup)

Notes

This branch is intended for iterative benchmark experimentation.
Follow-up commits can extend metrics/reporting without affecting mainline production flow.

PR Commands — comment on this PR to trigger (requires collaborator approval):

Command	Description
`/benchmark -f <framework>`	Run all benchmark tests
`/benchmark -f <framework> -t <test>`	Run a specific test
`/benchmark -f <framework> --save`	Run and save results (updates leaderboard on merge)

Always specify -f <framework>. Results are automatically compared against the current leaderboard.

Run benchmarks locally

You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.

./scripts/validate.sh <framework>
./scripts/benchmark-lite.sh <framework> baseline
./scripts/benchmark-lite.sh --load-threads 4 <framework>

Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk, ghz) are built as self-contained Docker images on first run.

kartikey321 · 2026-05-06T03:44:10Z

/benchmark -f dart-io

github-actions · 2026-05-06T03:44:20Z

👋 /benchmark request received. A collaborator will review and approve the run.

kartikey321 · 2026-05-06T03:44:55Z

/benchmark -f dart-zig

github-actions · 2026-05-06T03:45:07Z

👋 /benchmark request received. A collaborator will review and approve the run.

kartikey321 · 2026-05-06T06:54:16Z

@MDA2AV I was trying to bring your or any other contributors attention to this pr

I have not created this pr for adding new backends or merging to main. I had created it just for benchmarking the difference between dart io and an experimental version where all the http parsing happens in zig.

What was the motivation behind it?
As dart is progressing deep into backend dev and need for same arch on frontend and backend increases, the demand for more scalable dart backends increases. So this is just an experimental framework, whose aim is for a proof of work and to raise issue in dart repo for a server side runtime.
And i am to keep this pr as a record of benchmark between both runtimes via a valid benchmarking framework

Why do i call it experimental?
It works fine and handles 2.5x more request per sec but is not handles in prod niether its testing cover is that good and also it only fully covers linux http 1 calls.

If the current motive of httparena does not match the motives of this pr to store benchmarking record diffs for experimental projects , just notify me , i will be happy to arrange another sources for generating and storing benchmarking reords

github-actions · 2026-05-06T08:44:23Z

Benchmark Results

Framework: dart-io | Test: all tests

Test	Conn	RPS	CPU	Mem	Δ RPS	Δ Mem
baseline	512	601,780	6534.1%	368MiB	NEW	NEW
baseline	4096	583,020	6684.2%	812MiB	NEW	NEW
pipelined	512	998,153	6660.8%	397MiB	NEW	NEW
pipelined	4096	997,674	6713.7%	485MiB	NEW	NEW
limited-conn	512	493,328	6191.3%	859MiB	NEW	NEW
limited-conn	4096	522,346	6474.4%	1.4GiB	NEW	NEW
json	4096	235,528	6542.9%	1.4GiB	NEW	NEW

Full log

  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  10
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   7.69ms   4.00ms   7.25ms   97.60ms   177.20ms

  2611651 requests in 5.00s, 2611732 responses
  Throughput: 522.19K req/s
  Bandwidth:  97.11MB/s
  Status codes: 2xx=2611732, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 2611724 / 2611732 responses (100.0%)
  Reconnects: 261534
  Per-template: 870643,870410,870671
  Per-template-ok: 870643,870410,870671
[info] CPU 6474.4% | Mem 1.4GiB

[run 2/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  10
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   7.97ms   4.63ms   5.82ms   92.70ms   119.90ms

  2544440 requests in 5.00s, 2544454 responses
  Throughput: 508.75K req/s
  Bandwidth:  94.61MB/s
  Status codes: 2xx=2544454, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 2544444 / 2544454 responses (100.0%)
  Reconnects: 255372
  Per-template: 848028,848341,848075
  Per-template-ok: 848028,848341,848075
[info] CPU 6638.7% | Mem 2.1GiB

[run 3/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  10
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   8.04ms   4.63ms   5.76ms   94.80ms   122.20ms

  2525887 requests in 5.00s, 2524803 responses
  Throughput: 504.82K req/s
  Bandwidth:  93.88MB/s
  Status codes: 2xx=2524803, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 2524790 / 2524803 responses (100.0%)
  Reconnects: 252247
  Per-template: 841549,841683,841558
  Per-template-ok: 841549,841683,841558
[info] CPU 6453.2% | Mem 2.4GiB

=== Best: 522346 req/s (CPU: 6474.4%, Mem: 1.4GiB) ===
[info] input BW: 40.35MB/s (avg template: 81 bytes)
[info] saved results/limited-conn/4096/dart-io.json
httparena-bench-dart-io
httparena-bench-dart-io

==============================================
=== dart-io / json / 4096c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 7
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   17.52ms   17.10ms   27.30ms   48.60ms   157.80ms

  1152748 requests in 5.00s, 1152620 responses
  Throughput: 230.44K req/s
  Bandwidth:  816.59MB/s
  Status codes: 2xx=1152620, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 1152614 / 1152620 responses (100.0%)
  Reconnects: 44505
  Per-template: 163874,164804,165310,165555,165490,163834,163747
  Per-template-ok: 163874,164804,165310,165555,165490,163834,163747
[info] CPU 6398.4% | Mem 998MiB

[run 2/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 7
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   17.09ms   16.40ms   27.40ms   54.80ms   179.50ms

  1180645 requests in 5.00s, 1177643 responses
  Throughput: 235.46K req/s
  Bandwidth:  835.02MB/s
  Status codes: 2xx=1177643, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 1177636 / 1177643 responses (100.0%)
  Reconnects: 45361
  Per-template: 167716,168492,169123,168592,167733,167919,168061
  Per-template-ok: 167716,168492,169123,168592,167733,167919,168061
[info] CPU 6542.9% | Mem 1.4GiB

[run 3/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 7
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   17.32ms   16.50ms   27.10ms   51.90ms   169.00ms

  1165830 requests in 5.00s, 1165832 responses
  Throughput: 233.09K req/s
  Bandwidth:  825.87MB/s
  Status codes: 2xx=1165832, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 1165827 / 1165832 responses (100.0%)
  Reconnects: 45032
  Per-template: 165609,166332,167600,167664,167360,166123,165139
  Per-template-ok: 165609,166332,167600,167664,167360,166123,165139
[info] CPU 6453.2% | Mem 2.0GiB

=== Best: 235528 req/s (CPU: 6542.9%, Mem: 1.4GiB) ===
[info] input BW: 11.23MB/s (avg template: 50 bytes)
[info] saved results/json/4096/dart-io.json
httparena-bench-dart-io
httparena-bench-dart-io
[info] skip: dart-io does not subscribe to json-comp
[info] skip: dart-io does not subscribe to json-tls
[info] skip: dart-io does not subscribe to upload
[info] skip: dart-io does not subscribe to api-4
[info] skip: dart-io does not subscribe to api-16
[info] skip: dart-io does not subscribe to static
[info] skip: dart-io does not subscribe to async-db
[info] skip: dart-io does not subscribe to crud
[info] skip: dart-io does not subscribe to fortunes
[info] skip: dart-io does not subscribe to baseline-h2
[info] skip: dart-io does not subscribe to static-h2
[info] skip: dart-io does not subscribe to baseline-h2c
[info] skip: dart-io does not subscribe to json-h2c
[info] skip: dart-io does not subscribe to baseline-h3
[info] skip: dart-io does not subscribe to static-h3
[info] skip: dart-io does not subscribe to gateway-64
[info] skip: dart-io does not subscribe to gateway-h3
[info] skip: dart-io does not subscribe to production-stack
[info] skip: dart-io does not subscribe to unary-grpc
[info] skip: dart-io does not subscribe to unary-grpc-tls
[info] skip: dart-io does not subscribe to stream-grpc
[info] skip: dart-io does not subscribe to stream-grpc-tls
[info] skip: dart-io does not subscribe to echo-ws
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536
[info] restoring CPU governor → powersave

github-actions · 2026-05-06T09:07:52Z

Benchmark Results

Framework: dart-zig | Test: all tests

Test	Conn	RPS	CPU	Mem	Δ RPS	Δ Mem
baseline	512	1,353,265	6789.6%	2.5GiB	NEW	NEW
baseline	4096	1,665,927	6764.4%	2.5GiB	NEW	NEW
pipelined	512	1,364,400	6789.1%	2.5GiB	NEW	NEW
pipelined	4096	1,477,162	6464.4%	2.6GiB	NEW	NEW
json	4096	288,008	6237.6%	2.6GiB	NEW	NEW

Full log

  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   44.13ms   34.80ms   67.90ms   84.10ms   905.90ms

  7404243 requests in 5.00s, 7385811 responses
  Throughput: 1.48M req/s
  Bandwidth:  147.88MB/s
  Status codes: 2xx=7385811, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 7385792 / 7385811 responses (100.0%)
[info] CPU 6464.4% | Mem 2.6GiB

[run 2/3]
gcannon v0.5.3
  Target:    localhost:8080/pipeline
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   46.41ms   37.10ms   70.30ms   96.20ms    1.11s

  7037167 requests in 5.00s, 7035118 responses
  Throughput: 1.41M req/s
  Bandwidth:  140.86MB/s
  Status codes: 2xx=7035118, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 7035102 / 7035118 responses (100.0%)
[info] CPU 5730.3% | Mem 2.2GiB

[run 3/3]
gcannon v0.5.3
  Target:    localhost:8080/pipeline
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  16
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency      0us      0us      0us      0us      0us

  16 requests in 5.00s, 0 responses
  Throughput: 0 req/s
  Bandwidth:  154B/s
  Status codes: 2xx=0, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 0 / 0 responses (0.0%)
  Reconnects: 4858983
  Errors: connect 4859120, read 0, timeout 0
[info] CPU 0.0% | Mem 0MiB

=== Best: 1477162 req/s (CPU: 6464.4%, Mem: 2.6GiB) ===
[info] saved results/pipelined/4096/dart-zig.json
httparena-bench-dart-zig
httparena-bench-dart-zig

==============================================
=== dart-zig / limited-conn / 512c (tool=gcannon) ===
==============================================
[info] waiting for server...
[warn] dart-zig did not come up for limited-conn; skipping
httparena-bench-dart-zig
httparena-bench-dart-zig

==============================================
=== dart-zig / limited-conn / 4096c (tool=gcannon) ===
==============================================
[info] waiting for server...
[warn] dart-zig did not come up for limited-conn; skipping
httparena-bench-dart-zig
httparena-bench-dart-zig

==============================================
=== dart-zig / json / 4096c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 7
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   13.70ms   5.53ms   7.79ms   307.40ms   521.50ms

  1433640 requests in 5.00s, 1433643 responses
  Throughput: 286.62K req/s
  Bandwidth:  988.84MB/s
  Status codes: 2xx=1433643, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 1433641 / 1433643 responses (100.0%)
  Reconnects: 56869
  Per-template: 204239,204638,205070,205225,205098,204459,204912
  Per-template-ok: 204239,204638,205070,205225,205098,204459,204912
[info] CPU 6189.7% | Mem 2.6GiB

[run 2/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 7
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   13.67ms   5.53ms   8.04ms   316.90ms   517.40ms

  1435229 requests in 5.00s, 1435101 responses
  Throughput: 286.90K req/s
  Bandwidth:  990.09MB/s
  Status codes: 2xx=1435101, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 1435094 / 1435101 responses (100.0%)
  Reconnects: 56777
  Per-template: 204083,204773,204865,205727,205880,205076,204690
  Per-template-ok: 204083,204773,204865,205727,205880,205076,204690
[info] CPU 6332.6% | Mem 2.6GiB

[run 3/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 7
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   13.65ms   5.58ms   8.14ms   297.50ms   508.90ms

  1440229 requests in 5.00s, 1440040 responses
  Throughput: 287.91K req/s
  Bandwidth:  992.20MB/s
  Status codes: 2xx=1440040, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 1440026 / 1440040 responses (100.0%)
  Reconnects: 57098
  Per-template: 204735,205668,206085,206774,206780,205736,204247
  Per-template-ok: 204735,205668,206085,206774,206780,205736,204247
[info] CPU 6237.6% | Mem 2.6GiB

=== Best: 288008 req/s (CPU: 6237.6%, Mem: 2.6GiB) ===
[info] input BW: 13.73MB/s (avg template: 50 bytes)
[info] saved results/json/4096/dart-zig.json
httparena-bench-dart-zig
httparena-bench-dart-zig
[info] skip: dart-zig does not subscribe to json-comp
[info] skip: dart-zig does not subscribe to json-tls
[info] skip: dart-zig does not subscribe to upload
[info] skip: dart-zig does not subscribe to api-4
[info] skip: dart-zig does not subscribe to api-16
[info] skip: dart-zig does not subscribe to static
[info] skip: dart-zig does not subscribe to async-db
[info] skip: dart-zig does not subscribe to crud
[info] skip: dart-zig does not subscribe to fortunes
[info] skip: dart-zig does not subscribe to baseline-h2
[info] skip: dart-zig does not subscribe to static-h2
[info] skip: dart-zig does not subscribe to baseline-h2c
[info] skip: dart-zig does not subscribe to json-h2c
[info] skip: dart-zig does not subscribe to baseline-h3
[info] skip: dart-zig does not subscribe to static-h3
[info] skip: dart-zig does not subscribe to gateway-64
[info] skip: dart-zig does not subscribe to gateway-h3
[info] skip: dart-zig does not subscribe to production-stack
[info] skip: dart-zig does not subscribe to unary-grpc
[info] skip: dart-zig does not subscribe to unary-grpc-tls
[info] skip: dart-zig does not subscribe to stream-grpc
[info] skip: dart-zig does not subscribe to stream-grpc-tls
[info] skip: dart-zig does not subscribe to echo-ws
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536
[info] restoring CPU governor → powersave

MDA2AV · 2026-05-06T09:36:28Z

@MDA2AV I was trying to bring your or any other contributors attention to this pr

I have not created this pr for adding new backends or merging to main. I had created it just for benchmarking the difference between dart io and an experimental version where all the http parsing happens in zig.

What was the motivation behind it? As dart is progressing deep into backend dev and need for same arch on frontend and backend increases, the demand for more scalable dart backends increases. So this is just an experimental framework, whose aim is for a proof of work and to raise issue in dart repo for a server side runtime. And i am to keep this pr as a record of benchmark between both runtimes via a valid benchmarking framework

Why do i call it experimental? It works fine and handles 2.5x more request per sec but is not handles in prod niether its testing cover is that good and also it only fully covers linux http 1 calls.

If the current motive of httparena does not match the motives of this pr to store benchmarking record diffs for experimental projects , just notify me , i will be happy to arrange another sources for generating and storing benchmarking reords

Yea this is perfectly fine and what exactly the engine section is for

kartikey321 added 3 commits May 3, 2026 11:53

fix fletch type to production

015fe32

dart-zig: add AOT + cluster runtime track and dart-io baseline framework

ab5c8a1

dart-io: align with fletch-style stable AOT docker flow

53d7a50

kartikey321 requested review from Kaliumhexacyanoferrat and MDA2AV as code owners May 6, 2026 03:33

dart-io: fix fletch-style entrypoint binary path

1bc8f0b

This was referenced May 9, 2026

Discussion: Server-runtime profile for Dart backend hot paths (dart-zig experiment) dart-lang/sdk#63352

Open

fix server_native callback wakeups and request uri handling kingwill101/routed#12

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmark Track: Stable dart:io (AOT, fletch-style) vs Experimental dart-zig#692

Benchmark Track: Stable dart:io (AOT, fletch-style) vs Experimental dart-zig#692
kartikey321 wants to merge 4 commits intoMDA2AV:mainfrom
kartikey321:dart-zig-aot-cluster

kartikey321 commented May 6, 2026

Uh oh!

kartikey321 commented May 6, 2026

Uh oh!

github-actions Bot commented May 6, 2026

Uh oh!

kartikey321 commented May 6, 2026

Uh oh!

github-actions Bot commented May 6, 2026

Uh oh!

kartikey321 commented May 6, 2026

Uh oh!

github-actions Bot commented May 6, 2026

Uh oh!

github-actions Bot commented May 6, 2026

Uh oh!

MDA2AV commented May 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

kartikey321 commented May 6, 2026

Purpose

What This PR Changes

How dart-zig Differs from dart:io

Benchmark Scope (current)

Validation

Notes

Uh oh!

kartikey321 commented May 6, 2026

Uh oh!

github-actions Bot commented May 6, 2026

Uh oh!

kartikey321 commented May 6, 2026

Uh oh!

github-actions Bot commented May 6, 2026

Uh oh!

kartikey321 commented May 6, 2026

Uh oh!

github-actions Bot commented May 6, 2026

Benchmark Results

Uh oh!

github-actions Bot commented May 6, 2026

Benchmark Results

Uh oh!

MDA2AV commented May 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

How `dart-zig` Differs from `dart:io`