Skip to content

Add web-framework-cc, web-framework-cpp, web-framework-csharp, web-framework-python#699

Open
LazyPanda07 wants to merge 40 commits into
MDA2AV:mainfrom
LazyPanda07:add-web-framework
Open

Add web-framework-cc, web-framework-cpp, web-framework-csharp, web-framework-python#699
LazyPanda07 wants to merge 40 commits into
MDA2AV:mainfrom
LazyPanda07:add-web-framework

Conversation

@LazyPanda07
Copy link
Copy Markdown

Description

Hi. My WebFramework supports few languages, so I add tests for them all.


PR Commands — comment on this PR to trigger (requires collaborator approval):

Command Description
/benchmark -f <framework> Run all benchmark tests
/benchmark -f <framework> -t <test> Run a specific test
/benchmark -f <framework> --save Run and save results (updates leaderboard on merge)

Always specify -f <framework>. Results are automatically compared against the current leaderboard.


Run benchmarks locally

You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.

./scripts/validate.sh <framework>
./scripts/benchmark-lite.sh <framework> baseline
./scripts/benchmark-lite.sh --load-threads 4 <framework>

Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk, ghz) are built as self-contained Docker images on first run.

@MDA2AV
Copy link
Copy Markdown
Owner

MDA2AV commented May 11, 2026

/benchmark -f web-framework-cc

@github-actions
Copy link
Copy Markdown
Contributor

👋 /benchmark request received. A collaborator will review and approve the run.

@github-actions
Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: web-framework-cc | Test: all tests

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 59,770 99.7% 52MiB NEW NEW
baseline 4096 59,251 98.9% 43MiB NEW NEW
pipelined 512 66,932 99.0% 23MiB NEW NEW
pipelined 4096 66,758 98.9% 21MiB NEW NEW
limited-conn 512 7,834 97.2% 56MiB NEW NEW
limited-conn 4096 7,906 96.3% 55MiB NEW NEW
json 4096 1,306 116.5% 567MiB NEW NEW
upload 32 2 3.8% 14MiB NEW NEW
upload 256 7 3.4% 24MiB NEW NEW
static 1024 8,360 183.8% 43MiB NEW NEW
static 4096 8,394 186.1% 43MiB NEW NEW
static 6800 8,331 183.2% 45MiB NEW NEW
Full log
[info] skip: web-framework-cc does not subscribe to api-4
[info] skip: web-framework-cc does not subscribe to api-16

==============================================
=== web-framework-cc / static / 1024c (tool=wrk) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
Running 5s test @ http://localhost:8080
  64 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    37.83ms  118.15ms   1.98s    98.00%
    Req/Sec   605.96    319.44     3.36k    78.52%
  42500 requests in 5.08s, 2.52GB read
  Socket errors: connect 0, read 0, write 0, timeout 110
Requests/sec:   8360.91
Transfer/sec:    508.18MB
[info] CPU 183.8% | Mem 43MiB

[run 2/3]
Running 5s test @ http://localhost:8080
  64 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    37.92ms  117.65ms   1.99s    98.03%
    Req/Sec   589.02    320.68     3.48k    80.70%
  40466 requests in 5.09s, 2.40GB read
  Socket errors: connect 0, read 0, write 0, timeout 108
Requests/sec:   7947.15
Transfer/sec:    483.41MB
[info] CPU 183.7% | Mem 85MiB

[run 3/3]
Running 5s test @ http://localhost:8080
  64 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    38.60ms  117.30ms   1.99s    98.02%
    Req/Sec   570.35    310.17     3.24k    79.59%
  38465 requests in 5.08s, 2.28GB read
  Socket errors: connect 0, read 0, write 0, timeout 105
Requests/sec:   7567.90
Transfer/sec:    459.99MB
[info] CPU 180.5% | Mem 126MiB

=== Best: 8360 req/s (CPU: 183.8%, Mem: 43MiB) ===
[info] saved results/static/1024/web-framework-cc.json
httparena-bench-web-framework-cc
httparena-bench-web-framework-cc

==============================================
=== web-framework-cc / static / 4096c (tool=wrk) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
Running 5s test @ http://localhost:8080
  64 threads and 4096 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    28.23ms   66.24ms   1.98s    99.56%
    Req/Sec     1.20k   801.26     3.77k    62.57%
  42812 requests in 5.10s, 2.54GB read
  Socket errors: connect 0, read 0, write 0, timeout 107
Requests/sec:   8394.47
Transfer/sec:    510.37MB
[info] CPU 186.1% | Mem 43MiB

[run 2/3]
Running 5s test @ http://localhost:8080
  64 threads and 4096 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    22.79ms   74.03ms   1.99s    99.85%
    Req/Sec     1.24k     0.86k    2.92k    55.68%
  21933 requests in 5.10s, 1.30GB read
  Socket errors: connect 0, read 0, write 0, timeout 176
Requests/sec:   4300.13
Transfer/sec:    261.34MB
[info] CPU 140.1% | Mem 153MiB

[run 3/3]
Running 5s test @ http://localhost:8080
  64 threads and 4096 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us    -nan%
    Req/Sec     0.00      0.00     0.00      -nan%
  0 requests in 5.10s, 0.00B read
Requests/sec:      0.00
Transfer/sec:       0.00B
[info] CPU 104.7% | Mem 216MiB

=== Best: 8394 req/s (CPU: 186.1%, Mem: 43MiB) ===
[info] saved results/static/4096/web-framework-cc.json
httparena-bench-web-framework-cc
httparena-bench-web-framework-cc

==============================================
=== web-framework-cc / static / 6800c (tool=wrk) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    30.46ms   53.43ms   1.98s    99.75%
    Req/Sec     1.29k     0.94k    4.27k    63.14%
  42492 requests in 5.10s, 2.52GB read
  Socket errors: connect 0, read 0, write 0, timeout 96
Requests/sec:   8331.45
Transfer/sec:    506.72MB
[info] CPU 183.2% | Mem 45MiB

[run 2/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.55ms  775.45us   6.27ms   62.60%
    Req/Sec    13.50     12.02    22.00    100.00%
  141 requests in 5.10s, 9.17MB read
  Socket errors: connect 0, read 0, write 0, timeout 18
Requests/sec:     27.64
Transfer/sec:      1.80MB
[info] CPU 106.1% | Mem 165MiB

[run 3/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us    -nan%
    Req/Sec     0.00      0.00     0.00      -nan%
  0 requests in 5.10s, 0.00B read
Requests/sec:      0.00
Transfer/sec:       0.00B
[info] CPU 103.9% | Mem 228MiB

=== Best: 8331 req/s (CPU: 183.2%, Mem: 45MiB) ===
[info] saved results/static/6800/web-framework-cc.json
httparena-bench-web-framework-cc
httparena-bench-web-framework-cc
[info] skip: web-framework-cc does not subscribe to async-db
[info] skip: web-framework-cc does not subscribe to crud
[info] skip: web-framework-cc does not subscribe to fortunes
[info] skip: web-framework-cc does not subscribe to baseline-h2
[info] skip: web-framework-cc does not subscribe to static-h2
[info] skip: web-framework-cc does not subscribe to baseline-h2c
[info] skip: web-framework-cc does not subscribe to json-h2c
[info] skip: web-framework-cc does not subscribe to baseline-h3
[info] skip: web-framework-cc does not subscribe to static-h3
[info] skip: web-framework-cc does not subscribe to gateway-64
[info] skip: web-framework-cc does not subscribe to gateway-h3
[info] skip: web-framework-cc does not subscribe to production-stack
[info] skip: web-framework-cc does not subscribe to unary-grpc
[info] skip: web-framework-cc does not subscribe to unary-grpc-tls
[info] skip: web-framework-cc does not subscribe to stream-grpc
[info] skip: web-framework-cc does not subscribe to stream-grpc-tls

==============================================
=== web-framework-cc / echo-ws / 512c (tool=gcannon) ===
==============================================
[info] waiting for server...
[warn] web-framework-cc did not come up for echo-ws; skipping
httparena-bench-web-framework-cc
httparena-bench-web-framework-cc

==============================================
=== web-framework-cc / echo-ws / 4096c (tool=gcannon) ===
==============================================
[info] waiting for server...
[warn] web-framework-cc did not come up for echo-ws; skipping
httparena-bench-web-framework-cc
httparena-bench-web-framework-cc

==============================================
=== web-framework-cc / echo-ws / 16384c (tool=gcannon) ===
==============================================
[info] waiting for server...
[warn] web-framework-cc did not come up for echo-ws; skipping
httparena-bench-web-framework-cc
httparena-bench-web-framework-cc
[info] skip: web-framework-cc does not subscribe to echo-ws-pipeline
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-6800.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-32.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536
[info] restoring CPU governor → powersave

@MDA2AV
Copy link
Copy Markdown
Owner

MDA2AV commented May 11, 2026

/benchmark -f web-framework-cpp

@github-actions
Copy link
Copy Markdown
Contributor

👋 /benchmark request received. A collaborator will review and approve the run.

@github-actions
Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: web-framework-cpp | Test: all tests

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 58,093 98.8% 37MiB NEW NEW
baseline 4096 57,490 98.8% 25MiB NEW NEW
pipelined 512 64,016 99.1% 23MiB NEW NEW
pipelined 4096 62,116 98.5% 25MiB NEW NEW
limited-conn 512 7,828 96.5% 54MiB NEW NEW
limited-conn 4096 7,872 96.4% 53MiB NEW NEW
json 4096 1,554 117.4% 22MiB NEW NEW
upload 32 2 3.7% 13MiB NEW NEW
upload 256 7 2.9% 24MiB NEW NEW
static 1024 7,889 181.8% 84MiB NEW NEW
static 4096 8,247 186.1% 44MiB NEW NEW
static 6800 8,099 184.4% 42MiB NEW NEW
Full log
httparena-bench-web-framework-cpp
[info] skip: web-framework-cpp does not subscribe to api-4
[info] skip: web-framework-cpp does not subscribe to api-16

==============================================
=== web-framework-cpp / static / 1024c (tool=wrk) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
Running 5s test @ http://localhost:8080
  64 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    38.36ms  115.56ms   1.98s    98.09%
    Req/Sec   581.80    328.06     2.52k    80.47%
  39814 requests in 5.08s, 2.36GB read
  Socket errors: connect 0, read 0, write 0, timeout 103
Requests/sec:   7833.27
Transfer/sec:    476.04MB
[info] CPU 186.1% | Mem 43MiB

[run 2/3]
Running 5s test @ http://localhost:8080
  64 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    37.98ms  117.60ms   1.99s    98.03%
    Req/Sec   600.59    312.15     3.45k    82.09%
  40124 requests in 5.09s, 2.38GB read
  Socket errors: connect 0, read 0, write 0, timeout 109
Requests/sec:   7889.05
Transfer/sec:    479.78MB
[info] CPU 181.8% | Mem 84MiB

[run 3/3]
Running 5s test @ http://localhost:8080
  64 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    38.81ms  117.28ms   1.98s    98.01%
    Req/Sec   573.10    313.57     2.99k    80.57%
  37953 requests in 5.08s, 2.25GB read
  Socket errors: connect 0, read 0, write 0, timeout 104
Requests/sec:   7469.78
Transfer/sec:    454.25MB
[info] CPU 180.3% | Mem 127MiB

=== Best: 7889 req/s (CPU: 181.8%, Mem: 84MiB) ===
[info] saved results/static/1024/web-framework-cpp.json
httparena-bench-web-framework-cpp
httparena-bench-web-framework-cpp

==============================================
=== web-framework-cpp / static / 4096c (tool=wrk) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
Running 5s test @ http://localhost:8080
  64 threads and 4096 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    28.46ms   66.97ms   2.00s    99.55%
    Req/Sec     1.26k     0.91k    4.03k    64.88%
  42068 requests in 5.10s, 2.50GB read
  Socket errors: connect 0, read 0, write 0, timeout 106
Requests/sec:   8247.48
Transfer/sec:    501.38MB
[info] CPU 186.1% | Mem 44MiB

[run 2/3]
Running 5s test @ http://localhost:8080
  64 threads and 4096 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    22.25ms   68.76ms   1.99s    99.87%
    Req/Sec     1.26k     0.86k    2.79k    57.06%
  21294 requests in 5.10s, 1.26GB read
  Socket errors: connect 0, read 0, write 0, timeout 180
Requests/sec:   4175.01
Transfer/sec:    253.97MB
[info] CPU 139.3% | Mem 156MiB

[run 3/3]
Running 5s test @ http://localhost:8080
  64 threads and 4096 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us    -nan%
    Req/Sec     0.00      0.00     0.00      -nan%
  0 requests in 5.10s, 0.00B read
Requests/sec:      0.00
Transfer/sec:       0.00B
[info] CPU 104.9% | Mem 217MiB

=== Best: 8247 req/s (CPU: 186.1%, Mem: 44MiB) ===
[info] saved results/static/4096/web-framework-cpp.json
httparena-bench-web-framework-cpp
httparena-bench-web-framework-cpp

==============================================
=== web-framework-cpp / static / 6800c (tool=wrk) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    30.99ms   56.94ms   2.00s    99.73%
    Req/Sec     1.37k     0.95k    3.31k    54.93%
  41315 requests in 5.10s, 2.45GB read
  Socket errors: connect 0, read 0, write 0, timeout 91
Requests/sec:   8099.65
Transfer/sec:    492.36MB
[info] CPU 184.4% | Mem 42MiB

[run 2/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us    -nan%
    Req/Sec     0.00      0.00     0.00      -nan%
  0 requests in 5.10s, 0.00B read
Requests/sec:      0.00
Transfer/sec:       0.00B
[info] CPU 106.8% | Mem 163MiB

[run 3/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us    -nan%
    Req/Sec     0.00      0.00     0.00      -nan%
  0 requests in 5.10s, 0.00B read
Requests/sec:      0.00
Transfer/sec:       0.00B
[info] CPU 103.8% | Mem 224MiB

=== Best: 8099 req/s (CPU: 184.4%, Mem: 42MiB) ===
[info] saved results/static/6800/web-framework-cpp.json
httparena-bench-web-framework-cpp
httparena-bench-web-framework-cpp
[info] skip: web-framework-cpp does not subscribe to async-db
[info] skip: web-framework-cpp does not subscribe to crud
[info] skip: web-framework-cpp does not subscribe to fortunes
[info] skip: web-framework-cpp does not subscribe to baseline-h2
[info] skip: web-framework-cpp does not subscribe to static-h2
[info] skip: web-framework-cpp does not subscribe to baseline-h2c
[info] skip: web-framework-cpp does not subscribe to json-h2c
[info] skip: web-framework-cpp does not subscribe to baseline-h3
[info] skip: web-framework-cpp does not subscribe to static-h3
[info] skip: web-framework-cpp does not subscribe to gateway-64
[info] skip: web-framework-cpp does not subscribe to gateway-h3
[info] skip: web-framework-cpp does not subscribe to production-stack
[info] skip: web-framework-cpp does not subscribe to unary-grpc
[info] skip: web-framework-cpp does not subscribe to unary-grpc-tls
[info] skip: web-framework-cpp does not subscribe to stream-grpc
[info] skip: web-framework-cpp does not subscribe to stream-grpc-tls

==============================================
=== web-framework-cpp / echo-ws / 512c (tool=gcannon) ===
==============================================
[info] waiting for server...
[warn] web-framework-cpp did not come up for echo-ws; skipping
httparena-bench-web-framework-cpp
httparena-bench-web-framework-cpp

==============================================
=== web-framework-cpp / echo-ws / 4096c (tool=gcannon) ===
==============================================
[info] waiting for server...
[warn] web-framework-cpp did not come up for echo-ws; skipping
httparena-bench-web-framework-cpp
httparena-bench-web-framework-cpp

==============================================
=== web-framework-cpp / echo-ws / 16384c (tool=gcannon) ===
==============================================
[info] waiting for server...
[warn] web-framework-cpp did not come up for echo-ws; skipping
httparena-bench-web-framework-cpp
httparena-bench-web-framework-cpp
[info] skip: web-framework-cpp does not subscribe to echo-ws-pipeline
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-6800.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-32.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536
[info] restoring CPU governor → powersave

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants