diff --git a/edge0-strata/docs/BENCHMARK_PROTOCOL.md b/edge0-strata/docs/BENCHMARK_PROTOCOL.md index 949f9d8f..9a708344 100644 --- a/edge0-strata/docs/BENCHMARK_PROTOCOL.md +++ b/edge0-strata/docs/BENCHMARK_PROTOCOL.md @@ -3,6 +3,10 @@ **Status: draft, implemented in `strata bench run`; pilot validation pending.** Protocol ID: `strata-qwen38-thinking-v1-draft.1`. Result format: `strata-standard-benchmark`, schema version `1`. +> **Availability in this repository:** the `edge0-strata` build here has no `bench` subcommand, so +> `strata bench run` cannot be used yet. Use the community scripts in +> [COMMUNITY_BENCHMARKS.md](COMMUNITY_BENCHMARKS.md) until the runner is ported. + This document specifies a repeatable community benchmark and the data a future website can consume. The required suite took **2 to 4 minutes after model loading** on the two machines it has been run on (RTX 3060 and 2x RTX A4000, IQ3_XXS, Strata 0.2.1: 3 min 22 s and 2 min 22 s; the optional concurrency stage added 1.5 to 2 minutes). diff --git a/edge0-strata/docs/COMMUNITY_BENCHMARKS.md b/edge0-strata/docs/COMMUNITY_BENCHMARKS.md index 1c518d6b..b75e64b2 100644 --- a/edge0-strata/docs/COMMUNITY_BENCHMARKS.md +++ b/edge0-strata/docs/COMMUNITY_BENCHMARKS.md @@ -9,6 +9,9 @@ sampling, twelve sanity checks, about 2 to 4 minutes after the model has loaded the hardware, versions and every request. For a quick look at your own machine, [`BENCH.bat` / `./bench.sh`](#run-the-benchmark) takes 3 to 10 minutes and writes a report in the format below. +> The `edge0-strata` build in this repository has no `bench` subcommand yet, so the standard benchmark +> cannot be run here; the community scripts above are the way to measure for now. + The protocol is still a draft until a pilot has run on two kinds of hardware. The reports listed below were made before it existed and are not standard results; they stay valid under this guide.