[Samiii777]

Summary

Review and overhaul of ML-Bench. The JSON result block each script prints now decides PASS/FAIL. Every number is labelled with the precision that actually ran and backed by a correctness check. Models and reporting are updated, and there is now a test suite.

Correctness fixes

• A benchmark that failed its own validation was still reported as PASS; now PASS/FAIL/SKIP comes from the script's result. Non-zero exit, missing result or timeout is FAIL, and a timeout kills the script's whole process group.
• The report invented memory numbers when nothing was measured. avg_latency_ms was really 1/throughput. Reports now show per-batch p50/p99 latency and flag noisy runs (CV > 10%).
• Precision labels match what ran:
  • fp32 turns TF32 off for both matmul and cuDNN.
  • tf32 is now an explicit opt-in.
  • bf16 runs real bf16.
  • Training reports AMP precision as amp_fp16/amp_bf16.
• GPU kernel benchmarks:
  • They now emit the GFLOPS/GB/s values the report reads, and count FLOPs and bytes correctly.
  • Small kernels are timed in batches so launch overhead doesn't dominate.
  • Bandwidth tests use tensors bigger than any GPU cache.
  • GEMM results are checked against an fp64 reference.
• Training measures GPU optimizer steps on a batch already on the GPU; before, it was limited by CPU image augmentation.
• LLM benchmarks:
  • Prompt and output lengths are fixed, so no padding is miscounted.
  • Time to first token, prefill and decode speed are measured separately, with a perplexity sanity check.
  • Ollama gets a warmup and unique prompts so its cache can't inflate prefill.
  • llama.cpp is read from llama-bench -o json.
• ComfyUI:
  • Reports device-wide memory; it previously measured the wrong process and got about 0.
  • Workflow errors now fail the run.
  • Server output is drained so the server can't stall.
• Silent fallbacks removed:
  • YOLO no longer swaps in a ResNet when loading fails.
  • The insecure ONNX download and dummy-model fallback are gone.
  • An ONNX provider that's listed but unusable becomes a SKIP with a reason.

Workloads

• YOLO11, Qwen3 (1.7B/8B), Llama-3.1-8B, SD1.5 (new Hugging Face location), SD3.5-Medium, FLUX.2-klein-4B and Z-Image-Turbo.
• One generic torchvision script for classification and training, and one gpu_ops script.
• llama.cpp is now connected to the main runner.

Reporting

• Each run writes JSON, CSV and Markdown, and appends to history.jsonl.
• compare.py handles metrics where lower is better and exits non-zero on regressions, so CI can use it.
• Dashboard rebuilt on the history file.

Tests

• pytest covers harness math, FLOP/byte accounting, orchestrator status rules, reports, and the Ollama and llama.cpp result parsing.
• GPU smoke tests run when a GPU is present.
• CI runs the suite plus an end-to-end CPU run.

Validation (RTX 4090)

• Default matrix: 48/48 PASS. Full pytest suite: 45 passed.
• fp16 GEMM 161.6 TFLOPS (spec 165). Memory copy about 920 GB/s (spec 1008). Llama-3.1-8B bf16 decode 52 tok/s (a pure memory-bandwidth limit would be about 63).

🤖 Generated with Claude Code