Benchmarks
Measured engine and gateway performance — direct-engine parity, concurrency scaling, tail latency, and lifecycle — with receipts.
How we benchmark
Every number on this page comes from a scripted, reproducible campaign against real engine binaries on real hardware — direct engine versus the same engine behind the Blazar gateway, identical model, quantization, context, sampling, and cache state. The full receipt tables, failed cells, and caveats live in BENCHMARK.md; this page shows the campaign charts and headline results.
Test bed (campaign 2026-09-29): Intel Core i7-14650HX (24 threads) · NVIDIA GeForce RTX 4070 Laptop, 8 GiB, driver 580.173.02 · 16 GiB RAM · Linux Mint 22.3, kernel 7.0.0-31-generic · model Qwen3.5-9B-Q4_K_M.gguf. Gateway measured at blazar 0.13.0; ratios travel better than absolutes.
Headline results
- ~2% gateway overhead — 40.6 t/s through the gateway vs 41.5 t/s direct
- 104.8 t/s aggregate at C=4 (40.0 → 65.9 → 104.8 across the sweep)
- 554 ms TTFT p99 at C=4 under sustained bursts
- 117 ms tool-call TTFT p50 vs 284 ms (ollama) — 2.4×
- 3.63 s idle-wake to first token
- 49.1 → 63.0 t/s adaptive reshape under sustained load, zero failed requests
Flagship GGUF campaign — Qwen3.5-9B-Q4_K_M
Single-stream decode throughput
Gateway overhead
Concurrency scaling
Concurrency tail latency
Resource cost vs concurrency
Long-context degradation
VRAM vs context
Cold start and idle wake
SGLang safetensors campaign
The same harness against the SGLang lane on quantized safetensors — a different engine, the same gateway behavior.
Reproduce
The harness and per-campaign reports ship in the repo
under bench-artifacts/ (summary layer: reports + INDEX.md); raw
cells (JSONL), charts, and media payloads live on the
bench-archive release.
The full methodology, checker-verified quality
suites, and every caveat are documented in
BENCHMARK.md.