10. SCIENTIFIC & NUMERICAL METHODS
Verified against Blazar v0.6.1 source (crates/blazar-core/src/gguf.rs, hardware.rs, engine_kind.rs, store.rs). Commands and configuration spot-verified against v0.19.0 (2026-10-03). INFERRED tags mark reasoned statements.
Algorithms
| Field |
Value |
| Functions |
parse_metadata(buf) -> (GgufMeta, usize) (gguf.rs:567), read_metadata_file(path) (gguf.rs:703) |
| Structures |
GgufMeta (gguf.rs:18), AttentionClass enum (gguf.rs:127), GgufValue enum (gguf.rs:368) |
| Source reference |
GGUF container format as implemented by ggml-org/llama.cpp (de-facto standard; no paper) |
| Algorithm |
binary walk of the GGUF header + metadata KV pairs into GgufValue variants; extracts architecture, block/layer counts, attention class, context length (ctx_train), parameter count, projector presence |
| Input assumptions |
valid GGUF file (magic GGUF); partial headers tolerated to the extent llama.cpp tolerates them |
| Output |
GgufMeta driving arch classification, KV sizing, and the models table columns (arch, params, ctx_train, shards, mmproj_path) |
| Numerical handling |
integer fields read at native width; no float math beyond passthrough |
| Limitations |
follows upstream format quirks; unknown architecture names still parse but classify generically |
2. KV-cache VRAM charge estimation
| Field |
Value |
| Functions |
estimate_kv_vram_charge (full f16, lane-independent), scale_kv_by_cache_type (q8_0 → KV/2, q4_0 → KV/4) |
| Equation |
KV bytes ≈ 2 (K+V) × n_layer × n_kv_head × head_dim × ctx × bytes_per_element(bytes); loop architectures multiply layers by GgufMeta.num_loops (clamped 1..=8) — e.g. nanbeige-class num_loops = 2 → child n_layer 44 = 22 × 2 |
| Units |
bytes (MiB convenience at display) |
| Valid domain |
ctx > 0; standard transformer KV layout |
| Known limitations |
f16 baseline only pre-scaling; MoE/MLA-style layouts follow the child’s own reporting (INFERRED for exotic layouts) |
3. Unified-KV context verdict
| Field |
Value |
| Function |
unified_ctx_verdict → {Fit, Refuse} |
| Equation |
Refuse iff weights + scale_kv_by_cache_type(KV_f16) + 700 MiB overhead > VRAM; 85–100 % occupancy zone = serve with fitter-owned gpu-layers + warning; floor check 2a-bis: (weights + 64) × 20 / 17 |
| Output units |
bytes vs bytes; verdict enum |
| Semantics |
budget-validated BEFORE spawn; unhostable requests get a single teaching 400 naming four levers (ctx, cache_type, gpu-layers, model size) |
| Limitations |
device pool from hardware.rs probe (Hardware :56, GpuInfo :5); concurrent-instance contention handled by supervisor census (invalidated at evict) |
Note: --kv-unified is one DEVICE-backed shared buffer (“CUDA0 KV buffer”), not host memory; --cache-ram caps the weights mmap.
4. Quantized-safetensors lane detection
| Field |
Value |
| Constant |
QUANTIZED_SAFETENSORS_TOKENS = [awq, gptq, fp8] (store.rs) |
| Algorithm |
signal = NOT .gguf path AND any word-boundary token of name/repo/path (lowercased, split on non-alphanumeric) ∈ tokens |
| Classification boundary |
word boundary is load-bearing: hawk stays clean (substring awq must not match); GGUF paths never match |
| Why |
models.quant is unreliable for lane choice (AWQ pulls record 4BIT, fp8-dynamic records BF16) |
5. Engine routing classification
| Field |
Value |
| Functions |
route_format(shape: FormatShape, installed, policy) (shape = FormatShape { diffusion, shards: ShardFormat }; ShardFormat::detect(is_dir, quantized, mlx) folds the routing cascade), serving_lane(...), missing_kinds() (engine_kind.rs) |
| Decision table |
MLX quant dirs → [Mlx] only (checked first — quantized-safetensors shape); quantized safetensors → [Sglang] only; GGUF → [LlamaCpp, MistralRs]; plain safetensors → Quality|Throughput [Sglang, MistralRs], Latency [MistralRs, Sglang]; diffusion component sets → [SdCpp] only (text rows never route to SdCpp — bidirectional domain gate) |
| Boundary rationale |
mistral.rs dequant of AWQ/GPTQ/FP8 produces empty 200s / garbage f16 (live-proven against upstream v0.9.3) — exclusion is correctness, not preference |
| Missing candidates |
teaching error (byte-identical strings are contract) |
6. Cache/spec hint estimation (EWMA)
| Field |
Value |
| Function |
hint_task (gateway, 60 s cadence) |
| Equation |
EWMA with α = 0.7 / 0.3 over Prometheus counters llamacpp:prompt_tokens_cached_total, prompt_tokens_total, spec_decode_num_accepted, draft_tokens_total scraped from each child’s /metrics |
| Output |
supervisor CacheHints{cache_hint, spec_accept} — dimensionless ratios in [0,1] |
| Limitations |
counter-based (monotonic resets handled by scrape semantics); child-metric availability dependent |
7. Hardware probing
| Field |
Value |
| Structures |
GpuInfo (hardware.rs:5), Hardware (hardware.rs:56) |
| Output |
GPU list (name/VRAM), system RAM — feeds max_loaded_models auto, cache_ram_mb 30 % RAM cap, fit previews, devices config |
Unit conventions
- Memory: bytes internally; MiB/GiB at user surfaces; VRAM percentages relative to probed total.
- Time: unix seconds in the store (
installed_at, pulled_at, ts); nanoseconds for load_duration; milliseconds for queue deadlines (x-blazar-deadline-ms); seconds for config durations.
- Tokens: engine-reported token counts; body-size heuristic
PREFILL_HEAVY_BYTES assumes ~4 bytes/token.
- Throughput: tokens/second (
tg_tokens_per_sec decode, pp_tokens_per_sec prefill) in bench_history.
Coordinate conventions
Not applicable — no spatial/coordinate systems. (GGUF tensor layouts follow upstream ggml conventions verbatim.)
Validation datasets and golden outputs
- Engine update gate: decode-throughput regression > 10 % vs incumbent → auto-rollback (measured by the bundled bench against stub/real engines, results persisted in
bench_history).
- Quantize gate:
--verify runs llama-perplexity with --max-degradation (default 10.0 %).
- Repository E2E harness
scripts/validate.py drives the full gateway against stub engines (tests/fixtures/stub_llama_server.rs, stub_llama_bench.rs, stub_llama_quantize.rs, test-util feature) — internal golden harness. (Tolerance tables internal to the harness; not part of the product interface.)
- Live-measured calibrations embedded in code/comments: multi-slot greedy flip rate (14/20 probes) motivating
deterministic; cache_ram_mb unclamped failure mode; mistral.rs batch-cap failure mode. These are qualitative citations of in-repo evidence, not published benchmarks.
Classification boundary logic and decision criteria
- Format boundary:
.gguf vs safetensors (path suffix) — first split of all routing.
- Quantization boundary: token-based detection (algorithm 4) — never the
quant column.
- Occupancy boundary (algorithm 3): Refuse > VRAM; warn-zone 85–100 %; floor
2a-bis.
- Policy boundary: routing order within candidate lists by
RoutingPolicy (Quality default).
- Priority boundary (gateway): body > 64 KiB demotes one class; explicit deadline overrides WFQ.
- Similarity boundary: semantic cache hit iff cosine ≥
threshold (default 0.90) on L2-normalized prompt vectors, same model+key+lane.
Improve this page on GitHub