Documentation

Architecture

Crate layout, daemon design, and the request lifecycle.

2. ARCHITECTURE

Verified against Blazar v0.6.1 source. Commands and configuration spot-verified against v0.19.0 (2026-10-03). INFERRED tags mark reasoned-but-not-directly-read statements.

Architectural style and rationale

Style: modular monolith (4-crate Cargo workspace) with a supervisor-orchestrated process pool. The blazar binary is CLI + daemon + gateway in one; inference itself is delegated to spawned child processes (official upstream engine binaries).

Rationale:

Zero-fork engine policy — official upstream llama-server, mistral.rs, sglang, sd-server binaries (and the mlx-lm/sglang pip venvs) are downloaded, sha256-verified, and run unmodified side-by-side (versioned tags with update/use/rollback). Engine fixes and quant formats arrive without porting work.
Crash isolation — each model runs in its own child process; a GPU driver fault or engine crash cannot take down the gateway. The supervisor restarts children on demand (next request respawns).
Translation, not emulation — the gateway owns wire-format translation (OpenAI ⇄ ollama ⇄ Anthropic ⇄ child dialects) so children stay stock.

Crate layout:

Crate	Responsibility
`blazar-core`	`config.rs` (TOML config model + validation), `store.rs` (SQLite), `catalog.rs`, `gguf.rs` (GGUF metadata parser), `profile.rs` (per-(model,engine) launch argv compiler), `hardware.rs` (GPU/RAM probe), `telemetry.rs` (tracing init)
`blazar-runtime`	engine installer + HF client, `supervisor.rs` (child lifecycle/eviction/circuit breakers/session bank), event bus, bench, quantize, daemon
`blazar-gateway`	`lib.rs` router + middleware stack, OpenAI (`openai.rs`), ollama (`ollama.rs`), Anthropic (`anthropic.rs`) surfaces, `queue.rs` (priority/EDF/WFQ), `sessions.rs`, semcache (semantic cache), `sentinel.rs` (quality observation), `whisper` lane, batch API, `keys.rs` (API key management), `proxy.rs`/`translate.rs`
`blazar-cli`	`main.rs` — the `blazar` binary; every user-facing verb

Request/response lifecycle

A non-streaming POST /v1/chat/completions passes through:

request_log (outermost): assigns/echoes trace id plm-<hex>-<seq> (header x-blazar-trace-id), emits access log (target blazar::access), opens OTLP span, emits audit line if audit lane active.
auth (iff any [[keys]] configured): resolves secret from Authorization: Bearer <secret> then x-api-key; failure → 401 {"error":{"message":"missing or invalid API key (Authorization: Bearer or x-api-key)","type":"blazar_error","code":401}}. On success stamps KeyCtx{name,weight} on request extensions. Concurrency-cap gate (max_concurrent lease rides the response body; freed when the client drains or drops it) runs before scope/rpm/tpm checks.
CORS — from cors_origins config; empty list = no Access-Control-Allow-Origin ever; "*" = any. Methods GET/POST/DELETE; exposes x-blazar-trace-id. CORS sits logically outside auth so preflights always get answered.
DefaultBodyLimit — 50 MiB.
session pin middleware (sessions::pin_mw, innermost): x-blazar-session header pins the request to a named session.
Handler → priority queue admission (queue.rs): x-blazar-priority (case-insensitive high/low, else Normal), x-blazar-deadline-ms for strict EDF. Oversized prefill (>64 KiB body, ~16k tokens at 4 bytes/token) demotes one class. Bounded wait → 503 all_slots_busy.
Proxy core (proxy_core_chat, ollama.rs:1381): model resolution (incl. model:tag colon semantics and remote:model remote routing), routing decision (see below), ensures the child instance is spawned/healthy, translates OpenAI → child dialect, dispatches, translates the reply back.
- Upstream child failure → reap_dead_children + 502 (the next request respawns instead of a 502 loop).
- Non-2xx child status → api_error(status, engine text).
- Empty choices + error object in child body → 502 with the engine’s message (mistral.rs batch-step error class).
- sentinel_enforce on → sentinel judge runs before translation; violations → 422 sentinel enforce: [code] detail; ....
- Streaming: gateway forces stream_options.include_usage, converts child SSE → client-format incremental NDJSON with boundary-safe UTF-8 line buffering; merges OpenAI tool-call fragment deltas into complete tool_calls entries; measures TTFT (first byte) and TPOT (inter-chunk) itself because streaming children report no timings.
Observation close: sentinel observation feed ends (Drop), semantic-cache store path may run, load_duration (ns) injected when load took >100 ms.

Engine routing decision (engine_kind.rs::serving_lane): model override pin (exact tag → kind) first; manual mode → global active engine; auto mode → route_format matrix: MLX quant dirs → mlx lane only (they are quantized-safetensors-shaped; the mlx axis is checked first so SGLang never shadows them, MlxUnserved teaches when the lane is absent); quantized safetensors → SGLang only; GGUF → [LlamaCpp, MistralRs]; safetensors → policy-dependent order. Within a kind, mainstream engines outrank fork lanes (newest-first inside each class — LaneClass, plumbed via EngineRow::lane_class from the manifest source field), so a capability fork only ever receives the architectures mainstream lacks. An unknown-architecture death is classified from the child’s stderr (classify_unknown_arch, unicode-safe) and re-routed exactly once to an installed advertising lane; no advertiser → bounded 502 whose body names both the fork-build command and any curated registry lane. Missing engines produce teaching errors (byte-identical LaneError strings flow into API bodies) that name the install command, e.g. blazar engine install --kind <kind>.

Major modules/services and their responsibilities

Module	Responsibility
`gateway/src/lib.rs`	router(), middleware ordering, bind/TLS, background tasks, graceful shutdown
`gateway/src/queue.rs`	`PriorityQueue`: priority classes, EDF deadlines (SLO tiers HIGH=2 s, NORMAL=30 s, LOW=120 s), WFQ per-key fairness (credit += 1/weight, buckets reset at 256), 503 `all_slots_busy`
`gateway/src/keys.rs`	key CRUD + rotate via config.toml round-trip, redacted listings, last-key protection
`gateway/src/sessions.rs`	named session pins (`session_keep_secs`, default 900; 0 disables), session list, close via `POST /api/session {"action":"close"}`
`gateway` semcache	opt-in semantic cache: L2-normalized prompt vectors, cosine similarity ≥ threshold (default 0.90), same chat model + API key + lane only; TTL 600 s default; never crosses API keys; streams and structured-output requests bypass both directions
`gateway/src/sentinel.rs`	warn-only quality observation: truncation (`finish_reason:length`), tool-arg JSON/hallucinated-name/schema violations, `json_schema`/`json_object` validation, empty response, reasoning-no-answer, stalled stream (`sentinel_stall_secs`, default 30); child header-phase bound (`child_header_timeout_secs`, default 120; diffusion lanes ride `sdcpp_child_header_timeout_secs`, default 900, because a diffusion child writes headers only when the whole frame is computed): a wedged child that never writes response headers is evicted and the request retried once in-band (504 terminal);
- Admission is class-aware: every lane entry classifies the request (`classify_work`) into Interactive / Tool / RawLong (tool-bearing chat = Tool; ollama_compat raw completions = RawLong; image traffic on ollama_compat models rides the child lane and classifies normally). The queue orders by (priority, class rank, deadline) — Tool admits ahead of equal-decode Interactive, RawLong last — tightens Tool deadlines to a 10 s SLO, and caps RawLong concurrency at `raw_ceiling` (full borrowing while no interactive traffic waits; `max(1, N - ceil(N/4))` once it does), so raw generations cannot starve tool calls on multi-slot models. never alters response bytes; `sentinel_enforce` turns judge verdicts into 422 on non-streaming chat
`runtime/src/supervisor.rs`	child spawn (`spawn_instance`), instance state machine (incl. `InstanceState::Evicted`), eviction (`evict`, `evict_model` — releases session pins, evicts every key grammar: plain, `#replica`, `@vision`, combined), crash circuit breakers, session bank save/restore, `run/<model>.pid` + `run/<model>.apikey` lifecycle
`runtime` engine installer	sha256-verified engine download/verify, regression gate, `local` pseudo-tag, build lane (CUDA/CPU), prune/rollback; capability-lane lifecycle: bounded registry fetch (1 MiB cap), immutable pinned-SHA fork checkouts (verified after fetch), one-time architecture mining per upstream build (10 s per-request cap), supersede stamps on full architecture coverage, retirement sweep and manual `engine rm` sharing one aside-first removal primitive (aside → row delete → discard, restore on failure — never a live row over a deleted dir, never a zombie server on an unlinked tree)
`core/src/config.rs`	full config model, defaults, validation
`core/src/store.rs`	SQLite schema v4 + migrations
`core/src/gguf.rs`	GGUF metadata parser (`parse_metadata`, `read_metadata_file`), attention class, arch params, KV estimation inputs
`core/src/profile.rs`	per-(model,engine) launch argv compiler from config + overrides + GGUF metadata
`cli/src/main.rs`	all CLI verbs, REPL, daemon control

Authentication and authorization flow

Auth optional by design. Auth is active iff any [[keys]] entry exists. Without keys the gateway is open (local-only trust model).
Open paths (never auth-gated): /healthz, /health.
Secret sources, in priority order: Authorization: Bearer <secret>, then x-api-key (Anthropic convention).
Scopes: a key with non-empty models list is scoped to those models; empty models = full access + admin semantics (can manage keys).
Key management authorization (management_authorized): an authless gateway refuses management (403 no keys configured: add the first [[keys]] entry in config.toml (or BLAZAR_KEYS env) and restart); only unscoped keys manage keys.
Quotas: max_concurrent (lease at auth, held until response body drained/dropped — held SSE streams are a bounded resource), then scope, then rpm/tpm (tpm counts prompt+completion tokens on chat-family routes), daily_tokens per UTC day. weight feeds WFQ admission under contention.
Child auth: per-child API keys (run/<model>.apikey) are auto-generated; TCP children are authenticated, UDS children are not (child_auth config default auto).
Remotes: [[remotes]] entries carry optional upstream keys; local gateway key still gates the client.

State management strategy

State	Location	Notes
Config	`~/.config/blazar/config.toml`	single source; runtime key mutations do a full parse→swap→validate→atomic-write round-trip (temp+rename, timestamped `.bak`) under a process-wide `PERSIST_LOCK`; registry rollback on persist failure
Durable catalog	SQLite `<data_dir>/blazar.db` (WAL)	engines, models, profiles, loras, key_usage, bench_history
Model weights	`<data_dir>/models/`	plain files, multi-shard GGUF native; zero-copy hardlink aliasing (`blazar cp`)
Runtime children	supervisor `DashMap` of instances	states incl. `Evicted`; census cache for free VRAM (invalidated on evict)
Sessions	`<data_dir>/sessions/`	slot KV checkpoints; `_auto-<ctx>` banks saved on evict (bounded 2 s, ≤512 MiB entries kept), restored on respawn, ctx-scoped so never restored across ctx shapes
Spec caches	`<data_dir>/speccache/`	persistent ngram lookup-cache-dynamic per model
Key usage	`key_usage` table	whole-day absolute state written per batch in one txn
In-flight request state	gateway queue + `InFlightGuard`	concurrency leases ride response bodies

Concurrency style: single async tokio runtime; shared maps are dashmap/parking_lot; child signals sent to single PIDs only (the supervisor never signals a process group).

Background jobs, queues, schedulers, workers

Job	Cadence	Behavior
`hint_task`	every 60 s	scrapes each child’s `/metrics` Prometheus counters (`llamacpp:prompt_tokens_cached_total`, `prompt_tokens_total`, spec-decode `spec_decode_num_accepted`/`draft_tokens_total`), EWMA 0.7/0.3, feeds supervisor `CacheHints` (`cache_hint`, `spec_accept`)
`otlp_task`	every 10 s	bounded POST batches to `otlp_endpoint` (opt-in)
engine check	`engine_check_secs` (default from `default_engine_check_secs()`; 0 = off)	check-only update availability check; never auto-installs; surfaces in doctor/ps/log
idle reaper	`idle_timeout_secs` (1800) / `idle_sleep_secs` (300)	evicts idle children; `keep_alive` honored from both APIs (N seconds, 0 = immediate, -1 = forever); `/api/ps` shows countdown
whisper lane	per-request	own model lifecycle; `whisper.shutdown()` on gateway shutdown
whisper idle reaper	60 s tick	kills the lazy whisper-server child after `whisper_idle_secs` (900) without traffic; 0 keeps it until teardown; the child respawns on the next request
audit drain	continuous	`AuditLine{ts,trace,key,method,path,model,status,ms,priority,queue_depth}` via try_send; saturation increments `audit_dropped` (never blocks the hot path); file rotates at 16 MiB
semantic cache TTL	per-entry ttl (600 s default)	eviction

Failure-mode diagram

client ──▶ gateway ──▶ queue(503 all_slots_busy on bounded wait)
              │
              ├─ child spawn fails ──▶ teaching error names install command
              ├─ child HTTP !2xx ──▶ api_error(status, engine text)
              ├─ child dies mid-request ──▶ reap_dead_children + 502;
              │                               next request respawns child
              ├─ child returns empty choices + error ──▶ 502 (engine msg)
              ├─ sentinel enforce violation (non-stream) ──▶ 422
              ├─ prompt unfit for ctx (preflight) ──▶ 400 (teaching,
              │        names levers; unified-KV unhostable = one 400
              │        naming four levers)
              ├─ late_chunking payload > late_chunking_max_tokens ──▶ 413
              ├─ auth/quota ──▶ 401 / 403 / 429-class rejections
              └─ bind conflict (port in use) ──▶ exit code 3
                        (EXIT_BIND_CONFLICT; systemd RestartPreventExitStatus)

Child crash handling: circuit breakers per instance; blazar ps --reset clears breakers; clean teardown resets crash history (unclean_dead removed on clean exit). A sleeping llama-server that ignores SIGTERM until grace expiry gets SIGKILL; the follow-up “engine crashed (signal 9)” ERROR log line is a known cosmetic misclassification (breaker-safe) — documented intentional.

Error handling strategy

Fail fast, teach: errors carry actionable remediation (missing engine errors include the exact install command + installed roster; unfit prompts name the levers: ctx, cache_type, kv quant, gpu layers).
Stable machine-readable shapes: OpenAI-style {"error":{"message","type","code"}}; 401 example above; 422 sentinel shape sentinel enforce: [code] detail; ...; 503 all_slots_busy with human wait string (Ns whole seconds else millis).
LaneError parity contract: routing error Display strings are byte-identical across CLI and API because the same strings flow into both — changing wording is a breaking change.
No silent fallbacks: quantized-safetensors models never silently fall back to a broken engine lane (mistral.rs dequant is known-broken on AWQ/GPTQ/FP8 — route to SGLang only, teach if absent).
Crash-proof hot path: audit drops are counted, never blocking; OTLP posts are bounded.
Exit codes: bind conflict exits 3 so systemd does not restart-loop into a permanent port war.

Improve this page on GitHub