Documentation

Setup

Installation, engines, models, and the configuration reference.

7. SETUP

Verified against Blazar v0.6.1 source. Commands and configuration spot-verified against v0.19.0 (2026-10-03). INFERRED tags mark reasoned statements.

Environment variables (NAME + PURPOSE only)

Product runtime (read by blazar itself)

Variable	Purpose
BLAZAR_KEYS	bootstrap API keys into config at startup
BLAZAR_REGISTRY_BASE	override the model registry base URL
BLAZAR_REGISTRY_TOKEN	access token for private HF namespaces
BLAZAR_GH_BASE	override the GitHub base URL for engine downloads
BLAZAR_REPO	override the release repository for downloads
BLAZAR_INSTALL_BASE_URL	override the engine artifact download base URL
BLAZAR_ENGINE_PATH	point at an engine binary location
BLAZAR_SYSTEMCTL	override the systemctl binary used for service management
BLAZAR_CAPABILITY_REGISTRY	override (or, when empty, disable) the capability-lane registry URL
BLAZAR_ENGINE_GATE	influence the engine update regression gate
BLAZAR_HOST	target host for CLI client verbs against a running daemon
BLAZAR_DEFAULT_CTX	override the default context size
BLAZAR_CHILD_TRANSPORT	override child_transport (tcp / unix)
RUST_LOG	log filter; wins over config log_level (escape hatch)

Installer (read by scripts/install.sh / bootstrap.sh / uninstall.sh)

BLAZAR_REPO, BLAZAR_VERSION, BLAZAR_INSTALL_BASE_URL, BLAZAR_INSTALL_ENGINE, BLAZAR_INSTALL_MODEL, BLAZAR_BOOTSTRAP, BLAZAR_FORCE_BOOTSTRAP, BLAZAR_AUTO_BOOTSTRAP, BLAZAR_AUTO_DRIVER, BLAZAR_CHECKOUT, BLAZAR_SUDO, BLAZAR_SYSTEM_BIN_DIR, BLAZAR_SYSTEMCTL, BLAZAR_UNIT_PATH, BLAZAR_UNIT_MEMORY_HIGH, BLAZAR_SERVICE_USER, BLAZAR_SERVICE_GROUP — all override the corresponding installer default (release source, version pin, download base, bootstrap behavior, driver install, checkout dir, privilege helper, unit-file location/memory limit, service user/group).

Internal test harness (NOT part of the product interface)

A BLAZAR_VALIDATE* / BLAZAR_STUB_* variable family plus BLAZAR_TEST_MTP_GGUF exists for the repository’s own validation harness (scripts/validate.py). Values and semantics are internal to the test suite and intentionally not documented.

Local development setup

1. Toolchain: Rust (stable). On a fresh source checkout, a zero-arg cargo build works; scripts/bootstrap.sh --minimal provisions a C compiler + Rust toolchain first (announced before acting).
2. Clone and build:

   git clone <repo> && cd blazar
   cargo build --release
   # binary: target/release/blazar
   

3. Paths the daemon uses (created on demand by dirs.ensure()):
   • ~/.config/blazar/config.toml — configuration
   • <XDG data>/blazar/ — blazar.db, models/, engines/, run/ (per-child pid + child auth secret files, daemon.log — the daemon’s own durable log, rotated to daemon.log.1 at 10 MiB — and per-child .sock under child_transport = "unix"), speccache/, sessions/
   • Engine rows in blazar.db record their server binary relative to the data dir (engines/<tag>/...), so a data dir move/copy stays relocatable; the out-of-tree local lane (BLAZAR_ENGINE_PATH) keeps its absolute path.
   • XDG variables absent → falls back to CWD with a loud stderr warning (never silent).
4. First engine + model: blazar engine update then blazar pull qwen3-0.6b.

Platform & hardware coverage (honesty rows)

What runs where — stated up front so “why doesn’t lane X work here” is a docs lookup, not a bug report:

• Windows is first-class for the app and the llama.cpp lane (winget, Scoop, scripts/install.ps1; Vulkan and CPU compute; CUDA when the driver census finds it). blazar doctor names what your machine actually serves each lane with.
• The SGLang lane needs Linux or WSL2. The upstream runtime depends on uvloop, which does not install on Windows — the lane’s install and doctor output say so instead of failing mid-pip. Other lanes are unaffected.
• The MLX lane serves MLX quant dirs (mlx-community). Linux install requires an NVIDIA GPU (mlx[cuda12] backend); Apple Silicon installs plain mlx. Windows: use WSL2 with GPU passthrough, same as the sglang lane’s guidance.
• NPU-class accelerators are not a lane yet. Vulkan already covers Intel Arc / integrated GPUs for compute today; NPU (OpenVINO-class) support is an upstream-engine question tracked in docs/research/2026-10-02-competitor-pain-points-audit.md.

Running the test suite

Item	Command / fact
Unit + integration tests	cargo test --workspace (per-crate: cargo test -p blazar-gateway etc.)
Integration suites	per-crate tests/ dirs — e.g. blazar-gateway/tests/ (gateway.rs, semcache.rs, sentinel.rs, structured_output.rs, sessions.rs, compat.rs, crash.rs), blazar-runtime/tests/ (supervisor.rs, engine_manager.rs, tuner.rs, mistralrs.rs, replicas.rs, quantize.rs, models_alias.rs, engine_build.rs), blazar-cli/tests/ (upgrade.rs, config_set.rs)
Engine stubs	workspace-root tests/fixtures/ — stub_llama_server.rs, stub_llama_bench.rs, stub_llama_quantize.rs (exposed behind the test-util feature; tests spawn stubs instead of real engines)
Full harness	scripts/validate.py — the repository’s end-to-end validation harness (internal; drives real gateway + stubs via the BLAZAR_VALIDATE* env family)
Matrix scripts	scripts/bench_matrix.py, scripts/engine_coverage.py
Interpreting results	standard cargo test output; suite count: 73 workspace test binaries at v0.19.0

Build process

• cargo build --release — workspace build (4 crates, resolver 2, edition 2024, MSRV 1.95).
• unsafe_code = deny, clippy pedantic in CI (INFERRED: enforced via repo lint config).
• Engine binaries are NOT built by the workspace by default — blazar engine update downloads official artifacts (sha256-verified, regression-gated); blazar engine build cuda|cpu builds them locally (auto GPU-arch detection via nvidia-smi, host-compiler matching for nvcc, installs bNNNN-cuda-style tags with llama-server/-quantize/-imatrix/-perplexity/-bench binaries).

Deployment flow

1. End-user install (recommended): curl

   sh one-liner from the published repo (override via BLAZAR_REPO); Windows via winget (rustup provisioning included) or scripts/install.ps1; also Homebrew and Scoop packages (packaging/).

2. Service: scripts/install.sh can install a systemd unit; unit sets RestartPreventExitStatus=3 so bind-conflict exits (code 3) never restart-loop.
3. Run: blazar serve (alias start) — binds 127.0.0.1:11435 by default (blazar’s own port, never ollama’s 11434; set port = 11434 for an ex-ollama drop-in); port 0 = ephemeral.
4. Upgrade: blazar upgrade [--version X] [--dry-run] — sha256-verified self-upgrade.
5. Rollback engines: blazar engine rollback / blazar engine use <tag> — side-by-side tags, never destructive.

GPU driver preflight (installer)

Before the engine lands, the installer censuses PCI GPU hardware (lspci, provisioned when missing) and — when the driver userspace is absent — installs it from first-party distro repos only (announce-then-act): NVIDIA via apt’s nvidia-driver / pacman’s nvidia nvidia-utils; dnf auto-installs akmod-nvidia only when RPMFusion is already enabled; zypper and all third-party-only lanes print the exact command instead of running it. AMD/Intel without a Vulkan ICD get the mesa Vulkan drivers. Every install ends with the loud chain: REBOOT, then blazar engine update — which auto-picks the newest CUDA build the installed driver supports (runtimes are bundled; no CUDA toolkit is ever installed). Opt out with BLAZAR_AUTO_DRIVER=0. Windows is advise-only: the PowerShell installer detects NVIDIA hardware without nvidia-smi and links the driver download. blazar doctor warns (nvidia driver / vulkan driver rows) whenever PCI GPU hardware is present but its driver is not.

Feature flags and configuration switches

All switches live in ~/.config/blazar/config.toml; manage via blazar config set/get/unset/list/defaults/edit. Defaults below are the live output of blazar config defaults (the binary’s own serialization — authoritative). Keys marked (optional) exist in config.rs but are omitted from the defaults dump while unset.

Grouped sections (optional spelling)

The flat keys above are the canonical form and what every tool reads and writes. For readability, related keys may instead be written inside one of the group tables below — the loader hoists them to the same flat keys before parsing:

[gateway]
host = "127.0.0.1"
port = 11435

[resources]
default_ctx = 16384
max_loaded_models = 2

Groups: gateway (bind/TLS/CORS/timeouts), routing (engine routing, router, failover), resources (context, model cap, idle ladder, RAM/NUMA), speculative (spec* + spec_draft_*), diffusion (sdcpp_*), whisper (whisper_*), vision (image_*, mtmd_*, mmproj_*), yarn (yarn_*), reasoning (reasoning*), observability (otlp_*). Rules: a key set both at the root and in a group is refused (keep one spelling); a key placed in the wrong group is refused with a hint naming its home group; unknown groups stay hard errors. blazar config set/get and the defaults dump use the flat spelling.

Serving & lifecycle

Key	Default	Purpose
host / port	127.0.0.1 / 11435	bind address (blazar’s own port; port = 11434 = opt-in ex-ollama drop-in; 0 = ephemeral)
default_ctx	16384	default context size
max_loaded_models	0	concurrent model cap (0 = auto from hardware probe, bytes-based VRAM admission)
idle_sleep_secs / idle_timeout_secs	300 / 1800	child-native GPU sleep → SIGTERM eviction ladder
agent	false	agent-oriented defaults
sessions	true	slot KV checkpoints (--slot-save-path)
session_bank	true	auto _auto-<ctx> bank on evict, restore on respawn (>512 MiB banks dropped)
session_keep_secs	900	session pin lifetime (0 = off)
router / router_max_models	false / 0	one child serves all models (--models-preset INI; 0 = upstream default 4)
engine_check_secs	86400	update-currency check cadence (0 = off; check-only, never auto-install)
models_autoload	(optional)	autoload model list at startup
model_load_timeout_secs	(optional)	per-kind defaults (180 s precompiled / 600 s sglang first-JIT); a pin overrides all lanes

Memory, KV & context

Key	Default	Purpose
cache_ram_mb	8192	weights-mmap cap (auto-capped at 30% physical RAM)
cache_type	""	KV quant: auto ladder f16→q8_0→q4_0 at 90% VRAM (with a K=q8_0/V=q4_0 differential rung between the symmetric grades), or explicit f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1
cache_type_k	""	Per-phase K cache quant — set side wins over cache_type for K; a lone set side mirrors to the other when no other grade exists
cache_type_v	""	Per-phase V cache quant — see cache_type_k; q8_0 K + q4_0 V recovers KV capacity at better quality than symmetric q4_0
kv_unified	(optional)	unified KV pool (device-backed shared buffer; auto-emits when slots > 1)
kv_unified_per_slot	0	per-slot unified pool sizing
swa_full	false	full-attention SWA override
ctx_checkpoints	0	context checkpoints
ctx_extend	0.0	YaRN context extension factor
no_kv_offload	false	disable KV offload
load_mode	""	mmap / mlock / direct-io
spawn_mem_guard	true	refuse load when MemAvailable < model/2 + 512 MiB
keep_tokens	0	KV keep-token floor
no_host	false	suppress host info in child
context_shift	false	emit --context-shift only when true (upstream default is disabled since b10833)

Slots, parallelism & determinism

Key	Default	Purpose
slots	0	parallel slots (0 = auto: capacity-aware clamp by KV budget + trained ctx)
deterministic	true→forces slots=1	default false; multi-slot perturbs near-tie logits
slot_prompt_similarity	0.0	slot pick by prefix similarity ((0..=1], 0 = engine default 0.1)
adaptive_slots	true	adopt slots+1 after ~60 s sustained concurrency (cap 4)
cache_idle_slots	true	keep idle slots warm
repack	true	weight repack for batch-1 decode
batch_size / ubatch_size	0 / 0	physical/max batch (0 = engine default; raising -ub above 512 = measured regression — leave alone)
cpu_moe_n / cpu_ffn_n	0	MoE experts / FFN layers on CPU

Multi-GPU sharding & admission

Automatic tensor parallelism: when weights plus KV exceed the best single card but fit the discrete cards per-rank, the SGLang lane plans the spawn across them (--tp-size N) and the fit ladder judges per-rank demand against the bottleneck card. The same planner awareness extends admission: a load that only makes sense sharded is judged against the combined GPU pool instead of the largest single card, and refusals say so (even sharded across N cards (X GiB per rank)). Manual pins (tp_size, dp_size, pp_size, ep_size, tensor split, device lists) always win — auto-sharding only fires when parallelism is unpinned. The llama.cpp lane has its own auto tensor-split for the same case, and the mistral.rs lane spans by layer mapping: it admits a load that exceeds the best single card yet fits the cards combined and lets the engine distribute layers itself (--device-layers omitted), which permits uneven splits a per-rank planner would refuse.

On all-NVIDIA boxes every sharded spawn is pinned: the child receives CUDA_DEVICE_ORDER=PCI_BUS_ID plus CUDA_VISIBLE_DEVICES naming the first N cards in PCI bus order, so ranks and cards stay aligned on mixed-memory pairs. Any doubt — non-NVIDIA or mixed-vendor census, probe short of ranks — leaves the child env untouched and the engine’s own device mapping in charge.

Admission and queueing: when a model is saturated, Blazar parks requests behind bounded admission rather than allowing uncontrolled concurrency to inflate latency and destabilize the workload. Loads that no amount of waiting can fix — the admission floor exceeds every GPU budget even with the box empty — are refused immediately with the numbers and the levers (blazar fit, smaller ctx/slots, a larger card) instead of timing out in a queue.

Speculative decoding

Key	Default	Purpose
spec	auto	family: off / mtp / mtp-adaptive (PR27210 adaptive MTP, n-max 8 capped at the head — pre-wired, teaches until an engine advertises draft-mtp-adaptive) / eagle3 / dflash / dspark / ngram* (target-side verified-lossless)
spec_auto_ngram	true	spec=auto with no usable draft pair (no pair in the catalog, capacity gate, pair not pulled) falls back to n-gram lookup self-speculation instead of running dense — zero extra VRAM, warning names the knobs to disable it; env BLAZAR_SPEC_AUTO_NGRAM (bench: +92% on patterned traffic, −3% on fresh prose, see bench-artifacts/20260927-spec-ngram-default)
spec_auto_manage	false	Per-model spec governor: two consecutive 60 s windows of queue saturation or draft-acceptance collapse (< 0.15) park speculation for that model (respawn dense via the reshape queue); a sustained quiet streak (10 windows) restores it. Per-request and overlay spec pins always win; the router child is exempt; gauge blazar_spec_governor{model,state}; env BLAZAR_SPEC_AUTO_MANAGE
ubatch_auto	false	Adaptive prefill shaping: two consecutive saturated 60 s windows raise that model’s micro-batch ceiling one rung (--ubatch-size 1024 → 2048, capped — larger buffers OOM small cards); a quiet streak (10 windows) steps back down one rung at a time. Only fills where ubatch_size is unset (a config pin always wins); router child exempt; llamacpp lanes; gauge blazar_ubatch_governor{model,tier}; env BLAZAR_UBATCH_AUTO

Best-of-N fan-out (per request)

best_of (2–4) is a request knob, not a config knob: send it as a body field on /api/chat, /api/generate, or /v1/responses bodies, or as the X-Blazar-Best-Of header on /v1/messages too. The gateway fans the request out to N parallel child candidates, judges them (schema-valid > finish_reason: stop > first), and returns the winner with usage summed across all candidates; the X-Blazar-Best-Of response header itemizes asked/used/usage. Streaming requests cannot be judged and are rejected with a teaching 400, as is the knob on /v1/chat/completions (byte-faithful passthrough lane — the buffered OpenAI lane is /v1/responses). The fan-out degrades to a single send under queue depth, no free decode slots, or bodies over 1 MiB (counters in /metrics: blazar_bestof_fanouts_total, blazar_bestof_degraded_total).

Agentic cascade routing (per request)

cascade is a request knob on /api/chat: send cascade: ["small-model", "big-model"] in the body or X-Blazar-Cascade: small-model, big-model as a header (header wins and pins the order). Candidates are tried in order and each answer is judged against the same ladder best-of-N uses (2xx + requested schema + clean finish); the first pass is served and later candidates are never sent — a request the small model answers costs exactly one small-model call. When the body names the chain and at least two candidates have measured decode rates (A2 telemetry), the chain is tried cheapest-first. Responses carry X-Blazar-Cascade: tried=… served=… reason=first-pass|escalated|none-passed and usage summed across every attempted candidate; nobody-passed serves the last (strongest) candidate’s answer. Streaming requests are rejected with a teaching 400 (a winner cannot be judged before completion), as are chains over 4 models and bodies over 1 MiB. Counters in /metrics: blazar_cascade_runs_total, blazar_cascade_escalations_total. | spec_cache | true | persistent ngram lookup-cache-dynamic per model | | spec_draft_cpu_range … spec_draft_* | see dump | draft-model placement: cpu_range, cpu_strict, device, ngl, threads, prio, prio_batch, cpu_strict_batch, threads_batch, type_k/type_v, override_tensor, n_cpu_moe, cpu_moe | | spec_draft_backend_sampling | true | sampling on the backend | | spec_draft_p_min / spec_draft_p_split / spec_draft_poll / spec_draft_poll_batch | (optional) | draft acceptance/split/poll tuning (p_split default 0.10) | | ngram_size_m / ngram_size_n / ngram_min_hits | 0 | ngram/ngram-map-k/k4v families | | ngram_mod_n_match / ngram_mod_n_max / ngram_mod_n_min | 0 / 0 / 0 | mod-n window (bounds 1..=1024-class) | | adaptive_decay / adaptive_target | 0 / 0.0 | adaptive spec tuning |

Reasoning & thinking

Key	Default	Purpose
reasoning	""	on / off / auto
reasoning_format	""	none / deepseek / deepseek-legacy
reasoning_effort	""	minimal / low / medium / high / xhigh / max (bridged into chat_template_kwargs, chat lane only)
reasoning_budget	-1	-1 unrestricted / 0 immediate end / N budget
reasoning_budget_message	""	message shown when budget hits

Vision & embeddings

Key	Default	Purpose
mmproj_policy	(optional)	lazy (default; respawn w/ projector on first vision request) / attach / skip; bool compat true=attach false=skip; explicit -mm in extra_args wins
mmproj_offload / mmproj_auto	true / true	projector offload / auto-attach
mmproj_device	""	projector device
image_max_tokens / image_min_tokens / mtmd_batch_max_tokens	0	vision token budgets
embd_normalize	0	embedding normalization
late_chunking_max_tokens	8192	late-chunking embed cap (413 beyond; per-model late_chunking=true opt-in)
video_ffmpeg_dir / video_fps / video_timestamp_interval	"" / 0.0 / 0.0	video input (ffmpeg-capable vision models)

YaRN

Key	Default	Purpose
yarn_orig_ctx / yarn_ext_factor / yarn_attn_factor / yarn_beta_fast / yarn_beta_slow	0 / -1.0 / 0.0 / 0.0 / 0.0	YaRN rope-scaling knobs

Scheduling & CPU affinity

Key	Default	Purpose
cpu_range / cpu_strict	"" / false	pin child CPU set
prio / prio_batch / poll / poll_batch	0 / 0 / 0 / (optional)	scheduler priority/poll knobs
threads_http / threads_batch	0	HTTP server / batch threads (0 = engine default)
numa	""	distribute / isolate
main_gpu / split_mode / tensor_split	-1 / "" / ""	multi-GPU: primary, none/layer/row/tensor, ratios e.g. “3,1” (per-model tensor_split overlay replaces; empty = inherit)
devices	[]	device pinning (names from blazar doctor)
override_tensor	[]	per-tensor backend overrides
tensor_preset	""	”” or “moe-cpu-offload” (experts→CPU)
op_offload	(optional)	op-level offload
check_tensors	false	tensor validation at load

Child transport & HTTP

Key	Default	Purpose
child_transport	tcp	child transport: tcp (loopback host:port) or unix (per-child run/<name>.sock, Linux/macOS/BSD only)
child_auth	(optional)	unset = auto (TCP children authed, UDS not) / true / false; per-child secret at run/<model>.apikey
sse_ping_interval	(optional)	SSE ping (-1 disables; upstream 30)
server_timeout_secs	(optional)	child HTTP timeout (upstream 3600)
chat_template_kwargs	(optional)	JSON object, shape-validated
cont_batching	(optional)	true→--cont-batching, false→--no-cont-batching, unset = engine default
reuse_port	false	SO_REUSEPORT
lora_init_without_apply	false	LoRA init behavior
warmup	true	post-spawn warmup pass
samplers	""	sampler chain (top_k;top_p;typical); --sampler-seq deliberately not a knob
engine_env	table	extra env for children + probes

child_transport = "unix" — spawn children on Unix-domain sockets instead of loopback TCP: each child listens on run/<name>.sock in the daemon’s run directory (next to its pidfile), and the gateway dials it over the same socket, skipping the network stack and per-request port allocation. Socket paths are deterministic per model, swept for stale files at daemon boot, and unlinked when the child is evicted. Path budget is the OS sun_path limit (~104B): a model name deep under a long data_dir fails fast with a teaching error naming the offending component. Not available on Windows (config validation rejects it there). tcp remains the default and is byte-identical to previous releases.

Engines, downloads & updates

Key	Default	Purpose
engine_asset	auto	asset flavor selection
update_channel	latest	latest (incl. prereleases) / stable — a pin, not a floor; llamacpp currency surfaces annotate upstream’s stable marker when the active build is the one it stamps (b11429 (stable v0.6.0))
download_connections	8	parallel byte-range download (≥32 MiB; 1 = single-stream resume; Range-rejecting servers forced single)
download_speed_limit_mb	0	total download speed cap in MB/s across all pull lanes (0 = unlimited); one shared token bucket (1 s burst) covers model pulls, registry pulls, TTS/whisper voices, and engine-binary assets — BLAZAR_DOWNLOAD_SPEED_LIMIT_MB env overrides
whisper_stream_chunk_ms	30000	whisper streaming (stream=true on /v1/audio/transcriptions or /translations): decode window in milliseconds, 1000..=120000; WAV files are split at PCM frame boundaries and each window is transcribed in order on the same lazy child — BLAZAR_WHISPER_STREAM_CHUNK_MS env overrides
whisper_vad_model	unset	Silero VAD for the whisper-server child (--vad --vad-model): skips silence/noise segments before decode — faster long-audio runs, fewer hallucinated spans. Value = a ggml file name (looked up in the whisper engine dir) or a path. Never auto-pulled; a missing file is a teaching error, unset = no VAD
whisper_beam_size / whisper_best_of / whisper_entropy_thold / whisper_logprob_thold / whisper_word_thold	unset	First-class decode knobs for the whisper-server child (--beam-size etc.): beam 5 sharpens accuracy on noisy audio at a decode-speed cost; the thresholds prune hallucinated spans. VAD tuning rides whisper_vad_threshold / whisper_vad_min_speech_duration_ms / whisper_vad_min_silence_duration_ms (require whisper_vad_model); whisper_diarize (stereo input) / whisper_tinydiarize (needs a tdrz model) gate speaker separation. All surface-gated: a knob flag missing from the installed engine is a teaching error at spawn, and double-setting one via whisper_extra_args is refused
whisper_extra_args	unset	extra boot flags for the whisper-server child, appended after the pinned loopback/port/model/VAD argv — the lever for engine options no first-class knob covers (e.g. --dtw tiny for token-level timestamps by default). Structure-validated at config load; at child compile each flag is gate-checked against the child’s own --help surface (a misspelling fails fast with a teaching error instead of a dead child); --host/--port/--model/--vad/--vad-model are reserved. BLAZAR_WHISPER_EXTRA_ARGS env overrides (comma-separated; spaces inside a segment are preserved, so values with commas/spaces belong in the TOML list)
rpc_servers	""	llama.cpp RPC worker endpoints for llama-server children (--rpc, comma-separated host:port): model weights and KV cache split across local + remote devices in proportion to each device’s free memory. Start a worker on a GPU box with blazar rpc worker --host 0.0.0.0 (secure loopback default; the worker accepts the connection, so bind it where the gateway can reach it). The RPC protocol has NO authentication — trusted networks only. Dead endpoints are refused at spawn with a teaching error (preflight probe); blazar doctor shows the whole fleet, and blazar rpc status adds per-worker protocol version + per-device free/total memory (queries the worker wire protocol, not the banner). Spawn-time fit verdicts (admission, the too-large refusal math, the spill-to-CPU warn) count every reachable worker’s device VRAM — a worker unreachable at planning is skipped with a warn and still refused at spawn if down; placement ratios are left to the engine’s own fleet distribution. A per-model rpc_servers overlay replaces this list (empty = inherit)
lazy_mode	auto	lazy spawn mode
fork_retire_days	7	days after full-architecture supersede before a curated fork lane auto-deletes (0 = never; user forks and pinned lanes never auto-delete)
keep_tags	2	engine builds retained per kind after an install — newest N survive auto-prune; local and the active tag stay protected on top. 2 = fresh build + one rollback anchor for blazar engine rollback; 1 = an update deletes the previous build outright. Also governs whisper/piper voice-dir retention
capability_registry_url	upstream default	curated capability-lane catalog; "" disables lane discovery entirely

Request-path behavior

Key	Default	Purpose
singleflight	true	coalesce identical NON-STREAM chat (5 s bound)
prompt_preflight	true	refuse unfit prompts (exact /tokenize when estimate >90% of ctx)
predictive_preload	false	predictive model load (LC1)
preload	[]	startup preload list — models spawned right after the listener binds (one at a time, warn-not-fail; env BLAZAR_PRELOAD=m1,m2)
idle_ram_warm	true	after an idle-ladder eviction, re-page the evicted weights into the OS cache (RAM tier) so a re-spawn mmaps warm; skipped under load_mode = "direct-io"
sentinel / sentinel_stall_secs / sentinel_enforce	true / 30 / false	quality observation / stall window / hard 422 mode (per-request X-Blazar-Enforce override)
child_header_timeout_secs	120	ceiling for awaiting child response headers. Buffered (non-stream) responses emit headers only at completion, so a long generation can look like a wedged child — on expiry the gateway probes the child’s /slots: slots still generating ⇒ teaching 504 (stream the request or raise this knob) and the child stays; slots idle ⇒ wedged child evicted + one in-band retry. 0 disables (blanket transport ceiling only)
sdcpp_child_header_timeout_secs	900	same ceiling scoped to the diffusion lane, where time-to-headers IS generation time (a frame arrives only when fully computed — minutes at native resolutions on offloaded hardware, longer for video); 0 disables
sdcpp_cache_mode	unset	step-caching family for sd-server children (--cache-mode): easycache/ucache/dbcache/taylorseer/cache-dit/spectrum — 1.4-2.2x faster renders with near-lossless quality; unset = exact per-step compute
sdcpp_cache_option	unset	cache-family tuning (--cache-option, key=value list, e.g. "threshold=0.25,reset=0"): thresholds/warmup/window for the active sdcpp_cache_mode
sdcpp_control_net / sdcpp_ip_adapter (+ sdcpp_clip_vision) / sdcpp_motion_module / sdcpp_photo_maker / sdcpp_pulid_weights / sdcpp_upscale_model	unset	component weight paths for sd-server children (--control-net, --ip-adapter requires --clip_vision — config validation enforces the pair, --motion-module = AnimateDiff SD1.5 video, --photo-maker, --pulid-weights Flux, --upscale-model Real-ESRGAN hires fix). Assets are never auto-pulled; paths point at files you provide
sdcpp_lora_model_dir / sdcpp_hires_upscalers_dir / sdcpp_embd_dir / sdcpp_audio_encoder / sdcpp_high_noise_diffusion_model / sdcpp_uncond_diffusion_model / sdcpp_backend	unset	scan directories and advanced components (--lora-model-dir, --hires-upscalers-dir, --embd-dir, --audio-encoder wav2vec2 Wan2.2 S2V, Ideogram4 CFG pair, --backend per-module placement like "clip=cpu,vae=cuda0,diffusion=vulkan0")
sdcpp_flash_attention	true	flash-attention kernels (--fa) for sd-server children: faster attention, less VRAM (measured 25% faster renders at identical quality; set false for legacy kernels)
sdcpp_sage_attn	false	8-bit SageAttention kernels (--sage-attn) in the diffusion model: roughly 2x over legacy kernels — but hard-requires a patched GGML CUDA backend (CUDA 12+, SM80+); off CUDA devices (Vulkan/Metal/CPU) blazar skips the flag with a warning instead of letting the child die at load. extra_args --sage-attn overrides for custom postures
sdcpp_vae_tiling	false	tiled VAE decode (--vae-tiling): fits larger-than-VRAM decodes at some speed cost
sdcpp_tae	unset	Tiny AutoEncoder path (--tae): fast low-quality VAE decode for previews/iteration
sdcpp_rpc_servers	[]	RPC worker endpoints (--rpc-servers, comma-joined), the diffusion twin of llamacpp rpc_servers; empty = local devices only
sdcpp_max_vram	unset	per-device VRAM budget in GiB (--max-vram): "8" or "cuda0=8,cuda1=6" bounding weight placement; unset = full-device budgets
sdcpp_params_backend	unset	explicit weight placement (--params-backend): "disk"/"cpu" or per-module ("diffusion=disk,clip=cpu"); pins placement, disabling auto-fit
sdcpp_split_mode	unset	multi-device weight distribution (--split-mode): row (matmul rows, CUDA) / layer (blocks per device) or per-module ("diffusion=row,te=layer")
sdcpp_conditioning_cache_size	unset	cross-request prompt-embedding cache depth (--conditioning-cache-size, engine default 4): repeated/shared-prefix prompts skip re-encoding; 0 disables
sdcpp_model_args	unset	model-builder key=value list (--model-args, e.g. qwen_image_2_1_prefix_cache=true): family-specific engine levers ahead of first-class knobs
sdcpp_qwen_prefix_cache_type	unset	Qwen-Image-2.1 prefix KV cache strategy (qwen_image_2_1_prefix_cache_type model-arg, auto or a type name; upstream master-929+): early prefix-cache scheduling — a large VRAM win on repeated compositions. Composes into sdcpp_model_args; an explicit user entry for the same key wins
sdcpp_extra_args	unset	extra boot flags for sd-server children, appended ahead of per-model extra_args (engine last-wins — a flag set in both tiers resolves to the model override, with a compile warning naming the duplicate). This is the lever for boot-time model addons the first-class knobs don’t cover: --control-net, --ip-adapter, --photo-maker, --motion-module, --upscale-model, --lora-model-dir, --hires-upscalers-dir, --audio-vae/--audio-encoder, and friends. Pin flags (--model, --vae, --threads, …) are reserved and the rest manifest-gated exactly like the overlay tier; a --backend here owns device posture (the auto weight-split stands down). BLAZAR_SDCPP_EXTRA_ARGS env overrides (comma-separated)
sdcpp_tensor_type_rules	unset	per-tensor quant overrides (--tensor-type-rules, regex=type list, e.g. model.=q6_k,vae.=f16): quantizes matching weight groups independently for tighter VRAM
decode_policy	free	ollama_compat lane decode constraint: free lets the model choose prose or a tool call; strict attaches a GBNF grammar confining tool-bearing completions to whitelisted <tool_call> envelopes (names must be request tools, arguments structurally valid JSON)
raw_lane_max_tokens	2048	server-side cap for ollama_compat raw-completion generations when the caller omits (or exceeds) a cap; 0 disables (overlay wins over global)
pii_scrub	true	redact emails/bearer secrets/IPv4 in why/watch (on by default; set false for verbatim output)
reasoning interplay	—	see Reasoning table

Federation: peers behind one gateway

[[remotes]] names other OpenAI-compatible servers (another blazar box, vLLM, an MLX server — anything speaking /v1/*). Requests for <remote-name>:<model> always route there explicitly. On top of that, with remote_fallback = true (default), a bare model name the local store does not know routes to whichever peer lists it: the gateway keeps a 60-second TTL cache of each peer’s /v1/models listing, prefers the least-busy live peer, and never sends your caller credentials along — the peer’s configured key rides instead, exactly as with explicit prefix routing. Locally-ambiguous names stay local (that is a naming problem), and a model no peer serves gets the ordinary local 404 teaching. Peer listings also merge into GET /v1/models (snapshot of the cache; the first fallback warms it). Set remote_fallback = false to roll the whole feature back to pure-prefix routing.

Fallback covers the OpenAI surfaces (/v1/chat/completions, /v1/completions, /v1/embeddings, /v1/images/generations), and the Ollama-compatible /api/chat (the request is translated to the peer and the reply back into ollama shape). /api/generate stays local-only: it refuses remote prefixes explicitly, so fallback honors the same contract. When a peer starts serving a model inside the TTL window, the first miss for that name triggers one throttled re-probe per peer (at most one every 10 seconds), so new peer models are found without per-request probe traffic.

Remote URL scheme ladder

The bearer key of a [[remotes]] entry must never travel in cleartext off-machine, so remote URLs are classified at boot:

URL shape	Boot behavior
https://… (any host)	accepted
http:// to 127.0.0.1 / localhost / [::1]	accepted (never leaves the machine)
http:// to a private/LAN address (10.*, 192.168.*, 172.16-31.*, 169.254.*, fd00::/8, fe80::/10)	accepted with a logged warning — the key rides unencrypted on the LAN
http:// to anything else (public IP or hostname)	refused with a teaching error

Hostnames other than localhost count as public (no DNS at validation time) — use an IP literal or https:// for private hosts with names. A lab that genuinely must reach a public cleartext endpoint opts in per remote:

[[remotes]]
name = "legacy-box"
url = "http://203.0.113.7:8000"
allow_insecure_http = true   # loud warning instead of boot refusal

Structured output across dialects

One guarantee, three request dialects: the response never silently violates the schema the caller asked for. Engines enforce schemas natively per request (llama.cpp / mistral.rs convert the schema to a GBNF grammar); blazar adds the cross-dialect plumbing and the post-hoc sentinel check. Every constrained response carries the receipt header x-blazar-structured-output (json_schema / json_object / gbnf) so clients can prove the constraint was active; unconstrained responses omit it.

Dialect	Caller asks via	Translation	Guarantee
OpenAI	response_format (json_object / json_schema)	pass-through to child	child grammar + sentinel 422 on violation (non-streaming, sentinel_enforce / X-Blazar-Enforce)
Ollama	format: "json" or a JSON-schema object	format → response_format (translate.rs)	same as OpenAI once translated
Anthropic	forced tool: tool_choice {"type":"tool","name":…} + tool input_schema	→ OpenAI forced-function tool_choice	child binds the named tool’s schema; nameless forcing is rejected 400 at the boundary

Failover chains ([[failover]])

An ordered list of local-or-remote targets published under one alias: a request naming the alias sticks to the first healthy target, and on failure (5xx, 404, or a blazar admission refusal like an honest fit rejection) the next target serves inline. A failed target is benched for min_residence_secs (default 30 — the anti-flap window); when every target is benched the sticky one keeps serving. All three dialect lanes honor chains (OpenAI, /api/chat, /v1/messages — the Anthropic lane is local-targets-only and teaches why). Observation: x-blazar-failover + x-blazar-served-model response headers, failover_switched events on /api/events, and /api/failover admin routes (list, detail, pin, unpin — a pinned target keeps serving even while failing). Keys admission re-charges per retry attempt (each attempt is real work).

[[failover]]
name = "voicecheck"
alias = "primary"
min_residence_secs = 30          # anti-flap bench window per failed target

[[failover.targets]]
model = "qwen3-8b:q4_k_m"        # local target (bare model)

[[failover.targets]]
remote = "bigbox"                # remote target (existing [[remotes]] name)
model = "qwen3-8b:q4_k_m"        # forwarded as bigbox:qwen3-8b:q4_k_m

MCP tool catalog ([[mcp]])

Declared MCP servers — local stdio processes or remote Streamable HTTP endpoints — contribute tools to any chat request, on an opt-in per-request basis: body field "mcp": "all" (or one server name) or header x-blazar-mcp. The gateway namespaces the tools (mcp__<server>__<tool>), injects them alongside caller tools, executes the model’s tool calls itself (MCP 2025-06-18: initialize → tools/list → tools/call), and feeds results back — every engine gets tool use with zero child-side MCP support. Bounds: 4 tool rounds per request, timeout_secs (default 30, max 600) per call, streaming + mcp refuses with a teaching 400 (the loop is buffered by design). GET /api/mcp lists the catalog report-only. stdio sessions shut down with the daemon (stdin close → wait → SIGTERM); HTTP sessions get a best-effort session DELETE. Security posture: the config owner declares the servers — per-request opt-in only EXPOSES them to keyed clients, it never lets a request declare a new command or URL; the gateway logs tool name + duration, never tool payloads.

Mixed turns (the model calls a caller tool and an mcp__ tool in the same response): the gateway executes its namespaced calls and answers the caller-owned ones with an explicit placeholder result naming who runs them — no tool_call is left dangling, so the next round stays protocol-complete.

For stock clients that cannot add a field or header (plain OpenAI SDK apps, third-party tools), set a config-wide default:

mcp_default = "all"    # "all", one server name, or "none"; validated at boot

Resolution order per request: x-blazar-mcp header > body "mcp" field > mcp_default > none. "none" is the explicit opt-out — requests that must not see the catalog send it even when a default is configured.

[[mcp]]
name = "fetch"
command = ["uvx", "mcp-server-fetch"]   # stdio transport
timeout_secs = 30                       # per tools/call ceiling

[[mcp]]
name = "calc"
command = ["python3", "-m", "mcp_server_calculator"]

[[mcp]]
name = "remote"                         # Streamable HTTP transport (set
url = "http://mcp.internal:8808/mcp"    # exactly one of command / url)

[mcp.env]                               # optional, stdio only — forwarded
CACHE_DIR = "~/.cache/mcp"              # to the process

An MCP server is a program the gateway launches and lets a model call — treat it as a trust boundary. By default a stdio child inherits the daemon’s whole environment; a hardened server starts from an empty one and re-inherits only what you name (stdio-only options, validated at boot):

[[mcp]]
name = "fetch"
command = ["/opt/mcp/fetch-server"]     # absolute path: PATH is scrubbed
env_clear = true                        # start from an EMPTY environment
allowed_env = ["PATH", "HOME"]          # the only vars it may re-inherit
cwd = "/opt/mcp"                        # pin the working directory
max_output_bytes = 1048576              # cap one protocol message at 1 MiB

[mcp.env]
FETCH_TIMEOUT = "30"                    # explicit env always wins

Scrubbed children can’t find their binary via PATH unless you allow it back — prefer an absolute command path, or add "PATH" to allowed_env (Windows also needs "SYSTEMROOT"). The gateway logs a bounded slice of each child’s stderr (first 32 KiB) at warn so server diagnostics surface without flooding the log.

Python SDK (sdk/python)

A zero-dependency client (stdlib urllib + raw-socket WebSocket only) for the native surface: chat() with tools / best_of / mcp, SSE streaming, ps(), explain(), failover(), mcp_servers(), completion metadata_update(), Anthropic-dialect batches (anthropic_batch_create/_get/_results/_cancel/_delete), realtime voice via blazar.realtime(model) → RealtimeSession.voice_turn(pcm) (PCM16 mono audio in and out), TTS via blazar.speech(voice, text, response_format=...) (returns (content_type, audio_bytes); the full wav/pcm/pcm16/mp3/opus/aac/flac family), voice discovery via blazar.voices(search=...), and transcription via blazar.transcribe(path, response_format=..., **knobs) (JSON family returns a dict — including two-speaker diarized_json on stereo input — text/srt/vtt return the raw string; list kwargs like timestamp_granularities are sent per-value). Teaching errors surface verbatim in BlazarError. Run the demos against a live gateway:

PYTHONPATH=sdk/python python3 sdk/python/examples/demo.py qwen3-0.6b:q4_0
BLAZAR_URL=http://127.0.0.1:11435 PYTHONPATH=sdk/python \
    python3 sdk/python/examples/realtime_voice.py qwen3-0.6b:q4_0

Realtime voice (/v1/realtime)

A WebSocket lane that chains the local whisper child, any chat model, and a piper voice into conversational voice turns — no external service. Connect with ?model=<chat-model>&voice=<piper-voice>&stt_model=<whisper-size> (defaults: first voice / whisper-1), authenticate with the same bearer key as every other route, then drive the protocol: input_audio_buffer.append (base64 PCM16 mono, or raw binary frames) → commit → input_audio_transcription.completed → response.audio_transcript.delta / .done → response.audio.delta chunks / .done. Session control: session.update (voice / stt_model / instructions — applies from the next turn), input_audio_buffer.clear, response.create (force a turn on buffered audio), response.cancel (honored between pipeline stages), and conversation.item.create (text items feed later turns as context, capped 16 items / 8192 chars). Bounds: 20 MiB audio buffer, 120 s idle timeout, 600 s hard session TTL. Prepare the lanes once: blazar whisper --pull tiny and blazar tts --pull en_US-amy-medium.

Optional system ffmpeg (on PATH) unlocks the lossy TTS output formats (mp3/opus/aac/flac on /v1/audio/speech, both buffered and "stream": true live streaming); without it the lane serves wav/pcm and answers lossy requests with a teaching 400 naming the install hint. Nothing else depends on it.

Security, TLS & network

Key	Default	Purpose
keys	[]	[[keys]]: name/key/models/rpm/tpm/daily_tokens/max_concurrent/weight — non-empty list activates auth
tls_cert / tls_key	""	PEM pair (both-or-neither)
cors_origins	[]	browser CORS policy: local origins (localhost/127.0.0.1/[::1]/0.0.0.0 any port, plus app:///file:///tauri:///vscode-webview:///vscode-file:// webviews) are always allowed; listed entries are added on top; "*" = any origin
remotes	[]	[[remotes]]: name/url/key; <remote>:<model> routing; duplicate names = pool (3 fails → 30 s markdown, least-inflight LB, prefix stickiness); see Federation: peers behind one gateway above for fallback semantics
server_tools	(optional)	EXPERIMENTAL server-side tools (CORS→localhost when set)
server_tools_runtime	(optional)	docker:/podman:/ssh: runtime for tools
mcp_servers_config / mcp_servers_json	(optional)	MCP server defs (file vs inline JSON, mutually exclusive)

Observability

Key	Default	Purpose
log_level	(optional)	INFO default; RUST_LOG wins
audit_log	true	identity+outcome audit jsonl (<data>/log/audit.jsonl, 16 MiB rotation; one append per generation request — on by default, set false to disable)
otlp_endpoint / otlp_service	"" / "blazar"	OTLP traces/metrics export
lookup_cache_static / lookup_cache_dynamic	(optional)	read-only / refreshed lookup caches (must exist)

Tables

Table	Purpose
[warm_peg]	default = true; per-engine overrides: sglang + mlx (blocking peg, JIT venv class), llamacpp (detached peg), mistral.rs and sdcpp never peg
[sdcpp]	(no section — by design) diffusion tuning rides request parameters (steps/cfg/seed/size on POST /v1/images/*), not config knobs; spawn flags (--vae/--llm/--t5xxl…) come from the model’s component set; extra_args passes manifest-probed flags (--backend clip=cpu,vae=cuda0,..., --lora-model-dir, --vae-tiling…)
[engine_routing]	mode = "auto"\|"manual" (auto = per-model routing; manual = byte-identical single active engine), policy = "quality"\|"latency"\|"throughput"
[semantic_cache]	enabled = false, ttl_secs = 600, threshold = 0.9, max_entries = 256 (opt-in; per-request X-Blazar-Cache overrides)
[mistralrs]	mistralrs_pa_memory_fraction (0.05..=0.95), mistralrs_paged_attn (false = classic KV sized from ctx — fits big vision models on 8 GB) + MistralrsTuning knobs; pa_memory_mb = absolute paged-KV MiB for shared GPUs — when set, every --pa-memory-fraction source stays silent (the engine refuses both on one command line); chat_template overrides the lane’s template; isq = in-process quantization at load ("q4k"/"q8_0"/numeric 2..8 — shrinks the resident set, admission stays checkpoint-based so it stays conservative), imatrix/calibration_file sharpen ISQ k-quants, isq_organization = default/moqe
[mlx]	MlxTuning knobs for the mlx-lm server lane: draft_model + num_draft_tokens (speculative decoding), kv_bits + kv_group_size + quantized_kv_start (KV quantization — upstream serves one request at a time with kv-bits set), prefill_step_size, prompt_cache_size/prompt_cache_bytes, decode_concurrency/prompt_concurrency, chat_template{,_args}, use_default_chat_template, trust_remote_code, adapter_path; all manifest-gated (warn-and-skip on engines lacking the flag); per-model via model_overrides.<name>.mlx
[sglang]	SglangTuning knobs (cache_report defaults ON for blazar-managed children — it powers ps HIT and per-model metrics; set false to opt out; radix_eviction_policy lru/lfu/slru/priority, session_radix_cache, mixed_chunk, …); VRAM ladder flags (--mem-fraction-static, --cpu-offload-gb, --kv-cache-dtype, --context-length) owned by the profile compiler; explicit mem_fraction_static/kv_cache_dtype pins win. Speculation (opt-in, default off): spec_algorithm = "ngram" (draft-free prompt-lookup; DFLASH is not offered — sglang 0.5.21 requires a draft-model path for it) + spec_num_steps / spec_num_draft_tokens; engage adds a verify-KV/graph memory reserve to the ladder. 2026-10-09 receipts (qwen3-1.7b, RTX 4070): ngram lost to dense at every concurrency (−16% C1, −44% C8, +12 s cold) — hence opt-in; the 2026-10-09 greedy-only constraint and attention_backend = "triton" pairing are retired (2026-10-17 live receipts: temperature 0.7-1.0 requests complete on both sampling backends after the CUDA compiler-trio install pin; flashinfer backends work when pinned, triton stays the toolchain-less default). The EAGLE family uses the llamacpp-style draft-pair lane instead (spec = "eagle3" + a pulled safetensors draft). Depth knobs (2026-10-10): speculative_eagle_topk, speculative_accept_threshold_single/_acc (0..=1, spec arms only), mm_attention_backend, grammar_backend, retraction_policy (length/priority — priority requires enable_priority_scheduling = true, enforced at config-validate), enable_two_batch_overlap, enable_tf32_matmul
[model_overrides.<name>]	per-model REPLACEMENTS: ctx, slots, deterministic, spec, lazy_mode, loras, extra_args (manifest-validated), cache_type, cache_type_k, cache_type_v, kv_unified, ctx_extend, cpu_moe_n, cpu_ffn_n, override_tensor, devices, warmup, reasoning_budget, reasoning_effort, reasoning, replicas (>1 = prefix-affinity routing), pin (skip in capacity-eviction victim filter), chat_template (XOR chat_template_file), sampler_defaults, spm_infill, late_chunking, rpc_servers, tensor_split, mmproj, sglang, mistralrs, engine (tag or kind; wins over both routing modes) — deny_unknown_fields: typos fail loudly

Full key grammar and examples: blazar config --help (CONFIG_EXAMPLES, CONFIG_SET_EXAMPLES), and blazar config defaults prints the live default file.

Improve this page on GitHub