Documentation

Usage

Day-2 workflows: sessions, cache, diagnostics, federation.

9. USAGE

End-user operations guide for Blazar. Verified against v0.6.1 source. Commands and configuration spot-verified against v0.19.0 (2026-10-03). Companion docs: 1.SYSTEM_OVERVIEW.md (what it is), 4.API_SPEC.md (full HTTP+CLI reference), 7.SETUP.md (env/config).

Installation

Platform	Method
Linux / macOS	curl -fsSL <published-repo>/install.sh \| sh (override source with BLAZAR_REPO)
Windows	winget install blazar (rustup provisioned via winget) or scripts/install.ps1
Package managers	Homebrew, Scoop (see packaging/)
From source	git clone → cargo build --release (zero-arg build works; scripts/bootstrap.sh --minimal provisions toolchain first)

Quickstart (copy-paste)

blazar engine update                 # install official llama.cpp engine (sha256-verified)
blazar pull qwen3-0.6b               # pull a model (registry shortname)
blazar serve &                       # start the gateway on 127.0.0.1:11435
curl http://127.0.0.1:11435/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-0.6b","messages":[{"role":"user","content":"hello"}]}'
blazar run qwen3-0.6b                # interactive chat REPL
blazar run qwen3-0.6b --no-draft "hi"  # single-shot, dense spawn (spec off for this run)

Existing ollama clients work after re-pointing: OLLAMA_HOST=http://127.0.0.1:11435 ollama list (or set port = 11434 in config.toml for a drop-in once ollama is removed).

Common tasks

Goal	Commands	Expected result
Install/upgrade an engine	blazar engine update (or --kind sglang / --kind mlx / --kind mistralrs / --kind sdcpp; all or --all walks every installed lane, skipping absent and current ones)	engine downloaded, verified, regression-gated; auto-rollback on >10% decode drop
Build CUDA engine locally	blazar engine build cuda	local build with auto GPU-arch + host-compiler matching; llama-server + tooling installed under a bNNNN-cuda-style tag
Serve an architecture mainstream lacks	blazar engine offers → blazar engine install --lane <id> --backend cuda	curated fork lane built from an immutable pinned SHA; auto-routes only the models that need it, auto-deletes after mainstream catches up (fork_retire_days); engine rm refuses while a child still serves it
Switch engines	blazar engine use <tag> or --kind <kind>; blazar engine list	active engine switches; auto routing stays default
Rollback an engine	blazar engine rollback	previous tag re-activated
Pull from HF	blazar pull ggml-org/Qwen3-8B-GGUF:Q4_K_M	owner/repo:QUANT resolved, sha256-verified, resumable
Adopt an ollama / LM Studio model without re-downloading	blazar import --scan → run the listed blazar pull / blazar import command	local ollama stores, the HF hub cache, and LM Studio models are discovered and listed; adopting is byte-verified (size + sha256) and hardlinked — the source runtime’s files are never modified
See what fits before downloading	blazar fit <target>	VRAM fit preview + quant alternatives (+ lane hint for safetensors)
Register a local GGUF	blazar import /path/to/model.gguf --name mymodel	model usable without copying (or --copy)
List / inspect models	blazar list / blazar show <model> (--json for JSONL)	flat model rows, quants, sizes
Check what’s running	blazar ps	instances with slots/ctx/ports/in-flight + keep-alive countdown
Load/unload now	blazar stop <model>; or request with keep_alive: 0 / -1	immediate unload / keep forever
Save session across unload	blazar session save <model> … blazar session restore <model>	KV checkpoint survives unload + restarts (also automatic via session bank)
Tune performance	blazar tune <model> (--search, --slots, --spec, --ngram, --spec-race, --load, --replicas, --cache-reuse)	best argv recorded into profiles; --ngram refines n-gram lookup (m, min-hits) on live servers; --spec-race races dense vs ngram vs dflash end-to-end and adopts a mode only when it beats dense by >5% (measured, never assumed) — GGUF models race llama-server children, safetensors dirs (sglang lane) race sglang-server children and print adopt commands instead of auto-adopting
Benchmark	blazar bench <model>	tg/pp tokens/s recorded into bench_history
Add API keys	blazar keys add or edit [[keys]] in ~/.config/blazar/config.toml	gateway starts requiring Authorization: Bearer plm_... / x-api-key
Debug a bad response	blazar why (or /api/why) with the trace id	sentinel detections + routing trail (PII-scrubbed if pii_scrub)
Health check	blazar doctor	grouped SYSTEM/GPU/ENGINES/MODELS/CHANNELS/RUNTIME findings with fixes
Transcribe audio	blazar whisper file.wav --model <size>	transcription via the whisper lane
Discover whisper models	blazar whisper --search [substr]	every upstream ggml size (tiny..large-v3-turbo, .en, q5/q8) with DISK bytes + pulled state
Transcribe long audio without blocking	POST /v1/audio/transcriptions with async=true form field	job handle → poll GET /v1/audio/jobs/{id}, abort the wait via /v1/audio/jobs/{id}/cancel
Speak text	blazar tts "hello" --voice en_US-amy-medium	offline TTS via the piper lane (tts --install / --pull <voice>)
Discover voices	blazar tts --search <substr>	full rhasspy/piper-voices catalog — by language (en), locale (en_GB) or name (amy), with DISK bytes + pulled state
Attach a LoRA always-on	blazar lora add <model> /path/to/adapter.gguf	every spawn of the model loads the adapter (rows: blazar lora list, remove: lora rm <id>)
Try a LoRA per request	request model+adapter (e.g. "model": "qwen3-8b+rsmma")	a variant instance with ONLY that adapter spawns beside the warm dense model — no restarts, no config
Point an agent CLI at Blazar	blazar launch <command>	OpenAI/Anthropic/ollama base-URL env set, daemon ensured, command exec’d
See cache reuse	engine metrics endpoint (llamacpp:prompt_tokens_cached_total / llamacpp:prompt_tokens_total) · blazar why (cache_bust_system)	prefix-cache hit ratio; busting-client detection with fix hint

Prefix-cache behavior worth knowing

• Repeated prompts are nearly free. The engine caches the prompt KV state, so a growing conversation re-prefills only the new suffix (ratio = prompt_tokens_cached_total / prompt_tokens_total on the engine metrics endpoint). A cancelled request is also cheap to retry: the prefix computed before the disconnect is reused (measured: a ~8,200-token prompt aborted mid-prefill, retried, served 75% from cache in 1.5s).
• A mutating system prompt busts the cache every turn. Some clients embed counters or timestamps in the system message; because the system prompt is the prefix of the prefix, any byte change makes the whole conversation re-prefill from scratch. Blazar watches for this pattern (cache_bust_system in blazar why) and tells you the fix: keep the system prompt byte-stable and move per-turn values into the last user message.

Every way a user invokes Blazar

1. CLI blazar — full command surface (see 4.API_SPEC.md CLI table: serve/run/pull/import/list/show/ps/rm/create/cp/push(refused)/keys/quantize/bench/tune/launch/engine/lora/search/fit/config/upgrade/session/doctor/why/watch/snapshot/migrate/mmproj/drafts/coreside/whisper/tts/stop/completions + refused signin/signout/login/logout).
2. HTTP gateway — OpenAI (/v1/*), ollama (/api/*), Anthropic (/v1/messages) endpoints on 127.0.0.1:11435; plus Blazar-only surfaces: /v1/realtime voice WebSocket, /api/failover chain admin, /api/mcp tool-catalog status; see 4.API_SPEC.md.
3. Existing clients — any OpenAI SDK / ollama CLI / Anthropic SDK by setting base URL (ollama: OLLAMA_HOST).
4. First-party Python SDK — sdk/python, zero dependencies: chat() with tools/best_of/mcp, SSE streaming, ps()/explain()/failover(), realtime voice (Client.realtime()), Anthropic batches, completion metadata_update(); BLAZAR_URL picks the gateway.
5. Direct model files — weights under ~/.local/share/blazar/models/ are plain .gguf/safetensors, usable by external tools.

Configuration an end user can set

Managed via ~/.config/blazar/config.toml or blazar config set <key> <value> (dotted keys; quoted segments stay one segment). Highlights — see 7.SETUP.md for the full switch table:

• Serving: host, port, default_ctx, max_loaded_models, slots, deterministic
• Memory/quality: cache_type, kv_unified, cache_ram_mb, ctx_extend (YaRN), load_mode
• Behavior: spec, session_bank, singleflight, prompt_preflight, sentinel*, semantic_cache.*, audit_log
• Access: [[keys]], tls_cert/tls_key, cors_origins, [[remotes]]
• Routing: engine_routing.mode/policy, model_overrides.<model>.engine, .pin, .replicas
• Networking/observability: otlp_endpoint, otlp_service

Troubleshooting / FAQ

Symptom	Cause → fix
Port already in use at startup	another server (e.g. ollama on 11434) → set port in config or stop the other service; Blazar exits 3 by design instead of restart-looping
401 missing or invalid API key	keys configured → send Authorization: Bearer <plm-key> or x-api-key
403 no keys configured on /api/keys	key management needs at least one [[keys]] entry first
409 removing a key	it’s the last key — the gateway refuses to go authless
503 all_slots_busy	all slots saturated → wait, raise slots/max_loaded_models, or send x-blazar-priority: high for urgent work
502 with engine message	child failed mid-request; the next request auto-respawns — if persistent, blazar doctor
400 teaching error naming an install command	model needs an engine lane → blazar engine install --kind <kind>; unknown-architecture GGUFs also teach the curated lane when one exists (blazar engine install --lane <id>) or the pinned-fork build command
400 prompt refused (preflight)	prompt cannot fit ctx → raise ctx/num_ctx, quantize KV (cache_type), or pull a smaller quant (blazar fit)
422 sentinel enforce	response failed quality gate (schema/truncation/empty) with sentinel_enforce on → fix prompt/schema or disable enforce
Model slow to first token after idle	cold reload — consider warm_peg / keep_alive: -1 / session bank
blazar run eats my prompt as flags	flags must precede the prompt; quote leading - or use --
Weird model name errors containing #	# is reserved (replica keys) — rename without it
XDG warning on stderr at startup	XDG dirs unavailable → running with CWD fallback (deliberately loud)

Deeper docs

• Product shape: 1.SYSTEM_OVERVIEW.md
• Internals & lifecycle: 2.ARCHITECTURE.md
• Database: 3.DATA_MODEL.md
• Full HTTP + CLI reference: 4.API_SPEC.md
• Rules & limits: 6.BUSINESS_RULES.md
• Install/build/deploy/env: 7.SETUP.md
• Invariants for contributors: 8.DO_NOT_BREAK.md
• Numerics (VRAM/KV/GGUF): 10.SCIENTIFIC.md

Improve this page on GitHub