Documentation

System Overview

What Blazar is, the problems it solves, and its core concepts.

1. SYSTEM Overview

Detected project type: hybrid (HTTP gateway — crates/blazar-gateway/src/lib.rs router() on axum — plus CLI binary — crates/blazar-cli/src/main.rs Cmd enum on clap).

Canonical documentation for Blazar. Statements below were verified against v0.6.1 source unless tagged INFERRED. Commands and configuration spot-verified against v0.19.0 (2026-10-03). This file is the canonical source of truth for the product overview.

Problem statement

Running large language models locally today fragments the user across incompatible runtimes:

Pain	Concrete symptom
Engine lock-in	ollama ships a single forked llama.cpp; users of mistral.rs or SGLang must run and learn each engine’s own server CLI separately
API lock-in	Each runtime exposes one API dialect; clients written for OpenAI, ollama, or Anthropic cannot freely mix
Opaque resource behavior	Context silently clamped (ollama’s default 2048 ctx), VRAM/KV budgets not visible, models evicted without explanation
Modelfule/blob duplication	ollama copies GGUF weights into opaque blob layers, doubling disk usage and breaking other tools
Telemetry/cloud coupling	Runtimes that phone home or require accounts
Fork drift	Vendored engine forks lag upstream fixes and quant formats

Blazar solves this: it is a local-first, multi-engine inference server that orchestrates official (unforked) llama.cpp (llama-server), mistral.rs, SGLang, stable-diffusion.cpp (sd-server, diffusion GGUF component sets), and mlx-lm (python -m mlx_lm.server, MLX quant dirs on CUDA Linux + Apple Silicon) behind a single gateway speaking OpenAI, ollama, and Anthropic wire APIs simultaneously. Local-first is a hard product property: no telemetry, no cloud endpoints, no accounts (login/logout commands are refused by design). The one deliberate extension is opt-in peer federation ([[remotes]]): route, warm, and replicate across machines you configure — inference never touches a third-party cloud.

Target users and roles

Role	Interaction
Local developer	Uses `blazar` CLI or any OpenAI/ollama/Anthropic client pointed at the local HTTP gateway
Agent/tooling operator	Uses `blazar launch` to exec agent CLIs with base-URL environment pre-wired; uses tool-calling, JSON schema outputs, sessions
API-key administrator	When one or more `[[keys]]` exist in `config.toml`, manages named keys, scopes, quotas (rpm/tpm/daily_tokens/max_concurrent/weight) via `/api/keys`
Operator/SRE	Observes via `/api/ps`, `/api/why`, `/metrics`, `blazar doctor`, audit log, OTLP traces

There is no multi-tenant cloud role: the gateway binds 127.0.0.1 by default and auth is optional (active iff any key exists).

Core user workflows (happy paths)

Install & first run blazar engine update (sha256-verified official engine download, regression-gated; auto-rollback on >10% decode regression unless --no-gate) → blazar pull qwen3-0.6b (ollama registry shortname) or blazar pull ggml-org/Qwen3-8B-GGUF:Q4_K_M (HuggingFace owner/repo:QUANT) → blazar serve (daemon) → blazar run <model> (auto-pulls if missing).
Point existing clients at it OpenAI clients → http://127.0.0.1:11435/v1/...; ollama clients → OLLAMA_HOST=http://127.0.0.1:11435 ollama list works unchanged (re-point from 11434; a drop-in on 11434 is opt-in via port = 11434 once ollama is removed); Anthropic clients → /v1/messages.
Chat/generate with a model — request hits gateway, is queued (priority/SLO), routed to a spawned per-model engine child process (llama-server, mistral.rs serve, python -m sglang.launch_server, python -m mlx_lm.server for MLX quant dirs, or sd-server for diffusion component sets), translated to the child dialect, and translated back to the requested wire API. Streaming and non-streaming both supported; image generation/editing serve POST /v1/images/* through the same admission path.
Vision / embeddings / rerank / audio — mmproj-carrying models serve images (projector lazily attached); /v1/embeddings, /api/embed (late chunking opt-in), /v1/rerank; Whisper transcription lane (blazar whisper).
Voice, tools, resilience add-ons — /v1/realtime WebSocket voice turns (whisper STT → chat → piper TTS); [[mcp]] tool catalog over stdio or Streamable HTTP (per-request opt-in, gateway-mediated tool loop on every engine); [[failover]] alias chains over ordered local/remote targets; completion metadata cards; blazar ps shows child sleep state (blazar_sleeping).
Fit-before-download — blazar fit <target> previews VRAM fit and quant alternatives before pulling.
Observability — blazar ps (slots/ctx/ports/in-flight, load countdown, serving engine per row), blazar why (trace investigation), /metrics Prometheus exposition, blazar doctor (grouped SYSTEM/GPU/ENGINES/MODELS/CHANNELS/RUNTIME checks).

Tech stack

Layer	Technology
Language	Rust (workspace, edition 2024, MSRV 1.95), `unsafe_code = deny`, clippy pedantic
Crates	`blazar-core` (config/store/catalog/gguf/profile/hardware/telemetry), `blazar-runtime` (engine installer, HF client, supervisor, event bus, bench, quantize, daemon), `blazar-gateway` (OpenAI+ollama+Anthropic APIs, queue, sessions, semantic cache, sentinel, whisper, batch), `blazar-cli` (binary `blazar`)
HTTP	axum 0.8 on axum-server 0.7 (TLS via rustls 0.23, ring provider), hyper 1, tower-http 0.6 (CORS)
Async runtime	tokio 1
Database	SQLite via rusqlite 0.32 (bundled), WAL mode
Networking	reqwest 0.12 (rustls) for engine/model downloads
CLI	clap 4, rustyline 15 (REPL), indicatif (progress)
Schema validation	jsonschema 0.53 (gateway structured outputs)
Concurrency primitives	dashmap 6, parking_lot 0.12
Config	TOML (toml 0.9) + serde
Packaging	Homebrew, Scoop, Winget; shell installer (`scripts/install.sh`, `install.ps1`); systemd unit on Linux
License	MIT OR Apache-2.0

High-level architecture explanation

            ┌────────────────────────────────────────────┐
  clients ──▶│  blazar-gateway (axum, 127.0.0.1:11435)   │
 (OpenAI /  │  layers: request_log → auth → CORS →       │
  ollama /  │           body-limit(50MiB) → session-pin  │
  Anthropic)│  priority queue (EDF + WFQ) → translation  │
            └───────┬────────────────────────────────────┘
                    │ spawn/monitor/evict per model
            ┌───────▼────────────────────────────────────┐
            │  blazar-runtime supervisor                │
            │  per-model child processes:                │
            │   • llama-server (llama.cpp)      [gguf]   │
            │   • mistral.rs serve              [gguf/f16]│
            │   • sglang.launch_server          [safetensors]│
            │   • sd-server (sd.cpp)            [diffusion gguf sets]│
            └───────┬────────────────────────────────────┘
                    │ metadata, catalogs, profiles
            ┌───────▼────────────────────────────────────┐
            │  blazar-core: config.toml, SQLite store   │
            │  (engines/models/profiles/loras/usage/bench)│
            │  GGUF metadata parser, hardware probe,     │
            │  VRAM/KV math, profile (argv) compiler     │
            └────────────────────────────────────────────┘

One gateway, many engines. The gateway terminates all client APIs; children speak only their native dialect. The blazar-cli binary drives everything (serve, pull, run, engine management).
Engine routing is data-driven (engine_kind.rs): quantized safetensors (AWQ/GPTQ/FP8) route to SGLang only; GGUF routes to llama.cpp then mistral.rs; plain safetensors route by policy (Quality/Throughput → SGLang first, Latency → mistral.rs first); diffusion component sets (flag-keyed DiT+VAE+text-encoder rows) route to sdcpp only, and text models can never route to sdcpp — a bidirectional domain gate. engine_routing.mode = "manual" pins a single active engine; model-level engine override wins over both.
State is explicit and local: SQLite at <data_dir>/blazar.db, config at ~/.config/blazar/config.toml, model files as plain .gguf/safetensors under ~/.local/share/blazar/models/ (usable by any external tool), engine binaries under engines/, per-child runtime secrets under run/.
Observability built in: trace IDs (plm-<hex>-<seq>), access logs, Prometheus /metrics, OTLP export (opt-in), sentinel quality observations, why trace investigation.

Improve this page on GitHub