The control plane for local AI inference.
Unified APIs. Intelligent routing. Resource-aware scheduling. Multi-engine serving. Blazar manages the model store, picks or pins the right inference engine, fits workloads to your hardware, and controls concurrency, lifecycle, and diagnostics.
Running in under a minute
One install script, one command to find a model, one to serve. Your existing OpenAI, Ollama and Anthropic clients keep working — point them at the local gateway.
# install (macOS, Linux, WSL) $ curl --proto '=https' --tlsv1.2 -fsSL \ https://raw.githubusercontent.com/santanu20/blazar/v0.27.0/scripts/install.sh | sh # windows powershell $ irm https://raw.githubusercontent.com/santanu20/blazar/v0.27.0/scripts/install.ps1 | iex # find, pull, serve, run $ blazar search qwen3 instruct # find models on Hugging Face $ blazar pull qwen3-0.6b # registry shortname $ blazar serve # gateway on http://127.0.0.1:11435 $ blazar run qwen3-0.6b # interactive REPL blazar 0.27.0 — powered by upstream llama.cpp, mistral.rs and SGLang
Why Blazar
A control plane, not another runtime. Blazar sits between your applications and every inference engine on your machine.
Multi-engine serving
llama.cpp, mistral.rs, SGLang and MLX serve text and embeddings; sd.cpp, whisper.cpp and Piper cover images, video, speech-to-text and TTS — all behind one endpoint.
Resource-aware scheduling
VRAM and KV-budget planning decides what fits, what queues, and what evicts — measured per model and per engine before the first token.
Reliability by default
Durable jobs survive restarts, request cards can be cancelled or interrupted mid-stream, and response chains persist on disk.
Federation
Peers share capacity signals; the gateway ranks warm models, replicates hot ones, and routes across a small fleet — explicit, operator-controlled.
Multimodal in one place
Image generation and upscaling, streaming transcription with Silero VAD, text-to-speech — same gateway, same API surface, same diagnostics.
Diagnostics built in
blazar explain shows the routing decision for any model, blazar doctor audits the install, and per-request telemetry is first-class.
Seven lanes, one binary
Blazar never bundles engines — it manages official upstream builds, picks the right lane per model, and upgrades them independently.
Architecture
Applications speak OpenAI-, Ollama- or Anthropic-shaped HTTP. Blazar translates, admits, routes, and supervises the engine processes.
Measured, not marketed
Direct-engine parity at C=1, orchestration wins as concurrency grows. Reproduce every number with the commands in the benchmark report.
Documentation
Ten deep dives — concepts to schema to every HTTP surface — plus the research notes that shaped the engine lanes.
System Overview
What Blazar is and the problems it solves
Architecture
The modular monolith and its processes
Data Model
SQLite schema v10, tables, migrations, invariants
API Specification
OpenAI-, Ollama- and Anthropic-compatible surfaces
Business Rules
Routing, admission and validation decisions
Setup
Install, configuration, hardware, deployment
Do Not Break
Maintainer invariants and release contracts
Usage
Operating Blazar day to day
Engineering & Numerical Methods
GGUF parsing, KV estimation, routing math
Star the repo, join the discussion
Blazar is developed in the open. Issues, ideas and benchmarks welcome.