blazar
Documentation

API Specification

Every HTTP surface with request and response examples.

4. API Specification

Blazar exposes one gateway with three API dialects (OpenAI, Ollama, Anthropic) plus a CLI. Every route below was verified in crates/blazar-gateway/src/lib.rs (router()); every CLI command against blazar --help.

Authentication

  • Auth is active iff at least one [[keys]] entry exists in config.toml (or BLAZAR_KEYS env). With no keys configured the gateway is open.
  • Secret sources, in priority order: Authorization: Bearer <secret> header, then x-api-key header (Anthropic convention).
  • Failure → 401 with body {"error":{"message":"missing or invalid API key (Authorization: Bearer or x-api-key)","type":"blazar_error","code":401}}.
  • Open (authless) paths: GET /healthz, GET /health.
  • Middleware layer order (outermost→innermost): request-log (trace id plm-<hex>-<seq>, echoed as x-blazar-trace-id) → auth → CORS → 50 MiB body limit → session pin.
  • Per-key limits enforced at admission on every generation lane (text, images, video, audio): max_concurrent lease (held until the response body drains), rpm, tpm (chat-family prompt+completion tokens), daily_tokens (UTC day), model scopes (models list; empty = all models + admin). Media lanes charge request-count with the lane’s own model as scope target (diffusion set / whisper size / piper voice id).

HTTP routes

OpenAI-compatible

Method + route Purpose
POST /v1/chat/completions chat (stream + non-stream; byte-stream proxy, singleflight on identical non-stream); multimodal parts (image, video, input_audio — typed or bare key) are detected and routed to the projector lane; n (choices per request, integer 1..=8, strict) is validated fail-fast (400 before admission), passed through, and verified on non-stream responses — an engine that silently ignores it gets a teaching 502; the serving lane’s own ceiling is checked first in the gateway’s voice (llama.cpp lane caps n at 2, live-probed b11339+b11344; unprobed lanes keep the plane-wide 8); a non-stream completion created with a metadata object persists a slim card (≤16 string keys × 512 chars, validated) addressable by the update route below; mcp (string, "all", a configured [[mcp]] server name, or "none" to opt out) opts the request into the gateway-mediated MCP tool loop — buffered, so stream:true + mcp gets a teaching 400; the field is stripped before forwarding; with mcp_default set in config, requests carrying no selector inherit it (x-blazar-mcp header > body field > default > none)
POST /v1/chat/completions/{id} update a completion card’s metadata post-hoc (OpenAI UPDATE semantics); returns the card {id, object: "chat.completion", created, model, metadata} — unknown ids (created without metadata, streamed, aged out of the 4096-card window) get a teaching 404
POST /v1/completions text completion
POST /v1/embeddings embeddings (late-chunking when enabled)
POST /v1/rerank, POST /v1/reranking, POST /rerank, POST /reranking rerank (Jina alias; bare spellings are the upstream-native forms)
POST /v1/messages Anthropic Messages (thinking/budget supported); usage carries cache_read_input_tokens from the child’s cached-token accounting on responses and the final stream message_delta
POST, GET /v1/messages/batches, GET, DELETE /v1/messages/batches/{id}, POST /v1/messages/batches/{id}/cancel, GET /v1/messages/batches/{id}/results Anthropic Batch API (upstream shapes: msgbatch_ ids, ISO timestamps, request_counts, jsonl results; items replay through /v1/messages; 24 h retention; DELETE archives)
GET /v1/realtime WebSocket voice turns (upgrade; ?model=, ?voice=, ?stt_model=): input_audio_buffer.append/commit → input_audio_transcription.completed → response.audio_transcript.delta/.done → response.audio.delta/.done; plus session.update (voice/stt_model/instructions — applies from the NEXT turn, session.updated echo), input_audio_buffer.clear (…cleared), response.create (force a turn on buffered audio), response.cancel (honored between pipeline stages — the chat stream itself is not mid-cancellable), conversation.item.create (text items feed later turns as context; bounded 16 items / 8192 chars, oldest evicted), input_audio_transcription.failed on STT errors; out-of-band events during a running turn get a teaching error; bounded 20 MiB buffer, 120 s idle / 600 s hard TTL; bearer-key middleware guards the upgrade
POST /v1/responses + POST /responses Responses API
GET /v1/responses/{id} stored-response retrieval (durable: 24 h TTL / 256-entry hot LRU write-through to the SQLite ledger; a miss in memory promotes from disk, so stored responses survive gateway restarts)
POST /v1/responses background: true background mode (requires store: true, non-stream): 202 with status: queued, the roundtrip runs durably as a /v1/jobs job (kind responses); poll GET /v1/responses/{id} — it serves the verbatim stored body once complete, the job status while in flight, the failure as status: failed if it fails. Stream+background is a 400 contract error
GET + DELETE /v1/conversations/{id} named conversation threads: responses filed under "conversation": "name" (inherited through previous_response_id chaining) list chronologically; DELETE removes the thread’s stored responses. A conversation exists once a stored response names one
POST /v1/batches, GET /v1/batches, GET /v1/batches/{id}, POST /v1/batches/{id}/cancel OpenAI Batch API (each line replays through auth/quota)
POST /v1/files, GET /v1/files/{id}, GET /v1/files/{id}/content file upload/download for batches
POST /v1/audio/transcriptions (+ /audio/transcriptions) whisper transcription (multipart); timestamp_granularities[] (word/segment, array or comma form) is consumed gateway-side: it forces verbose_json (an explicit different format is a teaching 400) and word switches the child into token-timestamp mode — the gateway derives per-segment words[] from token timings (SentencePiece-aware grouping, never interpolated; x-blazar-word-timestamps: derived\|unavailable header states which); engine knobs whitelisted verbatim: decode shaping (temperature, prompt, beam_size, best_of, …), timing (token_timestamps, suppress_non_speech, suppress_nst), the VAD suite (vad_simple, vad_threshold, vad_min_speech_duration_ms, vad_min_silence_duration_ms, vad_max_speech_duration_s, vad_speech_pad_ms, vad_samples_overlap), and the stereo pair (diarize, no_language_probabilities — drops language_probabilities from verbose_json); response_format=diarized_json maps stereo diarization into {duration, text, segments[].{id,start,end,text,speaker}} (one voice per channel; (speaker N) prefixes stripped; 502 teaching if the child returns nothing mappable — never a wrong shape under the name); cloud-only fields teach 400 with the local menu (include, keywords); diarized_json on the stream lane teaches 400; async=true form field returns a gateway-owned job handle instead (see /v1/audio/jobs/{id}); stream=true returns progressive SSE (see Streaming transcriptions below — granularities apply to the buffered lane)
POST /v1/audio/translations (+ /audio/translations) whisper translation — same multipart shape and knob surface as transcriptions (granularities, VAD/timing fields, teaching 400s), forced translate=true; verbose_json adds task/language/segments; same async=true and stream=true knobs
GET /v1/audio/jobs/{id} poll an async audio job (queued/running/completed/failed/cancelled); result carries the upstream body; live handles die with the gateway process, but every job is also written to the durable ledger — after a restart the completed record is served from SQLite
POST /v1/audio/jobs/{id}/cancel abort the wait of a queued/running audio job (in-flight child inference runs to completion internally; the result is discarded)
GET /v1/audio/capabilities gateway-derived audio lane menu: engine presence, pulled models, live child (loaded size), endpoints, async + stream knobs, idle_timeout_secs; boots nothing
POST /v1/audio/speech offline TTS (piper): model = piper voice id; response_format wav (default — documented deviation, OpenAI defaults to mp3), pcm/pcm16 (streamed PCM16), or mp3/opus/aac/flac (buffered WAV transcoded via optional system ffmpeg, 60 s budget → 504, ffmpeg failure → 502, absent → teaching 400 naming wav/pcm and the install hint); "stream": true streams the lossy family live — each sentence is synthesized raw (--output_raw, the voice’s own sample rate) and fed into one persistent ffmpeg pipe, bytes flowing as produced (first audio ≈ first sentence); wav+stream teaches 400 (a WAV header needs total lengths up front — use pcm), pcm streams exactly as before, and the default (no stream flag) stays a fully-buffered byte-identical body; speed 0.25..=4.0 plus native piper knobs speaker, noise_scale (0..2), noise_w (0..2), sentence_silence (0..10 s); instructions gets a teaching 400 (voices are fixed personalities — pick a different voice); remote name:model forwards
GET /v1/audio/voices voice menu: installed piper voices ({object: voice, id, language, name, quality, installed}) plus optional ?search= remote HuggingFace voice search (URL-encoded, length-capped, 20 s budget, results capped at 50 with truncated flag) — a failed/timed-out search still returns the installed list with a search_error note

Streaming transcriptions (stream=true)

Progressive transcription over server-sent events, decoded gateway-side: the uploaded WAV is split at PCM frame boundaries into whisper_stream_chunk_ms windows (default 30 s, range 1-120 s) and each window is transcribed in order on the same lazy whisper child — a local lane feature (remote audio targets ignore stream). Because the child serializes inference anyway, total throughput is unchanged; partial text simply arrives while later audio is still decoding.

  • Request: multipart with file, model, and stream=true (1 accepted). stream and async on one body is a 400 request-shape error (checked before lane availability). Window-shaping fields the child would honor per-chunk (response_format, no_timestamps, duration, offset_t) are dropped and response_format=verbose_json is forced so every chunk reports segments.
  • Non-WAV uploads stream as a single chunk (no progressive split possible).
  • SSE events (content-type text/event-stream, Cache-Control: no-cache):
    • chunk.completed — {index, offset_ms, text, segments} per window; segment start/end are rebased to whole-file time.
    • transcript.completed — final {duration_ms, language, text, segments, chunks}.
    • error — teaching message, then the stream closes.
    • : keep-alive comments every 15 s while a chunk decodes.
  • Client hangup stops decoding at the next chunk boundary (spent child time is bounded by one window). | POST /v1/images/generations | image generation (sdcpp lane): model = diffusion checkpoint/component set, prompt + size WxH + steps + n, PNG b64_json out. "async": true returns a job handle instead (poll_url/cancel_url); "stream": true answers SSE progress frames; rich native fields (negative prompt, cache, lora, guidance, vae_tiling_params, seed) are shallow-translated to sd-server’s native dialect. OpenAI knobs consumed gateway-side: quality (low/medium/high/xhigh/max/auto + standard/hd synonyms) scales sample_steps relative to the family’s live capabilities default (explicit steps + quality is a teaching 400 — two dials for one knob); style (vivid/natural) presets guidance.txt_cfg for curated families only (else teaching 400 with the native knob named); response_format accepts b64_json (url teaches — no asset hosting); output_format png/jpeg/webp passes through; moderation/partial_images/background/input_fidelity teach 400 honestly | | POST /v1/images/edits | instruction image edits (multipart): diffusion sets with a vision encoder (--llm_vision) | | POST /v1/images/variations | img2img variations (multipart): one image file → native init_image with a random seed (prompt optional — neutral conditioning when absent, digit prompts stay strings); typed fields coerced (n, seed, strength, output_format, output_compression, clip_skip, control_strength, ip_adapter_strength); mask and other file parts teach toward /v1/images/edits; same admission, teaching, and async/stream job modes as generations | | POST /v1/images/upscale | standalone ESRGAN upscale (sd-server ≥ master-929): sync, no diffusion model load — upscaler weights come from the engine’s --hires-upscalers-dir; body {image (base64/data-URL, required), upscaler, repeats 1-4, tile_size, output_format png/jpeg/webp, output_compression}; ?model=NAME boots a diffusion child on demand, else serves from a live one | | GET /v1/images/jobs/{id} | poll an async image job; resolves live sd-server children, then falls through to the durable ledger (completed records survive child/gateway death); a non-terminal row whose child is gone (eviction, crash or restart) is closed as failed with a resubmit hint on first poll — rows never wedge in queued/in_progress | | POST /v1/images/jobs/{id}/cancel | cancel a queued/generating job (verdict relayed from the child); with no live child, open ledger rows close cancelled directly | | GET /v1/images/capabilities | live sampler/scheduler/cache/limits catalog from the running diffusion child; ?model=NAME narrows to that model’s child (400 with the live menu when the name matches none) | | POST /v1/videos/generations (also POST /v1/videos) | video generation (sdcpp lane, video families e.g. Wan 2.1 T2V): model + prompt + native video fields (video_frames, fps, width/height) shallow-translated to sd-server’s /sdcpp/v1/vid_gen; seconds (Sora spelling — "4"/"8"/"12" or int) converts to frames at the effective fps, taken from the child’s vid_gen capabilities default when unset and normalized to the engine’s 4n+1 cadence (the completed frame_count in the result is the truth); VRAM scratch admission refuses over-budget requests and teaches the "vram_overcommit": true lever — which operators can centrally disable with BLAZAR_VRAM_OVERCOMMIT=0 (requests using the lever then get a teaching 400, and the refusal names the remaining levers only); remixed_from_video_id gets a teaching 400 (no local image-to-video-from-video); "async"/"stream" job modes as on images | | GET /v1/videos | Sora-style listing over the durable video-jobs plane (kind=video): ?limit= default 10 (1..100), newest-updated first, {object: "list", data[], first_id, last_id, has_more} | | GET /v1/videos/{id} | Sora video object: {id, object: "video", status} with queued/running → in_progress, completed, failed, cancelled; progress (0..100) when the live child reports it (probed on the current sd-server build: the child exposes no mid-run fraction — 0 while queued, 100 at completion, omitted in between; partial previews are therefore not offered, probed-negative); prompt/size/seconds echoed from the request; completed rows carry seconds = frame_count/fps of the finished result and completed_at; 404 for non-video or unknown ids | | GET /v1/videos/{id}/content | finished container bytes with the job’s own mime (video/webm; animated webp serves image/webp; avi serves video/x-msvideo); 404 while in progress (poll {id}), 410 when the artifact was pruned | | DELETE /v1/videos/{id} | Sora delete: queued/running rows get a best-effort child cancel + close cancelled; completed/failed rows have their render purged (result dropped, /content 404s afterward, spilled artifact file removed). {id, object: "video.deleted", deleted: true} returns (idempotent on every state) | | GET /v1/videos/jobs/{id} | poll an async video job; resolves live sd-server children, then falls through to the durable ledger (completed records survive child/gateway death); orphaned non-terminal rows close as failed with a resubmit hint on first poll (same crash-window closure as the images plane — Sora GET /v1/videos/{id} applies it too) | | POST /v1/videos/jobs/{id}/cancel | cancel a queued/generating video job (verdict relayed from the child) | | GET /v1/videos/capabilities | live catalog from the running diffusion child (video modes advertised per loaded family); ?model=NAME narrowing as on images | | GET /v1/jobs | unified durable job plane across every async lane (audio/image/video/doctor): filters state kind limit (default 50, max 1000), newest-updated first | | GET /v1/jobs/{id} | one job: status, kind, model, result (≤ 1 MiB inline) or artifact stub, error with teaching text on failures | | POST /v1/jobs/{id}/cancel | cancel (idempotent on terminal rows; routes to the owning lane live handle when in-flight) | | GET /v1/jobs/{id}/events | append-only audit trail (created / running / probe verdicts / terminal), seq-ordered, bounded 1000 | | GET /v1/jobs/{id}/artifact | serve the job’s input artifact or oversized result file from disk (410 if pruned) | | GET /v1/requests | live request cards for generation routes: filters model state; in-flight first, then the bounded per-boot terminal ring; API-key-scoped when keys are enabled | | GET /v1/requests/{id} | one card: trace_id, model, engine, state (queued/running/done/cancelled/interrupted/disconnected), tokens when known, queue/compute timings, bounded 64 KiB partial | | POST /v1/requests/{id}/cancel | stop a running generation from another connection (499 to the original client, stream ends) | | POST /v1/requests/{id}/interrupt | stop generation but KEEP the partial output (stream ends with a clean dialect terminal frame) | | POST /infill | FIM infill | | POST /completion | native llama-server prompt completion (no /v1 prefix) | | POST /embeddings | native llama-server token-level embeddings (pooling=none batch dialect — distinct from /v1/embeddings) | | POST /v1/chat/completions/control | steering vectors | | POST /v1/chat/completions/input_tokens, POST /v1/responses/input_tokens, POST /responses/input_tokens, POST /v1/messages/count_tokens | token counting | | POST /v1/systemone | TypeSafe-compatible System One (typed questions about a state) — llamacpp lane only, no streaming; the child implements the semantics | | POST /tokenize, POST /detokenize, POST /apply-template | tokenizer/template utilities | | GET /v1/models | model list (resolved engine per model; capabilities array: vision when a projector is attached — blazar-native shape mirroring /api/show) | | GET+POST /v1/adapters | LoRA adapter management | | GET /props, GET /slots, POST /slots/{id} | engine-scoped surfaces (model from X-Blazar-Model > ?model= > single hot child; /props requires an engine whose manifest carries --props — the spawn emits it when supported) | | GET+DELETE /v1/stream, POST /v1/streams/lookup | stream retrieval |

Native llama-server dialect notes: /infill, /v1/systemone, /completion, and /embeddings bodies carry no model field upstream (it is a router-mode-only field there). Blazar resolves the target with the same chain as the engine-scoped surfaces — X-Blazar-Model header > ?model= query > the single hot child — and OpenAI-dialect paths (/v1/chat/completions, /v1/completions, …) keep the hard missing-model 400. Upstream surface parity is pinned in docs/upstream-api-inventory.json and enforced by scripts/validate.py --phase=inventory.

Ollama-compatible

Method + route Purpose
GET /api/version version
GET /api/tags installed models (+ resolved engine)
POST /api/show model details (capabilities: completion always, vision with mmproj)
POST /api/delete delete model
GET /api/ps live instances (slots/ctx/ports/in-flight, keep-alive countdown, effective spec mode + draft via blazar_spec/blazar_draft, serving engine tag via blazar_engine, lifetime gateway-observed cache-hit ratio via blazar_cache_hit — null until classified; blazar_sleeping when a llama.cpp child reports its idle-sleep state via /props — absent means unknown; remotes lines carry a cached capacity summary — devices, total/free VRAM, residents, snapshot age — absent when the peer is not a Blazar gateway)
POST /api/pull pull model
POST /api/chat chat (NDJSON streaming; logprobs/top_logprobs mapped onto ollama shape; options.spec = per-request spec mode; tool-result messages may carry tool_name — mapped onto the child’s name slot, explicit name wins)
POST /api/embeddings, POST /api/embed, POST /api/rerank embeddings/embed/rerank
POST /api/generate generate (full parity: system, images[], think)
POST /api/evict evict a model now
POST+GET /api/session session pin open/close + status
GET /api/sessions list active session pins
GET /api/why sentinel record ring (per trace)
GET /api/capacity per-GPU truth: live nvidia-smi census (total/free VRAM, utilization) joined with resident children (held VRAM per child via compute-apps) and foreign compute processes (external); CPU-only hosts get an honest empty census note
GET /api/fabric the ComputeFabric read-model: one JSON inventory of CPU (cores, RAM), GPU devices (census-backed VRAM total/free, utilization, per-device resident children + foreign compute apps), assignments (every live instance with engine/device/ctx/slots), installed engines (tag, kind, active), remote peers (devices + residents from the capacity cache), and honesty notes; placement decisions stay with the spawn planner — this is inventory, never a scheduler
GET /api/quantiles latency + cache observability: TTFT/TPOT p50/p95/p99 in ms with observation counts (null when empty — a fresh boot has no data), warm/cold TTFT split, prompt-cache tokens + hit rate, semantic-cache hits/misses/stores
GET /api/benchmarks stored measurement inventory: one row per (model, engine lane) merging durable blazar bench results (measured date + test names) with tuned profiles (score + tuned date); pure read of stored records — nothing re-runs
GET /ui the web console: a zero-dependency, offline, self-contained HTML page (no CDN, no external refs) with Dashboard / Models / Engines / Compute / Benchmarks / Jobs / Sessions tabs polling the read-only APIs every 5s; same bind + reachability as every other endpoint
GET /api/explain/{model} effective-config card with provenance: engine lane + routing reason, requested vs effective context WITH source, slots (live vs sized-at-spawn), speculation, KV cache types, semantic-cache scope, admission counters, residents with warnings; name:quant forms resolve through the canonical ladder. The cache block’s kv_advisory states the current KV grade, next ladder step, and why, plus a measured array of perplexity receipts when the exact checkpoint has them — one entry per measured context window, per-rung grades, compiled from crates/blazar-gateway/registry/kv-quant-quality.json, engine/corpus/ctx-stamped, null when unmeasured)
POST /api/warm make a model resident on this gateway. Default mode = the pull-notify contract (detached, policy-gated, answers {"status":"ok"}); "wait": true = synchronous ensure through admission (low priority, never starves traffic) that reports lane, engine, load_ms and the live resident state. Vision projector attaches on first vision request; the idle TTL governs residency after
POST /api/replicate remote replication: fan a synchronous warm out to named peers (or every configured remote), concurrently, each bounded by timeout_secs (default 300, max 900); per-peer results verbatim — non-Blazar peers are called out as having no /api/warm; capacity cache refreshed for touched peers
GET /api/route/{model} cross-node scheduling explainer: local residency, per-peer tier (warm/unknown/cold), wait estimate, free VRAM, breaker state, and the decision a request would take (local / remote:<peer> / none with teaching reason — never a fabricated target)
POST /api/model-doctor run bounded capability probes (chat/stream/JSON schema/tool call/embeddings; static vision+think) through the REAL gateway path as a durable /v1/jobs job; certificate lands in the store
GET /api/model-doctor/{model} last capability certificate (PASS/FAIL/N/A per capability with receipts)
GET /api/watch live SSE sentinel tail
GET /api/mcp MCP tool-catalog status plane — lists configured [[mcp]] servers (name, transport stdio/http, command or url, timeout) report-only; never spawns a server to answer
GET /api/failover, GET /api/failover/{chain}, POST /api/failover/{chain}/pin, POST /api/failover/{chain}/unpin failover-chain admin plane: list chains with targets/sticky/benched state + bounded switch history, per-chain detail (by name or alias), pin a target index (a failing pinned target keeps serving — no switch), release the pin
GET /api/events event stream

Keys management [ADMIN]

Authless gateways REFUSE management (403, teaching message). Only unscoped keys (empty models = admin) may manage.

Method + route Purpose
GET /api/keys list keys (secret redacted to prefix8+…; usage per UTC day)
POST /api/keys add: name charset [A-Za-z0-9._-] else 400; 409 on duplicate; rpm/tpm/daily_tokens/max_concurrent validated non-negative (default 0), weight default 1; secret = plm_ + 32 hex, shown ONCE
DELETE /api/keys?name= remove; 404 unknown; 409 refusing to remove the last key (would leave the gateway authless)
POST /api/keys/rotate?name= new secret (old dies immediately); name/scopes/limits/usage kept

Persistence: full-file config.toml round-trip (parse→swap→validate→atomic temp+rename, timestamped .bak) under a process-wide lock; registry rollback on persist failure.

Discovery & observability

Method + route Purpose
GET /.well-known/blazar capability manifest: name/version/engine identity/apis/endpoints census/headers/features (keys, tls, otlp, remotes, singleflight, session_bank, prompt_preflight, session_pins, semantic_cache)
GET /healthz, GET /health liveness (authless, plain ok)
GET /metrics Prometheus counters
GET /v1/stream family see OpenAI table

Request headers (blazar extensions)

Header Purpose
Authorization: Bearer / x-api-key auth secret
x-blazar-trace-id echoed trace id
X-Blazar-Priority high/low (case-insensitive; else Normal) — EDF queue tiers 2 s/30 s/120 s
X-Blazar-Deadline-Ms strict deadline (never WFQ-delayed)
X-Blazar-Num-Ctx per-request ctx on the OpenAI path (restart-once semantics)
X-Blazar-Spec per-request spec mode on the OpenAI path (off/auto/eagle3/… — same semantics as options.spec; same-shape = no restart)
X-Blazar-Enforce 1/0 per-request sentinel hard mode (422 on violations, non-stream chat)
X-Blazar-Session session pin (holds eviction for session_keep_secs)
X-Blazar-Model engine-scoped surface model selection
X-Blazar-Cache semantic cache override: on|off, -ttl 1..=86400, -threshold 0.01..=1.0
x-blazar-remote names the remote-pool member that served
x-blazar-structured-output response receipt: json_schema / json_object (from response_format or ollama format) / gbnf (grammar passthrough) — present only when the request was constrained
x-blazar-failover names the chain that served an alias request
x-blazar-mcp rounds=N receipt on gateway-mediated MCP tool loops (request opt-in: body "mcp" field or x-blazar-mcp header; config default mcp_default applies when neither is present)
x-blazar-served-model the local:/remote: target actually serving (failover lanes)

On-demand LoRA variants (model+adapter)

Any chat/generate/completions/embeddings request may name model+adapter instead of a bare model: the stem matches the file stem of an adapter attached with blazar lora add <model> <file> (colon was rejected as the separator — it is ollama’s model:tag). Semantics:

  • The variant spawns as its OWN instance (key model+adapter), so the dense model stays resident and warm — swap per request, no restarts.
  • The variant loads ONLY the named adapter (base spelling resolves through the usual exact/:-swap/prefix ladder first).
  • Unknown or duplicated stems 404 with the blazar lora add / blazar lora rm remedy instead of guessing.
  • blazar ps shows the variant as model+adapter; KV session banks, prefix heat, and keep_alive matching key the plain model name.
  • Spawn-shape options (options.num_ctx, options.spec) apply to the next spawn of the model family — variant or dense, whichever spawns first.
  • Router mode does not spawn per-model children: plus-syntax requests 404 with the /v1/adapters alternative (the router child’s dynamic adapter surface).

Error model

  • 401 invalid/missing key (shape above). 403 management on authless gateway.
  • 413 audit body over sniff cap; late-chunking beyond late_chunking_max_tokens.
  • 422 sentinel enforce (non-stream chat; codes + detail + fix hints).
  • 502 engine failure (engine text surfaced; dead children reaped so the next request respawns). 503 all_slots_busy (bounded queue wait). 400 teaching errors ( unfit ctx pins name all four levers; unknown routes/flags; sglang admission gates — quant_method outside the lane’s loader set, or a multimodal config missing preprocessor_config.json, both name the artifact and the levers pre-spawn). 507 honest memory-floor refusal (spawn_mem_guard: MemAvailable < model + floor; message names the numbers and the opt-out).
  • Port already bound → daemon exits with code 3 (RestartPreventExitStatus for systemd).

Tool calling and reasoning control

Tool calling

  • OpenAI-compatible: tools and tool_choice on /v1/chat/completions bodies; the preflight only treats a body as tool-bearing when it actually carries a tools member.
  • n choices vs best_of: n returns ALL N completions in choices[] (completions routes); best_of judges N generations and returns ONE winner (/v1/responses, Anthropic, ollama chat). Different routes, no knob collision. Token accounting: llama-server reports completion tokens per-choice — the gateway multiplies the sniffer charge by n for llamacpp lanes (live-probed); streaming and >ENFORCE_BODY_CAP responses skip the count-verify (documented limit). Per-lane engine ceilings surface as the engine’s own honest 4xx — live-probed llama-server b11339 caps n at 2; the gateway range stays the API contract.
  • Ollama-compatible: tools array on /api/chat (and /api/generate where supported).
  • Anthropic-compatible: tools are mapped to the engine dialect; tool_choice.type = "tool" must carry a non-empty tool_choice.name (rejected with a teaching error otherwise).
  • Tool-call deltas and arguments are translated back into each client’s native response shape, so OpenAI, Ollama, and Anthropic SDKs see their own convention.

Reasoning budget

  • OpenAI-compatible: top-level reasoning_effort is bridged into the engine’s chat_template_kwargs.reasoning_effort; an explicit per-request value wins over model config.
  • Anthropic-compatible: thinking.type = "enabled" with budget_tokens maps to the engine thinking budget; max_tokens must exceed budget_tokens (teaching 400 otherwise). Prior-turn thinking text is preserved across turns where the dialect carries it.
  • Per-model default: reasoning_budget in the model configuration; unset means the engine template default (auto).
  • Responses API routes pass reasoning through to the engine unmodified — no cross-dialect translation on that surface (chat completions and message routes translate).
  • Raw <think>...</think> leakage: when a request does not ask for thinking (no thinking controls present), the gateway strips raw think blocks from chat completions responses — buffered and SSE, all three client dialects. Ask explicitly (think: true, chat_template_kwargs.enable_thinking, enable_thinking, or a non-off reasoning_effort) and the reasoning ships intact.
  • mistral.rs engines expect top-level enable_thinking (bool) and reasoning_effort (off|low|medium|high|xhigh), and default thinking ON when both are omitted. On chat routes the gateway injects enable_thinking: false for control-free requests (fast answers by default), aligns top-level controls with chat_template_kwargs when both are present, and maps OpenAI effort names (minimal→off, max→xhigh).

CLI

blazar <COMMAND> — grouped as in --help. Flags before the prompt for run; --json where noted emits machine-typed JSONL on pure stdout.

Serve & chat

Command Purpose
serve (alias start) run the daemon in the foreground (all three APIs on one port)
run <model> [prompt...] chat REPL / one-shot; auto-pulls missing model; flags must precede prompt (leading - needs quoting or --); -v eval counts; --max-tokens single-shot only; REPL slash-commands: /exit /clear /model /sysinfo /profile
ps live instances: state, ctx, in-flight, idle countdown (--reset breaker clear, --json)
stop [model] stop daemon; with model = unload that model now
launch COMMAND exec an agent CLI against the daemon (sets base-URL env; --warm, --key)
session save/restore/rm/list slot KV checkpoints per model (survive unload/restart)

Model management

Command Purpose
pull <target> owner/repo[:QUANT] or catalog shortname; --force format-flip replace; --verify re-hash the pulled file against its recorded sha256 instead of pulling (single-file rows; dir rows refused)
import <path> import existing GGUF (hardlink default; --name --quant --copy --mmproj)
cp zero-byte hardlink alias
create Modelfile alias (FROM + PARAMETER num_ctx + ADAPTER → config overlay, no blob copy)
rm <models...> remove (refuses while running)
list / ls list models (--json JSONL)
show <model> details, active profile, last benchmark (--json)
quantize derive quantization via engine llama-quantize (-t, --imatrix, --verify perplexity gate, --max-degradation, --allow-requantize)
mmproj attach vision projector (refuses while running)
search HF search (every format --format any, --quant <q>, --json)
fit <target> pre-download VRAM/RAM fit preview + quant alternatives (--json)
lora add/rm/list LoRA adapter management

Tuning & benchmarks

Command Purpose
bench <model> pp/tg benchmark (history feeds tune gates)
tune <model> --search grid argmax; live probes --slots/--ngram/--load/--replicas/--cache-reuse; --ctx, --spec
drafts <model> speculative draft candidates (EAGLE3/MTP + small same-family)
coreside VRAM co-residency plan (hot-first greedy)

Engine & config

Command Purpose
engine update install + activate upstream engine (sha256-verified, regression-gated; --kind, --tag, --check, --no-gate)
engine build cuda\|cpu source build (auto GPU-arch, host-compiler match; --arch, --jobs); --fork owner/repo@<sha> / --repo + --ref builds a pinned capability lane (immutable commit, architectures mined from source, provenance in engine list)
engine offers curated capability-lane registry catalog (--arch, --json, BACKENDS column; --backend on install must be one the lane advertises; disabled via capability_registry_url = "" or BLAZAR_CAPABILITY_REGISTRY="")
engine install --lane <id> [--backend cpu\|cuda] build a curated registry lane (validators re-run, curated trust, backend must be advertised, auto-retire after fork_retire_days once upstream covers its archs; user pins protect)
engine list/use/rm/prune/rollback/local/install roster management (--json; rm refuses the active engine AND any engine with a live child serving it — names the models and teaches blazar stop <model>; with the daemon down it refuses on leftover child pidfiles and teaches the boot orphan sweep; prune by retention policy; local = pseudo-tag). Fork lanes are additive: never pruned, no retention budget; --json rows gain source + provenance + superseded_by
config list/get/set/unset/defaults/edit config.toml management (dotted keys; quoted segments stay one segment; root-scope-only unset)
keys list/add/rm/rotate key management against the running daemon
whisper transcription + whisper lane management (--install --pull --list --pin --model --tag)
doctor grouped diagnostics SYSTEM/GPU/ENGINES/MODELS/CHANNELS/RUNTIME (--flat, --json)
migrate legacy api_keys → [[keys]] (idempotent, backup)
snapshot config+store+sessions backup to timestamped dir
upgrade self-update (sha256-verified; --version, --dry-run)
completions bash\|zsh\|fish\|powershell shell completions to stdout

Observability

Command Purpose
why [trace] sentinel detections for a trace (or latest) with fix hints (--code, --model, --limit, --watch)
watch live tail of sentinel detections

Refused by design (no cloud services)

push, signin, signout, login, logout — all refused: no registry push, no cloud accounts.

Cross-references

Improve this page on GitHub