Documentation
Business Rules
Routing decisions, scheduling and admission policy.
6. BUSINESS RULES
Verified against Blazar v0.6.1 source. Commands and configuration spot-verified against v0.19.0 (2026-10-03). INFERRED tags mark reasoned statements.
Role → permission mapping
| Role | How acquired | Permissions |
|---|---|---|
| Anonymous (local) | Default — no [[keys]] configured |
Full model access on all APIs; NO key management (gateway refuses: 403 no keys configured: add the first [[keys]] entry in config.toml (or BLAZAR_KEYS env) and restart) |
| Scoped key | [[keys]] entry with non-empty models |
Generation on its scoped models only; no key management |
| Admin key (unscoped) | [[keys]] entry with empty models |
All models + key management (/api/keys add/remove/rotate/list) |
| Remote upstream | [[remotes]] entry |
Receives routed <remote>:<model> requests with its optional upstream key |
Validation rules
| Rule | Source of truth | Failure |
|---|---|---|
Key name charset [A-Za-z0-9._-] |
keys.rs | 400 |
| Key numeric limits must be non-negative integers (rpm/tpm/daily_tokens/max_concurrent; weight) | keys.rs | 400 |
| Duplicate key name | keys.rs | 409 |
| Removing the last key | keys.rs | 409 would leave the gateway authless |
models.name must not contain # |
store.rs CoreError::Catalog |
rejected at upsert |
Unknown engine kind string |
EngineKind::from_str |
teaching error naming supported kinds llamacpp, mistralrs, sglang, sdcpp, whisper, mlx |
Unknown tensor_preset name |
config validation | parse/validation failure |
ModelOverride unknown field (deny_unknown_fields) |
config.rs | hard config parse error (typos fail loudly) |
tls_cert/tls_key both-or-neither |
config.rs | validation failure |
mcp_servers_config XOR mcp_servers_json |
config.rs | mutually exclusive |
chat_template_kwargs must be a JSON object |
config.rs | shape-validated |
reasoning_effort ∈ {minimal, low, medium, high, xhigh, max} |
REASONING_EFFORT_LEVELS |
validation failure |
Semantic-cache per-request -ttl 1..=86400, -threshold 0.01..=1.0 |
gateway semcache | 400 malformed |
Anthropic thinking.budget_tokens >= max_tokens |
anthropic.rs | 400 invalid_request_error |
late_chunking payload > late_chunking_max_tokens (default 8192) |
gateway | 413 |
Audit body over AUDIT_BODY_SNIFF_LIMIT on audited lanes |
gateway | 413 PAYLOAD_TOO_LARGE |
extra_args unknown flags |
profile compile (manifest-validated) | error at profile-compile |
chat_template XOR chat_template_file, file must exist at load |
config.rs | validation failure |
Spec-draft numeric ranges (spec_draft_*: poll 0..=100, prio -1..=3, ngram_* 1..=1024-class bounds) |
config.rs | validation failure |
Limits, caps, thresholds
| Limit | Value | Notes |
|---|---|---|
| Request body | 50 MiB | gateway DefaultBodyLimit |
| Prefill-heavy demotion | > 64 KiB body (~16k tokens @ 4 B/token) | one priority class down |
| Queue SLO deadlines | HIGH 2 s / NORMAL 30 s / LOW 120 s | EDF within class |
| WFQ bucket reset | 256 | credit += 1/weight per serve |
| Singleflight window | 5 s | identical NON-STREAM chat coalesce |
| Prompt preflight | estimate > 90% of ctx → exact /tokenize check |
refuse unfit (400 teaching) |
| Cache RAM auto-cap | 30% of physical RAM | cache_ram_mb default 8192 |
| Session bank entry cap | 512 MiB | larger banks dropped |
| Bank save bound | 2 s | on evict, idle instances only |
| Audit log rotation | 16 MiB | <data>/log/audit.jsonl |
| Engine update regression gate | > 10% decode drop → auto-rollback | --no-gate skips |
| Quantize degradation gate | default max 10.0% | --max-degradation, --verify perplexity gate |
mistralrs_pa_memory_fraction |
0.05..=0.95 | |
slot_prompt_similarity |
(0.0..=1.0], 0 → engine default 0.1 | |
| Engine check | engine_check_secs, 0 = off |
check-only, never auto-install |
Time-based rules
| Rule | Value |
|---|---|
| Idle eviction | idle_timeout_secs 1800 default; idle_sleep_secs 300 |
keep_alive semantics |
N seconds / 0 = unload immediately / -1 = keep forever; honored from both OpenAI and ollama APIs; /api/ps shows countdown |
| Session pins | session_keep_secs 900 default; 0 disables |
| Key usage accounting day | UTC YYYY-MM-DD boundary |
| Semantic cache TTL | 600 s default, per-request -ttl override |
| Hint task cadence | 60 s (EWMA 0.7/0.3) |
| OTLP export cadence | 10 s bounded batches |
| Model load timeout | per-kind: 180 s precompiled (llamacpp/mistral.rs/sdcpp), 600 s SGLang first JIT, 600 s mlx first JIT (venv class; CUDA kernels compile at first generation); a pin overrides all lanes |
| Sentinel stall | sentinel_stall_secs 30 default; 0 disables stall check |
| Child header stall | child_header_timeout_secs 120 default; 0 disables — a child that accepts a request but never writes headers is evicted (J5 lane), the request retried once in-band, terminal 504. Diffusion (sdcpp) lanes ride sdcpp_child_header_timeout_secs (900 default, 5..=3600) instead — their first header arrives only when the full frame/video is computed |
| Media job wait | media_job_wait_secs 900 default; 0 = no gateway cap — sync/stream image+video generations give up (504 / SSE error event) when exceeded; raise it or use "async": true job handles. Job polls (sync, stream and async job GET) refresh the child’s activity, so an in-flight job holds the child against idle eviction |
| Graceful shutdown | SIGTERM/SIGINT; TLS listener drains 10 s |
Conditional logic and edge cases
- Auth activation is data-driven: zero keys → open gateway; ≥1 key → everything except
/healthz+/healthrequires a valid secret. - Concurrency lease ordering:
max_concurrentgate runs BEFORE scope/rpm/tpm; the lease is held until the response body drains (a held SSE stream occupies a slot — bounded by design). - Engine routing matrix: MLX quant dirs (is_mlx marker + directory shape) → mlx lane only, checked before the quantized-safetensors arm (
MlxUnservedteaching when absent); quantized safetensors (AWQ/GPTQ/FP8) → SGLang only, never a mistral.rs fallback (its dequant path is known-broken upstream v0.9.3); GGUF → llama.cpp then mistral.rs; plain safetensors → by policy (Quality/Throughput: SGLang first; Latency: mistral.rs first); diffusion component sets (rows with a flag-keyedcomponentslist, e.g.--vae+--llm) →sdcpponly (DiffusionUnservedteaching names the install command when absent), and no text row ever routes tosdcpp. Model-levelengineoverride beats both routing modes. Within a kind, mainstream engines outrank fork lanes regardless of install recency — a capability fork only receives architectures mainstream lacks; an unknown-architecture death re-routes once to an advertising lane (pin remembered), then teaches. - Fork-lane retirement: a lane is stamped superseded only on full architecture coverage upstream, and only curated lanes auto-delete (
fork_retire_days, default 7; 0 = never). User-built forks and lanes protected by a model pin are never auto-deleted; retirement warns and names the registry rebuild path. - Engine retention: an install keeps the newest
keep_tagsbuilds per kind (default 2 = fresh build + one rollback anchor forblazar engine rollback);localand the active tag are always protected on top. Whisper/piper voice dirs mirror the same policy with pin protection. The knob is read at prune time, so an edit applies to the next install with no restart; a broken config fails the prune loudly rather than deleting blindly. - Colon semantics:
owner/repo:QUANTfor pulls;model:tagat request time resolves onto existing flat rows only (colon→dash swap if that row exists; miss = name verbatim). - Replica keys:
model#Ninstance keys are internal; the store rejects#in user-facing names. - Vision grammars:
@visionsuffixed instance keys exist alongside plain/replica keys;evict_modelreleases pins and evicts every grammar. - Determinism tradeoff:
deterministic = trueforcesslots = 1— multi-slot serving perturbs near-tie logits (greedy probes flip). - Thinking-template gating:
think == Some(true)on a template without thinking markers → teaching 400 (fail-open if markers unreadable); absentthinkon marker-carrying templates →think:falseinjected (local lane only, before translation). - reasoning_effort bridge: chat completions only (completions/responses excluded); Anthropic
thinkingmaps intochat_template_kwargs. - Stream rendering: TTY chat renders thinking inline dimmed; piped output omits it.
- Singleflight: duplicates ride the leader’s warm prefix (non-stream only).
- Semantic cache bypass: streams bypass; structured-output requests bypass in BOTH directions; embed-model failures bypass (never block generation); never shared across API keys.
- Warm pegs: SGLang peg is BLOCKING, llama.cpp peg DETACHED (fire-and-forget, bounded 90 s + 60 s), mistral.rs and sdcpp never peg.
- Clean vs unclean child death: clean teardown resets crash history; SIGKILL-after-grace on a sleeping child logs a cosmetic “engine crashed (signal 9)” ERROR that is breaker-safe (documented misclassification).
- XDG fallback: when XDG dirs are unavailable the tool falls back to CWD with a loud stderr warning.
Rules enforced ONLY on the backend (gateway/runtime)
All of the above are backend-enforced; clients cannot opt out of: auth activation, quota ordering, queue discipline (priority/EDF/WFQ), preflight refusals, routing matrix (no client-selected engine except configured pin/engine override), eviction/session-bank lifecycle, circuit breakers, sentinel observation (bytes never altered regardless of sentinel_enforce), audit identity capture, quantized-safetensors lane protection, #-in-name rejection, key persistence round-trip (atomic write + .bak + rollback), last-key protection, anonymous-gateway management refusal.