typesafe-local
Local System One-compatible server that reads option probabilities from a small MLX LLM, with measured calibration.
Overview
typesafe-local asks a local language model typed questions about a document and returns probability distributions instead of text, exposing the same /v1/systemone API shape as TypeSafe so the same client works against either. It runs on Apple Silicon with MLX and small Qwen3 models. The author built it to see how much of Jev's behavior comes from the model and how much from serving structure.
The finding is that most of the speed is serving structure: encode the document once, keep its KV cache, append each question as a short suffix and read logits at one position without generating. The README reports about 200 ms for four questions on an M4 Pro. The harder part is the numbers: raw probabilities were badly miscalibrated (for example p(yes)=0.96 for an irrelevant question), and a two-parameter calibration on 96 labeled examples raised held-out accuracy from 0.40 to 0.88 and cut ECE from 0.53 to 0.20 while AUC stayed 0.886. It also measures how option order can flip judgment calls even at confidence 1.0.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Did the customer request a refund? | yes · no |
| Choice | Which department should handle this? | billing · technical · sales |
| Score | How frustrated is the customer? | — |
How it works
- Encode the document once and cache its KV state.
- Append each question as a suffix and read logits at the answer position.
- Apply a fitted calibration to the raw probabilities.
- Return System One-shaped answers.
Strengths
- No API key; runs locally.
- About 200 ms for four questions on an M4 Pro, per the README.
- Measured calibration and option-order effects rather than assertions.
- Compatible with System One clients.
Caveats
- Raw probabilities need calibration per model.
- Answers can depend on option order.
- Apple Silicon (MLX) only.
Getting started
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d @examples/support_ticket.jsonAt a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- MLX local HTTP server
- Has code
- yes
Use cases and integrations
How curated lists describe it
Inspired by TypeSafe Ai, Ask a local LLM typed questions, get calibrated probabilities instead of text. Structured output without generation or parsing. MLX / Apple Silicon.
hellogumbo/awesome-jev
Frequently asked questions
Is this Jev?
No, it serves a small open model with Jev's API shape; it is unaffiliated with TypeSafe.
Why calibrate?
Raw probabilities were badly scaled; a two-parameter fit greatly improved accuracy and ECE.
What is the option-order problem?
Reordering options can change the answer on judgment calls while still reporting high confidence.
Related projects
llama.cpp-based C++ server that turns any LLM into a System One decision model, reading answers from next-token logits.
ProjectbetaC++Decision-forensics library that explains a Jev answer by ablating parts of the input and measuring how far the probability moves.
ProjectbetaPythonAdapter that turns existing LLMs into Jev-style decision functions with option scores, optional calibration and review thresholds.
ProjectprototypePythonDecision sidecar and Copilot CLI pre-turn hook that mimics Jev choice, noul and score using Azure Foundry Cohere Rerank.
HookprototypePythonPython library that imitates Jev-style typed decisions (choice, score, yes/no) on local LLMs via Ollama, vLLM or llama.cpp.
ProjectprototypePythonMLX library that turns an existing local model into a decision engine by reading candidate logits, with no fine-tuning, on Apple Silicon.
ProjectprototypePython