jev-on-a-laptop
Unofficial study reproducing Jev-style parallel typed decisions on small stock models on an Apple Silicon laptop, with benchmarks.
Overview
Jev-style decisions on a laptop is an unofficial research repository that reproduces the parallel constrained decoding idea behind Jev on untrained small models. Instead of generating JSON token by token, the schema is a set of typed fields: the context is prefilled once, the KV cache is broadcast across one batch row per field, all fields are evaluated in a single forward pass, and logits are sliced and normalised to pick each value with a probability. JSON is assembled in code, so it cannot be malformed.
The headline claim is that a 1.5B model that cannot reliably write 28-field JSON can still make 28 schema-valid decisions in about 0.4 s. The author wrote a runner on MLX, benchmarked three model sizes on a 16 GB MacBook Air, and documents what is real versus marketing. A separate parallel-decisions library covers NVIDIA GPUs, where CUDA Graphs reduced latency in warmed tests. Demos include fraud and support-triage presets, and a static Hugging Face Space shows recorded outputs.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Value for each schema field (for example, fraud risk level or support category)(inferred) | — |
How it works
- Define a schema of typed fields
- Prefill context and schema once
- Broadcast the KV cache and evaluate all fields in one batched pass
- Softmax over allowed values to pick each field and its probability
- Assemble JSON programmatically
Strengths
- Schema-valid output by construction
- Fast multi-field decisions on a laptop
- Detailed notes separating evidence from marketing
Caveats
- Uses untrained stock models; probabilities are not calibrated by design
- Unofficial, not affiliated with TypeSafe
- Live Gradio demo needs a Hugging Face PRO account
Getting started
git clone https://github.com/rorshopping/jev-on-a-laptop && cd jev-on-a-laptop && ./setup.sh && ./run_benchmark.shAt a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- MLX on Apple Silicon (Torch backend for GPUs)
- Has code
- yes
Use cases and integrations
How curated lists describe it
Unofficial study: Jev-style parallel typed decisions on stock 1.5B-8B models on an Apple Silicon laptop. Benchmarks, research notes, and a Hugging Face Space demo.
hellogumbo/awesome-jev
Frequently asked questions
What technique does it study?
Parallel constrained decoding: one prefill, then all schema fields decided in a single batched forward pass.
How fast is it?
The README reports 28 schema-valid decisions in about 0.4 s with a 1.5B model on a MacBook Air.
Does it use TypeSafe's API?
No. It runs open models locally.
Related projects
llama.cpp-based C++ server that turns any LLM into a System One decision model, reading answers from next-token logits.
ProjectbetaC++MLX library that turns an existing local model into a decision engine by reading candidate logits, with no fine-tuning, on Apple Silicon.
ProjectprototypePythonLocal server (xks) implementing Jev's /v1/systemone contract on an open-weight LLM, accelerated with Xeon Phi cards.
ProjectresearchRustEducational C11 library and CLI that turns DiffusionGemma 26B into a CPU-first OpenJev-style decision engine reading candidate logits.
ProjectresearchCExperimental engine that makes an existing LLM do Jev-like classification and scoring by reusing context and building JSON.
ProjectresearchPythonReads typed decisions from one forward pass of PrismML's 1.75-bit Bonsai 27B under llama.cpp, using the OpenJev/SemIf logit method.
ProjectresearchPython