jev-mini
Runnable yardstick that tests Jev's three headline claims on a small local model: typed output, no hallucination, calibration.
Overview
jev-mini puts Jev's three marketed claims on trial using a small open model on your own GPU. It is explicitly not a Jev clone: it answers the same questions two ways with the same weights, by constrained scoring (append each option, read its log-probability, normalise) and by autoregressive JSON generation, and measures the difference. One command downloads a 0.5B model via ModelScope and runs in about two minutes on a 6 GB laptop GPU.
The data is 120 hand-labelled support tickets and 60 reviews, annotated before any model was run and stratified by difficulty, with ambiguous items included because calibration only matters where a model should hesitate. Reported results: constrained scoring had 137-154 ms median latency against 4.5-5.3 s for JSON generation and zero type errors versus up to 32 of 120. The author's verdict is that typed output holds, zero hallucination is only a type guarantee, and calibration is poor out of the box, with the larger model consistently worse, cross-checked on two public benchmarks.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which category does this support ticket belong to?(inferred) | — |
| Other | Is constrained option scoring calibrated compared with generated JSON? | — |
How it works
- Download a small open model (0.5B, optionally 1.5B)
- Score each allowed option by log-probability (no tokens generated)
- Generate JSON answers with the same model for comparison
- Measure latency, type errors, accuracy and calibration
- Cross-validate calibration on public benchmarks with bootstrap intervals
Strengths
- Independent, inspectable test of vendor claims
- Runs on modest hardware in minutes
- Labels fixed before running any model
Caveats
- Uses small open models, not Jev itself
- Small hand-labelled dataset
- Results measured on one Windows laptop
Getting started
python quickstart.pyAt a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- PyTorch scripts on local GPU
- Has code
- yes
Use cases and integrations
How curated lists describe it
Put Jev.
wh000wh000/awesome-jev-live
Frequently asked questions
Does this test Jev itself?
No. It tests the System One mechanism on a small open model you can inspect.
What did it find about calibration?
Out-of-the-box calibration was poor, and the bigger model was consistently worse calibrated.
How fast is constrained scoring?
The README reports 137-154 ms median versus several seconds for JSON generation.
Related projects
Local Jev-style decision model on Qwen3.5-4B with four stress tests that show where one-pass decisions break.
ProjectresearchPythonLogitTrail (formerly Mini Jev): local workbench that reads typed Choice, Noul and Score decisions from a frozen LLM's logits.
Projectresearch- TypeAR★ 946
TypeLLM: Jev-inspired type-safe decoding for existing autoregressive LLMs on SGLang, with schema-guaranteed typed outputs.
ProjectbetaPython Decision-forensics library that explains a Jev answer by ablating parts of the input and measuring how far the probability moves.
ProjectbetaPythonAdapter that turns existing LLMs into Jev-style decision functions with option scores, optional calibration and review thresholds.
ProjectprototypePythonDecision sidecar and Copilot CLI pre-turn hook that mimics Jev choice, noul and score using Azure Foundry Cohere Rerank.
HookprototypePython