mini-jev
Preregistered study: reading a frozen Qwen3-4B's option-letter logits matches grammar-constrained JSON accuracy, and runs faster.
Overview
mini-Jev asks how much of Jev's typed-decision interface an ordinary frozen model already provides. Instead of having a model write a JSON object under a grammar, it turns each closed-choice field into a lettered multiple-choice question, runs one forward pass and reads the model's scores for the option letters at the answer position; strings and numbers are still generated. Everything runs on Qwen3-4B-Instruct with xgrammar for the constrained baseline and CLINC150 as the task, and the experiment was preregistered, with every number recomputed from stored run records.
Results: reading letters lost no accuracy versus JSON (0.907 versus 0.909 on 6,750 paired intent observations, confidence interval covering zero at every option count from 2 to 16); the most likely next token was an option letter in all 13,600 questions; it was about 4 times faster on short texts and 1.4 to 2.4 times faster on 2,048-token texts with a shared-prefix cache; and letters beat writing option names by 10 to 13 points. It loses on out-of-scope texts, picking 'none of the above' less often than JSON, and on dependent fields. A teaching bench is included.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which intent (lettered option) fits this text? (CLINC150) | — |
| Yes / no | Boolean field read from letter logits | — |
How it works
- Convert each closed-choice schema field to a lettered question
- Run one forward pass per field (with shared-prefix cache)
- Read option-letter logits at the answer position
- Compare accuracy and time with grammar-constrained JSON
Strengths
- Preregistered with reproducible numbers
- Shows no accuracy loss and large speedups
- Explains where the approach loses
Caveats
- Single model and dataset
- Weaker on out-of-scope detection
- Experiment rather than a library
At a glance
- Type
- Model
- Status
- research
- Language
- Python
- Runtime
- research scripts (transformers, xgrammar)
- Has code
- yes
- Stars
- 59
Use cases and integrations
How curated lists describe it
Local reproduction: implements Jev's typed-decision interface on top of a local LLM.
yibie/awesome-jevmini-Jev: what a Jev-style typed-decision interface looks like on a frozen Qwen3-4B — read the option letter's logits instead of generating JSON. Preregistered experiment, results, teaching bench.
wh000wh000/awesome-jev-liveMini-Jev: what a Jev-style typed-decision interface looks like on a frozen Qwen3-4B — read the option letter's logits instead of generating JSON. Preregistered experiment, results, teaching bench.
hellogumbo/awesome-jev
Frequently asked questions
Is reading letters less accurate than JSON?
No, 0.907 versus 0.909 with a confidence interval covering zero.
Is it faster?
About 4 times on short texts and up to 2.4 times on long texts with prefix caching.
Where does it fall short?
On out-of-scope inputs and fields that depend on earlier answers.
Related projects
- TypeAR★ 946
TypeLLM: Jev-inspired type-safe decoding for existing autoregressive LLMs on SGLang, with schema-guaranteed typed outputs.
ProjectbetaPython Reads Jev-style choice, yes/no and score probabilities from masked diffusion language models with shared Yes/No weights.
ProjectresearchPythonOpen Jev-class decision models on Qwen3.5 with candidate-logit readout; the 4B model reports 80.1% on JevBench's 231 items.
ProjectresearchPythonOpen Jev replica on Gemma 4 E2B and Gemma 3 270M served on Modal, with eight demos and a TypeSafe eval head-to-head.
ProjectresearchPythonSmall experiment: reordering enum options changed a local Jev clone's answers on its old default, while hosted Jev's did not.
DemoresearchPython- JevK5★ 150
Apache-2.0 open-weight Jev alternative that answers yes/no, choice and score questions with per-option probabilities in one pass.
LibrarybetaPython