Evaluating and Benchmarking the System One Model Jev
Paper evaluating zero-shot Jev 1.13 on 37 datasets against Qwen and Gemma option probabilities, with code and raw responses released.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
This arXiv paper evaluates TypeSafe's Jev 1.13 zero-shot across 37 datasets using 346,009 requests, at a reported total cost under $10. It compares Jev with option probabilities read from Qwen3.8-27B and Gemma-4-E4B.
According to the list summary, Jev beats Qwen on 27 of the 37 datasets and Gemma on all 37. Its choice probabilities are well calibrated, but binary (noul) thresholds are poorly placed, meaning a default 0.5 cut is often not the best operating point. The authors release code and raw responses for reproduction.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Dataset-specific classification questions across 37 benchmarks(inferred) | — |
| Yes / no | Binary benchmark questions (threshold placement studied)(inferred) | yes · no |
How it works
- Pose each dataset's items to Jev as typed questions
- Read option probabilities from Qwen and Gemma for comparison
- Compare accuracy and calibration per dataset
- Release code and raw responses
Strengths
- Large-scale, low-cost evaluation
- Calibration analysis, not just accuracy
- Raw data released
Caveats
- List-only entry; paper not reviewed
- Zero-shot only
Getting started
See https://arxiv.org/abs/2609.37647.At a glance
- Type
- Paper
- Status
- research
- Has code
- yes
Use cases and integrations
How curated lists describe it
Zero-shot Jev 1.13 on 37 datasets (346,009 requests for under $10) against Qwen3.8-27B and Gemma-4-E4B option probabilities: Jev beats Qwen on 27 of 37 and Gemma on all 37, with well-calibrated choice probabilities but poorly placed binary thresholds. Code and raw responses released.
AnotiaWang/awesome-decision-models
Frequently asked questions
How did Jev compare?
It beat Qwen3.8-27B on 27 of 37 datasets and Gemma-4-E4B on all 37, per the summary.
Is Jev well calibrated?
Choice probabilities were well calibrated; binary thresholds were poorly placed.
Can I reproduce it?
Code and raw responses are released.
Related projects
Pre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPythonOpen System-1 (OpenJev): Apache-2.0 research models giving calibrated per-option probabilities, best is a 4.9B exp9a.
ProjectresearchPythonLightweight reproduction of the Jev shape on a ModernBERT encoder and a diffusion LLM, comparing speed, accuracy, calibration and cost.
ProjectresearchPythonInternLM's open multimodal decision models (0.8B-4B) trained on Qwen3.5, with calibration tooling and a 96-case calibration benchmark.
ModelresearchPythonVerdict: 151M ModernBERT/GLiClass non-autoregressive decision models with calibration heads, reporting wins over Jev and Laya.
ModelresearchPythonReproducible MLX and Colab recipes that turn Qwen decoders into Jev-like typed decision models, from 0.5B on a laptop to a noncausal 27B.
ModelresearchPython