decision-model-audit
Pre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
Overview
decision-model-audit is a reproducible comparison of decision models (zero-shot classifiers such as TypeSafe Jev, Fastino GLiNER2.5-Decide and Laya) against small LLMs with structured outputs and supervised baselines. It uses fresh data from September 2026, including arXiv metadata, Federal Register documents and research-funding project data, to avoid contaminated public sets.
It asks six questions: accuracy and calibration, sensitivity to label wording, behaviour on inputs longer than the encoder window, behaviour when no label fits, the cost of a rationale, and cost and latency per 1,000 decisions. A pre-registered protocol logs every deviation, and findings, including negative ones, are recorded chronologically. Later parts added Cloudflare's Clef, thirteen more decision models, a German paired test and a fine-tuning comparison. All responses are cached, so the analysis reruns without API access. Write-ups are available in English and German.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which label fits this document (from a fixed label set)?(inferred) | — |
How it works
- Collect fresh evaluation data and constructed probes
- Run each model family through OpenRouter, Cloudflare or local servers
- Compute accuracy, calibration, robustness, cost and latency
- Publish reports and cached raw outputs
Strengths
- Pre-registered protocol with deviations logged
- Fresh, uncontaminated data
- Reproducible from cached outputs
- Covers many decision models and baselines
Caveats
- Snapshot in time; models change
- Domains limited to the chosen datasets
Getting started
uv sync, then uv run python -m dma.analysis.report to rebuild results from the committed raw outputs.At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- analysis scripts
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Reproducible field test of decision models (TypeSafe Jev, GLiNER2.5-Decide, Laya) against small LLMs and supervised baselines on fresh September 2026 data.
hellogumbo/awesome-jev
Frequently asked questions
Which models are compared?
Jev, GLiNER2.5-Decide, Laya, Clef and others against small LLMs and supervised baselines.
Can I reproduce it without API keys?
Yes, the analysis reruns from committed cached outputs.
Where are the findings?
In results/REPORT.md, REPORT-part2.md and the linked blog posts.
Related projects
Open System-1 (OpenJev): Apache-2.0 research models giving calibrated per-option probabilities, best is a 4.9B exp9a.
ProjectresearchPythonLightweight reproduction of the Jev shape on a ModernBERT encoder and a diffusion LLM, comparing speed, accuracy, calibration and cost.
ProjectresearchPythonInternLM's open multimodal decision models (0.8B-4B) trained on Qwen3.5, with calibration tooling and a 96-case calibration benchmark.
ModelresearchPythonVerdict: 151M ModernBERT/GLiClass non-autoregressive decision models with calibration heads, reporting wins over Jev and Laya.
ModelresearchPythonReproducible MLX and Colab recipes that turn Qwen decoders into Jev-like typed decision models, from 0.5B on a laptop to a noncausal 27B.
ModelresearchPython- Verdict-open-jev★ 110
OpenJev: 151M ModernBERT decision model with RLCD-style calibration, a JevBench audit and an in-browser WebGPU playground.
BenchmarkresearchPython