qwen-rlcd-jev-repro
Reproducible MLX and Colab recipes that turn Qwen decoders into Jev-like typed decision models, from 0.5B on a laptop to a noncausal 27B.
Overview
This repository publishes the experiments behind a talk on Jev-like decision models. The common idea is to turn a decoder model into a decision model: serialize the state and a typed question, score only valid candidates rather than generating, train those scores with classification objectives, and calibrate on held-out data. It is an independent reconstruction and explicitly does not reproduce TypeSafe's architecture, weights or data.
Recipes range from a basic MLX run with Qwen2.5-0.5B 4-bit on an M1 Pro (typed choice/score/noul, candidate-token scoring, LoRA, temperature calibration), through a Nimble recipe with contrastive candidate classification and option-order controls, to AutoJev-style Colab notebooks at 9B and 27B with a 255-way learned readout. The latest noncausal 27B LoRA/readout model is reported at 57.70 on the Decision Index 0.3 public suite, with weights on Hugging Face. The README also reports negative results, such as an RL continuation that did not improve accuracy, and notes that training/benchmark overlap has not been fully audited. Slides from the talk are included.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Typed choice questions scored over valid candidate tokens | — |
| Score | Ordered-level score questions | — |
| Yes / no | Yes/no noul questions | — |
How it works
- Serialize state and a typed question for a Qwen decoder
- Score only valid candidate answers (token scoring or a learned readout)
- Train with LoRA and classification objectives
- Calibrate on held-out data and evaluate on JevBench / Decision Index
Strengths
- Runnable at several scales, including a laptop MLX smoke test
- Reports negative results and caveats openly
- Weights and prediction records published
Caveats
- Training/benchmark overlap not fully audited
- Banking comparison is 128 narrow synthetic cases
- Large recipes need a 98 GB GPU
Getting started
git clone https://github.com/jsaurabh/qwen-rlcd-jev-repro.git && cd qwen-rlcd-jev-repro && .venv/bin/python mlx_jev_repro.py --out runs/mlx-smokeAt a glance
- Type
- Model
- Status
- research
- Language
- Python
- Runtime
- MLX, Colab (PyTorch)
- Has code
- yes
Use cases and integrations
How curated lists describe it
Reproducible MLX and Colab experiments for Jev-like typed decision models with…
wh000wh000/awesome-jev-live
Frequently asked questions
Does this reproduce Jev?
No; the README says it is an independent reconstruction from public work, not TypeSafe's architecture or weights.
What score did the best model get?
The README reports 57.70 on the complete Decision Index 0.3 public suite for the noncausal 27B LoRA/readout run.
Can I run it without a big GPU?
Yes, the basic MLX recipe runs Qwen2.5-0.5B 4-bit on an M1 Pro.
Related projects
Pre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPythonOpen System-1 (OpenJev): Apache-2.0 research models giving calibrated per-option probabilities, best is a 4.9B exp9a.
ProjectresearchPythonLightweight reproduction of the Jev shape on a ModernBERT encoder and a diffusion LLM, comparing speed, accuracy, calibration and cost.
ProjectresearchPythonInternLM's open multimodal decision models (0.8B-4B) trained on Qwen3.5, with calibration tooling and a 96-case calibration benchmark.
ModelresearchPythonVerdict: 151M ModernBERT/GLiClass non-autoregressive decision models with calibration heads, reporting wins over Jev and Laya.
ModelresearchPython- Verdict-open-jev★ 110
OpenJev: 151M ModernBERT decision model with RLCD-style calibration, a JevBench audit and an in-browser WebGPU playground.
BenchmarkresearchPython