ReJev
Independent reproduction of Jev-style decision-model post-training on MiniCPM5-2B with LoRA, a sealed holdout and stated limits.
Overview
ReJev reproduces a Jev-style decision-model training pipeline on MiniCPM5-2B to test whether a small model can learn stable structured decisions without free-form generation. The task shape is state plus question plus options leading to one decision; stage one is single-letter Choice classification. The pipeline covers LoRA post-training, a sealed holdout with leakage checks, constrained versus unconstrained decoding evaluation, significance testing with grouped bootstrap, and cost tracking.
On a sealed holdout of 1,892 items in 924 groups, the README reports accuracy rising from a 51.11% baseline to 80.50% for ReJev-2B, with 0% invalid output, and notes the $5.31 figure is cumulative app billing, not one run. It is careful about what it does not claim: no equivalence with Jev, no out-of-distribution extrapolation, and a previously cited result voided after contamination was found. Some experiment paths are excluded from the repo, and a technical report walks through the method. Code is MIT.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Given the state and question, which lettered option is correct? | — |
How it works
- Prepare state, question and option data with leakage checks
- LoRA post-train MiniCPM5-2B
- Evaluate on a sealed holdout with constrained and unconstrained decoding
- Report significance and validity boundaries per experiment
Strengths
- Clear experiment design with sealed holdout and bootstrap
- +29.39 pp over baseline on its holdout per the README
- Limitations and corrections published openly
Caveats
- Covers Choice only (stage 1)
- Results limited to the in-distribution exam
- Some experiment records excluded
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- PyTorch on Modal
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Jev-style decision-model training reproduction on MiniCPM5-2B (stage 1: single-letter Choice classification).
hellogumbo/awesome-jev
Frequently asked questions
What did ReJev achieve?
The README reports 80.50% versus a 51.11% baseline on a sealed 1,892-item holdout, with no invalid outputs.
Is it equivalent to Jev?
No; the authors explicitly do not claim equivalence.
Which question types are covered?
Stage one covers single-letter Choice classification.
Related projects
- TypeAR★ 946
TypeLLM: Jev-inspired type-safe decoding for existing autoregressive LLMs on SGLang, with schema-guaranteed typed outputs.
ProjectbetaPython 33-chapter Chinese tutorial on agentic RL and GRPO, including a chapter on when Jev-style external decision models beat training with RL.
ProjectresearchPythonPre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPythonReads Jev-style choice, yes/no and score probabilities from masked diffusion language models with shared Yes/No weights.
ProjectresearchPythonOpen Jev-class decision models on Qwen3.5 with candidate-logit readout; the 4B model reports 80.1% on JevBench's 231 items.
ProjectresearchPythonOpen System-1 (OpenJev): Apache-2.0 research models giving calibrated per-option probabilities, best is a 4.9B exp9a.
ProjectresearchPython