jev-calibration-audit
Exact-target audit finding Jev's Choice probabilities are a step function on stated base rates while per-option Noul tracks them closely.
Overview
This jev-calibration-audit by Mohit Velu, with an accompanying Zenodo paper titled Calibrated Decisions Are Not Calibrated Probabilities, audits the probabilities returned by Jev (jev-1.13.0) and three open decision models against targets whose correct probability is known exactly: a hidden fair coin with witnesses of stated reliability, and stated reference-class base rates. It then checks conclusions against human vote distributions in ChaosNLI and DICES-350.
Main findings: with no evidence, Jev's Choice gives a fair coin 0.83-0.93 on heads, a pull that follows the word heads rather than key or position and mostly vanishes after one weak witness. On evidence stating a probability, Choice behaves like a step function, rising by 0.59-0.89 when a base rate moves from 45% to 55% where the correct change is 0.10, and neither temperature scaling nor a flexible monotone map fixes it. Noul tracks stated base rates to within 0.014-0.029, and asking one Noul per option and normalizing cuts held-out Brier from 0.067 to 0.003. On real human disagreement, normalized Noul is 1.3-2.1 times closer than Choice on Brier and TVD, though neither beats uniform on the most contested items. Code is MIT; third-party data keeps its own terms.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Did the hidden coin land heads or tails? | heads · tails |
| Yes / no | Is this option true? (one Noul per option, then normalized) | yes · no |
How it works
- Construct exact-probability targets (coins, stated base rates)
- Query Jev's Choice and Noul primitives
- Compare to exact targets and to human vote distributions
- Test recalibration methods
Strengths
- Exact targets remove label noise
- Actionable fix: per-option Noul then normalize
- Published paper with versioned DOI
Caveats
- Synthetic targets may not reflect real tasks
- Some datasets are non-commercial
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- research scripts
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Exact-target audit of the probabilities returned by calibration-trained…
wh000wh000/awesome-jev-live
Frequently asked questions
Are Jev's Choice probabilities calibrated?
Not on stated base rates; the audit finds them step-like and not fixable by temperature scaling.
What works better?
One Noul per option, normalized, which tracked base rates within 0.014-0.029.
Is there a paper?
Yes, on Zenodo (DOI 10.5281/zenodo.23179064).
Related projects
Research replicating Jev-style typed decisions with open LLMs via logprobs; Qwen3.5-9B reached 94% with good calibration in ~1 s.
ProjectresearchPythonLocal reproduction of Jev-style inference that reads option-token logits from a small LLM in one pass, with an honest failure log.
ProjectresearchPythonOpen decision models that derive calibrated confidence from recurrent looped-layer dynamics instead of RL, with early exit and a Jev API.
ProjectresearchPythonLaptop-scale reimplementation of RL for Calibrated Decisions: parallel constrained decoding trained with GRPO and a Brier reward.
ProjectresearchPython- AnyJev★ 1,255
Nokia's training-free library that turns open LLMs into Jev-style deciders robust to option order, plus self-distilled Tacit models.
ModelresearchPython Jev-style replica: frozen Qwen3-1.7B plus a ~0.8M-parameter trained head returns calibrated Choice, Boolean and Score distributions.
ServiceresearchPython