jev-calibration-audit
Independent API-only audit of what Jev's probabilities mean: abstain removal, Korean, coherence, option order, interference and caching.
Overview
jev-calibration-audit is an independent audit of the calibration of Jev using only public HTTP calls and public datasets, with no weights, fine-tuning or distillation. It starts from the observation that published measurements disagree, finding Jev near-calibrated on one corpus, under-confident on another and over-confident on post-cutoff phishing data, without confidence intervals, and that TypeSafe publishes no reliability curve or explanation of its confidence field.
Seven experiments, about 7,000 calls costing under a dollar, report: removing the abstain option drops accuracy on unanswerable items from 0.950 to 0.000, raises the stereotype rate from 0.03 to 0.79 at 0.79 confidence, and ECE from 0.023 to 0.793; Korean lowers accuracy by 6.5 points on identical items but calibration is unchanged; probabilities are not a coherent property of the judgment, with complements summing to 0.71-1.42 and a Noul versus a two-option Choice differing by 0.125 on average; there is no option-order bias (zero argmax flips in 400); bundling 16 questions barely changes answers; identical requests are not cached (50 identical requests gave 15 distinct answers); and latency from Seoul is 280 ms p50. Code is MIT and raw records are published.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Do Jev's probabilities mean what they say across domains, wording, languages and bundling? | — |
How it works
- Design experiments on public datasets
- Run controlled API calls with variants (abstain, language, order, bundling)
- Measure ECE, coherence, flips and latency
- Publish raw records and findings
Strengths
- Covers coherence, invariance and interference, not just accuracy
- Very cheap and reproducible
- Includes Korean, previously unevaluated
Caveats
- API-only; cannot explain internal causes
- Specific to model version tested
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- research scripts
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Audits Jev calibration, option-wording effects and Korean judgments through public APIs and datasets.
logicrw/awesome-jev-projects
Frequently asked questions
What happens without an abstain option?
Jev answers unanswerable items confidently and stereotypically; ECE jumps from 0.023 to 0.793.
Is there option-order bias?
No, zero argmax flips in 400 tests.
Are repeated identical requests deterministic?
No, 50 identical requests produced 15 distinct answers.
Related projects
Pre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPythonBlack-box reverse-engineering archive studying how the Jev decision model behaves, with evidence matrices and calibration analysis.
ProjectresearchPythonResearch framework (DART) testing whether Jev's probabilities are a reliable signal to answer, escalate or abstain.
ProjectresearchPythonEvidence ledger auditing Jev claims, recomputing published calibration and running an 800-item difficulty-gradient study.
AppresearchPythonRunnable yardstick that tests Jev's three headline claims on a small local model: typed output, no hallucination, calibration.
ProjectresearchPythonReproduction and stress test of a paper asking whether Jev-style probabilities stay calibrated after routing on selected examples.
ProjectresearchPython