jev-exploration
Evidence ledger auditing Jev claims, recomputing published calibration and running an 800-item difficulty-gradient study.
Overview
jev-exploration began as a launch-week deep dive and is now maintained as a ledger of claims about Jev, each with status and source, including other people's benchmarks. The author argues the classifier-shaped interface is reproducible with open models, so the claim worth testing is calibration.
It recomputed published Jev ECE values from each study's own data against the noise floor for its sample size, then ran an 800-item difficulty gradient on jev-1.13.0. Reported findings: probabilities are not calibrated (ECE 2.1 to 2.5 times the noise floor in all tiers, even at 97.5 percent accuracy); the distortion is compression toward the middle, with low probabilities overstated and high ones understated; because of that, a p of at least 0.9 rule gave a perfect hit rate at 21.5 to 32.5 percent coverage here, while on another phishing benchmark that is overconfident the same rule gave 73.9 percent; a widely cited 60-item benchmark cannot measure calibration at all; and a single-parameter correction fixes most of the error.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Are Jev's probabilities calibrated, and does thresholding them work?(inferred) | — |
How it works
- Collect published claims and benchmarks
- Recompute ECE against sample-size noise floors
- Run a controlled 800-item difficulty gradient
- Record findings and their evidence in the ledger
Strengths
- Statistically careful audit of calibration claims
- Explains when thresholding is safe
- Maintained over time
Caveats
- Some findings rely on non-significant gradients, as the author notes
- Results specific to tested versions and tasks
Getting started
Read docs/claims-audit.md in the repository.At a glance
- Type
- App
- Status
- research
- Language
- Python
- Runtime
- Analysis scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
A research repository tracking Jev claims and limitations, with calibration experiments and runnable examples.
logicrw/awesome-jev-projectsJev (TypeSafe) exploratory thread: claim audit, live demos, and runnable code.
hellogumbo/awesome-jev
Frequently asked questions
Are Jev's probabilities calibrated?
This study reports they are not, with compression toward the middle.
Is thresholding still useful?
Here, p of at least 0.9 was always correct at modest coverage, but the author shows this depends on the direction of miscalibration.
Can it be fixed?
The README says one parameter corrects most of the error.
Related projects
Pre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPythonBlack-box reverse-engineering archive studying how the Jev decision model behaves, with evidence matrices and calibration analysis.
ProjectresearchPythonIndependent API-only audit of what Jev's probabilities mean: abstain removal, Korean, coherence, option order, interference and caching.
ProjectresearchPythonResearch framework (DART) testing whether Jev's probabilities are a reliable signal to answer, escalate or abstain.
ProjectresearchPythonRunnable yardstick that tests Jev's three headline claims on a small local model: typed output, no hallucination, calibration.
ProjectresearchPythonReproduction and stress test of a paper asking whether Jev-style probabilities stay calibrated after routing on selected examples.
ProjectresearchPython