jev-medhallu-benchmark
Pre-registered benchmark of Jev as a medical hallucination check on MedHallu, versus fast LLMs, on accuracy, latency and cost.
Overview
jev-medhallu-benchmark compares TypeSafe's Jev 1.13 with current fast LLMs and two open models on medical tasks, adding them to Stanford CRFM's published MedHELM results. The main study uses MedHallu: given a PubMed abstract, a question and a proposed answer, is the answer hallucinated? All models are called through OpenRouter so latency includes the same gateway hop.
On the 1,000-item test split, the README reports Jev at 92.9% accuracy with its chosen question, against 92.4-95.1% for four fast hosted LLMs, at a 204 ms median and $0.03 per 1,000 checks. Three LLMs were significantly more accurate by about 2 points, while Jev was 2.6-6.5 times faster and 5-35 times cheaper. Jev's probability had an AUROC of 0.974, and letting Jev settle the 37% of items where it was at least 90% confident kept each LLM's accuracy with 37% fewer LLM calls.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Would the authors of the study described in knowledge say that answer misrepresents their findings? | yes · no |
How it works
- Choose a plain Jev question and threshold on dev data
- Run Jev and LLMs on the 1,000-item MedHallu test split via OpenRouter
- Compare accuracy with confidence intervals and McNemar tests
- Measure median and tail latency and cost per 1,000 checks
- Test a cascade where Jev settles high-confidence items and LLMs handle the rest
Strengths
- Pre-registered design with statistical tests
- Shows a practical cascade that cuts LLM calls without losing accuracy
- Reports latency tails and cost, not just accuracy
Caveats
- Jev was not the most accurate model on the same question
- Results are specific to MedHallu and the tested model versions
- Not a clinical tool
At a glance
- Type
- Benchmark
- Status
- research
- Runtime
- Benchmark harness via OpenRouter
- Has code
- yes
Use cases and integrations
How curated lists describe it
Medical AI: pre-registered test of Jev as a hallucination check on Stanford MedHELM's MedHallu (1,000 test items), asking one `Noul` on whether an answer misrepresents its PubMed abstract; Jev scored 92.9% against 92.4–95.1% for four fast LLMs at a 204 ms median and USD 0.03 per 1,000 checks, and letting Jev settle the 37% of items where it was at least 90% sure kept each LLM's accuracy with 37% fewer LLM calls.
yibie/awesome-jev
Frequently asked questions
How accurate was Jev on MedHallu?
The README reports 92.9% with its chosen question, about 2 points below the best fast LLMs.
Why use Jev if it is less accurate?
It was much faster and cheaper, and its confidence allowed a cascade that kept LLM accuracy with 37% fewer LLM calls.
How were models called?
All through OpenRouter, so every latency includes the same gateway hop.
Related projects
Pre-registered study testing Jev's calibration claims on clinical diagnosis cases against Claude Opus 5 and Opus 5.5.
ProjectresearchPythonEvaluation of Jev versus DeepSeek and Qwen on healthcare decision tasks: triage, diagnosis, medication, nursing and trial screening.
ProjectresearchPythonNext.js site with six live Jev demos, each compared side by side with Gemini or Claude on latency, cost and agreement.
DemobetaTypeScriptExperiment screening ADHD review abstracts as include or exclude with Jev, scored against Cohen et al. 2006 abstract triage labels.
AppresearchPythonResearch framework for biomedical decision models on Qwen3.5, studying domain pretraining, NLI training, calibration and abstention.
BenchmarkresearchPython100-question-per-cell check of Jev on Korean versus English reading, paraphrase and medical exams, with recorded responses.
BenchmarkresearchPython