Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-clinical-calibration

Pre-registered study testing Jev's calibration claims on clinical diagnosis cases against Claude Opus 5 and Opus 5.5.

ProjectresearchMIT
View on GitHub ↗Live demo ↗

Overview

jev-clinical-calibration holds the code, data and results for a pre-registered study, registered on OSF on 26 September 2026 before the first model call, asking whether calibrated-probability claims hold up in clinical diagnosis. It compares Jev 1.13, a single-pass decision model returning a probability for every option, with Claude Opus 5 with reasoning disabled and Opus 5.5 with adaptive reasoning, on three diagnostic case sets.

On DDXPlus (1,000 cases), top-1 accuracy was 72.0% for Jev, 79.5% for Opus 5 and 87.4% for Opus 5.5, with ECE of 0.116, 0.073 and 0.109; Jev assigned exactly 0% to the true diagnosis in 62 cases, versus none for either Claude model. On 80 ophthalmic complaints, accuracy was 88.8%, 91.2% and 88.8%, and Jev had the lowest ECE (0.065 versus 0.122 and 0.146). On 45 Semigran vignettes, accuracy was 84.4%, 88.9% and 93.3%. Every harness version, Colab notebook, case file, sign-off and model call is published. Code is MIT, with data under their own licences; a preprint link is pending.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich diagnosis is most likely for this case?—

How it works

  1. Pre-register the study
  2. Render case sets with option lists
  3. Query each model for probabilities over diagnoses
  4. Measure accuracy, ECE and zero-probability failures

Strengths

  • Pre-registered with full version history
  • Includes a clinically important failure mode (zero probability on the true answer)
  • All calls and results published

Caveats

  • Research only; not for clinical use
  • Small ophthalmic and vignette sets

At a glance

Type
Project
Status
research
Language
Python
Runtime
research harness (Colab)
Has code
yes
License
MIT

Use cases and integrations

How curated lists describe it

Frequently asked questions

How did Jev compare on DDXPlus?

72.0% top-1 versus 79.5% and 87.4% for the Claude models, with worse calibration than Opus 5.

Where did Jev do well?

On ophthalmic complaints it had the lowest ECE (0.065).

What is the most concerning finding?

Jev gave the true diagnosis exactly 0% probability in 62 DDXPlus cases.

Related projects

  • jevcheck

    pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.

    ProjectbetaPython
  • jevoss

    Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.

    ServicebetaPython
  • can-jev-play

    Experiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.

    Projectresearch
  • Jev × LexGLUE

    Zero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.

    ProjectresearchPython
  • Jev vs GPT-4.1 on a synthetic survey

    Pre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.

    ProjectresearchPython
  • jev-acento

    Pre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.

    CLIresearchPython