Smoking-history extraction benchmark
Benchmark of Jev versus OpenAI structured outputs extracting smoking history from 1,000 synthetic notes.
Overview
This benchmark compares TypeSafe Jev 1.13.0 with OpenAI structured outputs (chat-latest) on extracting smoking history from 1,000 synthetic outpatient progress notes. Both receive the same note, annotation policy, candidate values and ten typed questions; OpenAI uses strict JSON-schema enums matching Jev's options. The repository contains the input and answer-key CSVs, an executable evaluation program and archived results, but not the generator of the synthetic data.
Results: smoking-status accuracy was 99.0% for Jev and 100.0% for OpenAI, and complete extraction of all ten fields 92.4% versus 98.7% (OpenAI produced 63 more fully correct records), with Jev weaker on pack-years. Jev's estimated API cost was $0.18 versus $24.54 (about 133 times less) and its mean request time 0.405 s versus 2.093 s (5.2 times faster). The README is careful that these are pipeline-level results on synthetic data, not general model rankings or evidence of clinical validity, that costs are list-price estimates, and that model aliases may change.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | What is the patient's smoking status?(inferred) | current · former · never |
| Choice | Which pack-years value applies? | — |
| Choice | Which quit date applies? | — |
How it works
- Load synthetic notes and the extraction policy
- Ask both providers the same ten typed questions per note
- Score against the answer key
- Report accuracy, estimated cost and latency
Strengths
- Paired, like-for-like comparison
- Archived decisions and probabilities
- Clear cost-versus-accuracy trade-off
Caveats
- Synthetic data; no clinical validity
- Costs estimated from list prices
- Moving model aliases limit reproducibility
Getting started
python -m pip install -r requirements.txt, set OPENAI_API_KEY and TYPESAFE_API_KEY, then run the evaluation program.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Evaluation scripts
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
1,000 synthetic notes: Jev vs OpenAI structured outputs on accuracy, cost, and latency
AnotiaWang/awesome-decision-models
Frequently asked questions
Which was more accurate?
OpenAI structured outputs: 98.7% complete extraction versus 92.4% for Jev.
Which was cheaper and faster?
Jev, at about 133 times lower estimated cost and 5.2 times lower mean latency.
Is this clinically validated?
No; it uses synthetic notes and makes no clinical claims.
Related projects
FGV Direito SP experiment comparing Jev with Gemini 3.8 Flash and GPT-5.6 Luna for annotating 12 variables in 120 TJSP court rulings.
ProjectresearchPythonPre-registered study testing Jev's calibration claims on clinical diagnosis cases against Claude Opus 5 and Opus 5.5.
ProjectresearchPythonJev applied to SemEval-2026 DimABSA dimensional sentiment tasks using only typed primitives, no generation or fine-tuning.
ProjectresearchPythonEvaluation of Jev versus DeepSeek and Qwen on healthcare decision tasks: triage, diagnosis, medication, nursing and trial screening.
ProjectresearchPythonExperiment screening ADHD review abstracts as include or exclude with Jev, scored against Cohen et al. 2006 abstract triage labels.
AppresearchPythonResearch framework for biomedical decision models on Qwen3.5, studying domain pretraining, NLI training, calibration and abstention.
BenchmarkresearchPython