jev-synergy-screening
Experiment screening ADHD review abstracts as include or exclude with Jev, scored against Cohen et al. 2006 abstract triage labels.
Overview
This demo applies Jev to systematic-review abstract screening. MEDLINE titles and abstracts are fetched by PMID for an ADHD pharmacologic review (851 records, 84 includes), and Jev answers a Choice of include or exclude together with atomic Noul questions encoding the eligibility criteria (population, listed drugs, design, outcomes, duration and publication-type exclusions). Code combines them into a verdict, and results are compared with Cohen et al. 2006 abstract triage gold labels.
The README is careful about fairness: gold labels come only from abstract triage, matching the title-and-abstract evidence. On the full 851 records it reports 94.6% accuracy, 83.3% recall and 75.3% F1 for the include class at about 520 ms mean latency and roughly $0.08 total. A head-to-head on 100 stratified records reports Jev at 72.7% F1 versus Claude Sonnet 4.5 at 72.0% F1, at $0.0065 versus $0.586, with the author warning not to headline that subset since it has only 10 positives.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Should this abstract be included or excluded under the review criteria? | include · exclude |
| Yes / no | Atomic eligibility checks (e.g. is the population eligible, is a listed drug studied, is the design eligible) | — |
How it works
- Fetch MEDLINE title and abstract by PMID
- Ask Jev an include/exclude Choice plus atomic eligibility Nouls
- Combine answers in code with the documented rule
- Compare against gold labels and report metrics, latency and cost
Strengths
- Evidence modality matches the gold labels
- README reports 83.3% recall on full set at about $0.08 total
- Comparable F1 to Claude Sonnet 4.5 at roughly 1% of the cost on the H2H subset
Caveats
- Single topic (ADHD) dataset
- H2H subset has only 10 positives
- Source data is not CC-BY and must be cited
Getting started
python -m src.run_cohen_eval --subset full --workers 8 --out-csv results/full/predictions.csv --out-metrics results/full/metrics.jsonAt a glance
- Type
- App
- Status
- research
- Language
- Python
- Runtime
- CLI / web UI
- Has code
- yes
Use cases and integrations
How curated lists describe it
A Jev title-and-abstract screening experiment compared with Cohen Abstract Triage labels for an ADHD review.
logicrw/awesome-jev-projectsJev (TypeSafe System One) × ASReview SYNERGY abstract screening demo…
wh000wh000/awesome-jev-liveJev (TypeSafe System One) × ASReview SYNERGY abstract screening demo — Choice/Noul vs gold labels.
hellogumbo/awesome-jev
Frequently asked questions
What does Jev decide?
Whether each abstract should be included in or excluded from the ADHD review, via a Choice plus atomic eligibility Nouls.
How did it compare with Claude?
On a 100-record subset the README reports 72.7% F1 for Jev versus 72.0% for Claude Sonnet 4.5, at far lower cost and latency.
Is it a general screening tool?
No, it is an experiment on one review topic with its own criteria.
Related projects
Pre-registered study testing Jev's calibration claims on clinical diagnosis cases against Claude Opus 5 and Opus 5.5.
ProjectresearchPythonEvaluation of Jev versus DeepSeek and Qwen on healthcare decision tasks: triage, diagnosis, medication, nursing and trial screening.
ProjectresearchPythonResearch framework for biomedical decision models on Qwen3.5, studying domain pretraining, NLI training, calibration and abstention.
BenchmarkresearchPython100-question-per-cell check of Jev on Korean versus English reading, paraphrase and medical exams, with recorded responses.
BenchmarkresearchPythonPre-registered benchmark of Jev as a medical hallucination check on MedHallu, versus fast LLMs, on accuracy, latency and cost.
BenchmarkresearchBenchmark of Jev versus OpenAI structured outputs extracting smoking history from 1,000 synthetic notes.
BenchmarkresearchPython