jev-healthcare-lab
Evaluation of Jev versus DeepSeek and Qwen on healthcare decision tasks: triage, diagnosis, medication, nursing and trial screening.
Overview
JEV Healthcare Lab evaluates Jev on concrete medical decision tasks such as triage urgency, contraindication checks, medication choice, next-test selection, nursing and clinical trial screening, comparing accuracy, sustained throughput and cost with DeepSeek V4.1 Flash, Qwen3.5-9B and Qwen3.8-27B. The current round covers 1,514 judgements across 26 tasks plus 64 trial-referral judgements for 8 simulated patients, using exam cases, simulated patients and public labels. It focuses on Choice tasks; Score, Noul and calibration are not yet validated.
The README reports mixed results: for example Jev at 77.78 percent versus 62.96 percent on patient contraindication, 83 versus 62 percent on medication choice, but 28 versus 41 percent on traditional Chinese medicine syndrome classification. Invalid options count as errors, throughput counts valid answers per second rather than correct ones, and account limits may have affected speed. It states results are for research and prototyping, not clinical validation. Documentation is in Chinese and English.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | How urgently does this patient need care? | — |
| Choice | Which medication or test is most appropriate? | — |
| Choice | Is this patient eligible for this clinical trial?(inferred) | — |
How it works
- Assemble fixed medical task sets with reference answers
- Run Jev and comparison models at fixed concurrency
- Score accuracy (invalid outputs count as wrong), throughput and cost
- Publish per-task results and raw data
Strengths
- Broad set of medical decision tasks
- Open data and scoring for recomputation
- Reports both strengths and weaknesses
Caveats
- Not clinical validation
- Choice tasks only; calibration untested
- Account limits may have affected throughput
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Evaluation scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Jev vs DeepSeek 医疗任务对比：12 类场景、96 项测试，效果、耗时与费用一览。 | Jev vs DeepSeek on 96 healthcare tasks across 12 scenarios: quality, latency, cost and open experiment data.
hellogumbo/awesome-jev
Frequently asked questions
What tasks are covered?
Triage, diagnosis, medication, testing, nursing and clinical trial screening, among 26 tasks.
How did Jev compare?
It led on some tasks such as medication choice and trailed on others such as TCM syndrome classification.
Can it be used clinically?
No, the README says results are not clinical validation.
Related projects
Pre-registered study testing Jev's calibration claims on clinical diagnosis cases against Claude Opus 5 and Opus 5.5.
ProjectresearchPythonExperiment screening ADHD review abstracts as include or exclude with Jev, scored against Cohen et al. 2006 abstract triage labels.
AppresearchPythonResearch framework for biomedical decision models on Qwen3.5, studying domain pretraining, NLI training, calibration and abstention.
BenchmarkresearchPython100-question-per-cell check of Jev on Korean versus English reading, paraphrase and medical exams, with recorded responses.
BenchmarkresearchPythonPre-registered benchmark of Jev as a medical hallucination check on MedHallu, versus fast LLMs, on accuracy, latency and cost.
BenchmarkresearchBenchmark of Jev versus OpenAI structured outputs extracting smoking history from 1,000 synthetic notes.
BenchmarkresearchPython