Jev-MedQA
Jev-style medical QA on Qwen3.5-4B that picks the answer option in one forward pass instead of generating text.
Overview
Jev-MedQA applies the Jev paradigm to medical multiple-choice and image questions without calling Jev itself. It reads Qwen3.5-4B's scores for the candidate options in one forward pass over the question and any images, returns the highest-scoring option and also exposes scores for the others. No extra training is involved.
Across 153,889 single-answer questions from nine medical QA benchmark sets, the README reports 69.42% accuracy against 67.41% for the same model generating its answer, with average time per question falling from 1.891 s to 0.182 s, a 10.37x speedup. Gains vary by set: on MedXpertQA-MM accuracy is roughly equal while the speedup is about 70x. The project targets researchers interested in fast option selection with open models, and documents evaluation and reproduction.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which candidate answer is correct for this medical question? | A · B · C · D |
How it works
- Feed the question, options and any images to Qwen3.5-4B
- Read the scores of each option letter in one forward pass
- Return the highest-scoring option with scores for the rest
- Compare against standard generation on nine benchmark sets
Strengths
- Large speedup over generating answers (10.37x reported overall)
- Slight overall accuracy gain in the reported results
- Handles text and image questions through one interface
- No additional training
Caveats
- Not the hosted Jev model; uses an open model's option scores
- Option scores are not necessarily calibrated probabilities
- Research benchmark, not for clinical use
Getting started
See docs/reproduce.md in the repository for reproducing the evaluation.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Model inference scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Medical QA: a Jev-style implementation on Qwen3.5-4B that selects answers to text and image questions in one forward pass, reporting 69.42% accuracy versus 67.41% for standard generation with a 10.37x speedup across 153,889 questions from nine medical QA benchmark sets.
yibie/awesome-jev
Frequently asked questions
Does Jev-MedQA call TypeSafe's Jev?
No. It applies the Jev idea of choosing from allowed answers using Qwen3.5-4B's option scores.
How much faster is it?
The README reports 0.182 s versus 1.891 s per question, a 10.37x speedup across nine sets.
Does it handle images?
Yes. The same method covers text questions and questions with one or more images.
Related projects
- TypeAR★ 946
TypeLLM: Jev-inspired type-safe decoding for existing autoregressive LLMs on SGLang, with schema-guaranteed typed outputs.
ProjectbetaPython Research decoder that treats Jev as a language model, assembling replies from repeated choices over phrases, words and letters.
ProjectresearchPythonReads Jev-style choice, yes/no and score probabilities from masked diffusion language models with shared Yes/No weights.
ProjectresearchPythonWord-level language model whose output layer is Jev: next-word choices, Noul-verified drafted chunks and bits-per-token evaluation.
ProjectresearchTypeScriptRunnable yardstick that tests Jev's three headline claims on a small local model: typed output, no hallucination, calibration.
ProjectresearchPythonOpen Jev-class decision models on Qwen3.5 with candidate-logit readout; the 4B model reports 80.1% on JevBench's 231 items.
ProjectresearchPython