jev-korean-benchmark
100-question-per-cell check of Jev on Korean versus English reading, paraphrase and medical exams, with recorded responses.
Overview
Jev in Korean is a small, frozen, reproducible sample check of whether Jev can be used directly on Korean text or whether content should be translated to English first. It draws 100 questions per cell from four public test sets (Belebele reading comprehension, PAWS-X paraphrase, MedQA and KorMedMCQA), records every response, and compares Jev with a Luna model. The author stresses roughly plus or minus 8 points of uncertainty at this size, so small differences are not findings.
The README reports Korean reading comprehension is fine (96 of 100 versus 97 in English), fine-grained paraphrase judgement is weak in both languages (76 Korean, 80 English), writing instructions in English instead of Korean buys nothing, the two medical scores (89 English, 80 Korean) are different tests and cannot be compared, and reordering input changes about 1 answer in 8 in both languages. It notes this is not clinical validation. Documentation is in English and Korean.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which answer is correct for this reading, paraphrase or medical exam question? | — |
How it works
- Sample 100 questions per task and language
- Run Jev and a comparison model, recording responses
- Compare with confidence intervals
- Test instruction language and input-order sensitivity
Strengths
- Directly answers whether to translate Korean
- Honest about sample-size limits
- Recorded responses for reproduction
Caveats
- Small samples (±8 points)
- Not clinical validation
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Evaluation scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Small Korean/English sample study with recorded responses, including medical-text questions; not a clinical validation.
cobanov/awesome-jevReproducible early-access evaluation of Jev on Korean understanding and medical text, with runtime and cost evidence.
hellogumbo/awesome-jev
Frequently asked questions
Should I translate Korean to English first?
The study suggests no; Korean reading scores were close to English.
Where is Jev weak?
Fine-grained paraphrase judgements in both languages.
Can I compare the medical scores?
No, they come from two different exams.
Related projects
Pre-registered study testing Jev's calibration claims on clinical diagnosis cases against Claude Opus 5 and Opus 5.5.
ProjectresearchPythonEvaluation of Jev versus DeepSeek and Qwen on healthcare decision tasks: triage, diagnosis, medication, nursing and trial screening.
ProjectresearchPythonExperiment screening ADHD review abstracts as include or exclude with Jev, scored against Cohen et al. 2006 abstract triage labels.
AppresearchPythonResearch framework for biomedical decision models on Qwen3.5, studying domain pretraining, NLI training, calibration and abstention.
BenchmarkresearchPythonPre-registered benchmark of Jev as a medical hallucination check on MedHallu, versus fast LLMs, on accuracy, latency and cost.
BenchmarkresearchBenchmark of Jev versus OpenAI structured outputs extracting smoking history from 1,000 synthetic notes.
BenchmarkresearchPython