thaiexam-jev-charts
Static charts comparing Jev with 110 other models on Thai standardized exams for accuracy, cost, speed and calibration.
Overview
thaiexam-jev-charts publishes two static chart pages from evaluating TypeSafe's Jev (jev-1.13.0) on Thai standardized exam questions alongside 110 other language models. One page plots cost and speed against accuracy for 111 models on the same 375 questions, highlighting thinking models; the other checks whether Jev's stated confidence predicts correctness over 567 questions. Both pages have English and Thai toggles and are single self-contained HTML files.
The README reports that Jev answered 567 questions at 70.7% accuracy, 0.35 s and $0.000029 per question, making it the fastest and second-cheapest model in the comparison and mid-pack on accuracy. Its expected calibration error was 7.5 percentage points, which the author says makes its confidence a useful routing signal even where its raw score does not lead. Data comes from an existing ThaiExam benchmark site and harness.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which answer is correct for this Thai exam question?(inferred) | — |
How it works
- Run ThaiExam questions through Jev with the existing harness.
- Compare accuracy, cost and latency with 110 other models.
- Bin Jev's confidence to measure calibration.
- Publish static chart pages.
Strengths
- Shows Jev's calibration on a non-English benchmark.
- Simple static pages with no build step.
Caveats
- Multiple-choice exams are a narrow test of decision tasks.
- No licence stated; data credited to upstream projects.
Getting started
View the charts at https://vehas.github.io/thaiexam-jev-charts/.At a glance
- Type
- Project
- Status
- research
- Language
- JavaScript
- Runtime
- static HTML (Chart.js)
- Has code
- yes
- Has tests
- no
Use cases and integrations
How curated lists describe it
Charts: TypeSafe Jev evaluated on Thai standardized exams vs 110 other models.
hellogumbo/awesome-jev
Frequently asked questions
How did Jev do?
The README reports 70.7% accuracy on 567 questions, mid-pack, while being the fastest and second-cheapest model.
Is Jev's confidence reliable here?
Its expected calibration error was 7.5 percentage points, which the author considers good enough for routing.
Where does the data come from?
From the ai-vs-thai-exams project and the thaiexamjs harness.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPythonPre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.
CLIresearchPython