jev-tmmluplus-eval
Harness that runs Jev on all 19,646 TMMLU+ Traditional Chinese questions as four-way Choices, reporting accuracy and calibration.
Overview
jev-tmmluplus-eval puts Jev through TMMLU+ v1.1, a 66-subject Traditional Chinese multiple-choice benchmark, and scores it the way the official leaderboard does. Every question becomes one Choice over A, B, C and D, so the options are the response schema: Jev cannot answer E, refuse or bury the answer in prose, giving a 100% parse rate by construction. Because Jev returns a probability for each option, the harness also reports calibration with ECE and a four-way Brier score, which a letter-only model cannot provide.
The README reports 77.57 on the category macro average (78.66 pooled) across all 19,646 questions with zero API errors, at a cost of $0.42 and 91 seconds with concurrency 64, and about 298 ms per question. ECE was 7.65%. The author places this mid-table on the public leaderboard and frames Jev as a decision model priced for throughput rather than a frontier reasoning model. Weak subjects include Taiwanese Hokkien, accounting and taxation.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which option (A/B/C/D) answers this exam question? | A · B · C · D |
How it works
- Load TMMLU+ questions by subject
- Send each as a four-way Choice to /v1/systemone
- Score with leaderboard-style macro averaging
- Report accuracy, ECE, Brier, latency and cost
Strengths
- No answer parsing; off-schema answers impossible
- Calibration metrics alongside accuracy
- README reports full benchmark for $0.42 in 91 s
Caveats
- Mid-table accuracy versus frontier LLMs
- Single benchmark language and format
Getting started
uv venv && uv pip install -e ".[dev]", export TYPESAFE_API_KEY, then use the jev-tmmlu CLI.At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- CLI
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Jev is not a chat model. You hand it a \`state\` plus a map of typed questions, and it returns one typed answer each — with calibrated probabilities and \*\*no generated text\*\*.
logicrw/awesome-jev-projects
Frequently asked questions
How did Jev score?
77.57 category macro on TMMLU+ v1.1, mid-table on the public leaderboard, per the README.
Why is the parse rate 100%?
The four options are the response schema, so Jev can only return A, B, C or D.
What did it cost?
$0.42 for all 19,646 questions, per the README.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonReliability testing for typed decision models: contracts, calibration metrics, metamorphic fuzzing and CI pass/warn/fail gates.
DemoprototypePython- reflexbench★ 6
Open benchmark for System One decision models on Choice, Score and Noul, reporting accuracy, calibration and robustness separately.
BenchmarkproductionPython Pre-registered audits of AI judges that measure how many decisions a judge can take alone at a given error rate, with calibration.
BenchmarkbetaPythonClaude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.
PluginbetaPython