jev-no-enem
Reproducible benchmark of Jev on Brazil's ENEM 2025 exam, measuring accuracy and calibration with an abstain option against open LLMs.
Overview
Jev no ENEM is a Portuguese-language benchmark that evaluates TypeSafe's Jev on the multiple-choice questions of Brazil's ENEM 2025 national exam (blue booklet, both days), compared with open generative LLM baselines. It frames the study as a move from autoregressive generation to typed System One decisions, and relates results to the item response theory model used by the exam authority, including distractor attractiveness.
Calibration is measured with expected calibration error and multiclass Brier score, and a raw transcribed state is compared with a structured, decomposed state. Two rounds were run: one where Jev could choose A-E or an INVALIDO escape, and one forcing A-E. The README reports 97.7% agreement between rounds on items Jev answered firmly, 78.8% accuracy on items it chose to answer, and 35.5% accuracy when forced on items it had rejected, which the authors read as the escape option filtering genuinely uncertain items. Data is on Kaggle and results feed a GitHub Pages dashboard.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which alternative answers this ENEM question? | A · B · C · D · E · INVALIDO |
How it works
- Load ENEM 2025 questions and answer keys from Kaggle
- Build raw and structured states per question
- Ask Jev a Choice over A-E with and without an INVALIDO escape
- Score accuracy, ECE and Brier, and compare with open LLMs
- Publish results to a dashboard
Strengths
- Public, reproducible benchmark with an experimental protocol
- Tests the value of an explicit abstain option
- Psychometric framing beyond plain accuracy
Caveats
- Portuguese-language documentation
- Single exam year
- Image-dependent questions may not transfer well to text state
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark scripts with GitHub Pages dashboard
- Has code
- yes
- Stars
- 0
Use cases and integrations
How curated lists describe it
Reproducible benchmark evaluating TypeSafe AI's Jev (System One paradigm) on Brazil's ENEM 2025 standardized exam. Evaluates typed decision-making, domain-specific accuracy, and RLCD uncertainty calibration against open LLM baselines with an interactive GitHub Pages dashboard.
logicrw/awesome-jev-projectsReproducible benchmark evaluating TypeSafe AI's Jev (System One paradigm) on Brazil's ENEM 2025 standardized exam. Evaluates typed decision-making, domain-specific accuracy, and RLCD uncertainty calibration against open LLM baselines with an interactive GitHub Pages dashboard.
wh000wh000/awesome-jev-live
Frequently asked questions
What does the benchmark test?
Jev's accuracy and calibration on ENEM 2025 multiple-choice questions versus open LLMs.
Does allowing abstention help?
The README reports 78.8% accuracy on answered items versus 35.5% when forced on items Jev had rejected.
Where is the data?
On Kaggle in the prova-enem-2025 dataset.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPythonPre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.
CLIresearchPython