snbt-jev-bench
Benchmark of Jev on 159 community-reconstructed questions from Indonesia's SNBT 2025 university entrance exam, all responses published.
Overview
snbt-jev-bench measures TypeSafe Jev on SNBT 2025, Indonesia's state university entrance test. Official papers are never released, so the 159 questions across all seven subtests were reconstructed from participants' memories in a community module; the README is explicit that wording may differ, some questions are flawed, and results measure Jev against this reconstruction, not the real exam. The README is in English and Indonesian.
Each question is one request to /v1/systemone, usually a Choice over options a to e; two yes/no tables are sent as three Nouls. Figures (charts, a geometry diagram) were transcribed to text by Claude and hand-checked, since Jev reads text only. Reference answers come from the module's key for three subtests (re-checked, as its authors call them uncertain) and from Claude Opus labels for the other four, which may be wrong. Every request payload is saved before sending, and scoring reports accuracy by subtest and confidence level plus Brier score, with raw responses published.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which option (a-e) answers this exam question? | a · b · c · d · e |
| Yes / no | Is this statement in the Ya/Tidak table true? | — |
How it works
- Convert recalled questions and passages to JSON
- Transcribe figures to text and check by hand
- Send one Jev request per question and save the payload
- Score accuracy by subtest and confidence, plus Brier
Strengths
- All requests and responses published
- Confidence-level breakdown
- Bilingual documentation
Caveats
- Not the official questions
- Some reference answers are model-generated labels
- Small dataset
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Jev on Indonesia's SNBT 2025 university entrance exam, whose papers are never released: 159 questions reconstructed from memory by volunteers, 156 of them scored, every response published.
hellogumbo/awesome-jev
Frequently asked questions
Are these the real SNBT questions?
No; they were reconstructed from memory, so results measure Jev on the reconstruction.
How were figures handled?
Transcribed to text by Claude and checked by hand, since Jev reads text only.
Where are the results?
In RESULTS.md, with raw responses in data/results/.
Related projects
Independent 52-page Chinese report on Jev that checks vendor claims and reproduces 50 Chinese tests, with traceable data.
ProjectresearchPythonLocal SvelteKit workbench and CLI for organizing, editing, running and comparing Jev experiments defined as JSON.
CLIbetaTypeScriptJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePythonIndependent field guide to Jev 1.13.0 behaviour across 11,621 requests: framing sensitivity, Snake and 3D city driving experiments.
ProjectresearchPython, TypeScriptInteractive teaching site where Jev races frontier LLMs in games, an arena, a sandbox and quizzes, with a keyless replay mode.
AppproductionTypeScriptNext.js site with six live Jev demos, each compared side by side with Gemini or Claude on latency, cost and agreement.
DemobetaTypeScript