jev-aita
Independent benchmark of Jev vs Sonnet 5, GPT-5 nano and local LLMs on 770 Reddit AITA verdicts: Brier score, latency and cost.
Overview
This independent benchmark tests whether models can predict the verdict Reddit gave on r/AmItheAsshole posts, with four options: YTA, NTA, ESH and NAH. It compares jev-1.13-20260917 with Sonnet 5, GPT-5 nano and two local open-weight models on 770 posts from 2025, scoring each model's probability for every verdict against the official flair with the Brier score. The author has no TypeSafe affiliation and paid about $2.75 for all calls.
In the original seven-setup benchmark Jev came second behind Sonnet 5 (Brier 0.369 vs 0.344), with median calls 6.3 times faster and 62 times cheaper than Sonnet, which the author notes is short of TypeSafe's 40x-200x speed claim. Scored against how commenters split, Sonnet's lead grows and unadjusted Jev no longer clearly beats base-rate guessing. A logistic-regression recalibration trained on 300 labelled 2023 posts brought Jev to 0.337, but when every model got the same adjustment Sonnet still led by an inconclusive 0.007. The author stresses there is no true answer, only Reddit's verdict.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Who is at fault in this conflict? | YTA · NTA · ESH · NAH |
How it works
- Collect 770 AITA posts with official verdict flairs
- Ask each model for probabilities over four verdicts
- Score with Brier against flair and against comment splits
- Compare latency and cost; test post-hoc recalibration
Strengths
- Independent, self-funded comparison
- Reports calibration (Brier), not just accuracy
- Tests recalibration fairly across models
Caveats
- Target is an opinion (Reddit verdict), not ground truth
- Single domain and dataset
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- benchmark scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Benchmark of TypeSafe's Jev against Sonnet 5, GPT-5 nano and local LLMs on 770 Reddit AITA verdicts: Brier scores, latency and cost.
hellogumbo/awesome-jev
Frequently asked questions
How did Jev do?
Second behind Sonnet 5 on Brier (0.369 vs 0.344), but 6.3x faster and 62x cheaper per median call.
Did recalibration help?
It lowered Jev's Brier to 0.337, but with equal adjustment for all models Sonnet still led slightly.
Is there a correct answer?
No; the benchmark measures agreement with Reddit's verdict.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPythonPre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.
CLIresearchPython