jev-headline-bench
Benchmark asking Jev to pick the winner of 10,984 real Upworthy headline A/B tests: 64.5% overall, 74.7% when decisive.
Overview
jev-headline-bench tests whether Jev can pick the winner of a real headline A/B test. Labels come from Upworthy's archive of randomised experiments on its own readers (same article and image, split audience, clicks counted), not from opinions about writing. Jev sees only the two headlines. Built on jev-go, the run on 16 September 2026 made 21,968 requests to jev-1.13.0 in about 8 minutes.
All design decisions were made on the exploratory split (2,295 pairs, 63.9 percent), and the confirmatory split was run once unchanged: 64.5 percent on 10,984 pairs. Accuracy rises with the true difference, from 53.1 percent (chance) when headlines performed the same, which serves as a placebo control, to 74.7 percent when the difference was decisive. Jev beat every simple rule of thumb tested, such as including a number (58.5 percent) or a question mark (35.5 percent).
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which of these two headlines got more clicks? | — |
How it works
- Load Upworthy A/B pairs with exploratory and confirmatory splits
- Ask Jev to choose the winning headline
- Tune on exploratory, run confirmatory once
- Analyse accuracy by true effect size and compare with heuristics
Strengths
- Real behavioural labels, not opinions
- Pre-registered style split and placebo control
- Large sample
Caveats
- Headlines only, without article context
- One model version
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Go
- Runtime
- Benchmark CLI (jev-go)
- Has code
- yes
Use cases and integrations
How curated lists describe it
Can Jev pick the winner of a real headline A/B test? 64.5% across 10,984 Upworthy randomized experiments, 74.7% when the difference was decisive.
hellogumbo/awesome-jev
Frequently asked questions
How accurate is Jev?
64.5 percent on 10,984 confirmatory pairs, 74.7 percent when the difference was decisive.
Is that better than chance?
Yes; on pairs with no real difference it scored 53.1 percent, as expected.
Does it beat rules of thumb?
Yes, every heuristic tested scored lower.
Related projects
Audit of 300 web pages for AI citability with nine Jev decisions each, checked against an LLM judge for agreement, cost and latency.
ProjectresearchPython- jev-arena★ 122
Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.
ProjectbetaTypeScript Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonAnalytics engine that rates published posts on pre-registered rubrics with Jev and tests ratings against real engagement.
ProjectbetaTypeScriptOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPython