jev-reward-model-evaluation
Reproducible evaluation of Jev 1.13 as a reward model, LLM judge and process verifier across eight benchmark tracks.
Overview
Jev 1.13 as a Reward Model is a reproducible evaluation of Jev's structured classification API used as a reward model, LLM judge and process verifier, with an interactive report and a 54-row comparison against state-of-the-art results from leaderboards and papers. Across eight tracks in seven benchmark families it processed 40,940 examples with no API errors, using 76.0M input tokens at an estimated $3.19.
Reported headline results: RewardBench v1 92.58% (2.53 points behind the leader), RewardBench 2 81.15%, RM-Bench structured pairwise 81.29%, RubricBench 76.02%, PPE human preference 64.40% with strong model-ranking correlation, PRMBench 66.38% and ProcessBench 69.51%. The author's conclusion is that Jev is a strong general preference signal, especially for safety, reasoning and aggregate ranking, while its main weakness is fine-grained verification: precise instruction following scored 50.63% and ProcessBench fell from 74.63% on GSM8K to 66.01% on OlympiadBench. Methodology and data files are published, with a Chinese report.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which of these two responses is better?(inferred) | A · B |
| Yes / no | Is this reasoning step correct?(inferred) | yes · no |
How it works
- Convert each benchmark item into typed Jev questions
- Run all eight tracks and record answers
- Score with each benchmark's official metric
- Compare with curated state-of-the-art references
Strengths
- Large-scale evaluation at low cost
- Clear strengths and weaknesses identified
- Interactive report and open data
Caveats
- Reference results come from different protocols and dates
- Single model version
- Weak on fine-grained verification
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Evaluation scripts with web report
- Has code
- yes
Use cases and integrations
How curated lists describe it
Jev 1.13 reward-model evaluation across 8 benchmark tracks, with an interactive report and 54-row SOTA comparison.
hellogumbo/awesome-jev
Frequently asked questions
How good is Jev as a reward model?
The report finds it strong on general preference (92.58% on RewardBench v1) but weaker on fine-grained verification.
How much did the evaluation cost?
About $3.19 for 40,940 examples.
Where is the weakness?
Precise instruction following and harder step-level maths verification.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonAudit of 300 web pages for AI citability with nine Jev decisions each, checked against an LLM judge for agreement, cost and latency.
ProjectresearchPython