Jevals.com
Independent leaderboard grading Jev and six LLMs on the same typed questions against human labels, with open per-decision data.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
Jevals.com is an independent benchmark website that asks hosted Jev and six LLMs the same typed questions and grades every answer against human labels. According to listings, it uses PubMedQA for Noul (yes/no), Banking77 for Choice and HelpSteer2 for Score, with 300 items and five runs each.
The reported findings are that Jev tied for first on PubMedQA yes/no at about one twenty-eighth of the top LLM's price, tied for second on Banking77, and that no model beat the label base rates on HelpSteer2. Every per-decision probability is published as CC BY 4.0 data in the companion jevals-data repository. The site itself was not reviewed here; see the jevals-data entry for the published tables.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | PubMedQA: does the abstract support yes for this question?(inferred) | — |
| Choice | Banking77: which intent is this message?(inferred) | — |
| Score | HelpSteer2: how helpful is this response?(inferred) | — |
How it works
- Ask Jev and six LLMs identical typed questions
- Run 300 items five times each
- Grade against human labels
- Publish leaderboards and per-decision data
Strengths
- Independent, ground-truth evaluation
- Open CC BY 4.0 per-decision data
Caveats
- Site not reviewed directly here
- Three datasets only
At a glance
- Type
- Benchmark
- Status
- research
- Runtime
- Website
Use cases and integrations
How curated lists describe it
Model evaluation: independent leaderboard that asks Jev and six LLMs the same Noul, Choice and Score questions and grades every answer against human labels (PubMedQA, Banking77, HelpSteer2; 300 items × 5 runs each), finding Jev tied for first on PubMedQA yes/no at 1/28 of the top LLM's price, tied for second on Banking77 and no model beating the label base rates on HelpSteer2, with every per-decision probability published as CC BY 4.0 data.
yibie/awesome-jevIndependent benchmark of hosted Jev and six LLMs on the same Noul, Choice and Score questions, graded against human labels (PubMedQA, Banking77, HelpSteer2), with per-decision logs as open data
AnotiaWang/awesome-decision-modelsIndependent benchmark that grades hosted Jev and six LLMs against human labels on the same Noul, Choice and Score questions (PubMedQA, Banking77, HelpSteer2), with per-decision logs in [jevals-data](https://github.com/Jevals/jevals-data).
cobanov/awesome-jev
Frequently asked questions
How did Jev do?
Tied for first on PubMedQA yes/no at about 1/28 the price, tied for second on Banking77, per the listing.
Did any model handle HelpSteer2 well?
No model beat the label base rates there, per the listing.
Is the data open?
Yes, per-decision logs are published as CC BY 4.0 in jevals-data.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPythonPre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.
CLIresearchPython