Vals AI's independent evaluation of Jev
Vals AI's independent benchmark of Jev: matched a frontier model on claim verification at a fraction of cost, but ranked last on LegalBench.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
Vals AI posted an independent evaluation of Jev on X. According to the list summary, Jev matched GPT-6 Astra's 97.5 percent on 400 claim-verification questions at roughly one five-hundredth of the cost, yet ranked last on a 12-task slice of LegalBench. When tuned to a 1 percent error budget, it landed at 1.6 percent error.
It is a useful, balanced data point: strong cost-efficiency on narrow verification, weaker on legal reasoning tasks, and some slippage when calibrated thresholds are pushed to tight error budgets.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this claim supported? (claim verification)(inferred) | yes · no |
| Choice | LegalBench task labels(inferred) | — |
How it works
- Run Jev and other models on claim verification and LegalBench.
- Compare accuracy and cost.
- Tune a threshold to a 1 percent error budget and measure actual error.
Strengths
- Independent third-party results.
- Covers both strengths and weaknesses.
Caveats
- Known only through a list summary of a social post.
- Limited task coverage.
At a glance
- Type
- Discussion
- Status
- discussion
- Has code
- no
- Has tests
- no
Use cases and integrations
How curated lists describe it
Independent benchmark: matches GPT-6 Astra's 97.5% on 400 claim-verification questions at roughly a five-hundredth of the cost, yet ranks last on a 12-task LegalBench slice and lands at 1.6% error when tuned to a 1% budget.
yibie/awesome-jev
Frequently asked questions
How did Jev do on claim verification?
It matched GPT-6 Astra's 97.5 percent at about 1/500 of the cost, per the summary.
Where was it weak?
It ranked last on a 12-task LegalBench slice.
What about calibration?
Tuned to a 1 percent error budget, it reached 1.6 percent error.
Related projects
Zero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPythonPre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.
CLIresearchPython