Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jevals-data

Open CC-BY-4.0 data behind jevals.com: leaderboards and per-decision logs grading Jev and LLMs on accuracy, calibration, cost and speed.

BenchmarkresearchCC-BY-4.0
View on GitHub ↗Live demo ↗

Overview

jevals-data holds the data behind jevals.com, an independent ground-truth benchmark of TypeSafe's Jev (called through Vercel AI Gateway) and LLMs answering the same typed questions: Noul on PubMedQA, Choice on Banking77 and Score on HelpSteer2 helpfulness. Every answer carries probabilities and is scored against human labels for a Decision Score (100 perfect, 0 equal to guessing label base rates, below 0 worse), accuracy, calibration, cost and speed.

In the 2026-09-18 release, Gemini 3.8 Flash scored 73.0 and Jev 69.0 on PubMedQA, which the site treats as tied for first, with Jev at $0.029 per 1,000 decisions and 653 ms p95. On Banking77, Gemini scored 74.1 and Jev 67.8, with GLM-5.3 at 66.8. On HelpSteer2, all models scored near zero (Jev 9.2, Gemini 4.6, several negative), meaning none beat guessing in a meaningful way. Other models in the cohort are GLM-5.3, DeepSeek V4.1 Flash, Qwen3.8 Flash, Mistral Medium 3.5 and Mercury 2.5. Boards and per-decision logs are released under CC BY 4.0.

What the decision model decides

TypeQuestionAnswers
Yes / noPubMedQA yes/no questions—
ChoiceBanking77 intent classification—
ScoreHelpSteer2 helpfulness rating—

How it works

  1. Ask each model the same typed questions
  2. Score answers against human labels
  3. Compute Decision Score, accuracy, calibration, cost and latency
  4. Publish boards and per-decision logs

Strengths

  • Open per-decision data for reanalysis
  • Decision Score normalised against label base rates
  • Includes cost and latency per 1,000 decisions

Caveats

  • Three datasets; results may not generalise
  • Jev called through a gateway, not directly

At a glance

Type
Benchmark
Status
research
Runtime
Data repository
Has code
no
Has tests
no
License
CC-BY-4.0

Use cases and integrations

How curated lists describe it

  • Independent benchmark data for TypeSafe's Jev (System One model) vs LLMs: accuracy, calibration, cost. Boards + per-decision logs, CC-BY-4.0.

    hellogumbo/awesome-jev

Frequently asked questions

What is the Decision Score?

100 is perfect, 0 equals guessing label base rates, and negative is worse than that.

How did Jev compare?

69.0 on PubMedQA versus 73.0 for Gemini 3.8 Flash, and 67.8 on Banking77 versus 74.1.

What is the licence?

CC BY 4.0.

Related projects

  • jevcheck

    pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.

    ProjectbetaPython
  • jevoss

    Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.

    ServicebetaPython
  • can-jev-play

    Experiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.

    Projectresearch
  • Jev × LexGLUE

    Zero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.

    ProjectresearchPython
  • Jev vs GPT-4.1 on a synthetic survey

    Pre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.

    ProjectresearchPython
  • jev-acento

    Pre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.

    CLIresearchPython