Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

decision-model-benchmark

Independent benchmark of Jev, constrained-output LLMs and baselines on intent and spam tasks: accuracy, latency, cost, calibration.

Benchmarkresearch
View on GitHub ↗

Overview

DMB compares APIs that choose one option from a list and return a confidence score. It measures accuracy, latency, cost, failures and how confidence relates to correctness, with contenders including TypeSafe's Jev, LLMs with structured output and deterministic baselines.

The README reports that on September 29, 2026, jev-latest scored 79.2% on Banking77's full test set and 88.6% on CLINC150 including out-of-scope queries, while on multi-label NLU++ micro intent F1 was 48.3% with only 4.3% of messages fully correct. Median latency was 267 to 316 ms per decision. The expanded run made 40,226 decisions with no failures for a recorded $1.71, using vendor-claimed rates rather than invoices. An earlier pilot compared Jev with several LLMs on banking and spam. The authors stress these results are a guide and you should evaluate on your own data.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich intent does this message express?—
Yes / noDoes this message have this intent (per-label binary for NLU++)?yes · no

How it works

  1. Load public intent and spam datasets
  2. Send each item to each contender
  3. Record answers, confidence, latency and cost
  4. Report accuracy and confidence-correctness relationship

Strengths

  • Large-scale run with zero failures reported
  • Includes calibration analysis
  • Transparent about cost assumptions

Caveats

  • Costs not invoice-verified
  • Expanded run tested only Jev
  • Results limited to tested datasets and settings

Getting started

See the repository for running suites against configured endpoints.

At a glance

Type
Benchmark
Status
research
Language
Python
Runtime
benchmark CLI
Has code
yes

Use cases and integrations

How curated lists describe it

Frequently asked questions

How did Jev score on Banking77?

The README reports 79.2% on the full official test set.

What was it weak on?

Multi-label NLU++, with 48.3% micro F1 and 4.3% complete-message accuracy.

How much did the run cost?

About $1.71 for 40,226 decisions by recorded usage and claimed rates.

Related projects

  • jev-baselines-eval

    Pre-registered independent evaluation of Jev on intent classification and confidence-gated cascades, with a transparent erratum.

    LibraryresearchPython
  • jev.sorter

    Benchmark harness pitting Jev against Claude Sonnet 5 and Haiku 4.5 on Banking77 intent sorting with a confidence gate.

    ProjectresearchPython
  • JevVsLLM

    Benchmark of Jev against low-cost LLMs on 130,000 classification decisions across five datasets for accuracy, cost and latency.

    ProjectresearchPython
  • Jev Phishing Bench

    Reproducible benchmark of Jev vs Claude Haiku 4.5 on 2,000 emails deciding whether an agent should click a link, with calibration audit.

    BenchmarkresearchPython
  • Jev vs Fable on 520 real social posts

    Blog review of a scheduler's pre-publish check where Jev answers four yes/no questions per social caption, compared with blind Fable labels.

    Articleproduction
  • Is Jev as Accurate as Frontier Models at Classification?

    OpenRouter benchmark of Jev 1.13 vs Claude Opus 5 on Banking77: 81.0% vs 84.4% accuracy at a fraction of the latency and cost.

    Articleresearch