Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

classifier-bench

Benchmark of Jev, OpenAI Decisions, Clef, Microsoft-Decision-1 and Claude on 2,321 labelled cases for latency, cost and calibration.

Benchmarkresearch
View on GitHub ↗

Overview

classifier-bench compares purpose-built typed classification APIs with LLMs asked for schema-constrained output: on the same requests, how fast, cheap, well calibrated and accurate is each? Classifiers include TypeSafe Jev, OpenAI Decisions, Cloudflare Clef and Clef Flash, Microsoft-Decision-1 and any Claude model, each given the same instructions and criteria in its native question shape.

It ships a frozen labelled corpus of 2,321 cases across 31 tasks (some datasets are download-only for licensing), measures latency tails, reliability, cost and calibration with confidence intervals, and can pad cases with unrelated text up to Jev's 32k state limit to test scale. A dry-run mode with synthetic providers needs no keys, and runs refuse to start if the cost estimate exceeds a cap. It uses only the Python standard library. The author discloses building other tools on Jev; corpus, prompts and scoring rules are public.

What the decision model decides

TypeQuestionAnswers
OtherHow do decision models and LLMs compare on speed, cost, calibration and accuracy for the same typed questions?—

How it works

  1. Choose classifiers and estimate cost
  2. Run on the frozen corpus (or dry-run)
  3. Optionally pad cases to test long inputs
  4. Analyze offline into metrics and a report

Strengths

  • Same requests across many providers
  • Zero dependencies; dry-run mode
  • Cost cap and full transparency

Caveats

  • Author builds on Jev (disclosed)
  • Some datasets must be downloaded separately

Getting started

pip install classifier-bench && classifier-bench run -c jev -c haiku --dry-run --out /tmp/x

At a glance

Type
Benchmark
Status
research
Language
Python
Runtime
CLI
Has code
yes

Use cases and integrations

How curated lists describe it

  • Head-to-head speed: benchmarks Jev against OpenAI Decisions and an LLM over a frozen corpus of 2,321 labelled cases across 31 tasks, reporting latency and registry-priced cost, and running offline in dry-run mode without API keys.

    yibie/awesome-jev

Frequently asked questions

Which classifiers are compared?

Jev, OpenAI Decisions, Clef, Microsoft-Decision-1 and Claude models.

Can I try it without keys?

Yes, with --dry-run using synthetic providers.

How much does a full Jev pass cost?

Roughly $0.02, per the README's estimate.

Related projects

  • jevcheck

    pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.

    ProjectbetaPython
  • jevoss

    Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.

    ServicebetaPython
  • can-jev-play

    Experiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.

    Projectresearch
  • Jev × LexGLUE

    Zero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.

    ProjectresearchPython
  • Jev vs GPT-4.1 on a synthetic survey

    Pre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.

    ProjectresearchPython
  • jev-acento

    Pre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.

    CLIresearchPython