jevbench (dhruvmehra)
Small reproducible benchmark of Jev against LLMs, fine-tuned BERT, Laya and zero-shot NLI on SST-2, AG News and Banking77.
Overview
jevbench (dhruvmehra) asks whether TypeSafe's Jev is a better text classifier than LLMs, BERT or open-weights alternatives. It runs six classifiers over seeded samples (500 by default) of three public datasets: SST-2 sentiment, AG News topics and Banking77 intents. The classifiers are Jev 1.13 via OpenRouter's Decisions API with one Choice per example, the open Laya System One model (421M parameters) running locally, a cheap LLM and a frontier LLM with JSON-schema enum output, a per-dataset fine-tuned DistilBERT, and BART-MNLI zero-shot. Every zero-shot model receives the same label descriptions.
It reports accuracy, macro-F1, error rate, ECE where probabilities exist (Jev, Laya and the BERT variants), p50 and p95 latency, throughput and cost per 1,000 from providers' own usage data, with results in a dated summary file. Caveats are stated plainly: fine-tuned BERT has seen thousands of labels while the others have not, the 95% interval at n=500 is about plus or minus 2.5 points so several rankings tie, and older datasets were likely seen in LLM pretraining. Responses are cached so re-runs are free, and 31 offline tests run without network.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which label (sentiment, topic or intent) applies to this text? | — |
How it works
- Draw seeded samples from held-out splits
- Run six classifiers with shared label descriptions
- Measure accuracy, F1, ECE, latency, throughput and cost
- Publish summary tables
Strengths
- Shared label descriptions across zero-shot models
- Includes local open alternatives and fine-tuned baselines
- Caching and offline tests
Caveats
- n=500 per dataset; many ties
- Possible pretraining contamination for LLMs
- Results live in a separate summary file
Getting started
git clone, uv sync --extra dev, put an OpenRouter key in .env, uv run pytest, then run the benchmark.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- CLI
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Benchmark TypeSafe JEV against LLMs, fine-tuned BERT, Laya and zero-shot NLI on text classification: accuracy, calibration, latency, throughput, cost.
hellogumbo/awesome-jev
Frequently asked questions
Which models are compared?
Jev, Laya, a cheap LLM, a frontier LLM, fine-tuned DistilBERT and BART-MNLI zero-shot.
Is calibration measured?
Yes, ECE for models that return probabilities: Jev, Laya and the BERT variants.
How precise are the results?
At n=500 the 95% interval is about plus or minus 2.5 points, so several rankings are ties.
Related projects
- jev-arena★ 122
Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.
ProjectbetaTypeScript Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePythonLlamaIndex notebook comparing Jev with open decoders, encoders and specialised tools on document-pipeline Choice tasks.
ProjectresearchPythonJev applied to SemEval-2026 DimABSA dimensional sentiment tasks using only typed primitives, no generation or fine-tuning.
ProjectresearchPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPython