LangWatch Jev benchmark
LangWatch comparison of Jev against seven open models under 1B on 15 decision tasks, with intervals, latency and contamination checks.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
LangWatch's Jev benchmark page compares Jev with seven open models under one billion parameters on 15 decision tasks. According to the listing, it reports results with 95% confidence intervals, latency, and checks for training-data contamination.
No README or page text was available, so the specific models, tasks and results are not confirmed here. It is useful as a third-party comparison focused on small open models, from an LLM evaluation and observability vendor.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which answer is correct for each of the 15 decision tasks? (measured)(inferred) | — |
How it works
- Select 15 decision tasks and seven small open models
- Run Jev and the open models on the same items
- Compute accuracy with 95% intervals and latency
- Check for contamination
Strengths
- Third-party comparison with confidence intervals
- Includes contamination checks
Caveats
- Page content not available to verify results
- Compares only sub-1B open models
At a glance
- Type
- Benchmark
- Status
- research
- Runtime
- web page
Use cases and integrations
How curated lists describe it
Jev against seven open models under 1B on 15 decision tasks, with 95% intervals, latency, and contamination checks
AnotiaWang/awesome-decision-models
Frequently asked questions
What does it compare?
Jev against seven open models under 1B on 15 decision tasks.
Who published it?
LangWatch.
Are uncertainty estimates given?
Yes, 95% intervals per the listing.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePythonLlamaIndex notebook comparing Jev with open decoders, encoders and specialised tools on document-pipeline Choice tasks.
ProjectresearchPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonReproducible measurements of Jev's real cost, latency, response shapes and accuracy, plus 13 decision models on the same 633 items.
ServiceresearchShort report comparing Jev and many other fast decision classifiers with an LLM on grading section relevance in technical documents.
Projectresearch