JevBench (metamorphic)
Metamorphic benchmark testing whether decision-model probabilities stay coherent under 50 laws, with no gold labels.
Overview
This JevBench, unrelated to the other benchmarks with the same name, tests coherence rather than correctness. Typed decision models return probability distributions over declared answers, so their outputs should obey laws such as: a question and its negation sum to one; merging two options merges their probabilities; a conjunction is never more probable than either part; removing an irrelevant option leaves the others' relative probabilities unchanged; and an answer should not depend on which other questions are asked alongside it.
JevBench turns such laws into 50 metamorphic relations across five dimensions, including representation consistency and batch independence, applied to 240 cases in 12 decision domains for 12,000 frozen tests (1,200 in the reported mini suite). Every test edits only the question side, never the input, so no correct answer is needed, and requests are frozen so every model answers the same tests. Twenty-one open models from 48M to 9.7B parameters were evaluated; Open-Jev-9B leads the snapshot with coherence 83.9, and probability-measure coherence is the weakest dimension for every model. It is on PyPI with a technical report, dataset and CI.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Do the model's probabilities across related question variants obey laws such as negation summing to one? | — |
How it works
- Build cases in 12 domains
- Apply 50 relations that edit only the question side
- Freeze all requests and run every model
- Score coherence per dimension
Strengths
- Needs no gold labels
- Catches incoherence that accuracy misses
- Frozen tests for fair comparison
- PyPI package and dataset
Caveats
- Coherence is not correctness
- Evaluated open models only in the snapshot
Getting started
pip install jevbenchAt a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark harness
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Tests whether a decision model's probabilities fit together, using 50 laws of probability and choice and no gold labels. Unrelated to the JevBench above.
MrJev/awesome-jev
Frequently asked questions
Does it need labelled data?
No, it checks whether probabilities fit together, not whether answers are right.
Which model was most coherent?
Open-Jev-9B at 83.9 in the 0.1.0 snapshot.
What was weakest overall?
Probability-measure coherence was the weakest dimension for every model.
Related projects
- JevBench★ 268
MIT-licensed benchmark and leaderboard for Jev-class decision models: accuracy, calibration, latency and cost on sealed tiers.
BenchmarkproductionPython Probability-aware evaluation of decision models: calibration, coverage at a fixed error budget and latency, Jev vs GLiNER2.5.
BenchmarkresearchPythonReproducible benchmark comparing Jev with hosted and open decision models on rule application, probabilities, latency and cost.
BenchmarkresearchPythonBenchmark of nine local Jev-style decision engines on Apple Silicon across English and Korean sets, with paired significance tests.
BenchmarkresearchPythonFirst Chinese calibration benchmark for Jev-class models: accuracy, ECE, option order and script robustness.
BenchmarkresearchPythonPre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPython