zh-decision-bench
First Chinese calibration benchmark for Jev-class models: accuracy, ECE, option order and script robustness.
Overview
zh-decision-bench evaluates System One decision models on Chinese tasks, measuring not just accuracy but whether reported probabilities can be trusted. Noting that public evaluations of Jev-class models were English-only, it provides a CC BY 4.0 dataset on Hugging Face (378 items per the list) and evaluates five models, including Jev, the open NeoHorse-Jev-4B, LAYA multilingual and a Qwen3.5-2B logit probe, with temperature refits and robustness tests for option order and Simplified versus Traditional Chinese.
Findings include: Jev led voice routing (0.960, ECE 0.035), urgency and scam detection (0.952, ECE 0.073) with strong order invariance; NeoHorse-Jev-4B tied Jev on voice routing and won customer-service routing and escalation outright but was weak on urgency; a Qwen logit probe did well on routing but collapsed on binary judgments; and LAYA was fast (24 ms) but needed fixed option order and refit temperature, with 28 percent flips otherwise. A decision guide recommends models per task. Raw predictions for all 2,335 model-questions are included.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which department should handle this Chinese customer request? | — |
| Yes / no | Is this message a scam? | yes · no |
| Score | How urgent is this request? | — |
How it works
- Build a Chinese dataset with routing, urgency, scam and escalation tasks.
- Run five models and record probabilities.
- Measure accuracy, ECE, option-order and script flips; refit temperatures.
- Publish a decision guide and raw predictions.
Strengths
- Fills a non-English evaluation gap.
- Measures calibration and robustness, not just accuracy.
- Open dataset and raw predictions.
Caveats
- Some task subsets are small (e.g. scam n=21).
- Results specific to Chinese tasks.
Getting started
load_dataset("CodyQin/zh-decision-bench")At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- evaluation scripts
- Has code
- yes
- License
- CC BY 4.0 (dataset)
Use cases and integrations
How curated lists describe it
Benchmarking: first Chinese-language calibration benchmark for Jev-class decision models (378 items / 5 models incl. NeoHorse-Jev-4B; accuracy, ECE, option-order and zh-CN/zh-TW robustness; CC BY 4.0 dataset on Hugging Face).
yibie/awesome-jevFirst Chinese-language calibration benchmark for Jev-class 'System One'…
wh000wh000/awesome-jev-liveFirst Chinese-language calibration benchmark for Jev-class 'System One' decision models: dataset (CC BY 4.0), 5-model eval incl. Jev & NeoHorse, temperature refit, robustness tests.
hellogumbo/awesome-jev
Frequently asked questions
Which model is best for Chinese?
Jev led several tasks; NeoHorse-Jev-4B is the strongest open option and won some outright.
Should I use LLM logit probing for binary judgments?
The benchmark found it collapsed on scam detection, so it advises against it.
Does option order matter?
Yes for some models; LAYA flipped 28 percent of answers without a fixed order.
Related projects
- JevBench★ 268
MIT-licensed benchmark and leaderboard for Jev-class decision models: accuracy, calibration, latency and cost on sealed tiers.
BenchmarkproductionPython Probability-aware evaluation of decision models: calibration, coverage at a fixed error budget and latency, Jev vs GLiNER2.5.
BenchmarkresearchPythonReproducible benchmark comparing Jev with hosted and open decision models on rule application, probabilities, latency and cost.
BenchmarkresearchPythonMetamorphic benchmark testing whether decision-model probabilities stay coherent under 50 laws, with no gold labels.
BenchmarkresearchPythonBenchmark of nine local Jev-style decision engines on Apple Silicon across English and Korean sets, with paired significance tests.
BenchmarkresearchPythonPre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPython