jev-triage-benchmark
Independent benchmark of Jev versus GPT-4.1-mini on 200 South African customer enquiries, showing how question wording changed results.
Overview
This benchmark measures TypeSafe's Jev against GPT-4.1-mini on 200 South African customer enquiries with known correct answers for department, spam, frustration and urgency. Every number is recomputable from raw results in the repository, measured from Johannesburg.
Its first finding is a correction to its own earlier version. The original reported Jev losing on urgency, 64.7% against 76.5%. After reading TypeSafe's guidance that Score levels should describe situations rather than degrees and stay to one dimension, the author rewrote only the three urgency level descriptions and re-ran. Jev's urgency accuracy rose to 85.0% and mean absolute error halved, while three untouched control questions did not move; GPT also improved, to 87.2%, but Jev was penalised roughly twice as hard by the bad wording. The second finding is that confidence tells you when to ask a human, but not when to escalate to a bigger model. The README concludes that a benchmark measures the question as much as the model.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which department should handle this enquiry? | — |
| Yes / no | Is this enquiry spam? | — |
| Score | How frustrated is the customer? | Calm · Frustrated · Very angry |
| Score | How urgent is this enquiry (levels rewritten as situations)? | — |
How it works
- Label 200 enquiries with known answers
- Ask Jev and GPT-4.1-mini the same questions
- Compare accuracy, cost, latency and confidence behaviour
- Rewrite poorly worded questions and re-run with controls
Strengths
- Raw results committed and recomputable
- Controlled demonstration of question-wording effects
- Practical guidance on using confidence for human review
Caveats
- 200 enquiries from one region
- Single comparison model
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- CLI
- Has code
- yes
Use cases and integrations
How curated lists describe it
Independent benchmark of TypeSafe.
wh000wh000/awesome-jev-live
Frequently asked questions
What changed Jev's urgency score?
Rewriting the three urgency levels as situations instead of degrees raised accuracy from 64.7% to 85.0%.
Is confidence useful for escalating to a bigger model?
The author found it tells you when to ask a human, not when to ask a bigger model.
Can I verify the numbers?
Yes, raw results for both runs are in the repository.
Related projects
Independent test of whether Jev's probabilities stay calibrated on 900 unseen rule-generated tickets plus three public benchmarks.
Benchmarkresearchpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPython