jev-ood-calibration
Independent test of whether Jev's probabilities stay calibrated on 900 unseen rule-generated tickets plus three public benchmarks.
Overview
Does Jev know when it doesn't know? This repository tests the premise behind confidence thresholds: that Jev's probabilities stay honest on inputs it has not been trained on. It runs Jev through Vercel AI Gateway on three public benchmarks it has probably seen (OpenBookQA, CommonsenseQA, HellaSwag) and on 900 support tickets generated by a rule-based script, each with choice, boolean and score questions. Every raw response is published, and the whole run costs about $0.06.
On the public benchmarks, accuracy was 86-94% with ECE close to a simulated noise floor, read as in-domain calibration. On the synthetic tickets, a 4-way queue choice scored 89% but was overconfident (refit temperature 3.29), an angry-customer boolean scored 91.7% and was underconfident (T 0.66), and a priority score based on an unknowable organisational rule reached 44.7% with high overconfidence. The author concludes that the sign of miscalibration flips by question type, and that out of distribution Jev is confidently wrong much like a fine-tuned classifier.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which queue should this ticket go to? | — |
| Yes / no | Is the customer angry? | yes · no |
| Score | What priority does this ticket have under the organisation's rule? | — |
How it works
- Generate 900 synthetic tickets with a rule-based script
- Ask Jev one question per call with option criteria
- Run three public benchmarks the same way
- Compute accuracy, NLL, ECE against a noise floor and refit temperature
- Publish raw responses
Strengths
- Out-of-distribution test that cannot be in training data
- Reports the sign of miscalibration, not just ECE
- Cheap and fully reproducible
Caveats
- Synthetic tickets are rule-generated and simple
- Gateway does not expose the exact model version
- Probabilities are quantised to two decimals
At a glance
- Type
- Benchmark
- Status
- research
- Runtime
- Evaluation scripts via Vercel AI Gateway
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Model evaluation: independent calibration test of Jev on 900 rule-generated support tickets it cannot have seen plus three public benchmarks, publishing every raw response, ECE against a simulated noise floor, temperature refit, and the per-type sign of miscalibration (Choice and Score overconfident, Boolean underconfident).
yibie/awesome-jevIndependent calibration test of TypeSafe's Jev on a task it cannot have seen: 900 rule-generated support tickets (choice / score / boolean) plus 3 public benchmarks via Vercel AI Gateway. Raw responses, ECE with noise floor, temperature refit, per-type sign of miscalibration. Reproducible for ~$0.06.
logicrw/awesome-jev-projectsIndependent calibration test of TypeSafe.
wh000wh000/awesome-jev-liveIndependent calibration test of TypeSafe's Jev on a task it cannot have seen: 900 rule-generated support tickets (choice / score / boolean) plus 3 public benchmarks via Vercel AI Gateway. Raw responses, ECE with noise floor, temperature refit, per-type sign of miscalibration. Reproducible for ~$0.06.
hellogumbo/awesome-jev
Frequently asked questions
Is Jev calibrated?
On familiar public benchmarks largely yes; on unseen rule-based tickets it was overconfident on choice and score and underconfident on boolean.
How much does it cost to reproduce?
About $0.06 of API calls.
What should I take away?
Thresholds tuned on one domain may not hold out of distribution, and the direction of error depends on question type.
Related projects
Independent benchmark of Jev versus GPT-4.1-mini on 200 South African customer enquiries, showing how question wording changed results.
BenchmarkresearchPythonpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPython