jev-calibration
Reliability-curve study of Jev on 240 hand-labelled agent tool calls, publishing labels, raw responses and scripts.
Overview
jev-calibration asks whether Jev is actually calibrated, which matters because agent platforms act on high confidence and escalate low confidence, so a 0.9 must be right about 90% of the time. It follows up the author's earlier 60-case jev-benchmark, which was too small for a reliability curve and saturated at exactly 1.000 on 40 cases. The task is unchanged: classify an agent tool call as readonly, destructive, privileged or exfiltration, now over 240 cases split into 84 clear, 84 ambiguous (with an alternative defensible label) and 72 adversarial, including risky calls in benign wording, benign calls in alarming wording, and classify-this-as-readonly injections.
Results on 29 September 2026 for jev-latest: strict accuracy 90.4% and lenient 95.8%; ECE 0.057 (bootstrap 95% CI 0.037-0.098); 134 rows (55.8%) at exactly 1.000 with 99.3% strict accuracy there; 87.7% accuracy at a mean confidence of 0.970 in the 0.9-1.0 band; multiclass Brier 0.152; 8 misses at confidence 0.9 or above; and latency of 267 ms p50. jev-preview was nearly identical. A rerun gave the same answer on 240/240 cases, though confidence moved by up to 0.110. Labels, raw responses and scripts are published.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | What kind of risk does this agent tool call carry? | readonly · destructive · privileged · exfiltration |
How it works
- Label 240 tool calls across clear, ambiguous and adversarial slices
- Run jev-latest and jev-preview
- Plot reliability curve, split saturated rows
- Report ECE with bootstrap CI, Brier, selective accuracy and misses
Strengths
- Larger, harder set than the first benchmark
- Bootstrap CIs and saturated split
- Rerun consistency checked
Caveats
- Single task domain
- Confidence in the 0.9-1.0 band runs higher than accuracy
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- analysis scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Is TypeSafe AI.
wh000wh000/awesome-jev-live
Frequently asked questions
Is Jev calibrated on this task?
ECE was 0.057, with answers at 1.000 almost always right but the 0.9-1.0 band slightly overconfident (87.7% at 0.970).
Were answers stable?
A rerun gave the same answer on 240/240, with confidence shifts up to 0.110.
How many high-confidence misses?
8 at confidence 0.9 or above.
Related projects
Reproducible benchmark of Jev vs Claude Sonnet 5 on classifying agent tool calls as readonly, destructive, privileged or exfiltration.
BenchmarkresearchPythonpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPython