Calibration and threshold setting with decision models

Python tool that measures where to route decisions between Jev and a larger model on your own data, then routes by confidence.

CLI that checks Jev questions against your labelled examples and grades each: act on it, only sort by it, or rewrite it.

Offline CLI that measures whether a classifier's confidence matches reality and turns mistake costs into a human hand-off threshold.

CLI that fits per-question confidence thresholds to a target accuracy on your labelled data, reports escalation share and fails CI on drift.

Pre-registered audits of AI judges that measure how many decisions a judge can take alone at a given error rate, with calibration.

OpenJev: 151M ModernBERT decision model with RLCD-style calibration, a JevBench audit and an in-browser WebGPU playground.

Curiosity project, not a benchmark: over a million questions from 312 sources put to Jev, mapped by topic and written up as experiments.

Pre-registered independent check of Jev's calibration and type safety on Banking77 and CLINC150, with a split verdict and sealed logs.

Benchmark of Jev, OpenAI Decisions, Clef, Microsoft-Decision-1 and Claude on 2,321 labelled cases for latency, cost and calibration.

Offline-reproducible benchmarks of Jev decisions in DSPy pipelines: calibration, selective risk, confidence gating and cost.

Independent benchmark of Jev vs Sonnet 5, GPT-5 nano and local LLMs on 770 Reddit AITA verdicts: Brier score, latency and cost.

Reproducible benchmark of Jev vs Claude Sonnet 5 on classifying agent tool calls as readonly, destructive, privileged or exfiltration.

Probability-aware evaluation of decision models: calibration, coverage at a fixed error budget and latency, Jev vs GLiNER2.5.

Reproducible benchmark comparing Jev with hosted and open decision models on rule application, probabilities, latency and cost.

Awesome-style map of Jev resources plus independent, reproducible benchmarks of primitives, confidence gating and fan-out latency.

Reproducible benchmark of Jev on Brazil's ENEM 2025 exam, measuring accuracy and calibration with an abstain option against open LLMs.

Independent test of whether Jev's probabilities stay calibrated on 900 unseen rule-generated tickets plus three public benchmarks.

Measures whether SQL ORDER BY over Jev probabilities gives a defensible ranking: inversions, ordinality, calibration, wording and ties.

Independent benchmark of Jev versus GPT-4.1-mini on 200 South African customer enquiries, showing how question wording changed results.

Open CC-BY-4.0 data behind jevals.com: leaderboards and per-decision logs grading Jev and LLMs on accuracy, calibration, cost and speed.

Metamorphic benchmark testing whether decision-model probabilities stay coherent under 50 laws, with no gold labels.

Small one-pass option scorer with n-gram encoder, rival-aware attention and temperature scaling, benchmarked head-to-head against jevlike.

Benchmark of nine local Jev-style decision engines on Apple Silicon across English and Korean sets, with paired significance tests.

Public benchmark of PadFlow's land-development SaaS decisions: schemas, anonymised seed rows and a runner for calibrated decision models.

Benchmark comparing Privatemode (GLM-5.3-Flash), TypeSafe Jev and Laya on speed, accuracy, calibration and cost.

First Chinese calibration benchmark for Jev-class models: accuracy, ECE, option order and script robustness.

Library of 43 field-tested Jev question patterns with templates, thresholds and failure modes, plus 10 anti-patterns; CC0, EN/ZH.

Curated list of the open System One ecosystem: open models, independent benchmarks, calibration and constrained-decoding tools.

Curated list of 138 independent tests of how Jev's answers shift with wording, order, repetition or injection, and of its calibration.

Evidence survey of Jev-style decision models: what the probabilities mean, when to act on them, where they fail, what open models release.