judge-audit
Pre-registered audits of AI judges that measure how many decisions a judge can take alone at a given error rate, with calibration.
Overview
judge-audit replays decisions your people have already made and tells you how much of that work an AI judge could take over at the error rate you accept, with the bound attached. It reports a safe automation rate at 1, 2, 5 or 10% error together with the confidence threshold to deploy and an out-of-sample check, plus calibration metrics (ECE, Brier score, reliability diagrams) with intervals. A check command fails CI when the judge drifts or the rate falls below a minimum, and it runs as a CLI, a GitHub Action with PR comments, and an MCP server for agents.
Its v0.5 findings compare confidence sources: on BANKING77 at 5% or less error, Jev's native probability could decide 41.0% of texts alone (27.0 to 45.1% over split seeds), Gemini Flash's verbalised confidence 14.3%, and Qwen3-8B token log-probability essentially none. Vote counting ranked errors worse than verbalised confidence in most tests. Caveats are stated beside results: public datasets possibly seen in pretraining, mostly single runs, unmeasured label noise. It contains no LLM inside the measurement, and numbers recompute from committed evidence.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | At what confidence threshold can this judge decide alone within the accepted error rate? (measured) | — |
| Choice | Which BANKING77 intent applies to this query? (judge task under audit) | — |
How it works
- Collect past human decisions as the reference
- Run the judge to get decisions and confidence
- Compute safe automation rate and threshold per error budget
- Measure calibration with intervals and out-of-sample checks
- Gate CI on drift or minimum rate
Strengths
- Answers a deployment question directly
- Pre-registered analysis with stated caveats
- CI, GitHub Action and MCP integration
- No LLM in the measurement
Caveats
- Published results use public datasets and few runs
- Held-out slice not yet run
- Your own data audit is needed for real decisions
At a glance
- Type
- Benchmark
- Status
- beta
- Language
- Python
- Runtime
- CLI, GitHub Action, MCP server
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Pre-registered calibration audits of AI judges, comparing Jev's native probabilities with LLMs' verbalised confidence and vote shares, caveats stated beside every result.
MrJev/awesome-jev
Frequently asked questions
What is the safe automation rate?
The share of decisions a judge can make alone at your accepted error, with the threshold to use.
How did Jev compare?
On BANKING77 at 5% error, Jev's native probability could decide 41.0% alone versus 14.3% for Gemini Flash's verbalised confidence.
Should I rely on these numbers?
The README says the number to act on is the same audit run on your own decisions.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonHarness that runs Jev on all 19,646 TMMLU+ Traditional Chinese questions as four-way Choices, reporting accuracy and calibration.
ProjectresearchPythonReliability testing for typed decision models: contracts, calibration metrics, metamorphic fuzzing and CI pass/warn/fail gates.
DemoprototypePython- reflexbench★ 6
Open benchmark for System One decision models on Choice, Score and Noul, reporting accuracy, calibration and robustness separately.
BenchmarkproductionPython Claude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.
PluginbetaPython