Eval Genius
Agent skill that tells you when you need an eval and how to build it, with an opt-in lane routing closed-label residue to Jev.
Overview
Eval Genius is a skill for AI coding agents (Claude Code, Codex, Droid or any agent that loads SKILL.md) that brings measurement-engineer judgment to evals: whether a change needs an eval at all, where it fits, how to build the harness, which metric and how many examples, and how to read results, including treating a crash as unmeasured rather than a pass.
An opt-in lane covers the part of an eval that is a plain typed decision, such as real defect or not, failure class, sentiment or brand-voice match. After deterministic checks run, that residue goes to Jev as a typed question. A judge cascade lets code clear what it can, Jev grade the rest, and confidence route sure calls automatically while uncertain ones escalate to a human or reasoning model. Four standard-library scripts compute confidence routing, choice routing, true cascade cost, and calibration of confidence to your own labels.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this a real defect? | yes · no |
| Choice | Which failure class does this output belong to? | — |
| Yes / no | Does this match the brand voice? | yes · no |
How it works
- Run deterministic checks on all outputs
- Send remaining closed-label cases to Jev
- Route by confidence: accept, reject or escalate
- Calibrate thresholds on your own labels and compute cascade cost
Strengths
- Pushes checks to code first
- Explicit escalation for uncertain cases
- Scripts for cost and calibration arithmetic
Caveats
- Jev lane is opt-in and needs a key
- Advice quality depends on the agent following the skill
Getting started
Install SKILL.md into your agent's skills directory (Claude Code, Codex or Droid).At a glance
- Type
- Benchmark
- Status
- beta
- Language
- Python
- Runtime
- agent skill + scripts
- Has code
- yes
- License
- Apache-2.0
Use cases and integrations
How curated lists describe it
Skill that tells a coding agent when a question needs an eval rather than a vibe, with an opt-in lane that routes the closed-label residue to a decision model and scripts for the cost and confidence arithmetic.
MrJev/awesome-jev
Frequently asked questions
Does Eval Genius require Jev?
No, the Jev lane is opt-in.
What does Jev grade?
The closed-label residue left after deterministic checks.
How are thresholds set?
jev_calibrate.py fits confidence to your own labels.
Related projects
Claude Code and Codex skills powered by Jev, published with a calibration curve measured on 4,995 real agent decisions.
ProjectEval-first MCP server for Jev: prototype questions, map them over many items, and measure accuracy and thresholds on labelled data.
MCP serverbetaTypeScriptAgent skills for placing, building, evaluating and improving decision-model systems, with Jev as the default hosted example.
Agent skillbetaClaude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.
PluginbetaPythonPre-registered audits of AI judges that measure how many decisions a judge can take alone at a given error rate, with calibration.
BenchmarkbetaPythonMCP server exposing Jev as one evaluate tool, plus a harness comparing Jev's judgments with general LLMs on shared datasets.
MCP serverbetaJavaScript