jev-regress-bench
Benchmark for agent regression after config edits: a Jev Choice decides which approved answers changed meaning versus wording.
Overview
jev-regress-bench addresses a re-certification problem: an agent has approved answers, someone edits its prompt, and you need to know which answers actually changed and which were only reworded, since every false flag costs a human review. It compares five detectors on the same pairs: exact match, embeddings with a cosine threshold, a three-tier stack (markers, then embeddings, then an LLM judge for ambiguous cases), the LLM judge alone (gpt-4o-mini), and one Jev Choice.
Ground truth is derived mechanically from what each config rule does to the answer, using one support-agent fixture with 16 questions and seven config changes (three cosmetic, four rule changes), giving 109 pairs with 19 real changes. Reported results: Jev caught all 19 with 13 false alarms at 0.41 s median and $0.0028, versus 33 false alarms for the three-tier stack and 18-19 caught with 19 false alarms for the LLM judge; embeddings caught only 7. A notable case: a rule change that changed nothing led the stack to flag 8 of 16 answers, Jev 4.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | How does the new answer differ from the approved one? | same · fact_differs · action_differs · specificity_differs |
How it works
- Generate approved answers from a fixture agent
- Apply cosmetic and rule config changes and regenerate
- Derive ground truth from each rule's effect
- Run all five detectors on the pairs
- Compare catches, false alarms, latency and cost
Strengths
- Mechanically derived ground truth
- Focuses on false alarms, the real cost
- Compares against realistic baselines
Caveats
- Single fixture agent and 109 pairs
- Run-to-run variation noted for LLM judge and Jev
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Agent regression testing: after a config edit, one `Choice` (same / fact_differs / action_differs / specificity_differs) decides which of an agent's approved answers changed meaning rather than wording, and on 109 before/after pairs whose ground truth is derived from what each config rule does to the answer, Jev catches all 19 real changes with 13 false alarms against 33 for a markers-then-embeddings-then-LLM stack and 19 for the LLM judge alone.
yibie/awesome-jev
Frequently asked questions
What does Jev decide?
Whether a regenerated answer is the same or differs in fact, action or specificity.
How did it compare?
It caught all 19 changes with 13 false alarms, fewer than the three-tier stack's 33.
How is ground truth set?
Derived from what each config rule mechanically does to the answer.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonSemantic test matchers for Vitest and Jest that ask Jev whether output satisfies a plain-English expectation and threshold the probability.
ProjectbetaTypeScriptpytest plugin for semantic assertions on LLM output, asking Jev whether plain-English claims hold with a 0.8 probability bar.
PluginbetaPythonDesktop tool for testing chat endpoints conversationally, with rule checks and judged checks including TypeSafe's System One.
ProjectprototypeTypeScript