jev-guardbench
Benchmark testing whether Jev or open Kev can replace an LLM judge in agent guardrail callbacks on 7,925 public examples.
Overview
jev-guardbench tests whether a System One model can replace the second LLM judge many agents use to check messages before and after the main model answers. It gave Jev (and variants including a decomposed version and open-source Kev) and LLM judges (Gemini 3.6 Flash, Claude Haiku 4.5) the same policy text and 7,925 rows from public datasets across four checks: PII, prompt injection, harmful requests and harmful responses.
The README reports Jev at about 250 ms median per check versus 672 to 1,189 ms for the judges, $0.02 to 0.03 per 1,000 checks versus $0.21 to 0.71, matching the best judge on PII (F1 0.93) and prompt injection (0.61) but trailing on harmful content, so it missed the strict bar set in advance. Thresholds were tuned on one split and scored on another, hypotheses were pre-registered, guards also ran inside a google-adk agent to measure added latency, and stress tests covered attacks, repeats, long inputs and eight languages.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Does this message contain PII? | — |
| Yes / no | Is this a prompt injection? | — |
| Yes / no | Is this request or response harmful under the policy? | — |
How it works
- Give every guard the same policy text and datasets
- Tune thresholds on one split and score on another
- Measure F1, latency and cost; run inside a real agent
- Compare against pre-registered hypotheses
Strengths
- Pre-registered success criteria
- Fair split and real-agent latency
- Includes an open model (Kev)
Caveats
- Jev missed the pre-set accuracy bar on harmful content
- Public datasets may differ from production traffic
Getting started
See RESULTS.md and HYPOTHESES.md in the repository.At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Benchmark harness with google-adk
- Has code
- yes
- Stars
- 2
Use cases and integrations
How curated lists describe it
Can a System One model (TypeSafe Jev, open-source Kev) replace an LLM-as-judge in agent guardrail callbacks?
wh000wh000/awesome-jev-liveCan a System One model (TypeSafe Jev, open-source Kev) replace an LLM-as-judge in agent guardrail callbacks?
hellogumbo/awesome-jev
Frequently asked questions
Can Jev replace an LLM judge?
It matched the best judge on PII and injection but trailed on harmful content in this test.
How much faster and cheaper?
About 3 times faster and 10 to 25 times cheaper, per the README.
Was the test fair?
Criteria were pre-registered and thresholds tuned on a separate split.
Related projects
Portuguese workshop: a LangGraph sales agent where each decision step can be made by Jev or a frontier LLM and measured side by side.
ProjectprototypePythonSame support agent and 25 rules guarded twice, by an LLM judge and by Jev, compared on accuracy, latency, cost and coverage.
ProjectresearchTypeScriptMeasurement-first lab deciding which classifier technique, Jev, rules or other models, earns a place in a coding-agent fleet.
AppresearchBenchmark of Jev versus Claude Haiku 4.5 as input safety classifiers on jailbreak, evasion and benign pseudo-harmful prompts.
BenchmarkresearchPythonBenchmark of Jev versus rules, local PII models and LLMs at finding personal information: accuracy, leaks, cost and latency.
BenchmarkresearchPythonPaper and benchmark testing Jev as a zero-shot detector of AI alignment failures across 44 benchmarks and ten failure types.
BenchmarkresearchPython