jev-guardrails
Same support agent and 25 rules guarded twice, by an LLM judge and by Jev, compared on accuracy, latency, cost and coverage.
Overview
This repository builds the guardrail layer of a mock carrier support agent twice behind one interface, once with an LLM judge returning JSON and once with Jev answering typed questions, so both can be measured on identical traffic with the same 25 rules and thresholds. With Jev, rules leave the system prompt entirely, shrinking it from 8,694 to 3,004 characters, and all 25 soft rules are checked in one request rather than the 4 to 8 an LLM judge samples for cost reasons.
On 51 labelled cases the README reports 48 of 51 correct for Jev (94 percent, median 177 ms) versus 46 of 51 for the LLM (90 percent, 1,193 ms), noting two LLM misses came from the repo's own content-filter handling, leaving genuine errors at 3 versus 3. Checking all 25 behavioural rules (evasive, over-promising, ungrounded, condescending, blame-shifting and so on) on one reply took about 200 ms and $0.000125 with Jev versus about 5.5 s and $0.0058 with the LLM.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Does this reply break each of the 25 behavioural rules (evasive, over-promising, ungrounded, condescending, blame-shifting...)? | — |
How it works
- Mock support agent generates a reply
- Guard backend (LLM or Jev) checks rules
- Apply the same thresholds
- Compare accuracy, latency, cost and coverage
Strengths
- Controlled A/B with one variable
- Full rule coverage per review
- Smaller system prompt
Caveats
- 51 labelled cases
- Mock agent
At a glance
- Type
- Project
- Status
- research
- Language
- TypeScript
- Runtime
- Mock agent harness
- Has code
- yes
Use cases and integrations
How curated lists describe it
Comparing LLM-as-judge vs TypeSafe Jev for agent guardrails: same rules, same…
wh000wh000/awesome-jev-liveComparing LLM-as-judge vs TypeSafe Jev for agent guardrails: same rules, same agent, measured on cost, latency, calibration and coverage.
hellogumbo/awesome-jev
Frequently asked questions
What does Jev decide?
Whether each reply breaks each of 25 behavioural rules.
Which was more accurate?
Accuracy was roughly even; genuine errors were 3 versus 3.
What is the cost difference?
About 45 times cheaper per full-coverage review with Jev, per the README.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonPortuguese workshop: a LangGraph sales agent where each decision step can be made by Jev or a frontier LLM and measured side by side.
ProjectprototypePythonBenchmark testing whether Jev or open Kev can replace an LLM judge in agent guardrail callbacks on 7,925 public examples.
ProjectresearchPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPython