jev-as-a-guardrail
Benchmark of Jev versus Claude Haiku 4.5 as input safety classifiers on jailbreak, evasion and benign pseudo-harmful prompts.
Overview
This benchmark compares TypeSafe Jev with Claude Haiku 4.5 as input guardrails for enterprise AI systems, measuring detection rate, false-positive rate, latency and cost. Prompts come from three pinned Hugging Face datasets: curated jailbreak prompts (Jailbreak-PromptBank), adversarially modified evasion prompts (Mindgard), and benign pseudo-harmful prompts that look suspicious but are harmless (PHTest), sampled in a 53/13/33 split with deduplication.
Jev is asked three Noul questions (prompt injection, jailbreak, harmful intent) and one Choice (safe, suspicious or dangerous), and flags an input unsafe if any Noul exceeds 0.5 or the risk level is dangerous. Haiku uses a tool call returning is_safe, risk level, reason and confidence. The README excerpt describes the setup but does not include results, which are presumably produced by running it with both API keys.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this a prompt injection attempt? | yes · no |
| Yes / no | Is this a jailbreak attack? | yes · no |
| Yes / no | Does this show harmful intent? | yes · no |
| Choice | What is the risk level? | safe · suspicious · dangerous |
How it works
- Sample attack, evasion and benign prompts from pinned datasets
- Classify each with Jev and with Haiku
- Flag unsafe using each system's rule
- Compare detection, false positives, latency and cost
Strengths
- Includes benign look-alikes to measure false positives
- Pinned dataset revisions for reproducibility
Caveats
- Results not shown in the README excerpt
- Fixed 0.5 threshold may not be optimal
Getting started
`git clone https://github.com/rahulbansalc6414/jev-as-a-guardrail.git && cd jev-as-a-guardrail && uv sync`, then set TYPESAFE_API_KEY and ANTHROPIC_API_KEY in .env.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- benchmark scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Benchmark: TypeSafe Jev vs Claude Haiku 4.5 as input guardrails.
wh000wh000/awesome-jev-live
Frequently asked questions
What is compared?
Jev and Claude Haiku 4.5 as input safety classifiers.
How does Jev flag an input?
Unsafe if any of three Noul scores exceeds 0.5 or the risk choice is dangerous.
Are false positives measured?
Yes, using benign pseudo-harmful prompts from PHTest.
Related projects
Paper introducing an adversarial benchmark for decision models: one appended opinion flips 12.1% of Jev 1.13 decisions.
PaperresearchTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonPortuguese workshop: a LangGraph sales agent where each decision step can be made by Jev or a frontier LLM and measured side by side.
ProjectprototypePythonBenchmark testing whether Jev or open Kev can replace an LLM judge in agent guardrail callbacks on 7,925 public examples.
ProjectresearchPythonSame support agent and 25 rules guarded twice, by an LLM judge and by Jev, compared on accuracy, latency, cost and coverage.
ProjectresearchTypeScriptResearch code showing that short, natural-looking context additions can flip decision models like Jev to a chosen wrong answer.
ProjectresearchPython