Jev Phishing Bench
Reproducible benchmark of Jev vs Claude Haiku 4.5 on 2,000 emails deciding whether an agent should click a link, with calibration audit.
Overview
Jev Phishing Bench compares Jev with Claude Haiku 4.5 on one security decision for 2,000 emails: should an email agent click the link? It measures accuracy, calibration, latency and cost. On Jev's own verdict, Haiku clearly wins accuracy (81.3% versus 62.6%, AUROC 0.837 versus 0.689), while Jev was faster (239 ms p50 versus 687 ms) and much cheaper ($0.038 versus $0.462 per 1,000 emails).
The interesting result is in five signal questions asked in the same Jev call: a free-hosting signal alone reached AUROC 0.96, and a logistic regression over the five signals reached about 95% accuracy. After review, controls were added: a non-AI list rule scored 91.6%, showing the dataset largely separates by construction; on a held-out half Jev's best single signal fell below the regex rule but its regression still beat the regex regression; and Haiku was asked the same five questions for comparison at higher cost. Full tables and charts are published.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Should the agent click the link in this email? | yes · no |
| Yes / no | Is the link hosted on a free hosting service? | yes · no |
| Yes / no | Is the sender generic? | yes · no |
How it works
- Ask Jev the click verdict plus five signal questions per email
- Ask Claude Haiku the same verdict
- Compare accuracy, AUROC, ECE, latency and cost
- Fit rules and regression on one half, evaluate on the other
- Compare with a non-AI baseline and LLM signals
Strengths
- Honest result: Jev loses on its own verdict
- Shows decomposed signals plus code can beat a single verdict
- Controls added after public critique
Caveats
- Dataset largely separable by simple rules
- Single dataset and date
- Signal selection required a split to avoid leakage
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
2,000 emails: Jev vs Claude Haiku 4.5 on click-or-not, with calibration, latency, and cost. Haiku wins accuracy here.
AnotiaWang/awesome-decision-modelsThe signal result above was challenged on three points: no non-AI baseline, selection and evaluation on the same emails, and no equivalent decomposition for the LLM. Three controls were added (\`bench/heuristics.py\`, \`bench/protocol.py\`, \`run\_llm\_signals.py\`); nothing above was changed. Full tables in \`results/report.md\`, chart in \`results/controls.png\`.
logicrw/awesome-jev-projectsJev (TypeSafe) vs Claude Haiku 4.5 on 2 000 phishing emails: accuracy, calibration, latency, cost. Reproducible benchmark.
hellogumbo/awesome-jev
Frequently asked questions
Is Jev better than Haiku at phishing?
Not on its direct verdict: 62.6% versus 81.3% accuracy, though it is faster and cheaper.
What worked better?
Combining Jev's five signal answers in a regression reached about 95% accuracy.
Is the dataset hard?
A simple non-AI host-list rule reached 91.6%, so it largely separates by construction.
Related projects
Independent benchmark of Jev, constrained-output LLMs and baselines on intent and spam tasks: accuracy, latency, cost, calibration.
BenchmarkresearchPythonBlog review of a scheduler's pre-publish check where Jev answers four yes/no questions per social caption, compared with blind Fable labels.
ArticleproductionDiscord bot where Jev flags spam and scam links in real time, drives an escalating timeout ladder and profiles members.
ProjectbetaPythonDry-run-first Proton Mail Bridge triage on macOS where Jev answers whether each message is spam or phishing.
ProjectbetaPythonScam text checker where code extracts link and sender facts, Jev answers eleven narrow questions, and a pure policy scores risk.
ProjectprototypeTypeScriptExploratory study of zero-shot ham, spam and phishing email classification with Jev, compared with trained TF-IDF baselines.
BenchmarkresearchPython