jev-spam-eval
Exploratory study of zero-shot ham, spam and phishing email classification with Jev, compared with trained TF-IDF baselines.
Overview
This repository studies how far TypeSafe's pretrained Jev can go on email classification with written category definitions and no labelled examples in its requests. On a 5,733-email ham/spam/phishing test, the README reports Jev at 98.64% accuracy versus 98.87% for a TF-IDF logistic regression trained on about 4,600 labelled messages per fold with the same evidence.
The main finding is that context matters more than wording: keeping the question and definitions fixed while adding link destinations, Reply-To and attachment metadata raised accuracy from 93.62% to 97.98% and phishing recall from 85.71% to 98.43%. On 853 newer phishing messages from 2024-25, enriched Jev caught 95.31% versus 75.26% for the regression baseline. The author is careful about caveats: some specifications were informed by earlier labelled errors, results use jev-1.13.0, and contamination of public corpora cannot be ruled out.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Is this email ham, spam or phishing?(inferred) | ham · spam · phishing |
| Yes / no | Yes/no category questions over the email (Noul variant) | — |
How it works
- Parse email: subject, sender, body, Reply-To, link text with destination hostnames, attachment metadata
- Send enriched state and category definitions to Jev
- Use returned category and probabilities directly
- Compare with TF-IDF logistic regression on the same evidence
Strengths
- No training or labelled demonstrations needed for near-baseline accuracy
- Shows context enrichment as the main lever
- Generalised better than the baseline on newer phishing per the README
Caveats
- Exploratory; some choices informed by earlier labelled errors
- Possible pretraining contamination of public corpora
- Newer phishing set measures recall only, not false positives
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- CLI / notebooks
- Has code
- yes
Use cases and integrations
How curated lists describe it
Spam filtering: zero-shot spam classification with Jev `Boolean` questions, benchmarked against TF-IDF baselines.
yibie/awesome-jevExploratory zero-shot spam study vs trained TF-IDF baselines, with post-hoc-tuning caveats.
AnotiaWang/awesome-decision-modelsZero-shot spam filtering with TypeSafe Jev Noul questions, compared with TF-IDF…
wh000wh000/awesome-jev-liveZero-shot spam filtering with TypeSafe Jev Noul questions, compared with TF-IDF baselines.
hellogumbo/awesome-jev
Frequently asked questions
How did Jev compare with a trained classifier?
The README reports 98.64% versus 98.87% for a TF-IDF logistic regression trained on labelled data.
What improved accuracy most?
Adding link destinations, Reply-To and attachment metadata, which lifted accuracy from 93.62% to 97.98% with the same question.
Is this production-ready spam filtering?
No, it is an exploratory study with stated caveats about tuning and possible contamination.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonMCP server exposing Jev as one evaluate tool, plus a harness comparing Jev's judgments with general LLMs on shared datasets.
MCP serverbetaJavaScriptOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPython