jev-vs-pii
Benchmark of Jev versus rules, local PII models and LLMs at finding personal information: accuracy, leaks, cost and latency.
Overview
jev-vs-pii asks how well Jev finds personal information in text compared with today's tools and current LLMs, across three datasets. On European Court of Human Rights judgments, where context decides what identifies someone, Jev tied Claude Haiku 4.5 for the best score (F2 0.83 versus 0.82), found 81% of details that identify someone only through context such as a relative, employer or town, answered in 0.66 s versus 3.8 s, cost about three quarters as much and let the least personal information through (11% unmasked versus 17%). On short synthetic text, larger LLMs stayed ahead.
It also shows question design matters: Jev initially masked the word Account in Account number because the question asked what each word is, and adding one answer option for the name of a kind of information raised its F2 from 0.84 to 0.91; fixes were chosen on a separate dev set. The author concludes fixed formats suit cheap local models while context-dependent policies need instruction-following models, and that decision models are not yet usable for anonymisation under GDPR until offered with open weights or compliant terms.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | What kind of information is this word or span (personal identifier, name of a kind of information, other)?(inferred) | — |
| Yes / no | Could this detail re-identify the person?(inferred) | — |
How it works
- Prepare three datasets including court judgments
- Run rules, local PII models, LLMs and Jev
- Fix question design on a dev set
- Score F2, leaks, calibration, latency and cost on test
Strengths
- Jev tied the best LLM on context-heavy text, faster and cheaper per the README
- Separate dev set for fixes
- Practical guidance on method choice
Caveats
- LLMs ahead on short synthetic text
- Paid APIs not chosen for GDPR terms
- Comparison of approaches, not a vendor shortlist
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- CLI / notebooks
- Has code
- yes
Use cases and integrations
How curated lists describe it
How good is Jev, a decision model, at finding personal information?
wh000wh000/awesome-jev-liveHow good is Jev, a decision model, at finding personal information? Head-to-head against rules, local PII models and LLMs on three datasets: accuracy, leaks, calibration, cost and latency.
hellogumbo/awesome-jev
Frequently asked questions
How did Jev compare with Claude Haiku 4.5?
It tied on court judgments (F2 0.83 versus 0.82), was about six times faster and leaked less personal information.
Where did Jev fall behind?
On short synthetic text, where larger LLMs stayed ahead.
Can I use it for GDPR anonymisation now?
The author says not yet, because the paid APIs were not chosen for GDPR or data-residency terms.
Related projects
Benchmark testing whether Jev or open Kev can replace an LLM judge in agent guardrail callbacks on 7,925 public examples.
ProjectresearchPythonLocal-first personal-data inventory tool with optional Jev classification per column, not per row, and dry-run payload review.
ProjectprototypePythonRust CLI of Jev experiments: HIPAA-style PHI detection, code-comment review, tone analysis and occupation classification.
CLIprototypeRustZero-shot named entity recognition that turns Jev's Choice answers into entity spans via nominate, verify and boundary questions.
LibrarybetaPythonX post comparing Jev with a hand-built regex on 544 Spanish data-protection resolutions: 98.2% agreement for about five cents.
Discussiondiscussion- jev-arena★ 122
Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.
ProjectbetaTypeScript