jev-test
Pre-registered benchmark where Jev makes every search and evidence decision for a 2B local model answering SimpleQA questions.
Overview
This repository tests whether a small local model (Gemma 4 E2B) can answer web questions without making things up when a separate decision model, Jev (jev-1.13.0), makes every call the small model is bad at: whether to search, which pages count, whether there is enough evidence, and whether each sentence is backed by its citation. SearXNG provides search and MemPalace provides verbatim memory. The judgment specification and thresholds were fixed before any run, and seven arms include open local judges.
On 500 SimpleQA questions, Jev's confidence on whether a passage holds the answer reached AUROC 0.899 across 9,075 passages, and it ranked passages better than Gemma judging itself. End to end, however, the judged pipeline scored 0.612 against 0.740 for the no-judge baseline, missed its main pre-registered bar and lost mainly by declining far more questions; the README reports this openly. It also ships an audit tool for checking any judge's confidence scores.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Does this passage contain evidence that answers the question? | — |
| Yes / no | Should the model search before answering? | — |
| Yes / no | Is there enough evidence to answer? | — |
| Yes / no | Is this sentence supported by the source it cites? | — |
How it works
- Jev decides whether to search; SearXNG retrieves pages
- Jev scores each passage for answer evidence
- Jev decides if evidence suffices; otherwise decline
- Gemma writes; Jev checks each sentence against its citation
Strengths
- Pre-registered spec and thresholds
- Strong passage-level signal (AUROC 0.899) per the README
- Honest reporting of a negative end-to-end result
- Reusable confidence audit tool
Caveats
- Judged pipeline underperformed the baseline end to end
- Over-abstention is an open problem
- Complex local setup (SearXNG, MemPalace, local model)
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- CLI harness
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Pre-registered benchmark: can a 2B local model (Gemma 4 E2B) answer web questions without making things up when a decision model (TypeSafe Jev) makes every call? SearXNG for search, MemPalace for verbatim memory, seven arms including open local judges. Spec and thresholds fixed before any run.
logicrw/awesome-jev-projects
Frequently asked questions
Did Jev make the small model better?
Not end to end: the judged pipeline scored 0.612 versus 0.740 for the baseline, mainly by declining more questions.
What worked well?
Jev's passage-evidence confidence reached AUROC 0.899 and ranked passages better than the small model itself.
Were thresholds tuned after seeing results?
No, the judgment spec and thresholds were fixed before the first run.
Related projects
Next.js site with six live Jev demos, each compared side by side with Gemini or Claude on latency, cost and agreement.
DemobetaTypeScriptPre-registered benchmark of Jev as a medical hallucination check on MedHallu, versus fast LLMs, on accuracy, latency and cost.
BenchmarkresearchPaper and benchmark testing Jev as a zero-shot detector of AI alignment failures across 44 benchmarks and ten failure types.
BenchmarkresearchPythonIndependent benchmark asking whether Jev can replace an LLM reranker, on natural-language search over 6,245 YC companies, with a live demo.
BenchmarkresearchTypeScriptExperiment using Jev Noul judgments on accumulated evidence to decide when Zilliz's DeepSearcher agent should stop searching.
ProjectresearchPythonAgent observability platform that finds and fixes agent failures, with an optional Jev preclassifier for conversation checks.
ProjectproductionTypeScript