jev-search-eval-report
Short report comparing Jev and many other fast decision classifiers with an LLM on grading section relevance in technical documents.
This repository is mostly documentation or write-up rather than runnable code.
Overview
This short research report (25 September 2026, updated 13 October 2026) asks whether a fast classifier can find the relevant sections of technical documents as reliably as a large language model. It compares Jev, simple-jev, Liquid AI D1, Fastino GLiDE and GLiNER-2.5-Decide, Cloudflare Clef and Clef-flash, Nace.ai Drex 1.5 and 2.0, SubQ Snap, vLLM-SR Decision 3.0 (d3-flash, local), offline D1 and Laya against an LLM (Gemma 4 31B) on grading question-section relevance.
The report is published as a web page; the repository README only summarises its scope, so the specific results are not reproduced here.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this section relevant to the question?(inferred) | yes · no |
How it works
- Pair questions with sections of technical documents
- Grade relevance with each classifier and the LLM
- Compare agreement and reliability
Strengths
- Broad comparison across many decision models
- Updated as new models appear
Caveats
- Results not summarised in the README
- Mostly a written report
Getting started
Read it at https://95pctai.github.io/jev-search-eval-report/At a glance
- Type
- Project
- Status
- research
- Runtime
- Static report
Use cases and integrations
How curated lists describe it
Short report: can fast classifiers grade section relevance as reliably as an…
wh000wh000/awesome-jev-live
Frequently asked questions
What does the report compare?
Jev and about a dozen other decision classifiers against Gemma 4 31B on section relevance grading.
Where can I read it?
At 95pctai.github.io/jev-search-eval-report.
What were the results?
They are in the report itself, not the README.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonLarge rubric-classification benchmark of Jev vs Claude Haiku 4.5, Sonnet 5, OpenJev and many open models, with full price tracking.
BenchmarkresearchPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePython