jev-papers
Classifies 1,000 arXiv AI papers into 24 topics with one Jev Choice each and checks agreement against an LLM judge.
Overview
Jev Papers classifies 1,000 recent arXiv AI papers into 24 topics with one Jev decision per paper, then checks those labels against a strong chat model on a seeded sample of 100. It is three scripts and one static page: fetch.py pulls papers from arXiv, classify.py asks a 24-way Choice per paper, eval.py runs the judge, and a topic map page shows the results with search and per-paper confidence.
The committed run reports total cost of $0.0585 for Jev versus $0.894 for the judge on 100 papers, and Jev latency of 56.9 ms median per paper (four papers per request). Jev and the judge agreed on 85 of 100; the judge's pick was Jev's first or second choice on 95. When Jev's confidence was 0.90 or higher they agreed 98% of the time, versus 63% below, and 14 of 15 disagreements had Jev confidence under 0.90. The README stresses the judge is another model, not ground truth.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which of 24 topics does this paper belong to? | — |
How it works
- Fetch recent arXiv papers
- Ask Jev a 24-way topic Choice per paper
- Ask an LLM judge the same question on a seeded sample
- Compare agreement by confidence band
- Render a static topic map
Strengths
- Very low classification cost
- Shows confidence predicts disagreement
- Resumable scripts with no dependencies
- Static site works from disk
Caveats
- Agreement with another model is not accuracy
- Sample of 100 for evaluation
- Requires an OpenRouter key
Getting started
export OPENROUTER_API_KEY=..., then python3 fetch.py && python3 classify.py && python3 eval.py, and open site/index.htmlAt a glance
- Type
- Project
- Status
- prototype
- Language
- Python
- Runtime
- Scripts plus static page
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
1,000 arXiv AI papers classified with one Jev decision each, checked against an LLM judge. Open rebuild, MIT.
logicrw/awesome-jev-projects
Frequently asked questions
What does Jev decide?
Which of 24 topics each arXiv paper belongs to.
How well does it agree with an LLM?
85 of 100 exactly, rising to 98% when Jev's confidence is 0.90 or higher.
How much did the run cost?
About $0.06 for 1,000 papers with Jev.
Related projects
Live race labelling 1,000 app reviews with four typed Jev questions versus Gemini 3.8 Flash JSON output.
AppprototypeJavaScript- jev-arena★ 122
Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.
ProjectbetaTypeScript Title and abstract screening for systematic reviews with Jev: include, exclude or maybe per record, with reasons, PRISMA counts and recall.
ProjectbetaPythonGoodWatch movie and TV recommendation monorepo, listed for an experiment comparing Jev question designs for attribute scoring.
ProjectresearchTypeScriptReproducible experiments and reports on what Jev can and cannot do, from skill routing to GPQA and housing prices.
ProjectresearchBun harness that runs Jev over datasets via OpenRouter, streams JSONL answers, and checks calibration, stability and regressions.
LibrarybetaTypeScript