jev-vs-ml-classifiers
Learning project benchmarking zero-shot Jev and GPT models against scikit-learn classifiers and ModernBERT on five datasets.
Overview
This repository compares two kinds of classifiers on five public datasets. Trained models are scikit-learn's classifier-comparison set at library defaults plus a fine-tuned ModernBERT; zero-shot models are TypeSafe's Jev, OpenAI's GPT-5-nano, GPT-5-mini and GPT-6-luna, and BART-MNLI, each called the way its documentation shows. Everything is scored on the same test rows beside a majority-class baseline.
Trained models scored highest where labels exist: QDA averaged 0.76 macro-F1 and ModernBERT 0.69. Among zero-shot models averaged over three prompt variants, GPT-6-luna scored 0.70, GPT-5-mini and Jev tied at 0.58, GPT-5-nano 0.53 and BART-MNLI 0.22, below the 0.37 baseline. Jev was the fastest and cheapest API model at about 168 ms and $0.033 per 1,000 predictions. Readmission was hard for everyone. The author flags that public datasets may be in pretraining data, API answers vary between runs, the ModernBERT recipe changed once, and that the code was heavily AI-assisted and is for learning, not research quality.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which class does this row belong to (e.g. readmitted or not, income bracket)?(inferred) | — |
How it works
- Load five datasets with shared test rows
- Train scikit-learn models and fine-tune ModernBERT
- Query zero-shot models with three prompt variants
- Compare macro-F1, latency and cost against a majority baseline
Strengths
- Broad comparison including a majority baseline
- Reports latency and cost per 1,000 predictions
Caveats
- Not research-quality code by the author's own account
- Possible pretraining contamination
- Little hyperparameter tuning
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- CLI / notebooks
- Has code
- yes
Use cases and integrations
How curated lists describe it
How well does the latest JEV model do against classical ML classifier models…
wh000wh000/awesome-jev-live
Frequently asked questions
How did Jev do?
It tied GPT-5-mini at 0.58 macro-F1 among zero-shot models, behind GPT-6-luna at 0.70.
Was it cheaper?
Yes, about $0.033 per 1,000 predictions at about 168 ms, the fastest and cheapest API model tested.
Did trained models win?
Yes, QDA averaged 0.76 macro-F1 where labelled data existed.
Related projects
Independent 52-page Chinese report on Jev that checks vendor claims and reproduces 50 Chinese tests, with traceable data.
ProjectresearchPythonBenchmark of BM25, embeddings, rerankers and Jev for picking the right MCP tool among 525, plus an experimental Jev search engine.
MCP serverresearchTypeScriptRecorded benchmark of Jev versus two frontier LLMs on 100 synthetic security-triage cases, with a static dashboard.
AppresearchTypeScriptPre-registered audits of AI judges that measure how many decisions a judge can take alone at a given error rate, with calibration.
BenchmarkbetaPythonBenchmark of Jev on 159 community-reconstructed questions from Indonesia's SNBT 2025 university entrance exam, all responses published.
BenchmarkresearchPythonIndependent benchmark asking whether Jev can replace an LLM reranker, on natural-language search over 6,245 YC companies, with a live demo.
BenchmarkresearchTypeScript