Jev vs. ML
Benchmark of Jev 1.13.0 against 11 classical classification pipelines on eight datasets under a published protocol.
Overview
Jev vs. ML compares Jev 1.13.0 with 11 classical classification pipelines across eight datasets, using three training seeds and a shared test set under a published V3 protocol, with an executed Kaggle notebook, raw and adjusted CSVs, a manifest with hashes and a validation record.
Jev's strongest result was IMDb sentiment classification at 96.3% raw balanced accuracy versus 88.4% for the best classical pipeline. Results elsewhere were mixed, and classical pipelines led on all four tabular datasets. The author describes it as a bounded-budget comparison and notes that threshold-adjusted Jev uses labelled policy data for binary tasks, so those scores are not zero-shot end to end, and that seed standard deviations are not confidence intervals. A V4 set of policy, temporal and iterative-decision experiments responding to Reddit feedback is in development.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which class does this example belong to (per dataset, e.g. positive or negative sentiment)?(inferred) | — |
How it works
- Prepare eight datasets with shared test sets
- Train classical pipelines across three seeds
- Ask Jev zero-shot, optionally threshold-adjusted
- Compare balanced accuracy in an interactive report
Strengths
- Published protocol, hashes and frozen bundle
- Clear win on text sentiment; honest about tabular losses
- Reproducible on Kaggle
Caveats
- Threshold-adjusted scores use labelled data
- Mixed results; classical models win on tabular data
- Per-example diagnostics unavailable in this release
Getting started
Upload notebooks/jev_benchmark_v3.ipynb to a Kaggle session with Internet and GPUs enabled.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Jupyter / Kaggle
- Has code
- yes
Use cases and integrations
How curated lists describe it
Compares a typed decision model with classical classification pipelines across eight datasets, with a published protocol and an interactive report.
MrJev/awesome-jev
Frequently asked questions
Where did Jev do best?
IMDb sentiment, at 96.3% raw balanced accuracy versus 88.4% for the best classical pipeline.
Where did it lose?
Classical pipelines led on all four tabular datasets.
Are the results zero-shot?
Raw scores are; threshold-adjusted binary scores use labelled policy data.
Related projects
- jev-arena★ 122
Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.
ProjectbetaTypeScript Jev applied to SemEval-2026 DimABSA dimensional sentiment tasks using only typed primitives, no generation or fine-tuning.
ProjectresearchPythonJapanese demo judging the same utterance with Jev and an LLM to compare emotion-shift outputs and speed side by side.
ServiceprototypeTypeScriptChinese 'emotion lie detector' demo comparing Jev's Choice, Score and Noul judgments on a message with a general LLM.
CLIprototypePythonBenchmark comparing Jev, other decision models and LLMs on four judgments over 60 synthetic candidate-experience reviews.
BenchmarkresearchSmall reproducible benchmark of Jev against LLMs, fine-tuned BERT, Laya and zero-shot NLI on SST-2, AG News and Banking77.
BenchmarkresearchPython