jev-eval
Write-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
Overview
This repository is a write-up of testing Jev on public and private data, framed around whether it works better as a classifier or a filter. The author argues that launch comparisons against frontier models miss the point, since high-volume pipeline decisions mostly run on small, fast models such as flash-lite class, gpt-5.4-mini or gpt-5.6-luna, so that is the tier Jev must beat.
It explains the API with a worked example: a sentiment Choice on a short review returned negative at 0.97 probability with 0.93 confidence, took 850 ms from a laptop and cost $0.000014 for 336 input tokens. The author notes that TypeSafe does not document how confidence is computed (it is not the top probability) and that gating on top probability gave a similar picture on public sets. The post defines a confident answer as a Noul within 0.1 of either end, or a Choice confidence of at least 0.9, and measures how often confident answers are correct.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Is this movie review positive or negative? | negative · positive |
| Yes / no | Is this page worth reading, or is this email a request? | — |
How it works
- Assemble public and private labelled datasets
- Run Jev and small-tier LLMs on the same items
- Define confident answers by Noul extremes or Choice confidence
- Compare accuracy overall and on confident subsets
Strengths
- Compares against realistic small-model baselines
- Explains the confidence field with a concrete example
- Includes private data, not just public sets
Caveats
- Private dataset results cannot be reproduced by others
- Confidence definition chosen by the author
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Evaluation scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
TypeSafe.
wh000wh000/awesome-jev-live
Frequently asked questions
What is Jev compared against?
Small, fast LLM tiers that teams actually use for classification, not frontier models.
What counts as confident?
A Noul within 0.1 of 0 or 1, or a Choice confidence of 0.9 or higher.
How is confidence computed?
The author notes TypeSafe does not document it and that it differs from the top probability.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonHarness that runs Jev on all 19,646 TMMLU+ Traditional Chinese questions as four-way Choices, reporting accuracy and calibration.
ProjectresearchPythonReliability testing for typed decision models: contracts, calibration metrics, metamorphic fuzzing and CI pass/warn/fail gates.
DemoprototypePython- reflexbench★ 6
Open benchmark for System One decision models on Choice, Score and Noul, reporting accuracy, calibration and robustness separately.
BenchmarkproductionPython Pre-registered audits of AI judges that measure how many decisions a judge can take alone at a given error rate, with calibration.
BenchmarkbetaPythonClaude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.
PluginbetaPython