Jev judge call vs dimension scores
Study comparing one direct Jev question per row with 12–14 Jev-scored dimensions and fitted weights on three classification tasks.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
This Agent Journal post compares two ways of using Jev for classification: one direct question per row versus 12 to 14 Jev-scored dimensions combined with locally fitted weights. It ran three classification tasks with 5,477 test rows and 25,174 Jev calls for $1.43, reporting token costs, confidence intervals and false-positive rates.
According to the list summaries, decomposition won on Japanese NLI (0.9076 versus 0.8373) but flagged about 25 times more hard benign rows as attacks (37.2 percent versus 1.5 percent) on the attack-detection task. The full post was not collected here.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Direct label for the row(inferred) | — |
| Score | Score on each of 12 to 14 dimensions(inferred) | — |
How it works
- Ask a direct question per row
- Separately score 12 to 14 dimensions per row
- Fit weights locally on dimension scores
- Compare accuracy, false positives and cost
Strengths
- Large sample with costs and intervals
- Shows trade-off between accuracy and false positives
Caveats
- Summary only
- Results vary by task
At a glance
- Type
- Article
- Status
- research
Use cases and integrations
How curated lists describe it
Model evaluation: tests one direct Jev question per row against 12–14 Jev-scored dimensions with locally fitted weights on three classification tasks, reaching 0.9076 against 0.8373 on Japanese NLI but flagging about 25× more hard benign rows as attacks.
yibie/awesome-jevIndependent measurement on three classification tasks: one direct Jev question per row against 12–14 Jev-scored dimensions with locally fitted weights, with token costs, confidence intervals, and false-positive rates.
AnotiaWang/awesome-decision-modelsCompared "one direct Jev question per task" with.
wh000wh000/awesome-jev-liveOne direct Jev question per row against 12–14 Jev-scored dimensions with locally fitted weights on three classification tasks: 5,477 test rows, 25,174 Jev calls, $1.43. Decomposition wins on Japanese NLI (0.9076 vs 0.8373) but flags about 25× more hard benign rows as attacks (37.2% vs 1.5%).
hellogumbo/awesome-jev
Frequently asked questions
Which approach won?
Decomposition won on Japanese NLI but produced far more false positives on attack detection.
How large was the study?
5,477 test rows and 25,174 Jev calls for $1.43.
Where is it?
On agentjournal.dev.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonAudit of 300 web pages for AI citability with nine Jev decisions each, checked against an LLM judge for agreement, cost and latency.
ProjectresearchPython