When a Judgment Layer’s Self-Reported Fields Lie
Independent paper testing Jev's self-reported fields against ground truth, finding a collapsed verdict vocabulary and misleading flags.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
This paper, archived on Zenodo, measures Jev's typed decisions as one of three judgment layers compared on a common item set, and tests the fields Jev reports about itself against ground truth rather than trusting them. According to the list summaries, it finds that Jev's verdict vocabulary reaches only three values where the description lists six, that a sufficiency field does not separate thin evidence from contradictory evidence, and that a truncation flag does not track the clamp actually in force.
It includes a reproduction artifact and a 59-check verification suite. The write-up is candid that one contradiction reading has no JSON artifact behind it, noting this in its own errata. It is useful for anyone relying on metadata fields in a decision-model response.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Do Jev's self-reported fields (verdict, sufficiency, truncation) match ground truth? | — |
How it works
- Run Jev and two other judgment layers on a common item set.
- Compare self-reported fields with ground truth.
- Publish a reproduction artifact and a 59-check verification suite.
Strengths
- Independent measurement with reproduction artifacts.
- Transparent errata.
Caveats
- Known through list summaries.
- One finding lacks an artifact, as the authors note.
At a glance
- Type
- Paper
- Status
- research
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Independent measurement: tests Jev’s self-reported access-layer fields against ground truth rather than trusting them, reporting a verdict vocabulary reaching three values where the description lists six and a sufficient field that does not separate thin evidence from contradictory evidence; the contradiction reading has no JSON artifact behind it and the write-up says so in its own errata.
yibie/awesome-jevIndependent measurement of Jev's typed decisions as one of three judgment layers compared on a common item set, with a reproduction artifact and a 59-check verification suite; reports Jev's verdict vocabulary collapsing to three reachable values and a truncation flag that does not track the clamp actually in force.
cobanov/awesome-jev
Frequently asked questions
What did it find?
Jev's verdict vocabulary reaches three of six described values, and some flags do not track what they claim to.
Is it reproducible?
It ships a reproduction artifact and a 59-check verification suite.
Are there caveats?
Yes, its errata note one reading without a JSON artifact.
Related projects
Paper auditing 28 early Jev papers and proposing a 14-item checklist for evaluating typed decision models.
PaperresearchPre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPythonBlack-box reverse-engineering archive studying how the Jev decision model behaves, with evidence matrices and calibration analysis.
ProjectresearchPythonIndependent API-only audit of what Jev's probabilities mean: abstain removal, Korean, coherence, option order, interference and caching.
ProjectresearchPythonResearch framework (DART) testing whether Jev's probabilities are a reliable signal to answer, escalate or abstain.
ProjectresearchPythonEvidence ledger auditing Jev claims, recomputing published calibration and running an 800-item difficulty-gradient study.
AppresearchPython