jev-verify
Tool that recomputes Jev's confidence identities on outputs published in public repositories to spot answers Jev never actually produced.
Overview
jev-verify checks whether a published Jev output was really produced by Jev. Jev's confidence is a deterministic function of its returned probabilities, an identity the README credits to Stanislav Yurin's large live study (confidence equals (N times p_max minus 1) over (N minus 1)), later replicated by primeline. Because of this, any published answer can be recomputed, along with an expected-score identity for Score answers.
Where earlier work tested the live API, this project looks at what the ecosystem publishes. Scanning public repositories with no API access, it separates vendor-channel examples (10 of 10 consistent), recorded responses (843 of 854 consistent) and hand-authored fixtures (only 115 of 296 consistent). All 121 outputs whose confidence equals the fractional part of their score, such as score 2.58 with confidence 0.58, occur in hand-authored fixtures in projects such as ax-llm/ax and rig, which the author identifies as generator artifacts rather than real model output. The author describes their derivation of the identity as an independent replication, not a discovery. It is useful for SDK authors and anyone using published examples as test data.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Does this published Choice or Score output satisfy Jev's confidence and expected-score identities? | — |
How it works
- Collect published Jev outputs from a repository
- Recompute confidence from probabilities and expected score from level probabilities
- Flag violations such as confidence equal to the score's fractional part
- Report by source category
Strengths
- Works without API access
- Separates real recorded responses from hand-authored fixtures
- Credits and replicates prior work
Caveats
- Detects inconsistency, not correctness
- Relies on identities that could change with model versions
At a glance
- Type
- Service
- Status
- research
- Language
- Python
- Runtime
- CLI
- Has code
- yes
Use cases and integrations
How curated lists describe it
Developer tooling: recomputes Jev's confidence and expected-score identities against outputs published in public repositories rather than live API calls, separating vendor-channel examples (10/10) and recorded responses (843/854) from hand-authored fixtures (115/296), where all 121 outputs whose confidence equals the fractional part of their score are concentrated.
yibie/awesome-jev
Frequently asked questions
What identity does it check?
That Choice confidence equals (N*p_max - 1)/(N - 1), plus an expected-score identity for Score answers.
What did it find?
Recorded responses were almost all consistent, while many hand-authored fixtures were not, including 121 with confidence equal to the score's decimal part.
Who discovered the identity?
The README credits Stanislav Yurin's live study, replicated by primeline; this project replicated it from published artifacts.
Related projects
Declarative compiler that turns state, decisions and labeled examples into a measured, frozen JSON program of typed Jev questions.
LibraryprototypePythonOptimises the instructions and criterion descriptions of a Jev Choice from labelled data with EvoPrompt or GEPA, scored by Brier.
ProjectbetaPythonResearch demo where Jev answers plain questions about text and a small learned head adds new questions from human feedback.
ProjectresearchPythonpijev: drop-in wrapper for the TypeSafe Python SDK that averages Jev answers across option orders in one request.
LibrarybetaPythonClaude Code and Codex skills powered by Jev, published with a calibration curve measured on 4,995 real agent decisions.
ProjectPython trust middleware for the Jev API that logs decisions, measures calibration from your recorded outcomes and signs the evidence.
ServicePython