jevcheck
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
Overview
jevcheck is described as pytest for Jev. Because probabilities and model versions move, a raw 0.94 is not a release decision. jevcheck records a production contract (a baseline model, fixtures and expected answers) and evaluates a candidate model against it, reporting unchanged answers, confidence regressions and answer flips with exact diffs such as billing to general or 0.94 to 0.71, and exiting non-zero on failure so it can block an upgrade in CI.
Version 0.1 evaluates a candidate against fixtures; version 0.2 adds two-model execution, recording a baseline model's answers and comparing a candidate against that snapshot or fetching both live. It rejects floating aliases such as jev-latest and jev-preview unless explicitly allowed, and the README notes that some version labels in its fixtures are unverified examples while jev-1.13.0 is a documented pin. A ten-minute offline tutorial and an examples pack are provided. It installs from PyPI, uses only TYPESAFE_API_KEY for live calls, mocks the network in unit tests, and is MIT licensed.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Do the candidate model's answers and confidences match the pinned production contract? | — |
How it works
- Write a contract: pinned model, fixtures, expected answers
- Ship on the pinned model
- Evaluate a candidate (or record baseline and compare)
- Report unchanged, regressions and flips; fail on breaking changes
Strengths
- Catches silent answer flips and confidence drops on upgrade
- Rejects floating aliases by default
- Offline tutorial and mocked tests
Caveats
- Fixture coverage determines what is caught
- Some example version labels are unverified
Getting started
pip install jevcheck, set TYPESAFE_API_KEY, write a contract and run `jevcheck eval`.At a glance
- Type
- Project
- Status
- beta
- Language
- Python
- Runtime
- CLI
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Probabilities and model versions move. A raw \`0.94\` is not a release decision. jevcheck records a \*\*production contract\*\* (baseline model + fixtures + expected answers) and evals a candidate against that fixture.
logicrw/awesome-jev-projects
Frequently asked questions
What does jevcheck report?
Unchanged answers, confidence regressions and answer flips with exact diffs.
Can I use jev-latest?
Floating aliases are rejected unless you pass --allow-unpinned.
Can it compare two models?
Yes, version 0.2 records a baseline and compares a candidate against it.
Related projects
Write-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonHarness that runs Jev on all 19,646 TMMLU+ Traditional Chinese questions as four-way Choices, reporting accuracy and calibration.
ProjectresearchPythonReliability testing for typed decision models: contracts, calibration metrics, metamorphic fuzzing and CI pass/warn/fail gates.
DemoprototypePython- reflexbench★ 6
Open benchmark for System One decision models on Choice, Score and Noul, reporting accuracy, calibration and robustness separately.
BenchmarkproductionPython Pre-registered audits of AI judges that measure how many decisions a judge can take alone at a given error rate, with calibration.
BenchmarkbetaPythonClaude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.
PluginbetaPython