jeval
Offline CLI that measures whether a classifier's confidence matches reality and turns mistake costs into a human hand-off threshold.
Overview
jeval answers two questions about any classifier that returns a probability: when it says 0.9, how often is it right, and given what a mistake costs, where should the line sit between the machine deciding and a human deciding. It works from labelled decision records on disk with no server, database, network call, account or token, and applies to hosted APIs, gateways, local models, logistic regressions or scoring rules, though it was built with Jev in mind.
Outputs are a self-contained HTML report showing whether confidences are trustworthy, where they break and what the current threshold is costing; a thresholds.yaml file the application reads, with the threshold to deploy, an uncertainty range and whether a segment needs its own line; and a labels file of decisions worth labelling next. It can also catch a model change in CI as drift. A one-line installer creates a private environment, a demo runs on a synthetic log, and the README warns that the unrelated jeval package on PyPI is not this tool. It has CI and versioned releases.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Given labelled decisions and mistake costs, what confidence threshold should hand off to a human? | — |
How it works
- Load labelled decision records from disk
- Measure calibration and where it breaks
- Compute the cost-based hand-off threshold with uncertainty
- Emit report.html, thresholds.yaml and labels to collect; check drift in CI
Strengths
- Fully offline; no network or account
- Works with any probability-returning classifier
- Deployable thresholds file with uncertainty range
- CI drift detection
Caveats
- Needs labelled historical decisions
- Not on PyPI; the PyPI name belongs to another project
Getting started
curl -fsSL https://raw.githubusercontent.com/rlaope/jeval/main/install.sh | sh, then `jeval demo --out-dir /tmp/jeval-demo`.At a glance
- Type
- Benchmark
- Status
- beta
- Language
- Python
- Runtime
- CLI
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Measures what your Jev classifier's confidence is really worth, and sets the human hand-off line from what a mistake costs.
hellogumbo/awesome-jevMeasures how well a classifier's confidence matches reality and turns the cost of a mistake into the threshold where a human should take over.
MrJev/awesome-jev
Frequently asked questions
Does it call Jev?
No, it analyses labelled decision records offline.
What do I get?
An HTML report, a thresholds.yaml for your app and a list of decisions worth labelling.
Can I install it with pip install jeval?
No, that installs an unrelated project; use the installer or the release wheel.
Related projects
Agent observability platform that finds and fixes agent failures, with an optional Jev preclassifier for conversation checks.
ProjectproductionTypeScriptClaude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.
PluginbetaPythonCLI that grades the coding-agent sessions on your disk (Claude Code, Codex, opencode, Gemini CLI) with Jev for cents.
CLIbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonHarness that runs Jev on all 19,646 TMMLU+ Traditional Chinese questions as four-way Choices, reporting accuracy and calibration.
ProjectresearchPython