jev-test
Python harness stress-testing Jev's confidence, its accuracy as a RAGTruth hallucination judge, and its use as a RAG reranker.
Overview
This test harness backs three blog-style investigations into TypeSafe's Jev. The first audits whether Jev's confidence means anything, using YAML batches of off-menu questions (where no option fits, with and without an escape option), deliberately ambiguous tickets and 90 synthetic calibration cases, writing raw responses, per-question results and calibration buckets for each run.
The second uses Jev as a Noul judge of whether a response's claims are supported by a source passage, measured against human labels from RAGTruth on a stratified 600-case sample and compared with an LLM judge (Claude Haiku 4.5 by default) on accuracy, calibration, cost and latency, including ECE, isotonic recalibration and run-to-run consistency near p=0.5. The third asks whether chunking needs tuning, whether Jev adds precision as a reranker, and whether a chunk's text can change its own rank through injection. The client wraps typesafe-sdk but always logs raw HTTP responses, and a content-addressed cache and dry-run cost estimates are included.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Are this response's claims supported by the source passage? | — |
| Choice | Off-menu and ambiguous classification cases (e.g. ticket tone and urgency) | — |
How it works
- Load batch files of cases with expected labels
- Run against Jev, logging raw responses
- Grade accuracy and calibration by confidence bucket
- Compare with an LLM judge and test reranking and injection
Strengths
- Raw responses logged independent of SDK parsing
- Dry-run cost estimates and caching
- Covers calibration, hallucination judging and RAG
Caveats
- Synthetic calibration cases
- LLM-judge comparison needs an Anthropic key
- Results live in separate write-ups
Getting started
.venv/bin/pip install -r requirements.txt, fill .env, then `.venv/bin/python run.py testcases/calibration.yaml --model jev-1.13.0`.At a glance
- Type
- Demo
- Status
- research
- Language
- Python
- Runtime
- CLI
- Has code
- yes
Use cases and integrations
How curated lists describe it
Stress-Testing Jev's Confidence.
hellogumbo/awesome-jev
Frequently asked questions
What does the confidence audit test?
Whether Jev's confidence tracks correctness, including when no option fits and on ambiguous cases.
Which hallucination dataset is used?
RAGTruth, MIT licensed, on a stratified 600-case sample.
Does running it cost money?
Live runs do; the hallucination runner defaults to a dry-run that only estimates cost.
Related projects
Python library that replaces hard-coded RAG thresholds with calibrated decisions for stopping, chunking, selection, abstention and caching.
ServicebetaPythonResearch implementation of Jev-style typed decisions over clinical evidence: supported, contradicted or unresolved.
ProjectresearchPythonEditorial QA tool where Jev judges per fact whether an edit preserved the original, with a confidence gate and human review fallback.
ProjectbetaJavaMCP server giving coding agents typed judgment tools (verify, screen, rerank, gate, score) backed by Jev, with auto/review/escalate policy.
MCP serverbetaPythonTyped-decision accuracy layer for RAG: six checks for intent, relevance, clarification, sufficiency, grounding and caveats.
ServicebetaPythonSmall stdio MCP server with ask, rank and verify tools backed by Jev, a local event log and installers for several agent harnesses.
MCP serverbetaTypeScript