JudgeJev
Learning lab for DeepEval with Jev as the judge: replayed evaluations, adjustable decision rules, release gates and drift alerts.
Overview
JudgeJev is a hands-on learning lab for using DeepEval with Jev as the evaluation judge. The public site replays 17 real, recorded Jev evaluations of fictional support cases, starting with a false shipping guarantee that scored 15.1% next to a correct control at 95.5%. You can inspect raw requests and responses, change weights, thresholds and strict mode on saved probabilities, and see why a candidate fails a release gate. The hosted site makes no model calls; its drift sequence is reconstructed and labels reused samples.
A separate Python app makes fresh provider calls with your own Jev key, streams progress and keeps a local evidence ledger with hashes that tests deliberately corrupt to show verification failing. Exercises cover DeepEval test cases and datasets, a native JevEval using Noul, ordinal Score and credited Choice questions through DeepEval's TypeSafe adapter, RAG diagnosis separating agreement with retrieved text from correctness against policy, release decisions and monitoring. The README clarifies that its RAG criteria are custom Jev questions and gate logic is application code.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is the answer consistent with the authoritative policy?(inferred) | — |
| Score | How well does the answer meet this ordinal criterion? | — |
| Choice | Which credited option describes the answer? | — |
How it works
- Build DeepEval test cases from support tickets and answers
- Jev answers Noul, Score and Choice criteria
- Scores are weighted and thresholded
- Release gate checks pass rate and mean drop
- Monitoring compares rolling evaluations with a frozen reference
Strengths
- Hosted lab runs without any model calls
- Evidence ledger with hash verification
- Clear separation of model judgments and gate logic
Caveats
- Educational, using fictional cases
- Live app needs your own Jev key
- Not DeepEval's built-in RAG metrics
Getting started
git clone https://github.com/shyamsridhar123/JudgeJev.git, create a Python 3.12 venv and follow the live lab instructions; or open the hosted site.At a glance
- Type
- Project
- Status
- prototype
- Language
- Python
- Runtime
- Python app with static demo site
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
The public site replays \*\*17 real, recorded Jev evaluations\*\* of fictional support cases. It starts with a false shipping guarantee that scored \*\*15.1%\*\*, alongside a correct control that scored \*\*95.5%\*\*. You can inspect raw requests and responses, adjust decision rules, and see why a five-pair candidate fails a release gate.
logicrw/awesome-jev-projects
Frequently asked questions
Does the hosted site call Jev?
No, it replays recorded evaluations; the Python app makes fresh calls.
What does Jev judge?
Support answers against criteria such as policy correctness, via DeepEval's TypeSafe adapter.
Are the gates part of DeepEval?
No, the README says gate and drift rules are application logic.
Related projects
Agent observability platform that finds and fixes agent failures, with an optional Jev preclassifier for conversation checks.
ProjectproductionTypeScriptSemantic test matchers for Vitest and Jest that ask Jev whether output satisfies a plain-English expectation and threshold the probability.
ProjectbetaTypeScriptpytest plugin for semantic assertions on LLM output, asking Jev whether plain-English claims hold with a 0.8 probability bar.
PluginbetaPython'Vitest for LLM outputs': CLI running YAML/JSON eval suites on Jev for RAG faithfulness, hallucination and agent safety in CI.
CLIprototypeTypeScriptNext.js site with six live Jev demos, each compared side by side with Gemini or Claude on latency, cost and agreement.
DemobetaTypeScriptPython toolkit that evaluates LLM, RAG and agent outputs and guards tool calls with typed decision-model questions.
BenchmarkbetaPython