pytest-jev
pytest plugin for semantic assertions on LLM output, asking Jev whether plain-English claims hold with a 0.8 probability bar.
Overview
pytest-jev lets you test what an LLM app's reply means rather than its exact words. In a test you call jev.expect(reply, holds=[...], lacks=[...]) with plain-English claims; all claims about one text go to Jev in a single request as Noul questions. A claim in holds passes at p >= 0.8 and one in lacks at p <= 0.2, so when Jev is unsure the test fails visibly. Failures print each claim's probability.
The fixture also offers choice (for routing checks such as which team handles a ticket) and score (rubric levels compared probabilistically, for example P(level >= warm)). Extra context such as a policy or retrieved documents can be passed and referenced in claims. Answers are cached in .pytest_cache for free, stable reruns, and tests are skipped without a key. The README reports that on its 12 example tests Jev matched Claude Sonnet 5's verdicts in 5.3 s versus 27.1 s and $0.00017 versus $0.0192 per run, while warning these are designed cases, not a general accuracy measure.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Does this reply hold the claim, e.g. 'apologizes to the customer'? | — |
| Choice | Which team should handle this ticket? | — |
| Score | How polite is this support reply? | — |
How it works
- Write claims about an LLM reply in a pytest test
- Send all claims in one Jev request
- Pass holds at p >= 0.8 and lacks at p <= 0.2; anything unsure fails
- Print per-claim probabilities on failure and cache answers for reruns
Strengths
- README reports 5x faster and over 100x cheaper than Sonnet 5 on its examples
- Unsure verdicts fail instead of passing on a coin flip
- Cached answers make reruns free and deterministic
- Works through OpenRouter or TypeSafe
Caveats
- Benchmark is 12 designed cases; the README says to measure on your own labelled cases
- Thresholds are fixed defaults that may need tuning
Getting started
pip install pytest-jev, export OPENROUTER_API_KEY (or TYPESAFE_API_KEY), then run pytest.At a glance
- Type
- Plugin
- Status
- beta
- Language
- Python
- Runtime
- pytest plugin
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
LLM app testing: a pytest plugin that asks one Jev Noul per plain-English claim about a reply (all claims in one request), passes a claim at p ≥ 0.8 and fails anything unsure, and adds Choice and Score checks; on its 12 example tests it matched Claude Sonnet 5's verdicts in 5.3 s vs 27.1 s at $0.00017 vs $0.0192 per run.
yibie/awesome-jevpytest plugin for semantic assertions on LLM output: each plain-English claim about a reply becomes a Jev Noul in one request, a claim passes at p ≥ 0.8, and failures print every claim's probability; `choice` and `score` cover routing and rubric checks
AnotiaWang/awesome-decision-modelsExtra state goes in \`context\`, such as a policy or the documents a RAG app retrieved. A claim can name it in backticks:
logicrw/awesome-jev-projectspytest plugin where Jev decides whether each plain-English claim about a test's text holds, and the test passes only when every claim clears 0.8 (or stays at or below 0.2 for claims that must not hold), with Choice and Score answers compared by probability.
cobanov/awesome-jevSemantic assertions for pytest: test what your LLM app's output means, judged by TypeSafe's Jev.
hellogumbo/awesome-jev
Frequently asked questions
What does pytest-jev decide?
Whether each plain-English claim about a test's text holds, plus choice and score checks for routing and tone.
How is it different from LLM-as-judge?
It gets a typed probability, fails unsure claims visibly, and per the README is much faster and cheaper on its examples.
What happens in CI without a key?
Tests using the jev fixture are skipped.
Related projects
Semantic test matchers for Vitest and Jest that ask Jev whether output satisfies a plain-English expectation and threshold the probability.
ProjectbetaTypeScriptAgent observability platform that finds and fixes agent failures, with an optional Jev preclassifier for conversation checks.
ProjectproductionTypeScriptDesktop tool for testing chat endpoints conversationally, with rule checks and judged checks including TypeSafe's System One.
ProjectprototypeTypeScript'Vitest for LLM outputs': CLI running YAML/JSON eval suites on Jev for RAG faithfulness, hallucination and agent safety in CI.
CLIprototypeTypeScriptLearning lab for DeepEval with Jev as the judge: replayed evaluations, adjustable decision rules, release gates and drift alerts.
ProjectprototypePythonPython toolkit that evaluates LLM, RAG and agent outputs and guards tool calls with typed decision-model questions.
BenchmarkbetaPython