jevtest
Semantic test matchers for Vitest and Jest that ask Jev whether output satisfies a plain-English expectation and threshold the probability.
Overview
This jevtest adds semantic matchers to Vitest and Jest. Instead of brittle string checks such as toContain('sorry'), you write await expect(reply).toSatisfy('apologizes politely'). The matcher asks Jev one narrow yes/no question about the output and turns the probability into pass or fail: toSatisfy passes at 0.85 or higher by default, and the negated form passes at 0.15 or lower, both adjustable.
toSatisfyAll checks several expectations in one API request, and toMatchSemanticSnapshot passes if output is identical to a baseline or the difference is classified as cosmetic with probability at least 0.8. Expectations can be refined with explicit yes and no descriptions to sharpen the boundary, plus context and model options. The README notes typical latency of 150 ms to 1.5 s per check, cheap enough for test suites. It is aimed at developers testing LLM-powered features such as support bots, where meaning matters more than exact wording.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Does this output satisfy the expectation (e.g. apologizes politely)? | — |
| Choice | Is the difference from the snapshot cosmetic or meaningful?(inferred) | cosmetic · meaningful |
How it works
- Install jevtest and register matchers in Vitest or Jest setup
- Write plain-English expectations in tests
- Each matcher asks Jev a yes/no question about the output
- Probability is compared with min or max thresholds to pass or fail
Strengths
- Assertions express intent rather than keywords
- Adjustable thresholds per assertion
- Several expectations in one request with toSatisfyAll
- Semantic snapshots tolerate cosmetic changes
Caveats
- Tests make network calls and need a key
- Probabilistic results can be flaky near thresholds
- Async matchers require await
Getting started
npm i -D jevtest, export TYPESAFE_API_KEY, and add jevtest/vitest/setup to setupFiles.At a glance
- Type
- Project
- Status
- beta
- Language
- TypeScript
- Runtime
- Vitest / Jest plugin
- Has code
- yes
Use cases and integrations
How curated lists describe it
Semantic test matchers for Vitest and Jest, powered by TypeSafe's Jev model. Write expectations in plain English, get calibrated probabilities back.
hellogumbo/awesome-jev
Frequently asked questions
What does toSatisfy check?
Whether Jev's probability that the output meets the expectation is at least the threshold, 0.85 by default.
How fast is it?
The README says usually 150 ms to 1.5 s per question.
Which test runners are supported?
Vitest and Jest.
Related projects
pytest plugin for semantic assertions on LLM output, asking Jev whether plain-English claims hold with a 0.8 probability bar.
PluginbetaPythonAgent observability platform that finds and fixes agent failures, with an optional Jev preclassifier for conversation checks.
ProjectproductionTypeScriptDesktop tool for testing chat endpoints conversationally, with rule checks and judged checks including TypeSafe's System One.
ProjectprototypeTypeScript'Vitest for LLM outputs': CLI running YAML/JSON eval suites on Jev for RAG faithfulness, hallucination and agent safety in CI.
CLIprototypeTypeScriptLearning lab for DeepEval with Jev as the judge: replayed evaluations, adjustable decision rules, release gates and drift alerts.
ProjectprototypePythonPython toolkit that evaluates LLM, RAG and agent outputs and guards tool calls with typed decision-model questions.
BenchmarkbetaPython