jev-judge
'Vitest for LLM outputs': CLI running YAML/JSON eval suites on Jev for RAG faithfulness, hallucination and agent safety in CI.
Overview
jev-judge positions itself as Vitest for LLM outputs. It runs declarative YAML, JSON or JSONL test suites using Jev Noul, Choice and Score decisions to check RAG faithfulness, detect hallucinations and test agent safety, with watch mode, parallel batching and GitHub Actions pull-request reporting, per the list summary.
The README argues LLM-as-judge in CI is slow, expensive and flaky, and that Jev's typed decisions avoid parsing and drift. Its sample output shows tests such as a grounded answer, a detected hallucination and refusal to exfiltrate a secret, with per-test cost estimates. Speed and cost claims (sub-100 ms, large savings versus a GPT-4o judge) are the author's, and the project is at version 0.1.0.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this answer grounded in the provided context?(inferred) | — |
| Yes / no | Does this agent action violate safety rules?(inferred) | — |
How it works
- Write eval suites in YAML, JSON or JSONL
- Run jev-judge locally or in CI
- Jev answers typed questions per test
- Report pass or fail with cost on PRs
Strengths
- Declarative test suites
- CI integration with PR reports
- Much cheaper than LLM judges per the author
Caveats
- Early version (0.1.0)
- Performance figures are the author's claims
At a glance
- Type
- CLI
- Status
- prototype
- Language
- TypeScript
- Runtime
- CLI with GitHub Actions
- Has code
- yes
Use cases and integrations
How curated lists describe it
CI/CD & LLM evaluation: runs declarative YAML/JSON/JSONL test suites natively on Jev Noul, Choice, and Score decisions for RAG faithfulness, hallucination detection, and agent safety with watch mode, parallel batching, and GitHub Actions PR reporting in sub-100ms at $0.00004 per decision.
yibie/awesome-jev
Frequently asked questions
What does Jev judge?
Test assertions such as groundedness, hallucination and agent safety.
Does it run in CI?
Yes, with GitHub Actions PR reporting.
Are the speed claims verified?
They come from the README's sample output.
Related projects
Agent observability platform that finds and fixes agent failures, with an optional Jev preclassifier for conversation checks.
ProjectproductionTypeScriptSemantic test matchers for Vitest and Jest that ask Jev whether output satisfies a plain-English expectation and threshold the probability.
ProjectbetaTypeScriptpytest plugin for semantic assertions on LLM output, asking Jev whether plain-English claims hold with a 0.8 probability bar.
PluginbetaPythonLearning lab for DeepEval with Jev as the judge: replayed evaluations, adjustable decision rules, release gates and drift alerts.
ProjectprototypePythonLocal PR reviewer: deterministic diff handling plus Jev or Laya judgments of severity, category and true-positive per hunk.
ProjectprototypePythonPython toolkit that evaluates LLM, RAG and agent outputs and guards tool calls with typed decision-model questions.
BenchmarkbetaPython