jev-as-a-judge
Experiment using Jev through langchain-typesafe as an agent-eval judge, compared with GPT-5.6 and Claude Sonnet 4.6 against human labels.
Overview
Jev as a judge for agent evals tests a third option between deterministic code evaluators and LLM-as-a-judge: using Jev, TypeSafe's decision model, to grade agent runs. Agent evaluation is framed as a decision task, so Jev's typed Noul, Score and Choice questions, such as whether the final answer is grounded in retrieved evidence, how useful it is, or whether the agent searched appropriately, can be asked in parallel against the same state.
The experiment builds a weather agent with LangChain's Deep Agents and Tavily search, defines five cases in a LangSmith dataset, and compares Jev with GPT-5.6 Luna, GPT-5.6 Terra and Claude Sonnet 4.6 on the same fixed runs, measuring binary accuracy, score reliability, cost and latency. The author reports that Jev's decision-first design coincided with lower latency, cost and variance, while stressing that a repeatable evaluator can still be consistently wrong, so each judge is compared with human labels and accuracy is kept separate from reliability. The MrJev list notes there is no licence file.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is the final answer grounded in the retrieved evidence? | yes · no |
| Score | How useful is the answer? | — |
| Choice | Did the agent search appropriately? | — |
How it works
- Run a fixed set of agent traces
- Ask each judge the same typed or rubric questions
- Compare with human labels for accuracy
- Measure reliability, cost and latency across repeats
Strengths
- Separates accuracy from reliability
- Integrates with LangSmith and langchain-typesafe
- Lower latency, cost and variance per the README
Caveats
- Very small case set (five request types)
- No licence file
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- LangChain / LangSmith
- Has code
- yes
Use cases and integrations
How curated lists describe it
Using Jev as an evaluator.
hellogumbo/awesome-jevUses Jev through `langchain-typesafe` as the judge in an eval suite, asking typed quality questions instead of asking a larger model to grade. No licence file.
MrJev/awesome-jev
Frequently asked questions
Can Jev replace an LLM judge?
The experiment found lower latency, cost and variance, but stresses checking accuracy against human labels.
What agent was evaluated?
A weather agent built with Deep Agents and Tavily search.
Which judges were compared?
Jev, GPT-5.6 Luna, GPT-5.6 Terra and Claude Sonnet 4.6.
Related projects
LangChain post comparing Jev with LLM judges for agent evaluation on accuracy, repeatability, latency and cost.
ArticleresearchEvaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonPortuguese workshop: a LangGraph sales agent where each decision step can be made by Jev or a frontier LLM and measured side by side.
ProjectprototypePythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPython