Typed Evals
Python toolkit that evaluates LLM, RAG and agent outputs and guards tool calls with typed decision-model questions.
Overview
Typed Evals is a Python package for evaluating responses and guarding actions with decision models. By default it uses Jev (jev-1.13.0 via the official typesafe-sdk); it can also use Microsoft Decision-1 on OpenRouter through the same backend, or OpenAI's Decisions API. Presets check things like faithfulness, answer relevancy and context relevance for RAG in one judge request.
Beyond response grading, it evaluates recorded agent executions and checks proposed tool calls before they run. Metrics and thresholds can be customized, and an optional calibration step fits judge outputs against human labels. An offline demo works without an API key. It targets teams who want cheaper, structured LLM-as-judge style evaluation. Presets can be extended with your own metrics and thresholds, and the same backend abstraction lets teams compare judges without rewriting evaluation code.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Score | Is the response faithful to the provided contexts? | — |
| Score | Is the response relevant to the input question? | — |
| Yes / no | Should this proposed tool call be allowed?(inferred) | yes · no |
How it works
- Pass input, response and contexts (or an agent trace) to evaluate.
- A preset sends several metric questions in one judge request.
- Scores are compared with thresholds to pass or fail each metric.
- Optionally calibrate judge scores against human labels.
Strengths
- Several metrics in one request.
- Backend-agnostic: Jev, Decision-1 or OpenAI Decisions.
- Optional calibration against human labels.
- Offline demo without a key.
Caveats
- Live use needs a hosted decision-model key.
- Judge accuracy depends on the backend and your data.
Getting started
python -m pip install typed-evals; export TYPESAFE_API_KEY='your-key'At a glance
- Type
- Benchmark
- Status
- beta
- Language
- Python
- Runtime
- library
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Python toolkit that puts evaluation of LLM, RAG and agent output as typed questions, with optional calibration against human labels.
MrJev/awesome-jev
Frequently asked questions
What does Typed Evals measure?
Response quality metrics like faithfulness and relevancy, agent executions, and whether tool calls should run.
Which backends are supported?
Jev by default, Microsoft Decision-1 via OpenRouter, and OpenAI Decisions.
Can I try it without a key?
Yes, there is an offline demo.
Related projects
Agent observability platform that finds and fixes agent failures, with an optional Jev preclassifier for conversation checks.
ProjectproductionTypeScriptSemantic test matchers for Vitest and Jest that ask Jev whether output satisfies a plain-English expectation and threshold the probability.
ProjectbetaTypeScriptpytest plugin for semantic assertions on LLM output, asking Jev whether plain-English claims hold with a 0.8 probability bar.
PluginbetaPython'Vitest for LLM outputs': CLI running YAML/JSON eval suites on Jev for RAG faithfulness, hallucination and agent safety in CI.
CLIprototypeTypeScriptLearning lab for DeepEval with Jev as the judge: replayed evaluations, adjustable decision rules, release gates and drift alerts.
ProjectprototypePythonChecks whether each cited paper supports its citing sentence: code fetches papers, Claude proves a quote, Jev scores it, a human decides.
Serviceproduction