jevals
Openlayer library packing agent evals and guardrails (tool choice, grounding, injection, PHI) into one decision-model request.
Overview
jevals runs agent evals and guardrails as typed questions to Jev-style decision models instead of an LLM judge. All evals for a trace go out in one request that costs a few thousandths of a cent and returns in a few hundred milliseconds, so they can run on every trace and inside the agent loop. The README's example evaluates eight checks in one request: tool choice, whether the tool result was used, groundedness, staying in scope, answer relevancy, completeness, indirect prompt injection and PHI, at 1,388 tokens, $0.00006 and 0.33 seconds.
It accepts OpenAI-format messages and tool schemas, and also Anthropic content blocks or LangChain messages. Backends include Jev through TypeSafe, Vercel or OpenRouter, Kev or Laya running locally on a Mac, Eikos on your own GPU, or a regular chat LLM as a slower, costlier fallback, plus a mock backend so the whole library runs without a key. The motivation is that most teams eval only a sample because frontier-LLM judges are expensive, and Ragas-style metrics need multiple LLM calls each. It comes from Openlayer and is MIT licensed.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Was the agent's tool choice correct? | — |
| Score | How grounded is the answer in the tool results? | — |
| Yes / no | Does a tool result contain an indirect prompt injection? | — |
| Yes / no | Does the output contain protected health information? | — |
How it works
- Pass the trace messages and tool schemas to evaluate()
- All selected evals become typed questions in one request
- Backend returns answers with probabilities and usage
- Use results for monitoring or as in-loop guardrails
Strengths
- One request for many evals per trace
- README example costs $0.00006 in 0.33 s
- Local backends and a keyless mock
- Accepts several message formats
Caveats
- Eval quality depends on the chosen backend
- Chat-LLM fallback is slower and costlier
Getting started
pip install jevals, then export AI_GATEWAY_API_KEY (or TYPESAFE_API_KEY / OPENROUTER_API_KEY).At a glance
- Type
- Benchmark
- Status
- beta
- Language
- Python
- Runtime
- Library
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Openlayer's agent evals and guardrails as decision-model questions: tool choice, grounding, relevancy, and injection checks for a trace go out in one request, to Jev or locally to Kev, Laya, or Eikos. `pip install jevals`.
AnotiaWang/awesome-decision-modelsAgent evals and guardrails as Jev decisions: one request per trace, a fraction of a cent, fast enough for the agent loop. Runs locally with Kev or Laya.
logicrw/awesome-jev-projectsAgent evals and guardrails as Jev decisions: one request per trace, a fraction…
wh000wh000/awesome-jev-liveAgent evals and guardrails in one request. Built on Jev, Kev and Laya.
hellogumbo/awesome-jevAgent evals and guardrails as typed questions instead of an LLM judge, packing every eval for a trace into one request. From Openlayer, with a mock backend so the whole library runs without a key.
MrJev/awesome-jev
Frequently asked questions
How many requests does an eval run take?
One request per trace for all selected evals.
Can it run locally?
Yes, with Kev or Laya on a Mac or Eikos on your own GPU.
Can I try it without a key?
Yes, a mock backend runs the whole library offline.
Related projects
Agent observability platform that finds and fixes agent failures, with an optional Jev preclassifier for conversation checks.
ProjectproductionTypeScriptCLI that grades the coding-agent sessions on your disk (Claude Code, Codex, opencode, Gemini CLI) with Jev for cents.
CLIbetaTypeScriptVideo demo code comparing Jev with Laya, Strands Decider and Claude on grading, fetching and prompt-injection resistance.
AppresearchPythonOffline CLI that measures whether a classifier's confidence matches reality and turns mistake costs into a human hand-off threshold.
BenchmarkbetaPythonExample of using Jev as a webhook guardrail inside the agentgateway proxy to inspect model requests and responses.
ProxyproductionRustPortuguese study using Jev as a real-time guardrail and as a pre-evaluator for LLM-as-judge on customer-service agents.
ProjectresearchPython