jev-mcp

Eval-first MCP server for Jev: prototype questions, map them over many items, and measure accuracy and thresholds on labelled data.

MCP serverbetaMIT

View on GitHub ↗

Overview

jev-eval-mcp is an MCP server for TypeSafe's Jev that focuses on whether a question can be trusted rather than just asking it. It exposes three tools: jev_ask prototypes a question against one state and shows the full distribution; jev_map runs a question set over many items and returns one compact row each; jev_eval measures question variants against labelled examples with accuracy, calibration, a threshold sweep and the worst misses.

It is aimed at agent users who want to triage hundreds of files, findings or tickets without reading them all into context, and who want thresholds chosen from data. Questions use Jev's three types (noul, choice, score) keyed by ids that are not sent to the model. The README recommends a loop of ask, eval, then map, picking the threshold from the sweep table.

What the decision model decides

Type	Question	Answers
Yes / no	Is this ticket urgent?	yes · no
Score	How severe is this?	trivial · minor · major · critical
Choice	Which team owns this?	billing · platform

How it works

Try two or three question wordings with jev_ask
Run promising wordings over labelled examples with jev_eval
Read the threshold sweep and worst misses, then pick a threshold
Apply the measured question to the real workload with jev_map

Strengths

Thresholds come from measured data instead of intuition
Bulk triage without loading every item into the agent's context
Several questions per call cost one round trip

Caveats

Requires a TypeSafe API key
Evaluation quality depends on the labelled examples you supply

Getting started

claude mcp add jev -e TYPESAFE_API_KEY=sk-... -- npx -y jev-eval-mcp

At a glance

Type: MCP server
Status: beta
Language: TypeScript
Runtime: Node MCP server
Has code: yes
License: MIT

Use cases and integrations

Infra / SDKs / Integrations537

How curated lists describe it

An evaluation-focused Jev MCP server for individual questions, batch processing and question or threshold comparisons.
logicrw/awesome-jev-projects
An eval-first MCP server for TypeSafe.
wh000wh000/awesome-jev-live
An eval-first MCP server for Jev, that returns typed judgments (noul, choice, score) with probabilities instead of generated text.
cobanov/awesome-jev
Eval-first MCP server that focuses on knowing whether Jev's answers can be trusted for your task.
MrJev/awesome-jev

Frequently asked questions

What makes this MCP server different?

Besides asking questions, it measures question variants against labelled data and sweeps thresholds.

Which clients does it support?

Any MCP client; the README gives configs for Claude Code, generic mcp.json and opencode.

What does it need?

Node 20+ and a TypeSafe API key; the model defaults to jev-latest.

Related projects

jev-pipeline-skill
Agent skill and CLI that chains Jev question design: find decision calls, check fit, draft, lint, calibrate thresholds and plan a cascade.
Agent skillprototypePython
Eval Genius
Agent skill that tells you when you need an eval and how to build it, with an opt-in lane routing closed-label residue to Jev.
BenchmarkbetaPython
jev-skills
Claude Code and Codex skills powered by Jev, published with a calibration curve measured on 4,995 real agent decisions.
Project
augustus
Agent skills for placing, building, evaluating and improving decision-model systems, with Jev as the default hosted example.
Agent skillbeta
Jev DSH Decision Engine
Decision plugin for agent harnesses where Jev recommends tools, skills or owners and scores output, for DeepSeek Harness and iPolloWork.
Agent skillbetaTypeScript
jev-mcp (arunav25)
MCP server exposing Jev as one evaluate tool, plus a harness comparing Jev's judgments with general LLMs on shared datasets.
MCP serverbetaJavaScript