Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-mcp (arunav25)

MCP server exposing Jev as one evaluate tool, plus a harness comparing Jev's judgments with general LLMs on shared datasets.

MCP serverbeta
View on GitHub ↗

Overview

This project has two halves. The first is an MCP server with a single evaluate tool: an agent passes state and one or more typed questions, and gets back typed answers with probabilities, so it can branch on a number instead of parsing a sentence. The second is an evaluation harness that asks the same question to Jev and to a general LLM over a shared dataset and compares them with Brier score, calibration and McNemar's test.

It is for developers who want to know whether Jev actually beats what they used before, not just call it. An install command registers the server with Claude Code, Claude Desktop and Codex, with a dry-run option. The package is not yet on npm and installs from GitHub.

What the decision model decides

TypeQuestionAnswers
Yes / nois_urgent: does the sender need a response today rather than this week?yes · no
Choicedepartment: which team should own this ticket?billing · technical · sales

How it works

  1. Install from GitHub and export TYPESAFE_API_KEY
  2. Run jev-mcp install to register with supported clients
  3. Agents call evaluate with state and typed questions
  4. Use jev-eval to compare Jev and an LLM on a labelled dataset

Strengths

  • Probabilities let agents distinguish a confident yes from a coin flip
  • Built-in statistical comparison against general LLMs
  • Client registration with dry-run preview

Caveats

  • Not yet published to npm
  • The install writes the API key value into client configs
  • Requires a TypeSafe API key

Getting started

export TYPESAFE_API_KEY=sk-... then npm install -g github:arunav25/jev-mcp && jev-mcp install

At a glance

Type
MCP server
Status
beta
Language
JavaScript
Runtime
Node MCP server and CLI
Has code
yes

Use cases and integrations

How curated lists describe it

  • Connect JEV to MCP clients and compare its judgments against general-purpose LLMs using shared datasets and measurable accuracy.

    hellogumbo/awesome-jev

Frequently asked questions

What does the evaluate tool return?

Typed answers with probabilities for each question you pass against the given state.

How does it compare Jev with LLMs?

The jev-eval harness asks both the same questions on a dataset and reports Brier score, calibration and McNemar's test.

Which clients are registered automatically?

Claude Code, Claude Desktop and Codex, skipping any that are not installed.

Related projects

  • jev-mode (tiffygk)

    Claude Code and Codex skills plus a study course for building with Jev, including Jevaluate, which rates how well projects use Jev.

    Projectbeta
  • jevriel

    Skill and plugin for Codex, Claude Code and other agents that helps them use Jev for bounded judgments and measure the effect.

    Agent skillbetaJavaScript
  • mcp-server-jev

    Community MCP server giving Codex, Claude and other clients two tools for typed Jev decisions: evaluate and classify.

    MCP serverbetaTypeScript
  • Jev DSH Decision Engine

    Decision plugin for agent harnesses where Jev recommends tools, skills or owners and scores output, for DeepSeek Harness and iPolloWork.

    Agent skillbetaTypeScript
  • jev-mcp

    Eval-first MCP server for Jev: prototype questions, map them over many items, and measure accuracy and thresholds on labelled data.

    MCP serverbetaTypeScript
  • decisions-judge-mcp★ 5

    MCP server exposing a judge tool for typed noul, choice and score questions over JSON state, backed by Jev.

    MCP serverbetaTypeScript