Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-experiments

Experiments testing Jev ideas against baselines, starting with Jev as a top-10 tool router for MCP agents on three benchmarks.

ProjectresearchMIT
View on GitHub ↗

Overview

jev-experiments tests one idea per folder against a baseline with reproducible benchmarks, calling Jev through a Decisions API on OpenRouter or Vercel AI Gateway. The first experiment uses Jev as a tool router for MCP agents: instead of loading all of roughly 250 tools, Jev picks the 5 to 10 an agent needs.

On MCP-Atlas, MCP-Bench and MCP-Universe, compared with OpenAI's decision model, the README reports Jev's top-10 contained the right tool 80 percent of the time on MCP-Atlas versus 64 percent, with better calibration (ECE 0.11 vs 0.33), at about half the cost and about 96 percent less tool context. Benchmarks are cloned at setup rather than redistributed and keep their own licences.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich tools does the agent need for this task?—

How it works

  1. Clone benchmark data at setup
  2. Ask Jev to rank tools for each task
  3. Keep the top 5 to 10 tools in the agent context
  4. Compare hit rate, calibration and cost with OpenAI's decision model

Strengths

  • Large reduction in tool context
  • Better calibration than the compared decision model per the README
  • One-command benchmarks

Caveats

  • Only one experiment published so far
  • Results depend on benchmark tool descriptions

Getting started

Each experiment folder has its own README with setup, keys and a one-command benchmark.

At a glance

Type
Project
Status
research
Language
Python
Runtime
Benchmark scripts
Has code
yes
License
MIT

Use cases and integrations

How curated lists describe it

Frequently asked questions

What does Jev decide?

Which few tools out of about 250 an MCP agent needs for a task.

How well did it do?

The README reports the right tool in Jev's top-10 80 percent of the time on MCP-Atlas versus 64 percent for OpenAI's decision model.

Which API is used?

A Decisions API on OpenRouter or Vercel AI Gateway.

Related projects

  • toolJev

    Code Mode for MCP where retrieval shortlists tools and Jev picks one or says none fits, benchmarked on several tool-use suites.

    BenchmarkresearchPython
  • jev-eval-agent

    Agent harness with 100 mocked tools measuring steps to finish tasks when the LLM picks tools versus when Jev picks them.

    ProjectresearchTypeScript
  • jev-tool-search

    Benchmark of BM25, embeddings, rerankers and Jev for picking the right MCP tool among 525, plus an experimental Jev search engine.

    MCP serverresearchTypeScript
  • what-the-jev

    Reproducible experiments and reports on what Jev can and cannot do, from skill routing to GPQA and housing prices.

    Projectresearch
  • playwright-cli-jev-bakeoff

    Reproducible bake-off of Playwright CLI alone, CLI plus Jev, and Playwright MCP on fuzzy browser goals, with honest results.

    MCP serverresearchPython
  • jev playground

    Weekly go-to-market workflows built on Jev, such as lead triage and deal risk, each tested against Claude.

    Benchmarkprototype