jev-eval-agent
Agent harness with 100 mocked tools measuring steps to finish tasks when the LLM picks tools versus when Jev picks them.
Overview
jev-eval-agent is a personal-assistant agent built on Vercel's eve framework with 100 mocked tools, served through OpenRouter. It asks one question: how many steps does the agent need to finish the same task when the LLM chooses tools itself versus when Jev chooses them? In llm-direct mode the LLM sees all 100 tools every step with reasoning on. In jev-classifier mode, Jev is called before every step with the conversation state as a Choice over 101 options, and only the chosen tool is exposed to the LLM, which just fills in arguments with minimal reasoning.
Everything else (instructions, catalogue, evals, UI) is identical between modes, controlled by one environment variable. A deterministic mock world covers calendar, contacts, flights and expenses. Six benchmark prompts with required tool families are evaluated, Jev's decisions are traced to JSONL, and a dashboard and slide deck present comparisons. Documentation is in English and Portuguese.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which of the 100 tools (or none) should the agent use for the next step? | — |
How it works
- Set AGENT_MODE to llm-direct or jev-classifier
- In Jev mode, ask Jev for the next tool before every step
- Expose only the chosen tool so the LLM fills arguments
- Run six eval prompts and compare steps and metrics in the dashboard
Strengths
- Clean A/B with one switch
- Shrinks tool context to one tool per step
- Traces every Jev decision
Caveats
- Mocked tools and six prompts
- Results not summarised in the excerpt
Getting started
See scripts/run-all.sh to run the evals in both modes and scripts/serve-ui.mjs for the dashboard.At a glance
- Type
- Project
- Status
- research
- Language
- TypeScript
- Runtime
- Vercel eve agent via OpenRouter
- Has code
- yes
Use cases and integrations
How curated lists describe it
Public eval harness for early Jev tests.
AnotiaWang/awesome-decision-modelsAgent tool evaluation harness comparing standard LLM tool selection against Jev routing across 100 mocked tools.
logicrw/awesome-jev-projectsCompares LLM tool selection with Jev routing in a personal-assistant harness containing 100 mocked tools.
cobanov/awesome-jevPersonal-assistant agent built on Vercel's eve with 100 mocked tools, measuring how many steps it takes when Jev picks the tool versus the LLM.
hellogumbo/awesome-jev
Frequently asked questions
What does Jev decide?
Which of 100 tools the agent should use at each step.
What is compared?
Steps to completion when the LLM picks tools versus when Jev does.
Are the tools real?
No, they are mocked against a deterministic world.
Related projects
Experiments testing Jev ideas against baselines, starting with Jev as a top-10 tool router for MCP agents on three benchmarks.
ProjectresearchPythonBenchmark of BM25, embeddings, rerankers and Jev for picking the right MCP tool among 525, plus an experimental Jev search engine.
MCP serverresearchTypeScriptReproducible experiments and reports on what Jev can and cannot do, from skill routing to GPQA and housing prices.
ProjectresearchIndependent evaluation of Jev on MetaTool, When2Call and BFCL V4 tool selection, abstention and tool-use decisions.
BenchmarkresearchPythonOpen framework and preprint benchmarking typed decision models such as Jev, pplx-decider, Liquid D1 and Clef in six clusters.
BenchmarkresearchPythonPython agent runtime where Jev Choice decisions pick tools and arguments, and low-confidence picks go to an LLM for review.
ServicebetaPython