jev-tool-search
Benchmark of BM25, embeddings, rerankers and Jev for picking the right MCP tool among 525, plus an experimental Jev search engine.
Overview
jev-tool-search benchmarks how to pick the right tool for an LLM agent, comparing BM25, Voyage embeddings, rerankers and Jev, and includes an experimental Jev search engine described as an Orama for Jev. It backs two articles on the author's blog. The main dataset is LiveMCPBench: 525 real MCP tools from 69 servers with 94 human-annotated tasks, where the metric is whether the first result is a tool the task needs.
In the reported runs, BM25 on the agent's query got the right tool first 32% of the time at no cost, Voyage embeddings 46% at $0.01 per 1,000 searches, a FastMCP-style two-stage Jev search 56% at $1.03, Voyage rerank-2.5 over the whole catalog 60% at $2.23, and embeddings top 100 followed by the experimental Jev engine 59% with 92% in the top five at $0.26. The author cautions that with 94 tasks, differences under about 10 points are within noise. The repository also includes 1,000- and 2,000-tool runs, a no-tool-fits run, a Claude end-to-end run and vitest tests.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this tool what the task needs?(inferred) | — |
| Choice | Which shortlisted tool best fits the task?(inferred) | — |
How it works
- Load tool catalogs and annotated tasks
- Retrieve with BM25, embeddings or a reranker
- Optionally rerank a shortlist with Jev
- Measure right-tool-first rate, top-5 and cost
Strengths
- Head-to-head comparison on real MCP tools
- Cost per 1,000 searches reported for each method
- Hybrid embeddings plus Jev was competitive at lower cost than a full reranker
Caveats
- 94 tasks; small differences are noise
- Results from different runs are not directly comparable
- Experimental engine defaults may not transfer between catalogs
Getting started
pnpm install (Node 22+), then run the benchmark scripts.At a glance
- Type
- MCP server
- Status
- research
- Language
- TypeScript
- Runtime
- Node
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Tool search for LLM agents: BM25 vs embeddings vs rerankers vs Jev on 525 real MCP tools, plus an experimental Jev search engine
logicrw/awesome-jev-projects
Frequently asked questions
Which method found the right tool most often?
Voyage rerank-2.5 at 60% and the embeddings-plus-Jev engine at 59%, within noise of each other per the author.
How does BM25 compare?
It got the right tool first 32% of the time, the lowest in the table but free.
What is the experimental engine?
A Jev-based search library prototype, described as an Orama for Jev.
Related projects
Independent benchmark asking whether Jev can replace an LLM reranker, on natural-language search over 6,245 YC companies, with a live demo.
BenchmarkresearchTypeScriptAgent harness with 100 mocked tools measuring steps to finish tasks when the LLM picks tools versus when Jev picks them.
ProjectresearchTypeScriptExperiments testing Jev ideas against baselines, starting with Jev as a top-10 tool router for MCP agents on three benchmarks.
ProjectresearchPythonReproducible experiments and reports on what Jev can and cannot do, from skill routing to GPQA and housing prices.
ProjectresearchFourteen-dataset reranking study of Jev vs Cohere Rerank 4, zerank-2 and chat models, with raw responses and bootstrap intervals.
BenchmarkresearchPythonGraded relevance study over 9,831 labelled pairs: Jev reranking alone does not beat embeddings, but fused with them it is best.
BenchmarkresearchPython