jev-decision-benchmarks
Independent evaluation of Jev on MetaTool, When2Call and BFCL V4 tool selection, abstention and tool-use decisions.
Overview
JEV Decision Benchmarks evaluates whether Jev selects the right tools, knows when to call a tool or abstain, and avoids choosing tool calls when none are available. It runs Jev 1.13 through a Choice adapter on three public benchmarks and places the results beside published baselines, including ChatGPT, Llama, Qwen, Vicuna and others from the original papers.
On MetaTool, the README reports Jev at 77.79 percent on similar-tool selection and 87.04 to 88.54 percent on abstention, ahead of the paper's ChatGPT numbers on those columns but behind on the multi-tool 'at most two' condition. On When2Call it reports 74.84 percent accuracy and 56.55 macro F1, alongside a 76.36 percent tool-hallucination rate that is higher than several small baselines. The README notes protocol differences, such as slightly different public set sizes versus the paper, and links full reports and CSVs. Documentation is in English and Chinese.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which tool (or none) fits this user request? | — |
| Choice | What should the next step be: answer directly, call a tool, ask for missing information, or decline? | answer directly · call a tool · ask for information · cannot complete |
How it works
- Convert each benchmark item into a Jev Choice question over candidate tools or next steps
- Run Jev 1.13 on the public sets
- Score with each benchmark's metrics (CSR, accuracy, macro F1, tool hallucination)
- Compare with baselines from the original papers
Strengths
- Tests Jev on established public agent benchmarks
- Reports both strengths and weaknesses, including tool hallucination
- Publishes CSVs and protocol notes
Caveats
- Baselines come from papers with different set sizes and answer extraction
- Single model version (1.13)
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Independent evaluation of Jev on MetaTool, When2Call, and BFCL V4 agent decision tasks, focusing on tool selection, abstention, and tool-use decisions. Includes comparison tables against Claude, Qwen, and GPT models.
cobanov/awesome-jev
Frequently asked questions
What is measured?
Tool selection, abstention and next-step decisions on MetaTool, When2Call and BFCL V4.
How did Jev do?
The README reports strong abstention and similar-tool scores on MetaTool, and 74.84 percent accuracy on When2Call with a high tool-hallucination rate.
Are comparisons exact?
No. Baselines come from papers that generate tool names and use slightly different set sizes.
Related projects
Agent harness with 100 mocked tools measuring steps to finish tasks when the LLM picks tools versus when Jev picks them.
ProjectresearchTypeScriptExperiments testing Jev ideas against baselines, starting with Jev as a top-10 tool router for MCP agents on three benchmarks.
ProjectresearchPythonBenchmark of BM25, embeddings, rerankers and Jev for picking the right MCP tool among 525, plus an experimental Jev search engine.
MCP serverresearchTypeScriptReproducible experiments and reports on what Jev can and cannot do, from skill routing to GPQA and housing prices.
ProjectresearchOpen framework and preprint benchmarking typed decision models such as Jev, pplx-decider, Liquid D1 and Clef in six clusters.
BenchmarkresearchPythonPython agent runtime where Jev Choice decisions pick tools and arguments, and low-confidence picks go to an LLM for review.
ServicebetaPython