jev-toolspace
Research code evaluating Jev as an independent per-tool relevance judge for building task-level tool spaces on TGB, BFCL and GTA.
Overview
jev-toolspace evaluates Jev (jev-1.13.0) as a relevance judge for deciding which tools an agent should keep. Jev answers one yes/no question per tool about a shared state and returns an independent probability for each, so one API call scores a whole menu and tools do not compete with each other. Three settings are compared: per request (the request and tool descriptions, run online), per task (averaging the judgment over a task's build requests into one tool space) and per task from traces (judging the executor's build traces of tool outputs and answers).
With a Qwen2.5-7B executor, the README reports that the trace-based setting reached 80.7 macro accuracy on TGB versus 69.5 for keeping all tools and 67.6 for the best baseline, while on BFCL multi-turn the per-task setting scored 17.3 against 11.3 for keeping all and 13.8 for Tool2Vec. Results on the GTA subsets are small-sample and mixed. The client caches calls, throttles to 600 requests per minute, retries on errors and has an offline mock mode. A paper write-up is included.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this tool relevant to the request, task or build trace? | — |
How it works
- Build state from a request, a task's build requests or execution traces
- Ask one Noul per tool in a single call
- Aggregate scores and keep the top-K tools per task
- Evaluate the executor with the reduced tool space
Strengths
- Independent per-tool probabilities in one call
- Trace-based tool spaces beat keep-all on TGB per the README
- Caching, throttling and mock mode for reproducibility
Caveats
- Mixed results across benchmarks and settings
- Small GTA subsets
- Raw API caches not included
At a glance
- Type
- Service
- Status
- research
- Language
- Python
- Runtime
- CLI research harness
- Has code
- yes
Use cases and integrations
How curated lists describe it
Jev answers yes/no (\`noul\`) questions about a shared state and returns an independent probability for each one. One API call scores every tool in a menu with one question per tool, and a tool's score does not compete with the others.
logicrw/awesome-jev-projects
Frequently asked questions
Why Noul per tool instead of a Choice?
Each tool gets an independent probability, so several can be relevant and they do not compete.
What was the best result?
Trace-based tool spaces reached 80.7 macro accuracy on TGB versus 69.5 for keeping all tools.
Did it help everywhere?
No, results vary by benchmark and setting, as the README's table shows.
Related projects
Python agent runtime where Jev Choice decisions pick tools and arguments, and low-confidence picks go to an LLM for review.
ServicebetaPythonToolkit where Jev condenses large MCP responses for Claude Code and maps Codex requests to reviewed, fixed repository operations.
ProjectbetaTypeScriptClaude Code plugin with three Jev tools: verbatim context compaction, per-edit model routing and per-prompt skill selection.
ProjectbetaPythonPi extension that strips the inlined skill list and asks Jev which single skill, if any, to load per turn.
Agent skillbetaTypeScriptPairs Jev with the get-fable lifecycle harness to give coding agents fast tool and skill routing and verbatim-preserving compaction.
ProjectprototypeTypeScriptGives an agent one tool that uses Jev to search a nested capability tree and return only the documented commands a task needs.
ProjectprototypeTypeScript