what-the-jev
Reproducible experiments and reports on what Jev can and cannot do, from skill routing to GPQA and housing prices.
Overview
what-the-jev is a framework and collection of reproducible experiments exploring Jev's capabilities and limits, with a live site in English and Chinese. It provides lightweight examples, complete experiments with analysis reports and a catalog of Jev-related models, tools and applications.
Selected findings from the README: choosing an agent skill (or none) from 126 real skills, Jev answered 488 of 500 prompts correctly (97.6 percent); on 198 GPQA Diamond science questions with four options it got 75.76 percent; and on Boston Housing it correctly picked which of two suburbs had higher prices 88.2 percent of the time, but estimating a single suburb's price range was much harder (9.7 to 31.4 percent). The mix of strengths and weaknesses makes it a useful reference.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which agent skill fits this request, or none? | — |
| Choice | Which answer is correct for this science question? | A · B · C · D |
| Choice | Which of two suburbs has higher housing prices? | suburb A · suburb B |
How it works
- Define an experiment task and dataset.
- Run Jev at scale with the framework.
- Score results reproducibly.
- Write an analysis report.
Strengths
- Reproducible experiments with reports.
- Shows both strengths and weaknesses.
Caveats
- Requires a TypeSafe key to reproduce.
- Some tasks are outside Jev's intended use.
At a glance
- Type
- Project
- Status
- research
- Runtime
- experiment framework
- Has code
- yes
Use cases and integrations
How curated lists describe it
What can Jev actually do? Reproducible experiments and research reports…
wh000wh000/awesome-jev-liveWhat can Jev actually do? Reproducible experiments and research reports exploring its capabilities and limits.
hellogumbo/awesome-jev
Frequently asked questions
How good is Jev at skill routing?
97.6 percent on 500 prompts over 126 skills, per the report.
Where does it struggle?
Estimating numeric ranges, such as a single suburb's price range.
Are the experiments reproducible?
Yes, that is the framework's goal.
Related projects
Agent harness with 100 mocked tools measuring steps to finish tasks when the LLM picks tools versus when Jev picks them.
ProjectresearchTypeScriptExperiments testing Jev ideas against baselines, starting with Jev as a top-10 tool router for MCP agents on three benchmarks.
ProjectresearchPythonBenchmark of BM25, embeddings, rerankers and Jev for picking the right MCP tool among 525, plus an experimental Jev search engine.
MCP serverresearchTypeScriptIndependent evaluation of Jev on MetaTool, When2Call and BFCL V4 tool selection, abstention and tool-use decisions.
BenchmarkresearchPythonOpen framework and preprint benchmarking typed decision models such as Jev, pplx-decider, Liquid D1 and Clef in six clusters.
BenchmarkresearchPythonPython agent runtime where Jev Choice decisions pick tools and arguments, and low-confidence picks go to an LLM for review.
ServicebetaPython