Jev search rerank eval
Graded relevance study over 9,831 labelled pairs: Jev reranking alone does not beat embeddings, but fused with them it is best.
Overview
jev-search-rerank-eval asks whether a TypeSafe Jev rerank beats embedding search, using a catalog of 33,047 agent skills, MCP servers and coding-agent tools with 164 real Chinese, English and mixed queries and 9,831 labelled query-skill pairs. Systems compared include the shipped ash keyword ranker, BM25, dense retrieval with bge-m3 and text-embedding-3-small, a Jev score rerank of each system's top 30 in one call per query, and reciprocal-rank-fusion hybrids, measured by NDCG@10 with paired bootstrap intervals.
The findings: Jev as a standalone reranker of bge-m3's top 30 did not beat bge-m3 (+0.012, interval crossing zero; -0.028 under labels from an unrelated model only). Fusing bge-m3 with the Jev ranking was the best system (+0.090 NDCG@10, still +0.064 with LLM-only labels), reaching 0.864. The apparent Jev advantage under Jev's own labels is judge circularity, which the study measures. On a weak lexical candidate list, Jev was the better reranker. The author concludes Jev's value here is as a second signal rather than the ranker.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Score | How relevant is this skill to the query (graded)?(inferred) | 0 · 1 · 2 · 3 |
How it works
- Build labelled query-skill pairs with graded relevance
- Run lexical, dense and Jev rerank systems
- Combine with reciprocal-rank fusion
- Compute NDCG@10, MRR and P@3 with bootstrap intervals
- Check judge circularity with alternative labels
Strengths
- Measures and removes judge bias
- Large labelled set with mixed languages
- Clear, actionable conclusion about fusion
Caveats
- Single catalog domain
- Labels partly produced by models
- Pinned to one Jev version
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Evaluation scripts
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
9,831 labelled pairs: Jev rerank vs BM25 / bge-m3, with judge-circularity measured. Fusion wins; Jev alone does not beat embeddings
AnotiaWang/awesome-decision-modelsRetrieval evaluation system measuring Jev reranking against lexical, embedding, and fusion baselines across 9,831 query-document pairs.
logicrw/awesome-jev-projectsChinese/English retrieval evaluation comparing Jev reranking with lexical, embedding, and fusion baselines, including judge-circularity analysis.
cobanov/awesome-jevDoes a TypeSafe Jev rerank beat embedding search? Graded relevance eval (9,831 pairs, 164 zh/en queries) over the Agent Skills Hub catalog, with the judge-circularity bias measured.
hellogumbo/awesome-jev
Frequently asked questions
Does Jev reranking beat embeddings?
Not alone; fused with embeddings it gave the best results.
What is judge circularity?
Jev looking better when its own labels are used; the study measures and removes this effect.
When is Jev the better reranker?
On weak lexical candidate lists, where it beat bge-m3 reranking.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonAudit of 300 web pages for AI citability with nine Jev decisions each, checked against an LLM judge for agreement, cost and latency.
ProjectresearchPython