Jev Rerank Bench
Fourteen-dataset reranking study of Jev vs Cohere Rerank 4, zerank-2 and chat models, with raw responses and bootstrap intervals.
Overview
jev-rerank-bench asks whether a decision model can do a reranker's job. The same 30 BM25 candidates (cut to 2,000 characters) go to several Jev setups, Cohere Rerank 4 Pro and Fast, ZeroEntropy zerank-2, a DeepSeek chat baseline, an open Jev-like 9B model, Qwen recipes and eight more open models. The main comparison covers eight English datasets, with five BRIGHT subsets, NevIR negation pairs and MIRACL French reported separately. Every raw API response is saved, and gaps have paired bootstrap ranges.
On the eight English datasets, Jev's four-level rubric averaged 0.692 nDCG@10 and Cohere Pro 0.691 with no clear winner, though weighting each question equally put Cohere ahead (0.756 versus 0.738). Jev was faster (about 0.4 s versus 0.8 s) and cheaper ($0.45 versus $2.51 per 1,000 queries), and did better on negation than the rerankers. Qwen3-Reranker-4B came within noise of Jev's one-at-a-time setup and beat every Jev setup on French. Follow-ups test passage-order sensitivity and keyword-stuffing injection.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Score | How useful is this passage for the query (4-level rubric)? | — |
| Yes / no | Is this passage relevant? | yes · no |
| Choice | Which passage best answers the query, or none? | — |
How it works
- Retrieve 30 BM25 candidates per query
- Score them with each reranker and Jev variant
- Compute nDCG@10, top-pick accuracy, negation and no-relevant AUROC
- Report bootstrap intervals and save raw responses
Strengths
- Broad dataset coverage with raw evidence
- Tests negation and empty-result detection
- Honest reporting of ties and losses
Caveats
- Results are study-specific
- Passages truncated to 2,000 characters
- Model versions as of September 2026
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark scripts
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Reranking comparison with raw provider responses, scoring code, uncertainty intervals, and documented limits.
AnotiaWang/awesome-decision-modelsCompares Jev, dedicated rerankers and chat models on the same retrieved passages.
logicrw/awesome-jev-projectsFourteen-dataset reranking study with saved raw responses, paired bootstrap intervals, order-sensitivity checks, and no-relevant-document tests; comparisons remain study-specific.
cobanov/awesome-jevCan a decision model beat dedicated rerankers? TypeSafe Jev vs Cohere Rerank 4 vs ZeroEntropy zerank-2 vs a chat-model baseline: 14 datasets, every raw API response, bootstrap ranges on every gap.
hellogumbo/awesome-jev
Frequently asked questions
Does Jev beat Cohere Rerank 4?
On dataset-averaged nDCG@10 it tied; per-question weighting favoured Cohere.
Where does Jev do better?
On negation pairs, and on latency and cost.
Are open models competitive?
Qwen3-Reranker-4B came within noise of one Jev setup and beat all Jev setups on French.
Related projects
Benchmark of BM25, embeddings, rerankers and Jev for picking the right MCP tool among 525, plus an experimental Jev search engine.
MCP serverresearchTypeScriptGraded relevance study over 9,831 labelled pairs: Jev reranking alone does not beat embeddings, but fused with them it is best.
BenchmarkresearchPythonReproducible Turkish RAG benchmark comparing Jev batch-Noul reranking with Cohere Rerank on quality, latency and cost.
BenchmarkresearchPythonReproducible Chinese RAG reranking benchmark for Jev on T2Reranking with hard negatives, staged and cost-capped, logs published.
BenchmarkresearchPythonAuditable reranking benchmark: Jev vs Qwen3-Reranker-0.6B on BEIR SciFact and NFCorpus for nDCG, cost per query and tail latency.
BenchmarkresearchPythonIndependent benchmark asking whether Jev can replace an LLM reranker, on natural-language search over 6,245 YC companies, with a live demo.
BenchmarkresearchTypeScript