rerank-bench-jev
Auditable reranking benchmark: Jev vs Qwen3-Reranker-0.6B on BEIR SciFact and NFCorpus for nDCG, cost per query and tail latency.
Overview
rerank-bench-jev, from the team at denser.ai, asks whether TypeSafe's Jev should be used as a production reranker instead of a dedicated cross-encoder. Both systems rerank a frozen BM25 top 100 in a single request at the same concurrency on two BEIR datasets, and every raw API response is committed so numbers can be recomputed without a key.
The conclusion is that neither ranks reliably better: Jev's nDCG@10 is 0.7699 versus 0.7481 on SciFact (significant, p=0.011, but not stable across repeat runs) and a tie on NFCorpus. Jev is about 1.8x faster at p50 and 2.4x at p95 (p95 under 0.8 s), while Qwen3-Reranker-0.6B is about 3.9x cheaper per query ($0.00045 versus $0.00178). The advice is to decide on latency or cost, not nDCG. The README also notes that packing all candidates into one request, following TypeSafe's rerank cookbook, is what makes Jev viable, and tests Noul relevance, task-specific rubric and 4-level Score variants.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this passage relevant to the query? | — |
| Score | How relevant is this passage (4 ordered levels)? | — |
How it works
- Retrieve top 100 passages with BM25
- Pack all 100 candidates into one Jev request with a relevance question
- Rank by P(relevant) or expected level
- Compare nDCG, cost and latency with a cross-encoder using bootstrap and Wilcoxon tests
Strengths
- Raw responses shipped for reproducibility
- Honest significance testing and repeat runs
- Clear latency vs cost trade-off
Caveats
- Only two BEIR datasets
- Quality difference not stable across runs
- Costs depend on list prices that may change
Getting started
python3 -m venv .venv && ./.venv/bin/pip install -r requirements.txt, then recompute results from the shipped run files.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Scripts
- Has code
- yes
- License
- Apache-2.0
Use cases and integrations
How curated lists describe it
Production reranker benchmark: TypeSafe Jev vs Qwen3-Reranker-0.6B on BEIR SciFact and NFCorpus — nDCG@10, cost/query and p50/p95 latency. Quality is indistinguishable; Jev is ~2x faster, Qwen ~4x cheaper. From the team at denser.ai.
hellogumbo/awesome-jev
Frequently asked questions
Is Jev a better reranker than Qwen3-Reranker?
Not reliably; quality was indistinguishable overall, Jev was faster and Qwen cheaper.
Can I check the numbers without an API key?
Yes, every raw response is in the repo.
How was Jev used?
As a Noul relevance question per passage, all 100 packed in one request.
Related projects
Benchmark of BM25, embeddings, rerankers and Jev for picking the right MCP tool among 525, plus an experimental Jev search engine.
MCP serverresearchTypeScriptFourteen-dataset reranking study of Jev vs Cohere Rerank 4, zerank-2 and chat models, with raw responses and bootstrap intervals.
BenchmarkresearchPythonGraded relevance study over 9,831 labelled pairs: Jev reranking alone does not beat embeddings, but fused with them it is best.
BenchmarkresearchPythonReproducible Turkish RAG benchmark comparing Jev batch-Noul reranking with Cohere Rerank on quality, latency and cost.
BenchmarkresearchPythonReproducible Chinese RAG reranking benchmark for Jev on T2Reranking with hard negatives, staged and cost-capped, logs published.
BenchmarkresearchPythonIndependent benchmark asking whether Jev can replace an LLM reranker, on natural-language search over 6,245 YC companies, with a live demo.
BenchmarkresearchTypeScript