jev-rag-benchmark
Reproducible Turkish RAG benchmark comparing Jev batch-Noul reranking with Cohere Rerank on quality, latency and cost.
Overview
jev-rag-benchmark runs reproducible, vendor-neutral experiments on whether TypeSafe Jev improves a small RAG system, making no assumption that Jev wins. The locked full benchmark uses 1,044 Turkish XQuAD questions over 240 passages, with all model and embedding calls through OpenRouter. First-stage retrieval fuses BM25 with BGE-M3; every reranker sees the same 20 candidates and the answer model gets the top five.
The README first reports retrieval ceilings (hybrid top-20 contains the gold passage for 99.5% of questions). Then, with generation disabled, Jev 1.13 batch-Noul reranking and Cohere Rerank 3.5 both reached 99.425% Recall@5, versus 97.6% for the raw hybrid order. Jev cost 60.6% less ($0.41 versus $1.04 total), while Cohere led by 0.529 nDCG points and 65.9 ms at the median. A frozen-context section compares answer generation with two models on the same Jev top-5 context. Raw rows and reports are published.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this candidate passage relevant to the question? | yes · no |
How it works
- Retrieve 20 candidates with BM25 plus BGE-M3 fusion
- Rerank with no reranker, Jev batch Noul or Cohere Rerank
- Measure Recall@5, nDCG@10, MRR@10, latency and cost
- Generate answers from frozen top-5 contexts and score them
Strengths
- Same candidates for every reranker
- Reports cost and latency alongside quality
- Raw results published
Caveats
- Single Turkish dataset with small corpus
- One run per configuration
- All calls through one gateway
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark scripts via OpenRouter
- Has code
- yes
Use cases and integrations
How curated lists describe it
Reproducible benchmark for measuring Jev reranking quality, latency, and cost in RAG.
hellogumbo/awesome-jevReproducible experiments on whether reranking with Jev improves a small RAG system, on a locked Turkish dataset, with quality, latency and cost reported together.
MrJev/awesome-jev
Frequently asked questions
Does Jev beat Cohere Rerank?
It tied on Recall@5 and cost 60.6% less, while Cohere was slightly better on nDCG and latency.
Does reranking help at all here?
Yes. Jev added 19 top-5 recoveries over the raw hybrid order.
Which dataset is used?
1,044 Turkish XQuAD questions over a 240-passage corpus.
Related projects
Benchmark of BM25, embeddings, rerankers and Jev for picking the right MCP tool among 525, plus an experimental Jev search engine.
MCP serverresearchTypeScriptFourteen-dataset reranking study of Jev vs Cohere Rerank 4, zerank-2 and chat models, with raw responses and bootstrap intervals.
BenchmarkresearchPythonGraded relevance study over 9,831 labelled pairs: Jev reranking alone does not beat embeddings, but fused with them it is best.
BenchmarkresearchPythonReproducible Chinese RAG reranking benchmark for Jev on T2Reranking with hard negatives, staged and cost-capped, logs published.
BenchmarkresearchPythonAuditable reranking benchmark: Jev vs Qwen3-Reranker-0.6B on BEIR SciFact and NFCorpus for nDCG, cost per query and tail latency.
BenchmarkresearchPythonIndependent benchmark asking whether Jev can replace an LLM reranker, on natural-language search over 6,245 YC companies, with a live demo.
BenchmarkresearchTypeScript