Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-rag-zh-benchmark

Reproducible Chinese RAG reranking benchmark for Jev on T2Reranking with hard negatives, staged and cost-capped, logs published.

BenchmarkresearchMIT
View on GitHub ↗

Overview

Jev RAG Chinese Benchmark measures Jev as a reranker on C-MTEB/T2Reranking, a Chinese dataset with hard negatives. It follows a frozen protocol: a pinned model version, staged spending with a dry run, a first stage capped at a few cents and a runtime cost breaker, single runs per variant, and full per-call JSONL logs published on Hugging Face and ModelScope.

The first stage (100 queries, each with one positive and nine hard negatives) reports Jev Score reaching nDCG@5 of 0.546 and nDCG@10 of 0.602, against 0.413 and 0.513 for jieba BM25, with bootstrap confidence intervals on the paired difference excluding zero and Recall@1 2.4 times higher. Jev Noul nearly tied Score, suggesting a binary question is enough for pure ranking. The whole stage cost $0.071 for 2,000 calls, with p50 latency of 0.47 s. Limitations listed include the small sample, single run, binary relevance and one provider channel. It links to a prompt-pattern library and pipeline skill by the same author.

What the decision model decides

TypeQuestionAnswers
ScoreHow relevant is this Chinese passage to the query?—
Yes / noIs this passage relevant to the query?yes · no

How it works

  1. Prepare a seeded sample of queries with candidate pools
  2. Run Score and Noul variants with a cost cap
  3. Compute nDCG, MRR and recall with bootstrap intervals
  4. Compare with BM25 and publish logs

Strengths

  • Pinned model and published per-call logs
  • Strict cost controls
  • Reports limitations next to results

Caveats

  • Only 100 queries in the reported stage
  • Single run per variant
  • Binary relevance labels

Getting started

uv run python scripts/prepare_data.py --n 100 --out s1, then uv run python scripts/run_bench.py --sample data/s1.jsonl --variant score-zh --max-cost-usd 0.15

At a glance

Type
Benchmark
Status
research
Language
Python
Runtime
Benchmark scripts via OpenRouter
Has code
yes
License
MIT

Use cases and integrations

How curated lists describe it

Frequently asked questions

Does Jev beat BM25 on Chinese?

Yes, by about 13 nDCG@5 points on hard negatives in this 100-query stage.

Score or Noul?

They nearly tied; a binary question was enough for ranking.

How much did it cost?

$0.071 for the whole first stage.

Related projects

  • jev-tool-search

    Benchmark of BM25, embeddings, rerankers and Jev for picking the right MCP tool among 525, plus an experimental Jev search engine.

    MCP serverresearchTypeScript
  • Jev Rerank Bench

    Fourteen-dataset reranking study of Jev vs Cohere Rerank 4, zerank-2 and chat models, with raw responses and bootstrap intervals.

    BenchmarkresearchPython
  • Jev search rerank eval

    Graded relevance study over 9,831 labelled pairs: Jev reranking alone does not beat embeddings, but fused with them it is best.

    BenchmarkresearchPython
  • jev-rag-benchmark

    Reproducible Turkish RAG benchmark comparing Jev batch-Noul reranking with Cohere Rerank on quality, latency and cost.

    BenchmarkresearchPython
  • rerank-bench-jev

    Auditable reranking benchmark: Jev vs Qwen3-Reranker-0.6B on BEIR SciFact and NFCorpus for nDCG, cost per query and tail latency.

    BenchmarkresearchPython
  • yc-jev-bench

    Independent benchmark asking whether Jev can replace an LLM reranker, on natural-language search over 6,245 YC companies, with a live demo.

    BenchmarkresearchTypeScript