Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

Jev-Reranking

Compares Jev's zero-shot reranking with specialist rerankers on MS MARCO passage and document benchmarks and a TREC WSJ transfer test.

Projectresearch
View on GitHub ↗

Overview

JEV Reranking Comparisons asks whether Jev's zero-shot general ability can match established specialist rerankers, measuring ranking quality, time and cost on fixed candidate sets. It uses MS MARCO passage and document reranking from TREC Deep Learning 2019, with TREC-1 WSJ as a transfer test, and includes an interactive simulator that replays the fixed WSJ BM25 ranking and completed Jev scores.

On the MS MARCO passage task (41,042 supplied query-passage pairs, 43 judged queries), the README reports Jev at nDCG@10 of about 0.683 versus 0.718 for a local monoBERT implementation, but the highest MAP (0.475) among the displayed methods, at an estimated API cost of about $0.76 for the run. Published TREC runs are listed as historical references, not reproductions. On WSJ, reranking the top 100 BM25 results raised nDCG@10 from 0.464 to 0.663 and full MAP from 0.252 to 0.306. The README is careful about metric scopes and which rows were run locally.

What the decision model decides

TypeQuestionAnswers
ScoreHow relevant is this passage or document to the query?(inferred)—

How it works

  1. Take fixed candidate lists from the benchmark
  2. Score each candidate with Jev
  3. Rerank and evaluate with trec_eval
  4. Compare with monoBERT and published TREC runs
  5. Replay results in the simulator

Strengths

  • Uses standard TREC benchmarks and metrics
  • Separates local runs from published references
  • Reports cost and timing

Caveats

  • Jev trailed monoBERT on nDCG@10 for passages
  • Per-query time was high in these runs
  • Candidate identity of published runs not audited

At a glance

Type
Project
Status
research
Language
Python
Runtime
Experiment scripts plus web simulator
Has code
yes

Use cases and integrations

How curated lists describe it

  • Search engine experimentation on the TREC collections. Currently focused on zero-shot reranking implementations with typesafe.ai's JEV model.

    hellogumbo/awesome-jev

Frequently asked questions

Does Jev beat monoBERT?

Not on nDCG@10 for MS MARCO passages (about 0.683 vs 0.718), though it had the highest MAP shown.

Does it help on older news text?

On TREC WSJ, reranking the top 100 raised nDCG@10 from 0.464 to 0.663.

Are the published rows reproduced?

No. They are historical references from the TREC DL 2019 overview.

Related projects

  • jev-rerank★ 17

    Jev as a calibrated reranker: query plus up to 30 documents in one call, one relevance Noul per document, benchmarked on BEIR.

    ProjectbetaPython
  • Vector Graph RAG

    Milvus-based multi-hop graph RAG library where two-stage Jev Noul scoring selects relations and reranks passages.

    LibrarybetaPython
  • Jev RAG

    Local-first knowledge search with BM25 shortlisting and Jev Noul reranking by default, seven selectable pipelines and cited answers.

    AppbetaPython
  • JevRanker

    Open Jev-style 0.6B decision model plugged into BlitzRank and Reranker-Guided Search, scoring k candidates per match in one pass.

    ServiceresearchPython
  • jevector

    Profile matching that encodes preference criteria as Clef or Jev decision vectors, retrieves with HNSW and checks rules both ways.

    BenchmarkresearchPython
  • Jev vs embeddings for page selection

    Sayora write-up comparing Jev with production embeddings for picking the right website page to answer caller questions.

    Articleproduction