Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

yc-jev-bench

Independent benchmark asking whether Jev can replace an LLM reranker, on natural-language search over 6,245 YC companies, with a live demo.

Benchmarkresearch
View on GitHub ↗

Overview

yc-jev-bench tests whether Jev can replace an LLM reranker. It runs natural-language search over all 6,245 YC companies, comparing Jev with Claude Haiku and a BGE embedding reranker, with Claude Opus as the judge on pooled results. A live search page routes queries with Jev, retrieves with BM25 plus embeddings, and reranks with Jev; a report page renders the benchmark results.

The repository documents the experiment spec, design rules and Jev API notes, plus scripts for picking prompt formulations on a dev split, reranking, routing, judging and scoring. It also documents deployment details on Vercel and a fallback to keyword-only retrieval if the embedding model fails, noting the measured loss in that case. Specific headline rerank results are in the report rather than the README excerpt.

What the decision model decides

TypeQuestionAnswers
ScoreHow relevant is this company to the search query?—
ChoiceHow should this query be routed?—

How it works

  1. Retrieve candidates with BM25 and embeddings.
  2. Rerank with Jev, Haiku or BGE.
  3. Grade pooled results with Opus.
  4. Compute and render scores in the report.

Strengths

  • Independent, reproducible comparison.
  • Live demo on real data.
  • Documented fallbacks and their costs.

Caveats

  • Judge is an LLM (Opus).
  • Headline results are in the report, not the README.

Getting started

bun install, cp .env.example .env.local (set TYPESAFE_API_KEY), bun run dev

At a glance

Type
Benchmark
Status
research
Language
TypeScript
Runtime
Next.js on Vercel
Has code
yes

Use cases and integrations

How curated lists describe it

  • Can Jev replace an LLM reranker? A benchmark of TypeSafe's Jev vs Claude Haiku and BGE on natural-language search over all 6,245 YC companies, with a live search demo.

    hellogumbo/awesome-jev

Frequently asked questions

What is compared?

Jev, Claude Haiku and BGE as rerankers over YC company search.

Who judges relevance?

Claude Opus on pooled results.

Is there a live demo?

Yes, the app's search page uses Jev for routing and reranking.

Related projects

  • jev-tool-search

    Benchmark of BM25, embeddings, rerankers and Jev for picking the right MCP tool among 525, plus an experimental Jev search engine.

    MCP serverresearchTypeScript
  • Jev Rerank Bench

    Fourteen-dataset reranking study of Jev vs Cohere Rerank 4, zerank-2 and chat models, with raw responses and bootstrap intervals.

    BenchmarkresearchPython
  • Jev search rerank eval

    Graded relevance study over 9,831 labelled pairs: Jev reranking alone does not beat embeddings, but fused with them it is best.

    BenchmarkresearchPython
  • jev-rag-benchmark

    Reproducible Turkish RAG benchmark comparing Jev batch-Noul reranking with Cohere Rerank on quality, latency and cost.

    BenchmarkresearchPython
  • jev-rag-zh-benchmark

    Reproducible Chinese RAG reranking benchmark for Jev on T2Reranking with hard negatives, staged and cost-capped, logs published.

    BenchmarkresearchPython
  • jev-test

    Pre-registered benchmark where Jev makes every search and evidence decision for a 2B local model answering SimpleQA questions.

    BenchmarkresearchPython