jevarena (chenmingtang830)
Open-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
Overview
JevArena (chenmingtang830) is an open-source arena for testing Jev against another judge. You enter a question and its possible answers (or start from an example), connect OpenRouter and pick any supported model to face Jev, compare the two hidden decisions, and then vote to reveal identities, latency, cost and available confidence data. Every live battle includes Jev. A vote records preference rather than verified correctness, and community-submitted questions are AI-screened and remain unverified until independently reviewed; you can opt out of contributing.
The site does not save your API key: it lives in the current tab's memory and OpenRouter calls go directly from the browser. Alongside the website, a Python package called JevJudge-Bench is the reproducible research harness: it transforms pinned JudgeBench, RM-Bench and RewardBench 2 data into one diagnostic pairwise protocol, with mock runs clearly labelled and paid runs requiring explicit cost limits and no hidden retries; the author notes its scores are not the upstream leaderboards. The web app has type checks, unit tests and end-to-end tests with synthetic responses.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | User-defined question: which of these possible answers is correct? (asked to Jev and an opponent model) | — |
How it works
- User enters a question and possible answers
- Jev and a chosen OpenRouter model answer blind
- User votes, then identities, latency and cost are revealed
- Optionally contribute to community questions; research harness runs pairwise benchmarks
Strengths
- Key never stored; browser calls OpenRouter directly
- Blind comparison reduces bias
- Reproducible research harness with explicit cost limits
- Tests including end-to-end
Caveats
- Votes reflect preference, not verified correctness
- Community questions unverified
- JevJudge-Bench scores are not upstream leaderboards
Getting started
cd web && npm ci && npm run dev; for the harness: uv sync && uv run jevjudge fetch.At a glance
- Type
- Project
- Status
- beta
- Language
- TypeScript
- Runtime
- Next.js web app + Python harness
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Open-source BYOK arena for Jev and other AI judges. Find failures, compare quality, cost, and latency.
hellogumbo/awesome-jev
Frequently asked questions
Does the site store my key?
No, it stays in the tab's memory and disappears on refresh.
Do votes mean an answer is correct?
No, they record preference, not verified correctness.
What is JevJudge-Bench?
A pairwise research harness built from JudgeBench, RM-Bench and RewardBench 2 data.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonNovelty demo that turns Jev answers to 22 Boolean gate questions into lookup tables for a simulated RISC-V CPU running compiled C.
ProjectprototypeBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonAudit of 300 web pages for AI citability with nine Jev decisions each, checked against an LLM judge for agreement, cost and latency.
ProjectresearchPython