ai-elo-ranker
Tournament engine that ranks texts with Swiss pairing, Elo and TrueSkill, using Jev or local models to judge each head-to-head duel.
Overview
AI Elo & TrueSkill Ranker ranks large sets of texts such as poems, startup pitches, lyrics, cold emails or ad hooks by running pairwise duels. A Swiss matchmaker pairs items with adjacent ratings instead of comparing all pairs, a pool of 16 parallel workers runs duels with position randomized to reduce bias, and ratings update with dynamic K-factor Elo and TrueSkill/OpenSkill. Results persist in SQLite and stream to a live web dashboard over WebSockets.
Each duel is judged by Jev or by local open-weight models (Qwen 2.5, Gemma 2, OpenJev, Laya via Ollama, vLLM or LM Studio), with ensemble and mock judges available. The README describes Jev returning a winner plus a rationale and critique, which does not match Jev's documented behaviour of returning only typed probabilities.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which of these two texts is better?(inferred) | A · B |
How it works
- Load a dataset of texts
- Swiss matchmaker pairs similarly rated items
- Judge each duel (Jev, local model or ensemble) with randomized order
- Update Elo/TrueSkill and stream to the dashboard
Strengths
- Avoids O(N^2) pairings
- Position-bias randomization
- Can run fully locally
Caveats
- README claims Jev returns rationales, which conflicts with Jev being a no-text model
- Marketing-heavy README; no licence stated
Getting started
Set TYPESAFE_API_KEY in .env (optional with local models or the mock judge) and run the tournament engine.At a glance
- Type
- Project
- Status
- prototype
- Language
- Python
- Runtime
- local service + web dashboard
- Has code
- yes
Use cases and integrations
How curated lists describe it
High-speed recursive AI Elo tournament engine powered by Jev and Swiss matchmaking.
hellogumbo/awesome-jev
Frequently asked questions
What does the judge decide?
Which of two texts wins a duel.
Can I avoid API costs?
Yes, local open-weight models or a mock judge can be used.
Why Swiss pairing?
It pairs similar ratings so far fewer comparisons are needed than round-robin.
Related projects
Claude Code plugin that logs your prompts locally and later has Jev score how well you prompt the coding agent over time.
PluginbetaTypeScriptChrome extension that grades readable page sections on ten rubric scores with Jev via Vercel AI Gateway and gives the page an A-E grade.
ProjectbetaTypeScriptCLI and agent skill that checks every line of a document against every rule with Jev and outputs flags for an AI agent to fix.
Agent skillbetaTypeScriptCLI that transcribes a video, has Jev score every sentence against rubric flags, and renders a 16:9 edit with a live meter.
AppbetaPythonFree web tool that checks any URL for 35 tells of AI-generated design and returns a slop score in about 250 ms.
DemoproductionAnalytics engine that rates published posts on pre-registered rubrics with Jev and tests ratings against real engagement.
ProjectbetaTypeScript