JEVals
Self-hosted playground running SystemOne requests through an emulator, Jev and an LLM judge to compare per-question fidelity.
Overview
JEVals (duberblock) is a self-contained playground for evaluating structured judgment requests. You edit and validate a SystemOneRequest payload, run it through a local emulator and real providers (a Jev baseline via TypeSafe, plus an independent answer and a judge from any OpenAI-compatible endpoint), and compare results question by question, with streaming progress.
Three screens organise the work. Principal shows per-question verdicts, Jev fidelity, question alignment, semantic divergence and independent verification. Investigation explains the why behind an execution with evidence, payloads and the full LLM exchange, with dedicated views for Noul (comparing probabilities, deltas and midpoint behaviour), Choice (whether systems agree) and Score (values, deltas and rubric alignment). Operation browses persisted history with provider timings, parallel execution and judge latency. It is built with a FastAPI API and a Next.js web app and requires Python 3.13+ with uv and Node.js 22+.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | User-supplied SystemOneRequest questions (Noul, Choice, Score) compared across emulator, Jev and an LLM judge | — |
How it works
- Edit and validate a SystemOneRequest
- Run through emulator, Jev and independent LLM providers
- Judge compares answers per question
- Inspect fidelity, divergence and timings; browse history
Strengths
- Per-question comparison across providers
- Full payloads and LLM exchanges for investigation
- Persisted execution history
Caveats
- Requires several provider credentials for full comparison
- Fidelity metrics depend on the judge model
Getting started
cp .env.example apps/api/.env, make install-api install-web, make dev, then open http://localhost:3000.At a glance
- Type
- App
- Status
- prototype
- Language
- Python
- Runtime
- FastAPI + Next.js
- Has code
- yes
Use cases and integrations
How curated lists describe it
Open-source playground for evaluating System One and JEV-compatible models with deterministic fidelity, semantic judging, and independent verification.
hellogumbo/awesome-jev
Frequently asked questions
What does it compare?
A local emulator, Jev and an independent LLM answer, with an LLM judge, for each question in a request.
What can I inspect?
Per-question verdicts, probability deltas, rubric alignment, payloads and timings.
What does it need to run?
Python 3.13+ with uv, Node.js 22+ and provider credentials in Settings.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonAudit of 300 web pages for AI citability with nine Jev decisions each, checked against an LLM judge for agreement, cost and latency.
ProjectresearchPython