Jev-Flywheel
Research demo where Jev answers plain questions about text and a small learned head adds new questions from human feedback.
Overview
Jev Flywheel asks a hosted model (Jev) a short list of plain questions about each piece of text, and a small readable model on top turns those answers into a verdict. When the verdict disagrees with a person, Jev is not retrained: the loop either reweights the answers or works out a new question worth asking and adds it. The goal is to find the unwritten parts of a scoring rubric.
On a dataset built so that labels secretly follow subject matter rather than sentiment, the loop identified the bias, stated it in plain English and gained about ten points of accuracy from 140 human labels. The README reports this happens about a quarter of the time, worth about 12 points when it does. It is candid that the labeller is a script and the bias was planted, so it demonstrates machinery and measurement rather than real human feedback. A recorded run replays offline with no keys.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Plain yes/no questions about each text, including newly proposed ones(inferred) | — |
How it works
- Ask Jev a list of plain questions per text
- Fit a small readable head that turns answers into a verdict
- On disagreement with labels, reweight or propose a new question
- Re-score and measure accuracy gain
Strengths
- Improves without fine-tuning the hosted model
- Explains what it learned in plain English
- Offline replay with make demo
Caveats
- Synthetic bias and scripted labeller
- Works only about a quarter of the time per the README
Getting started
`make demo` replays the recorded run offline.At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Research scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Jev plus a decision head that learns from feedback: can a loop name a bias we planted in our own dataset?
hellogumbo/awesome-jev
Frequently asked questions
What does Jev decide?
Answers to plain questions about each text, which a small model combines into a verdict.
Does it retrain Jev?
No, it changes weights or adds new questions instead.
Does it work on real feedback?
Not yet shown; the demo uses a planted bias and a scripted labeller.
Related projects
Claude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.
PluginbetaPythonOptimises the instructions and criterion descriptions of a Jev Choice from labelled data with EvoPrompt or GEPA, scored by Brier.
ProjectbetaPythonTool that recomputes Jev's confidence identities on outputs published in public repositories to spot answers Jev never actually produced.
ServiceresearchPythonToolkit that turns Laya into a task specialist: LLM-generated and verified training data, RLCD training, calibration and longer context.
ModelbetaPythonCLI and Claude Code skill that turns one typed decision into a fine-tuned, calibrated, evaluated Laya checkpoint you can serve.
ModelbetaPythonClaude Code and Codex skills powered by Jev, published with a calibration curve measured on 4,995 real agent decisions.
Project