generative-ai-jev-rlcd-calibrated-decisions
Tiny pre-registered experiment testing whether a confidence-scoring reward (RLCD-like) yields confidences safe to automate on at 0.9.
Overview
This repository is a small, dependency-free check of one claim from a Generative AI article about Jev and RLCD, TypeSafe's training method for calibrated decisions. Since TypeSafe has not published the objective, it trains one tiny policy on a simulated refund desk several ways, changing only the reward: correct-only (standing in for RLVR), a reward that also likes assertive confidence, a Brier-penalised scored reward standing in for RLCD, the scored reward with language added to the input, plus ideal and flat reference arms. It does not call Jev or any API.
The question was registered before training: does scoring stated confidence give confidences you can auto-approve on at 0.9, including on a queue with a different language mix? Over five seeds, all arms give the same answers but different confidences. The scored arm stayed under 10% error on auto-approved tickets on the training mix (5.7%) but not on a 50% Portuguese queue (14.2%); adding language to the input fixed the mixed queue (6.6%) but not a Portuguese-only one.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Should this refund ticket be approved (with stated confidence)?(inferred) | yes · no |
How it works
- Define a simulated world with known true probabilities
- Train the same policy under different rewards
- Auto-approve at confidence ≥ 0.9
- Measure accuracy, ECE and auto-approval error under distribution shift
Strengths
- Pre-registered question and stop/continue thresholds
- Exact metrics from a known world
- Clear demonstration of calibration under shift
Caveats
- Toy simulation, not Jev itself
- RLCD objective approximated with a Brier penalty
Getting started
Run the scripts in the repository; no API keys or network needed.At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- scripts
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
No-key trial: three rewards on one tiny policy, and whether its 90% confidence…
wh000wh000/awesome-jev-live
Frequently asked questions
Does it call Jev?
No, it trains a tiny policy offline to test the RLCD idea.
What was found?
Confidence-scored rewards improved auto-approval safety on the training mix but not under a strong language shift.
Is RLCD's real objective used?
No, it is unpublished; a Brier penalty stands in.
Related projects
- cbjev★ 4
Self-hosted encoder model giving Jev-compatible choice, score and noul answers in one pass, pitched as a better-calibrated Laya successor.
ProjectresearchPython Joke decision model trained to answer wrongly with total confidence: 1.9% accuracy at 0.957 mean confidence, via a Jev-style API.
ProjectresearchPythonPre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPython11 MB CPU-only model that answers choice, noul and score questions with fitted calibration and no LLM at inference.
ProjectresearchPythonResearch code for an image-and-text System One model that reads option-letter logits from a multimodal LLM, with training and calibration.
ProjectresearchPythonExact-target audit finding Jev's Choice probabilities are a step function on stated base rates while per-option Noul tracks them closely.
ProjectresearchPython