decision-head-rlcd
Experiment showing a RLCD-trained LoRA decision head on Qwen3.5-4B generalises only where training data covers the task structure.
Overview
This repository tests where a decision model's generalisation comes from. It attaches a 4.9M-parameter LoRA to Qwen3.5-4B and trains it with RLCD (Gaussian noise on option-letter logits as the action, a leave-one-out baseline over eight samples) on 32,000 typed decisions in one pass. Answers come from one forward pass as a softmax over the declared option letters. Base model, algorithm and hyperparameters are held fixed across three seeds.
Evaluation is grouped by how held-out sets relate to the training data, plus four external suites. The README reports that generalisation tracks training-data coverage in both directions: on covered structures the RL model approaches Jev, while on uncovered structures such as long policies or different material it lands below the untrained base, consistently across seeds. The conclusion is that data, not the algorithm, is the variable, and narrow data is a liability.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which declared option letter answers this typed decision? | — |
How it works
- Attach a LoRA to Qwen3.5-4B
- Train with RLCD on 32,000 typed decisions
- Evaluate on held-out sets grouped by relation to training data
- Compare base, three seeds and Jev
Strengths
- Controlled experiment isolating data from algorithm
- Three seeds with consistent sign
- Released checkpoint and report
Caveats
- Single base model and training mix
- Some data from synthetic commercial-API items
Getting started
See the repository report and released seed-1 checkpoint.At a glance
- Type
- Model
- Status
- research
- Language
- Python
- Runtime
- PyTorch training
- Has code
- yes
- License
- Apache-2.0
Use cases and integrations
How curated lists describe it
Where does a decision model.
wh000wh000/awesome-jev-live
Frequently asked questions
What is RLCD?
Reinforcement learning for calibrated decisions; here, noise on option-letter logits with a leave-one-out baseline.
What is the main finding?
Generalisation follows training-data coverage; where data is absent, RL makes the model worse than the base.
Does it compare to Jev?
Yes, on covered tasks the RL model approaches Jev's scores; on long policies it falls well short.
Related projects
- cbjev★ 4
Self-hosted encoder model giving Jev-compatible choice, score and noul answers in one pass, pitched as a better-calibrated Laya successor.
ProjectresearchPython Joke decision model trained to answer wrongly with total confidence: 1.9% accuracy at 0.957 mean confidence, via a Jev-style API.
ProjectresearchPythonPre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPython11 MB CPU-only model that answers choice, noul and score questions with fitted calibration and no LLM at inference.
ProjectresearchPythonTiny pre-registered experiment testing whether a confidence-scoring reward (RLCD-like) yields confidences safe to automate on at 0.9.
ProjectresearchPythonResearch code for an image-and-text System One model that reads option-letter logits from a multimodal LLM, with training and calibration.
ProjectresearchPython