NAACL27_Jev
Reproduction and stress test of a paper asking whether Jev-style probabilities stay calibrated after routing on selected examples.
Overview
NAACL27_Jev reproduces and stress-tests the study Can We Trust Jev After Routing?. The concern is routing-conditional calibration: when a system routes only the top-scoring examples onward, are the decision model's probabilities still calibrated on that routed subset? The pipeline prepares Civil Comments and GoEmotions, obtains typed decision probabilities (from OpenJev and OneJev predictions), runs a score-conditioned audit with paired bootstrap uncertainty and BH-FDR correction, and adds random and oracle controls.
Robustness analyses include absolute calibration, bin-count sensitivity, spline calibration curves, within-bin permutation tests, soft labels, prompt sensitivity and a second model. A CPU-only strengthen command adds score balancing, null binning diagnostics and held-out recalibration across all 12 directed label pairs without new model runs. Every underspecified choice is recorded in a config whose hash is written to each run manifest. It is for researchers checking calibration claims in routed pipelines.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this comment toxic (Civil Comments, binarised at 0.5)?(inferred) | yes · no |
| Choice | Which emotion label applies (GoEmotions)?(inferred) | — |
How it works
- Prepare Civil Comments and GoEmotions.
- Collect typed decision probabilities from the models.
- Route the top ceil(r*N) rows by score and audit calibration per score bin.
- Bootstrap, apply BH-FDR and generate tables and figures.
Strengths
- Every methodological choice is pinned in a hashed config.
- Rich robustness checks and controls.
- Strengthening audit runs on CPU without re-running models.
Caveats
- Research reproduction, not a reusable library.
- Depends on prior OpenJev/OneJev prediction files.
Getting started
bash scripts/run_strengthening_full.sh configs/reproduce.yaml results/strengthening_v1 8At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- CLI
- Has code
- yes
Use cases and integrations
How curated lists describe it
Reproduction pipeline for routing-conditional calibration in OpenJev NLP…
wh000wh000/awesome-jev-live
Frequently asked questions
What question does this study ask?
Whether decision probabilities remain trustworthy on the subset of examples a router selects.
Which datasets are used?
Civil Comments and GoEmotions.
Does it need a GPU?
The strengthening audit is CPU-only and reuses existing predictions.
Related projects
Pre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPythonBlack-box reverse-engineering archive studying how the Jev decision model behaves, with evidence matrices and calibration analysis.
ProjectresearchPythonIndependent API-only audit of what Jev's probabilities mean: abstain removal, Korean, coherence, option order, interference and caching.
ProjectresearchPythonResearch framework (DART) testing whether Jev's probabilities are a reliable signal to answer, escalate or abstain.
ProjectresearchPythonEvidence ledger auditing Jev claims, recomputing published calibration and running an 800-item difficulty-gradient study.
AppresearchPythonRunnable yardstick that tests Jev's three headline claims on a small local model: typed output, no hallucination, calibration.
ProjectresearchPython