JevOut
Research code showing that short, natural-looking context additions can flip decision models like Jev to a chosen wrong answer.
Overview
JevOut accompanies the paper 'JevOut: Natural Context Can Flip Decision Models', accepted as a poster at Agenthon @ NeurIPS 2026. It asks whether decision models used in routers, evaluators and agents can tell useful context from distraction. The authors use probability-guided context optimisation to build short additions that read as ordinary background or procedural details, do not change the task, options or correct answer, and yet steer an initially correct decision to a wrong option fixed in advance.
The headline metric is Targeted Flip Rate. With up to 64 accepted target evaluations per decision across seven datasets, the README reports TFR of 61.4% for Jev, 72.6% for OpenSourceJev, 73.2% for Von and 64.9% for plain Qwen. The study also covers cross-model transfer, human validation, repeatability and learned context generation. The code supports Jev, HTTP endpoints and Python callbacks, and an offline demo uses a deterministic synthetic model with no keys. It is for researchers and practitioners assessing decision-model robustness.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Does an answer-preserving context addition flip the model's correct choice to a targeted wrong option? (measured as Targeted Flip Rate) | — |
How it works
- Select items where the decision model is initially correct
- Pick a target wrong option in advance
- Optimise natural-looking context additions guided by the model's probabilities
- Measure how often the decision flips to the target within a budget
- Check transfer, human validity and repeatability
Strengths
- Quantifies a realistic attack on decision models
- Compares hosted and open decision systems on the same method
- Offline demo needs no keys
Caveats
- Research code, not a defence
- Results depend on the optimisation budget used
- Hosted model results may change as models update
Getting started
git clone https://github.com/xzx34/JevOut.git && cd JevOut && uv sync --frozen && uv run context-opt demo --output-dir outputs/demoAt a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- research CLI
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Code and evaluation tools for JevOut: Natural Context Can Flip Decision Models.
hellogumbo/awesome-jev
Frequently asked questions
What did JevOut find?
Short natural additions flipped 61 to 73% of initially correct decisions to a chosen wrong answer across the four systems tested.
Is this prompt injection?
The additions read as background details, not instructions, which is what makes them hard to filter.
Can I run it without an API key?
Yes, the demo uses a synthetic model; testing Jev needs a key.
Related projects
Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonPaper testing Jev, Laya, Decider and Nimble against classifiers and LLM judges on prompt injection and risk screening.
Paperresearchpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPython