Evaluating System One Models for Agent Security Decisions
Paper testing Jev, Laya, Decider and Nimble against classifiers and LLM judges on prompt injection and risk screening.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
This arXiv paper evaluates System One decision models, Jev, Laya, Decider and Nimble, against specialised classifiers and LLM judges on agent security decisions, specifically prompt-injection detection and risk screening.
According to the list summary, the key finding is that good aggregate calibration can hide confident failures concentrated in particular attack groups. For anyone using a decision model as a security gate, this means overall calibration metrics are not enough; you need to check performance per attack family.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Does this input contain a prompt injection?(inferred) | yes · no |
| Choice | What risk level does this agent action carry?(inferred) | — |
How it works
- Assemble prompt-injection and risk-screening datasets
- Evaluate decision models, classifiers and LLM judges
- Analyse calibration overall and per attack group
Strengths
- Highlights hidden confident failures
- Compares several decision models head to head
Caveats
- List-only entry; paper not reviewed
Getting started
See https://arxiv.org/abs/2609.33401.At a glance
- Type
- Paper
- Status
- research
Use cases and integrations
How curated lists describe it
Jev, Laya, Decider, and Nimble against specialized classifiers and LLM judges on prompt injection and risk screening: good aggregate calibration can hide confident failures concentrated in particular attack groups
AnotiaWang/awesome-decision-models
Frequently asked questions
Which models were tested?
Jev, Laya, Decider and Nimble, plus specialised classifiers and LLM judges.
What is the main finding?
Good aggregate calibration can hide confident failures in specific attack groups.
Why does it matter?
Security gates need per-attack-family evaluation, not just overall metrics.
Related projects
Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonResearch code showing that short, natural-looking context additions can flip decision models like Jev to a chosen wrong answer.
ProjectresearchPythonPre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPythonBlack-box reverse-engineering archive studying how the Jev decision model behaves, with evidence matrices and calibration analysis.
ProjectresearchPythonIndependent API-only audit of what Jev's probabilities mean: abstain removal, Korean, coherence, option order, interference and caching.
ProjectresearchPythonResearch framework (DART) testing whether Jev's probabilities are a reliable signal to answer, escalate or abstain.
ProjectresearchPython