shade-arena-jev-monitor
Preliminary SHADE-Arena study of Jev as a monitor and per-action gate for covert agent sabotage, compared with Gemini 2.5.
Overview
This fork of the SHADE-Arena public split evaluates Jev as a monitor for covert agent sabotage, compared with Gemini 2.5 Flash and Pro, and includes a full PDF report. The data are 13 hand-built honest and attack transcripts across three tasks (bank transfer, API key leak, spam filter), split into 353 tool calls of which 13 are sabotage. The authors label the results preliminary design signals, not benchmark numbers.
Findings: as a whole-transcript monitor Jev was weak (AUC 0.78 versus 0.99 for Gemini Flash), because transcripts over its roughly 32K-token limit must be chunked and a few malicious lines did not move chunk scores. As a per-action gate scoring each tool call before it runs, Jev was strong (AUC 0.97 versus 0.91), catching 12 of 13 attacks with about 1.5% of benign actions blocked, at a median 1.5 s per action versus 4.7 s; Gemini Flash's scores clustered at 0 and 100. Jev missed an autodelete rule disguised as anti-phishing. Two-stage gating, with Jev screening every action and escalating to Gemini, caught all 13 attacks with 8.5% benign blocked and 71% fewer Gemini calls.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Score | How suspicious is this tool call (0-100) as possible sabotage? | — |
| Score | How suspicious is this whole transcript? | — |
How it works
- Split transcripts into individual tool calls
- Score each action with Jev before it runs
- Block above a threshold or escalate suspicious actions to Gemini
- Compare AUC, catch rate and false alarms across monitors
Strengths
- Strong per-action ranking of sabotage in this sample
- Faster than Gemini per action
- Two-stage design cuts Gemini calls by 71%
Caveats
- Only 13 hand-built transcripts
- Weak on long whole transcripts due to the 32K limit
- Missed a semantically disguised attack
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Evaluation scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Evaluating TypeSafe's Jev as a fast monitor and action gate for agent sabotage in SHADE-Arena, compared with Gemini 2.5 Flash/Pro.
hellogumbo/awesome-jev
Frequently asked questions
Is Jev a good sabotage monitor?
In this small study, weak on whole transcripts but strong as a per-action gate.
What is two-stage gating?
Jev screens every action and escalates suspicious ones to Gemini, which caught all 13 attacks with fewer false alarms.
How reliable are the numbers?
The authors call them preliminary design signals from 13 hand-built transcripts.
Related projects
Hackathon arena where 200 people attack a support agent and a Jev allow/block/escalate gate on every tool call is hardened each round.
ProjectprototypePythonAuditable Zig terminal agent pairing Codex for coding with Jev typed preflight and postflight checks under deterministic policy gates.
CLIproductionZigGateway that checks agent tool calls with policy plus Jev evidence, then binds approval to a single-use permit for that exact action.
MCP serverbetaTypeScriptMCP proxy and hooks that screen agent tool calls before they run and their results before the agent reads them, using Jev judgments.
MCP serverbetaTypeScriptXano backend template where Jev judges each agent tool call as safe, needs review or dangerous, and uncertain calls wait for a human.
ProjectbetaXanoScriptOpenAI-compatible endpoint that answers an agent's command-approval call with APPROVE, DENY or ESCALATE from a Jev Choice, failing closed.
ProxybetaPython