jev-context-filter-harness
Paper code where Jev screens agent tool output for relevance and prompt injection before the agent reads it.
Overview
This repository holds the code, data and run records for a paper under double-blind review, 'System One Tool-Output Screening for LLM Agents for Safe and Efficient Outcomes'. It is an agent harness with two filters between the tools and the agent, both asking Jev typed questions about each chunk of tool output. The relevance filter keeps chunks that answer the agent's own question about the call; the injection filter withholds lines that try to give the agent a task the user did not request.
The experiments use three agent models (Gemma 4 31B, Mistral Small 3.2 24B, DeepSeek V4.1 Flash) on three search-heavy test beds (ContextBench, LegalBench-RAG, DocFinQA) in clean and planted versions, plus AgentDojo for injection. Measures include task score, attacks followed, prompt tokens, compactions, Jev tokens and wall time, with paired intervals. All analysis scripts rebuild the numbers offline.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Does this chunk help answer the agent's question about the call?(inferred) | — |
| Yes / no | Does this line try to give the agent a task the user did not ask for?(inferred) | — |
How it works
- Agent calls a tool and states what it wants from the output
- Split the output into chunks and ask Jev the relevance and injection questions
- Keep relevant chunks, withhold injection lines
- Compare against an unfiltered control across 1,200 runs per agent
Strengths
- Fewer agent prompt tokens from dropping irrelevant output
- Injection screening before the agent sees text
- Full run records and offline analysis for reproducibility
Caveats
- Research code tied to a paper under review
- Results specific to the chosen agents and test beds
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Research harness
- Has code
- yes
Use cases and integrations
How curated lists describe it
Screening LLM agent tool output for relevance and prompt injection with Jev, a…
wh000wh000/awesome-jev-live
Frequently asked questions
What does Jev decide?
Whether each chunk of tool output is relevant, and whether each line is a prompt injection.
How was it evaluated?
Three agent models on code, legal and finance test beds in clean and attacked versions, plus AgentDojo.
Can I reproduce the numbers?
Yes, the repo includes every run record and scripts that rebuild results offline.
Related projects
DeepSeek Harness plugin where Jev prunes long tool output, screens fetched pages for injected instructions and gates completion claims.
Agent skillbetaTypeScriptClaude Code mod that hands small decisions (routing, skill choice, screening, gating, trimming) to Jev or another decision backend.
ProjectbetaTypeScriptTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonDeterministic evidence selector for noisy Codex investigations; its earlier Jev subagent router is kept for the record, not recommended.
ProjectresearchResearch code evaluating Jev as an independent per-tool relevance judge for building task-level tool spaces on TGB, BFCL and GTA.
ServiceresearchPythonResearch code showing that short, natural-looking context additions can flip decision models like Jev to a chosen wrong answer.
ProjectresearchPython