openpoke-meets-jev
OpenPoke fork sending yes/no decisions to Jev instead of Claude Sonnet, with an A/B on cost and latency and an injection-gate attack study.
Overview
openpoke-meets-jev forks OpenPoke, an open reimplementation of the Poke assistant whose important-email watcher sends every new message to Claude Sonnet with a tool schema whose only required field is a boolean. The fork sends those decisions to Jev instead, keeping the LLM for writing the notification.
An A/B harness gives both arms the same state and question dictionaries, hashed for verification. On 36 screens, the README reports mean latency of 424 ms for Jev against 2,452 ms for Sonnet, input cost of $0.0018 against $0.1667, 18 distinct probabilities against 6, and agreement on 33 of 36 at a 0.75 bar, while cautioning that agreement is not accuracy, the cost ratio uses list rates on unequal token counts, and latency came from one machine. The author then attacked the prompt-injection gate with a 43,776-request adversarial run and documents a contamination finding in their own defence.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this email important enough to notify the user? | yes · no |
| Yes / no | Does this content contain a prompt injection? | yes · no |
How it works
- Watcher pulls new emails each minute.
- Ask Jev the typed email questions.
- Notify via the LLM only when Jev says important above the bar.
- Screen inputs with the injection gate.
Strengths
- Large cost and latency reductions per decision.
- Paired, hash-verified A/B harness.
- Candid adversarial evaluation.
Caveats
- No labelled set; reports agreement, not accuracy.
- Single-machine latency measurements.
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- FastAPI backend
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Fork of OpenPoke where the yes/no decisions go to Jev instead of Claude Sonnet, with a same-inputs A/B against the replaced LLM decision and a 43,776-request adversarial run on the injection gate.
hellogumbo/awesome-jev
Frequently asked questions
What does Jev replace?
Claude Sonnet calls that only produced a yes/no decision about each email.
How much cheaper is it?
Per the README, $0.049 versus $4.63 per 1,000 screens in input cost, using list rates.
Is accuracy reported?
No, the harness refuses to print accuracy without labelled data.
Related projects
Example of using Jev as a webhook guardrail inside the agentgateway proxy to inspect model requests and responses.
ProxyproductionRustLarge catalog of Claude Code agents, commands, hooks and MCPs, listed with community Jev mods for guardrails and subagent models.
CLIproductionJavaScriptx402 pre-payment risk-check provider combining sanctions lists, phishing feeds and simulation with Jev checks for injected instructions.
ProjectproductionAutomated PR review: an LLM agent finds issues, a verifier refutes them, and Jev decides how to triage, route and label.
ProjectproductionTypeScriptCisco's security scanner for agent skills, with a System One analyzer as an advisory tier that cannot emit findings or change severity.
Agent skillproductionPythonMCP proxy and hooks that screen agent tool calls before they run and their results before the agent reads them, using Jev judgments.
MCP serverbetaTypeScript