siege
Hackathon arena where 200 people attack a support agent and a Jev allow/block/escalate gate on every tool call is hardened each round.
Overview
SIEGE puts a live customer-support agent with real tools (refunds, address changes, discounts, store credit, order and profile lookups) in front of a room of 200 people trying to make it misbehave. The agent, running gpt-oss-20b on W&B Inference, is deliberately eager to help. Every proposed tool call passes a defender-written Python prefilter and then TypeSafe System One, which answers Choice{allow, block, escalate} with calibrated probabilities in roughly 150-300 ms, given policy rules, the authenticated customer, the action, facts, recent messages and recent attack examples. Only allow executes.
A deterministic policy oracle labels each call allowed or forbidden, so breaches (gate allowed, oracle forbidden, executed) and benign blocks are exact. After each round a defender loop reads breach traces, has a red-team model generate variants, has DeepSeek rewrite the gate policy and prefilter, validates code in a W&B Sandbox, and ships the new policy only if a Weave Evaluation shows the catch rate rising while the benign allow rate stays at or above 0.9. The agent itself is never changed. The provider chain falls back to an LLM and then a heuristic mock. It was built at CoreWeave Hacks.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Should this proposed tool call be allowed, blocked or escalated under the current policy? | allow · block · escalate |
How it works
- Agent proposes a tool call
- Prefilter runs, then Jev answers allow/block/escalate
- Only allow executes; the oracle labels the outcome
- Defender loop rewrites policy from breaches and ships it only if evaluation improves
Strengths
- Inline gate fast enough for every tool call
- Exact breach labels from a deterministic oracle
- Policy improves without modifying the agent
Caveats
- Hackathon project; public site is a labelled simulation
- Depends on several hosted services (W&B, DeepSeek)
At a glance
- Type
- Project
- Status
- prototype
- Language
- Python
- Runtime
- Backend + Node web arena; marimo notebook
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
SIEGE: 200 people vs one agent. A typed action gate (TypeSafe System One) that learns from every breach, evaluated by W&B Weave, hardened by a defender loop. Built at CoreWeave Hacks: Agent Loops 2026.
hellogumbo/awesome-jev
Frequently asked questions
What does Jev decide in SIEGE?
Whether each tool call the support agent proposes is allowed, blocked or escalated.
How does the gate improve?
A defender loop rewrites its policy from breaches and ships it only if evaluation shows improvement.
What if TypeSafe is down?
It falls back to an LLM structured-output judge, then a heuristic mock.
Related projects
Preliminary SHADE-Arena study of Jev as a monitor and per-action gate for covert agent sabotage, compared with Gemini 2.5.
ProjectresearchPythonAuditable Zig terminal agent pairing Codex for coding with Jev typed preflight and postflight checks under deterministic policy gates.
CLIproductionZigGateway that checks agent tool calls with policy plus Jev evidence, then binds approval to a single-use permit for that exact action.
MCP serverbetaTypeScriptMCP proxy and hooks that screen agent tool calls before they run and their results before the agent reads them, using Jev judgments.
MCP serverbetaTypeScriptXano backend template where Jev judges each agent tool call as safe, needs review or dangerous, and uncertain calls wait for a human.
ProjectbetaXanoScriptOpenAI-compatible endpoint that answers an agent's command-approval call with APPROVE, DENY or ESCALATE from a Jev Choice, failing closed.
ProxybetaPython