switchboard
Stdlib-Python guardrail and model router for LLM gateways on Jev, with an independent accuracy, calibration and latency evaluation.
Overview
This switchboard is a guardrail and model router for LLM gateways built on Jev, plus an evaluation of whether Jev's probabilities can be trusted. Each POST /v1/chat/completions triggers one Jev call with four parallel questions: Nouls for injection, harmful content and PII, and a Choice of tier (small, large or unclear). Pure-code policy blocks with 403 at risk of 0.80 or above, allows but flags between 0.40 and 0.80, and routes to the small backend only if confidence is at least 0.60. If the request carries context, a second call checks grounding (supports, contradicts, says nothing) and leakage. Prometheus metrics report requests, actions, routes, latency and Jev spend. If Jev is unreachable it fails open to the large model and flags the request, or closed if configured.
The evaluation runs the injection question over 662 labelled prompts from deepset/prompt-injections against a regex baseline. The README reports ROC-AUC 0.990, F1 0.938 at a 0.1 threshold but recall of only 0.529 at 0.8 (precision 1.000), latency p50 226 ms, and $0.017 per 1,000 requests, with ECE 0.122 and a reliability table showing where probabilities are off. A committed cassette reproduces every number without a key.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this request a prompt injection? | — |
| Yes / no | Is this request harmful? Does it contain PII? | — |
| Choice | Which model tier should serve this request? | small · large · unclear |
| Choice | Does the context support, contradict or say nothing about the answer? | supports · contradicts · says_nothing |
How it works
- Gateway receives a chat completion request
- One Jev call answers injection, harm, PII and tier
- Policy blocks, flags or routes to small/large backend
- Optional grounding and leak check when context is present
Strengths
- Reproducible evaluation with calibration analysis
- No dependencies (90-line stdlib client)
- Clear fail-open or fail-closed behaviour
Caveats
- High thresholds miss many injections (recall 0.53 at 0.8)
- ECE 0.122 shows calibration gaps
- Single-dataset evaluation
Getting started
python3 eval_guardrail.py reproduces results from the cassette; TYPESAFE_API_KEY=... python3 eval_guardrail.py --record for a live run.At a glance
- Type
- Service
- Status
- research
- Language
- Python
- Runtime
- Stdlib HTTP gateway
- Has code
- yes
Use cases and integrations
How curated lists describe it
Guardrail + model router for LLM gateways on TypeSafe's Jev (System One model), with an independent accuracy/calibration/latency evaluation. Stdlib Python.
hellogumbo/awesome-jev
Frequently asked questions
What does Jev decide?
Whether a request is an injection, harmful or contains PII, which tier serves it, and whether context supports the answer.
How well did it detect injections?
ROC-AUC 0.990 on 662 prompts, with recall dropping at high thresholds.
What if Jev is down?
It fails open to the large model and flags the request, unless FAIL_MODE=closed.
Related projects
Large catalog of Claude Code agents, commands, hooks and MCPs, listed with community Jev mods for guardrails and subagent models.
CLIproductionJavaScriptJanus (formerly calibre): measures on your data the confidence threshold at which a small model like Jev should defer to a larger one.
ProjectbetaPythonClaude Code plugin where Jev decides and Claude writes: sort piles of text, calibrate thresholds, route prompts and gate Bash commands.
PluginbetaPortuguese-language agent skill for millisecond Jev decisions via OpenRouter, with validation, an MCP mode and calibration evals.
Agent skillbetaJavaScriptClaude Code mod that hands small decisions (routing, skill choice, screening, gating, trimming) to Jev or another decision backend.
ProjectbetaTypeScriptTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPython