jev-lab
Portuguese study using Jev as a real-time guardrail and as a pre-evaluator for LLM-as-judge on customer-service agents.
Overview
Jev Lab is a practical study, in Portuguese, of applying Jev to business conversational agents in a customer-service domain where public measurements in Portuguese were lacking. It asks what becomes possible when evaluating a message costs about 500 ms and a tiny fraction of a cent, through two runnable experiments.
The first uses Jev as a real-time guardrail before the agent's LLM call, filtering prompt injection, out-of-scope questions and customers pasting card numbers, so the filter costs a fraction of the call it avoids instead of adding another LLM. The second uses Jev as a pre-evaluator so that 100 percent of turns can be screened, with only failed or borderline traces sent to an LLM judge, rather than sampling 1 to 5 percent. The excerpt states the motivation; measured results were not included in the collected portion.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this message a prompt injection, out of scope or containing card data? | — |
| Score | Does this agent turn pass quality criteria, or does it need the LLM judge? | — |
How it works
- Incoming message is screened by Jev before the LLM
- Block or redirect flagged messages
- Every agent turn is pre-evaluated by Jev
- Send failed or borderline traces to an LLM judge
Strengths
- Guardrails on every message at low cost
- Full coverage of traces instead of sampling
- Portuguese-language measurements
Caveats
- Results not visible in the collected excerpt
- Single domain
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Experiment scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
O Jev (TypeSafe AI) como guardrail em tempo real e como pré-avaliador de…
wh000wh000/awesome-jev-live
Frequently asked questions
What does Jev decide?
Whether messages should be blocked before the agent, and which turns need an LLM judge.
Why pre-evaluate?
So all turns can be screened cheaply and only some go to an expensive judge.
What language is it in?
Portuguese.
Related projects
Large catalog of Claude Code agents, commands, hooks and MCPs, listed with community Jev mods for guardrails and subagent models.
CLIproductionJavaScriptMCP proxy and hooks that screen agent tool calls before they run and their results before the agent reads them, using Jev judgments.
MCP serverbetaTypeScriptXano backend template where Jev judges each agent tool call as safe, needs review or dangerous, and uncertain calls wait for a human.
ProjectbetaXanoScriptEffect-based safety gate for coding-agent shell commands: fast rules first, then Jev decides allow/deny plus eleven risk checks.
ProjectbetaTypeScriptDeepSeek Harness plugin where Jev classifies risk, irreversibility, task fit and injection for every tool call before it runs.
PluginbetaTypeScriptOpen-source agent guardrails that use Jev or LAYA to check prompts, retrieved content, tool calls and outputs for injection and risk.
ProjectbetaPython