Will-It-Jev
Rust five-tier cascade router that sends prompts to Jev or an LLM, escalating only ambiguous requests.
Overview
Will-It-Jev is an in-process Rust routing engine that classifies incoming prompts and routes them either to a discriminative engine (Jev) or a generative LLM. It is a five-tier deterministic fallback cascade where each tier resolves the request if confident enough or passes it on: static fuzzy rules and regex guardrails; a learned decision layer with margin-scaled confidence gating; micro-transformer or embedding evaluators using labeled exemplars at runtime; Jev self-evaluation, where Jev judges whether the request is within its own capability; and finally a generative LLM that always produces an outcome.
It targets p99 decision latency under 5 ms for tiers 1 to 3, with native Rust, C++ bindings, a custom allocator and zero-copy ingress. Evaluators are pluggable, behavior is deterministic for fixed inputs and artifacts, and the README covers benchmarking, observability, failure modes and known constraints.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Should this prompt go to Jev or a generative LLM? | Jev · LLM |
| Yes / no | Can Jev handle this request? (Tier 4 self-evaluation) | yes · no |
How it works
- Tier 1 rules and guardrails try to resolve the prompt.
- Tier 2 learned classifier commits if confidence clears the threshold.
- Tier 3 exemplar evaluators, then Tier 4 Jev self-evaluation.
- Tier 5 LLM handles everything else.
Strengths
- Sub-5 ms p99 target for most traffic.
- Deterministic and pluggable.
- Expensive tiers only see ambiguous requests.
Caveats
- Complex system with model artifacts to maintain.
- Latency targets are design goals; verify with the included benchmarks.
At a glance
- Type
- Project
- Status
- beta
- Language
- Rust
- Runtime
- in-process library
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Ultra-low-latency, 4-tier prompt router in Rust using exact/fuzzy pattern…
wh000wh000/awesome-jev-live
Frequently asked questions
How many tiers are there?
Five, ending with a generative LLM that always decides.
How is Jev used?
As a destination for discriminative tasks and as a self-evaluator in tier 4.
What latency does it target?
p99 under 5 ms for tiers 1 to 3.
Related projects
Local-first routing engine for long-running agent tasks that uses a Jev decider to pick a model group per turn.
PluginproductionRust, TypeScriptOpen-source AI gateway for 100+ LLMs that can use Jev to classify requests for complexity routing and to compact tool results.
ProxyproductionPythonOpen-source workspace for running and reviewing AI coding agents across devices, with an optional Jev-based automatic model router.
ProjectproductionTypeScript- jev-router★ 593
Per-turn model routing for Claude Code and Codex: wraps the real CLI and lets Jev send simple turns to fast models and hard ones to strong.
ProxybetaJavaScript - jev-codex-router★ 274
Per-call router for Codex where Jev picks the model and thinking effort for every model call, including tool continuations.
ProjectbetaPython Model router where Jev classifies each request's task family and code picks the cheapest capable model on a cost Pareto chain.
ProjectbetaTypeScript