LiteLLM
Open-source AI gateway for 100+ LLMs that can use Jev to classify requests for complexity routing and to compact tool results.
Overview
LiteLLM is a widely used open-source AI gateway and Python SDK that gives a single OpenAI-format interface to more than 100 LLM providers, including OpenAI, Anthropic, Gemini, Bedrock and Azure. It can run as a self-hosted proxy server for teams, with features such as unified authentication, cost tracking and routing.
Per the listings, LiteLLM includes Jev integrations: its complexity-based model router can use Jev to classify each request's complexity and pick a model tier, and an optional TypeSafe guardrail hook judges relevance to compact tool results before they return to an agent. The README excerpt is LiteLLM's general overview and does not describe these features, so their exact configuration is not confirmed here. Its adoption by a major gateway signals use of typed decision models for request routing.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | How complex is this request, and which model tier should handle it?(inferred) | — |
| Yes / no | Is this part of a tool result relevant enough to keep? (compaction guardrail)(inferred) | — |
How it works
- Request arrives at the LiteLLM proxy
- Optional Jev classification of request complexity
- Router sends the request to the matching model
- Optional guardrail compacts tool results by relevance
Strengths
- Integrated in a widely used gateway
- Cheaper routing than an LLM classifier
- Optional, configurable features
Caveats
- Jev features not described in the main README
- Requires a TypeSafe key
- Custom licence
Getting started
pip install litellm or deploy the LiteLLM proxy; see its docs for routing and guardrail configuration.At a glance
- Type
- Proxy
- Status
- production
- Language
- Python
- Runtime
- SDK and proxy server
- Has code
- yes
- Has tests
- yes
- License
- Custom license
Use cases and integrations
How curated lists describe it
LiteLLM can use Jev to classify requests for its complexity-based model router.
logicrw/awesome-jev-projects`litellm/types/proxy/guardrails/guardrail_hooks/typesafe.py` implements a…
wh000wh000/awesome-jev-liveJev-backed complexity routing and an optional relevance guardrail for compacting tool results before they return to an agent.
cobanov/awesome-jev
Frequently asked questions
How does LiteLLM use Jev?
For complexity-based model routing and an optional relevance guardrail, per the listings.
Is Jev required?
No, these are optional features.
Where is it documented?
In LiteLLM's documentation and source rather than the main README.
Related projects
Claude Code mod that picks reasoning effort per prompt with a Haiku judge, optionally Jev, and shows cache and context state.
ProjectbetaTypeScriptHermes Agent plugin that picks the model each turn with a direct TypeSafe Jev Choice and enforces trim-then-compress context handling.
PluginbetaPythonClaude Code plugin that picks model tier and effort per turn with Jev and weighs each switch against cost, with Jev-based compaction.
ProjectbetaTypeScriptClaude Code and Codex plugin where Jev judges each turn for model routing, delegation, evidence search and compaction.
ProjectbetaTypeScriptClaude Code mod using Jev to pick model and effort per turn, size subagents, and prune stale context verbatim instead of summarising.
ProjectbetaTypeScriptLocal-first routing engine for long-running agent tasks that uses a Jev decider to pick a model group per turn.
PluginproductionRust, TypeScript