Jevstiller
Proxy that distils a repeated Jev classification task into a local model on the fly and answers most requests itself once it agrees.
Overview
Jevstiller sits in front of a repeated Jev classification call. At first every request goes to Jev. From Jev's own answers and full probability distributions, it trains a small local model on your traffic, checks that the local model agrees with Jev within a budget you set, and then answers most requests itself. Uncertain or novel inputs, plus a permanent random audit slice, keep going to Jev so agreement stays measured.
It runs as a Docker container that speaks the same API, so you point the TypeSafe SDK's base URL at it and keep your key; nothing else changes. The motivation is workloads where a roughly 300 ms hosted call and vendor rate limits hurt: agent loops, game ticks, classify-then-act pipelines, tight latency budgets or machines without egress. The README reports that on Banking77, replayed against recorded Jev answers, the local model answered 71.9% of held-out messages at 99.5% agreement with Jev, and a demo of 5,000 real messages saw the local model take over after about 4,000 requests.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which label applies to this input? (the same question your app asks Jev) | — |
| Yes / no | Is the local model confident enough to answer, or should this go to Jev?(inferred) | — |
How it works
- Point the SDK's base URL at Jevstiller
- Requests initially pass through to Jev and are recorded with distributions
- A local model is trained on Jev's answers
- Once agreement meets the budget, confident requests are answered locally
- Uncertain, novel and audit-sample requests still go to Jev
Strengths
- Drop-in proxy; no application code changes
- Lower latency and fewer hosted calls once trained
- Continuous audit against Jev
Caveats
- Only suits repeated, stable classification tasks
- Needs enough traffic before taking over
- Local answers may drift from Jev between audits
Getting started
docker run -d -p 8080:8080 -v jevstiller-data:/data ghcr.io/tomerglick57/jevstiller, then export TYPESAFE_BASE_URL=http://localhost:8080.At a glance
- Type
- Model
- Status
- beta
- Language
- Python
- Runtime
- Docker proxy service
- Has code
- yes
- Has tests
- yes
- License
- Apache-2.0
- Stars
- 68
Use cases and integrations
How curated lists describe it
Distill a repeated Jev classification task into a local model, on the fly — same answers, your hardware.
logicrw/awesome-jev-projectsDistill a repeated Jev classification task into a local model, on the fly — same answers, your hardware.
wh000wh000/awesome-jev-liveDistill a repeated Jev classification task into a local model, on the fly — same answers, your hardware.
hellogumbo/awesome-jevSits in front of a repeated classification call, trains a local model on Jev's own answers and their distributions, checks the two agree within a budget you set, and then answers most requests itself.
MrJev/awesome-jev
Frequently asked questions
Do I need to change my code?
No, set TYPESAFE_BASE_URL to the proxy; your service keeps its own key.
How accurate is the local model?
On Banking77 the README reports 71.9% of held-out messages answered locally at 99.5% agreement with Jev.
Does it stop calling Jev entirely?
No, uncertain and novel inputs and a random audit slice always go to Jev.
Related projects
Docker service exposing local Laya, Von, a CLM head or hosted Jev through one TypeSafe-compatible HTTP and MCP endpoint.
MCP serverbetaPythonConverts open Jev reproductions (Laya, Kev, NanoJev, PlayJev) to INT8 ONNX for offline CPU decisions at about 15.6 ms per question.
ServicebetaPythonNative Go runtime serving Gemma 4 12B as a Jev-like decision model with hand-tuned PTX kernels and no CGo or CUDA toolkit.
ProjectbetaGoGo harness for System One models: hosted Jev, open Kev in-process via llama.cpp, and Cloudflare Clef, with typed helpers and a CLI.
ServicebetaGoGo gateway exposing the Jev /v1/systemone contract over ordinary LLM backends by reading single-token label logprobs.
ServicebetaGoRuns Intern-Decision-0.8B on Apple Silicon behind a local /v1/systemone endpoint that accepts screenshots for agent decisions.
ServicebetaPython