stuntdouble
Zero-dependency Node proxy that shadows Jev with local decision models and reports whether you can safely swap, per question and policy.
Overview
stuntdouble is a drop-in proxy for TypeSafe's /v1/systemone API that answers the question of whether a free local decision model can replace Jev on your own traffic. You change your app's base URL; the proxy forwards every request to the 'star' (Jev) and returns its reply byte for byte, while also asking one or more 'doubles' (Kev, Laya or anything with the same wire shape) the same question and recording all answers.
A report then shows label agreement per question and primitive, confidence-gated agreement, latency and cost differences, and a swap verdict: the lowest confidence threshold that keeps disagreement within a budget you choose, the share of traffic it covers and the saving per 1,000 decisions. Uniquely, with suites that include application policy code such as jev-by-example, it measures policy agreement, whether your app would have acted differently; on those cases two small models diverged on 6 of 28 application decisions, including a retry-or-reconcile case where one would retry a write that may already have succeeded. A replay mode reruns recorded traffic against new checkpoints without paying Jev again, and suites can be imported from JevBench and Kev. It is MIT licensed, independent and has zero dependencies.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Would the local double have answered each question the same way as Jev? | — |
| Other | Above which confidence threshold can traffic be routed to the double within the disagreement budget? | — |
How it works
- Configure Jev as the star and local models as doubles
- Point the app's SDK at the proxy and run real traffic
- Record star and double answers for every request
- Run the report for agreement, policy agreement, cost and a swap verdict
Strengths
- Measures swap safety on your own traffic
- Policy-level agreement, not just labels
- Replay without new Jev calls
Caveats
- Doubles add local compute during shadowing
- Policy agreement needs suites with policy code
Getting started
npx stuntdouble serve --config stuntdouble.json, set the SDK baseURL to http://127.0.0.1:8010, then npx stuntdouble report records/*.jsonl.At a glance
- Type
- Proxy
- Status
- beta
- Language
- JavaScript
- Runtime
- Node 20.17+ proxy and CLI
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Drop-in /v1/systemone proxy that shadows Jev with local decision models (Kev, Laya) and reports whether you can swap
logicrw/awesome-jev-projects
Frequently asked questions
Does my app see the local model's answers?
No; it always gets Jev's reply byte for byte while doubles are recorded in the background.
What is policy agreement?
Whether your application's own code would have acted differently given the double's answer.
Can I test new checkpoints cheaply?
Yes, replay reruns recorded traffic without calling Jev again.
Related projects
Reproducible measurements of Jev's real cost, latency, response shapes and accuracy, plus 13 decision models on the same 633 items.
ServiceresearchIndependent measurements of Jev versus open Laya on typed decisions, plus a local-first cascade that matches Jev at 1.8x speed.
ServiceresearchPythonApple Silicon workbench running Laya, Decider, Kev and Jev side by side with accuracy, calibration, latency and memory benchmarks.
BenchmarkbetaPythonReproducible 2,976-request measurement of whether batching questions in one Jev call bills the state once and keeps answers stable.
BenchmarkresearchPythonLocal benchmark comparing Jev with Laya-MLX on classifying GitHub issues for beginner-friendliness, CV fit and difficulty.
BenchmarkresearchPythonSpeed and accuracy benchmark of hosted Jev 1.13 versus local Laya 421M on 1,470 synthetic items across eight decision tasks.
BenchmarkresearchPython