jev-test
Small harness calling Jev via OpenRouter's Decisions API, with 17 doc-derived cases and a latency and cost benchmark against two LLMs.
Overview
jev-test is a compact Python test harness for Jev (~typesafe/jev-latest) via OpenRouter's alpha Decisions API. It wraps the endpoint in a decide(state, questions) helper, includes a quickstart using all three question types, and runs 17 test cases (23 checks) taken from TypeSafe and OpenRouter documentation, reporting a confidence band and total cost. The README also serves as a guide to the request shape and to Noul, Choice and Score semantics.
A benchmark script runs the same cases on Jev, GPT-5.6 Terra and Claude Opus 5. In the author's single run on jev-1.13, all three models passed all 23 checks; Jev's median latency was 452 ms versus 1,505 ms and 3,394 ms, and total cost for 17 calls was $0.00027 versus $0.00971 and $0.05223. The author notes the set is small and clear-cut, latency varies, and token counts are not directly comparable, so cost is the fairer comparison.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is the customer requesting a refund? | — |
| Other | Doc-derived Choice and Score cases (e.g. routing, urgency) | — |
How it works
- Define cases with state, questions and expectations
- Send each case to Jev via OpenRouter Decisions API
- Check expectations and report confidence and cost
- Optionally run the same cases on general LLMs for comparison
Strengths
- Reproducible with uv in a few commands
- README reports Jev about 3-7.5x faster and 35-190x cheaper than the compared LLMs on these cases
- Documents error codes and request shape
Caveats
- Small, clear-cut case set; single run
- OpenRouter alpha API may change
Getting started
uv sync, put OPENROUTER_API_KEY in .env, then `uv run run_cases.py` or `uv run benchmark.py`.At a glance
- Type
- Benchmark
- Status
- prototype
- Language
- Python
- Runtime
- CLI
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Model benchmarking: reproducible test harness evaluating TypeSafe Jev Noul, Choice, and Score decisions via OpenRouter's Decisions API, comparing latency and accuracy against LLM prompt-and-parse baselines.
yibie/awesome-jev
Frequently asked questions
Was Jev more accurate than the LLMs?
No; all models passed all checks, so the differences were speed and cost.
How is Jev called?
Through OpenRouter's alpha Decisions API with an OPENROUTER_API_KEY.
How much did the test suite cost?
About $0.0003 for the full 17-case run on Jev, per the README.
Related projects
One-page playground and test bench for Jev with TypeSafe, OpenRouter, LLM-baseline and mock backends behind one /v1/systemone contract.
ServiceprototypeTypeScriptRust port of TypeSafe's system-one-adapter that answers System One questions with OpenAI or Anthropic LLMs for comparison with Jev.
ProjectbetaRustLocal SvelteKit workbench and CLI for organizing, editing, running and comparing Jev experiments defined as JSON.
CLIbetaTypeScriptJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePythonIndependent field guide to Jev 1.13.0 behaviour across 11,621 requests: framing sensitivity, Snake and 3D city driving experiments.
ProjectresearchPython, TypeScriptIndependent 52-page Chinese report on Jev that checks vendor claims and reproduces 50 Chinese tests, with traceable data.
ProjectresearchPython