jev-eval
Third-party check of Jev versus GPT-4o-mini and Claude Sonnet 4.5 routing 60 multilingual photo-shoot booking inquiries.
Overview
jev-eval compares Jev with LLMs (openai/gpt-4o-mini and anthropic/claude-sonnet-4.5) under identical conditions on one task: routing booking inquiries to a photo-shoot service for international tourists in Japan, using 60 synthetic messages in four languages. All models are called through one Vercel AI Gateway key.
The main run makes 60 cases times 3 rounds with Jev and gpt-4o-mini called serially, discards one warm-up call per model, and uses no retries; a second run swaps in Claude Sonnet 4.5. Raw responses are saved, scored into summary and per-case files, and charted. The README reports the main run cost about $0.03 and the Claude run about $0.62, and warns that the free Gateway tier hits rate limits. Results are in a separate RESULTS.md.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Where should this booking inquiry be routed?(inferred) | — |
How it works
- Run a smoke test against Jev and the LLMs via the Gateway
- Measure 60 cases over 3 rounds per model
- Score the run into summary and per-case results
- Generate charts for comparison
Strengths
- Identical conditions across models
- Multilingual inputs
- Low cost to reproduce
Caveats
- Synthetic dataset of 60 messages
- Requires paid AI Gateway credits for a full run
Getting started
Put AI_GATEWAY_API_KEY in .env.local, `npm install`, `npx tsx src/smoke.ts`, then `npm run measure` and `npm run score`.At a glance
- Type
- Project
- Status
- research
- Language
- TypeScript
- Runtime
- Node scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Third-party check: compares Jev against GPT-4o-mini and Claude Sonnet 4.5 under identical conditions on the same judgment task.
yibie/awesome-jev
Frequently asked questions
What task is used?
Routing 60 synthetic booking inquiries in four languages for a photo-shoot service.
What does a run cost?
About $0.03 for the Jev plus gpt-4o-mini run and $0.62 with Claude Sonnet 4.5, per the README.
Where are the results?
In RESULTS.md in the repository.
Related projects
C# System One client and benchmark comparing Microsoft-Decision-1, Jev and open-jev on the same labelled support tickets.
LibraryprototypeC#Learning experiment comparing Jev with an OpenAI model on GitHub issue ownership and on AG News, TREC and SNIPS classification.
ModelresearchPythonWeb app that sends identical ticket-routing and alert-triage questions to OpenAI Decisions and Jev side by side.
BenchmarkprototypePythonSmall labelled support-triage benchmark of jev-1.13 on OpenRouter covering noul, choice, score, routing, consistency and cost.
BenchmarkresearchPythonIndependent test of whether Jev's probabilities stay calibrated on 900 unseen rule-generated tickets plus three public benchmarks.
BenchmarkresearchIndependent benchmark of Jev versus GPT-4.1-mini on 200 South African customer enquiries, showing how question wording changed results.
BenchmarkresearchPython