Decision Models Showdown
Web app that sends identical ticket-routing and alert-triage questions to OpenAI Decisions and Jev side by side.
Overview
Decision Models Showdown compares two decision-model APIs on the same questions: OpenAI Decisions (gpt-6-luna) and TypeSafe Jev (jev-latest). Both answer typed questions with probabilities instead of prose; the app sends identical requests in parallel and plain code makes the final decision using thresholds you control.
It contains two demo projects. A support ticket router for a fictional SaaS asks team (choice), security_sensitive (yes/no) and customer_impact (score 0 to 3), and routes to a team queue, a security queue or human review, over 20 hand-labelled tickets. A cloud alert triage takes a JSON monitoring alert, asks severity, recommended_action and likely_caused_by_deploy, and decides between ignore, watchlist, paging on-call and rollback, over 15 labelled alerts. Without keys it runs in a clearly labelled mock mode with fake probabilities.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | team: which team should handle this ticket? | — |
| Yes / no | security_sensitive: is this ticket security-sensitive? | yes · no |
| Score | severity: how severe is this alert? | 0 · 1 · 2 · 3 |
| Yes / no | likely_caused_by_deploy: was this alert likely caused by a deploy? | yes · no |
How it works
- Enter a ticket or alert (or pick a labelled sample)
- Send identical typed questions to both providers in parallel
- Show both probability sets side by side
- Code applies thresholds to route, page or roll back
Strengths
- Direct comparison on identical inputs
- Mock mode for exploring without keys
- Shows decisions acting on both free text and JSON state
Caveats
- Very small labelled samples (20 and 15)
- Mock numbers are meaningless
- Needs keys for both providers for a real comparison
Getting started
python3 -m venv .venv, activate it, pip install the requirements, add keys to .env and start the app.At a glance
- Type
- Benchmark
- Status
- prototype
- Language
- Python
- Runtime
- local web app
- Has code
- yes
Use cases and integrations
How curated lists describe it
Head-to-head: puts OpenAI Decisions and Jev on the same questions in two demo apps, a support ticket router over 20 labelled tickets and a cloud alert triage over 15 labelled alerts, where the code acts on the typed answers.
yibie/awesome-jev
Frequently asked questions
Which APIs are compared?
OpenAI Decisions (gpt-6-luna) and TypeSafe Jev (jev-latest).
Can I run it without keys?
Yes, in mock mode, but the probabilities are fake.
Who makes the final decision?
Plain code applying thresholds you control.
Related projects
Third-party check of Jev versus GPT-4o-mini and Claude Sonnet 4.5 routing 60 multilingual photo-shoot booking inquiries.
ProjectresearchTypeScriptExperiment testing whether a local Qwen read via label log-probabilities can replace Jev on synthetic AIOps alert decisions.
ProjectresearchPythonC# System One client and benchmark comparing Microsoft-Decision-1, Jev and open-jev on the same labelled support tickets.
LibraryprototypeC#Learning experiment comparing Jev with an OpenAI model on GitHub issue ownership and on AG News, TREC and SNIPS classification.
ModelresearchPythonSmall labelled support-triage benchmark of jev-1.13 on OpenRouter covering noul, choice, score, routing, consistency and cost.
BenchmarkresearchPythonIndependent test of whether Jev's probabilities stay calibrated on 900 unseen rule-generated tickets plus three public benchmarks.
Benchmarkresearch