llm-as-decision-model
Experiment testing whether a local Qwen read via label log-probabilities can replace Jev on synthetic AIOps alert decisions.
Overview
llm-as-decision-model asks whether a general LLM, read as a decision model, can replace a dedicated System One model like Jev. A local Qwen is never asked to write text: each decision is one forward pass, reading the probability on each label's token and renormalising over allowed labels, giving output in the same shape as Jev's so both are scored identically.
The data is 310 synthetic AIOps alerts, with made-up services and hosts, each needing three decisions: category (5 classes), priority (P1/P2/P3) and whether to page a human. Arms cross Jev, Qwen3.8-27B and Qwen3.5-4B with three prompt variants: plain, a written priority rubric with a tie-break, and a 'policy' variant adding two rules found by error analysis on the same alerts, which the README notes is an upper bound rather than a fair arm. A TF-IDF plus logistic regression baseline gives a no-LLM floor. The Qwen runner checks at startup that the chat template renders as expected and every label is a single token, refusing to run otherwise. Measurements were on an M5 Max.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which category does this alert belong to? (5 classes) | — |
| Choice | What priority is this alert? | P1 · P2 · P3 |
| Yes / no | Should a human be paged? | — |
How it works
- Load 310 synthetic alerts and the shared question schema
- Run Jev and local Qwen models with plain, rubric and policy variants
- Read Qwen label log-probabilities via llama-server
- Compare with a TF-IDF baseline
- Score and plot results
Strengths
- Identical wording for all models
- Safety checks on tokenisation before scoring
- Includes a no-LLM baseline
- Honest about the upper-bound policy arm
Caveats
- Synthetic data only
- Policy arm tuned on test data
- Single hardware setup
Getting started
uv sync, then uv run python -m src.jev --variant plain (with JEV_API_KEY) and run llama-server for the Qwen arms.At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- scripts with llama.cpp server
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Can a small local LLM, read as a decision model via label logprobs, replace a…
wh000wh000/awesome-jev-live
Frequently asked questions
How is Qwen used as a decision model?
By reading the probability on each label's single token and renormalising over allowed labels.
Is the data real?
No, all alerts are synthetic.
Is the policy arm a fair comparison?
No, the README calls it an upper bound because its rules were found on the same alerts.
Related projects
Study of Jev's choices in multi-armed bandits, testing whether Bayesian instructions and supplied statistics improve its exploration.
ProjectresearchLocal Jev-style decision model on Qwen3.5-4B with four stress tests that show where one-pass decisions break.
ProjectresearchPythonLogitTrail (formerly Mini Jev): local workbench that reads typed Choice, Noul and Score decisions from a frozen LLM's logits.
ProjectresearchWeb app that sends identical ticket-routing and alert-triage questions to OpenAI Decisions and Jev side by side.
BenchmarkprototypePythonExperimental harness for typed JSON decisions from DiffusionGemma by picking the likeliest allowed tokens from final logits.
BenchmarkresearchPython- TypeAR★ 946
TypeLLM: Jev-inspired type-safe decoding for existing autoregressive LLMs on SGLang, with schema-guaranteed typed outputs.
ProjectbetaPython