jev-local-research
Research replicating Jev-style typed decisions with open LLMs via logprobs; Qwen3.5-9B reached 94% with good calibration in ~1 s.
Overview
Local Jev: typed decisions with open LLMs asks whether Jev can be replicated locally without an API key and how well it works. The answer it gives is yes: Jev's approach can be reproduced with any LLM that exposes log probabilities, by labelling options with letters, asking for only the letter, and reading the probability of each letter at the first position as the decision. This takes one forward pass, gives explicit probabilities for threshold routing, and allows options defined at runtime.
It compares SemIf with other local setups on authored144, a 144-decision English set from the SemIf repo covering evidence interpretation, rule application and candidate selection, using the same prompt for every system. The README reports that SemIf adds nothing over reading logprobs directly, and that the best local result was Qwen3.5-9B at 94 percent accuracy with good calibration and about one second per decision.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Does the evidence support, contradict or not settle the claim? | support · contradict · not settled |
| Yes / no | Does the rule apply? | — |
| Choice | Which candidate meets the criterion? | — |
How it works
- Label options with letters in a fixed prompt
- Run local LLMs and read first-token letter probabilities
- Score accuracy and calibration on authored144
- Compare SemIf with direct logprob reading
Strengths
- Shows a simple local replication path
- Same prompt across systems
- Accuracy and calibration reported
Caveats
- Single 144-item test set
- Latency depends on local hardware
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Local LLMs (MLX, Docker Model Runner)
- Has code
- yes
Use cases and integrations
How curated lists describe it
Replicating Jev-style typed decisions with open LLMs via logprobs: SemIf vs…
wh000wh000/awesome-jev-live
Frequently asked questions
Can Jev be replicated locally?
The study says its approach can, using logprobs from any open LLM.
Which local model did best?
Qwen3.5-9B at 94 percent with good calibration.
Does SemIf help?
The README reports it adds nothing over reading logprobs directly.
Related projects
Exact-target audit finding Jev's Choice probabilities are a step function on stated base rates while per-option Noul tracks them closely.
ProjectresearchPythonLocal reproduction of Jev-style inference that reads option-token logits from a small LLM in one pass, with an honest failure log.
ProjectresearchPythonOpen decision models that derive calibrated confidence from recurrent looped-layer dynamics instead of RL, with early exit and a Jev API.
ProjectresearchPythonLaptop-scale reimplementation of RL for Calibrated Decisions: parallel constrained decoding trained with GRPO and a Brier reward.
ProjectresearchPython- AnyJev★ 1,255
Nokia's training-free library that turns open LLMs into Jev-style deciders robust to option order, plus self-distilled Tacit models.
ModelresearchPython Jev-style replica: frozen Qwen3-1.7B plus a ~0.8M-parameter trained head returns calibrated Choice, Boolean and Score distributions.
ServiceresearchPython