mini-jev
Local Jev-style decision model on Qwen3.5-4B with four stress tests that show where one-pass decisions break.
Overview
mini-jev builds a local Jev-style decision model from a frozen open LLM (Qwen3.5-4B running on MLX) that answers by reading option probabilities instead of generating text. The main contribution is a set of four stress tests probing the limits of single-pass decisions, with results recorded in RESULTS.md.
The README reports that accuracy on chained facts falls from 0.995 at one hop to 0.285 at four hops while the model still reports about 0.56 confidence; paraphrased attributes with 255 options drop accuracy to 0.330; and when every candidate SQL query is wrong the model says None only 80% of the time for JOINs and 61% for GROUP BY/HAVING. Look-alike exact-ID lookups showed no break. The repo also includes a decision server, unit tests, a sanity eval on SST-2 and AG News, and temperature calibration. It is useful for anyone deciding how far to trust single-pass decision models on multi-step or abstention-heavy questions.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which option answers the question, read from next-token option probabilities(inferred) | — |
| Choice | Which candidate SQL query is correct, or None | — |
How it works
- Load Qwen3.5-4B frozen with MLX on Apple Silicon.
- Score each option by its probability instead of generating text.
- Fit a temperature for calibration.
- Run the four stress-test suites and compare accuracy against reported confidence.
Strengths
- Concrete, reproducible failure analysis with 200 items per level.
- Shows confidence can stay high when accuracy collapses, which informs threshold choices.
- Runs fully locally with a small HTTP decision server.
Caveats
- Requires Apple Silicon with 32 GB unified memory.
- Full stress suite takes about 165 minutes.
- The write-up link is still a placeholder in the README.
Getting started
python3.12 -m venv .venv && .venv/bin/pip install -r requirements.txt, then run .venv/bin/python -m server.app --port 8008 to start the decision server.At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- MLX on Apple Silicon, HTTP server
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Where one-pass decision models break: a local Jev-style model + 4 stress tests.
wh000wh000/awesome-jev-live
Frequently asked questions
What does this project measure?
Where a single-pass decision model fails: multi-hop fact chaining, paraphrased options, none-of-these SQL candidates and look-alike facts.
What hardware does it need?
The README targets Apple Silicon with 32 GB unified memory and Python 3.12.
Is the model calibrated?
A temperature calibration step is included, but the results show confidence stays near 0.56 even when four-hop accuracy is near chance.
Related projects
LogitTrail (formerly Mini Jev): local workbench that reads typed Choice, Noul and Score decisions from a frozen LLM's logits.
ProjectresearchRunnable yardstick that tests Jev's three headline claims on a small local model: typed output, no hallucination, calibration.
ProjectresearchPythonpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonStudy of Jev's choices in multi-armed bandits, testing whether Bayesian instructions and supplied statistics improve its exploration.
ProjectresearchExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
Projectresearch