jevbetter
Small one-pass option scorer with n-gram encoder, rival-aware attention and temperature scaling, benchmarked head-to-head against jevlike.
Overview
jevbetter trains a small model that picks among a changing list of text options in a single forward pass, returning one probability per option instead of decoding an answer. It uses the same JSONL data format as the open jevlike scorer so datasets transfer unchanged, and it is an independent project not affiliated with TypeSafe. Such a model suits routing, ranking, open-label classification and game controllers.
Compared with jevlike, it swaps raw byte embeddings for hashed character n-grams, adds a two-layer transformer context encoder, lets options attend to each other so near-miss rivals sharpen the decision, uses a gated MLP scoring head, trains with a cosine schedule, warmup, early stopping and label smoothing, and fits temperature scaling on validation. It also adds harder synthetic data and richer evaluation. On 800 held-out hard menus with matched budgets on CPU, the README reports top-1 accuracy of 0.916 versus 0.873 and ECE of 0.0182 versus 0.0367, with a shuffled-context control at 0.335 confirming the model reads context; the trade-off is throughput, 40 menus per second versus 4,608.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which of these N text options matches the context? | — |
How it works
- Generate or load JSONL menus
- Train with n-gram encoder and rival-aware head
- Fit temperature scaling on validation
- Evaluate top-k, MRR, ECE and throughput; predict on new menus
Strengths
- Higher accuracy and half the calibration error of jevlike per the README
- Compatible data format
- Shuffled-context control
Caveats
- Much lower throughput than jevlike
- Evaluated on synthetic data
Getting started
pip install -e '.[dev]', jevbetter-data --output data/synthetic, then jevbetter-train and jevbetter-eval.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- PyTorch (CPU)
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Improved scorer: a stronger one-pass scorer over a variable list of text options, using a hashed n-gram encoder, rival-aware attention, and gated heads.
yibie/awesome-jevA stronger one-pass scorer over a variable list of text options: hashed n-gram encoder, rival-aware attention, gated head, temperature scaling, benchmarked against jevlike.
hellogumbo/awesome-jev
Frequently asked questions
How does it compare with jevlike?
0.916 versus 0.873 top-1 and about half the ECE on hard synthetic menus, but far lower throughput.
Is it related to TypeSafe?
No, it is an independent project.
Can I use jevlike datasets?
Yes, the JSONL format is the same.
Related projects
Exact-target audit finding Jev's Choice probabilities are a step function on stated base rates while per-option Noul tracks them closely.
ProjectresearchPythonResearch replicating Jev-style typed decisions with open LLMs via logprobs; Qwen3.5-9B reached 94% with good calibration in ~1 s.
ProjectresearchPythonLocal reproduction of Jev-style inference that reads option-token logits from a small LLM in one pass, with an honest failure log.
ProjectresearchPythonOpen decision models that derive calibrated confidence from recurrent looped-layer dynamics instead of RL, with early exit and a Jev API.
ProjectresearchPythonLaptop-scale reimplementation of RL for Calibrated Decisions: parallel constrained decoding trained with GRPO and a Brier reward.
ProjectresearchPython- AnyJev★ 1,255
Nokia's training-free library that turns open LLMs into Jev-style deciders robust to option order, plus self-distilled Tacit models.
ModelresearchPython