metask-jev
Open Jev-class decision models on Qwen3.5 with candidate-logit readout; the 4B model reports 80.1% on JevBench's 231 items.
Overview
Metask Jev-Lab, documented in Chinese, explores the best open-weight typed-decision (Jev-class) models: given a state and a bounded rubric, a single forward pass outputs a calibrated probability for each option, with no generation or parsing. Its main release, metask-jev-4b on Qwen3.5-4B, is published on Hugging Face; a 0.8B variant is unreleased.
The README reports a 13-subset macro average of 79.6% and 80.1% on JevBench's 231 items (at 4,096 context) for the 4B model, compared with 74.8% and 63.5 for Bespoke Nimble-9B and 76.0% and 75.3 for Jev 1.13.0 under the same protocol (JevBench v1.2 plus 13 human-labelled public subsets totalling 3,880 items). Inference takes about 24 ms per decision on an RTX 4090 via a unified candidate-logit scorer that adapts to MPS or CUDA. The repository includes a model registry, training recipe cards, a data pipeline for view augmentation and teacher soft-label synthesis, and a release pipeline for Hugging Face and ModelScope, though the teacher-distillation details are kept internal. Code is Apache-2.0 and weights follow the Qwen3.5 terms.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which option fits the state? | — |
| Yes / no | Boolean question about the state | — |
| Score | Rubric score for the state | — |
How it works
- Install inference requirements
- Load metask-jev-4b from Hugging Face
- Send state and schema (choice, boolean or rubric score)
- Read calibrated probabilities from candidate logits
Strengths
- Reported to exceed Jev on JevBench under the same protocol
- Fast single-pass inference
- Reproducible inference demo
Caveats
- Teacher distillation recipe not public
- 0.8B model unreleased
- Documentation primarily Chinese
Getting started
git clone https://github.com/metask-ai/metask-jev && cd metask-jev && pip install -r inference/requirements.txt && python inference/demo.py --model wayfind/metask-jev-4b-policy-mixAt a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- PyTorch inference (CUDA or MPS)
- Has code
- yes
- License
- Apache-2.0
Use cases and integrations
How curated lists describe it
Metask-Jev: calibrated typed-decision models (Jev-class). Single forward pass, candidate-logit readout. metask-jev-4b beats Bespoke Nimble-9B and Jev on JevBench.
hellogumbo/awesome-jev
Frequently asked questions
How does it compare with Jev?
The README reports 80.1% on JevBench's 231 items versus 75.3 for Jev 1.13.0 under its protocol.
How fast is it?
About 24 ms per decision on an RTX 4090.
Is the training recipe public?
Recipe cards are, but teacher-distillation details are internal.
Related projects
- TypeAR★ 946
TypeLLM: Jev-inspired type-safe decoding for existing autoregressive LLMs on SGLang, with schema-guaranteed typed outputs.
ProjectbetaPython Reads Jev-style choice, yes/no and score probabilities from masked diffusion language models with shared Yes/No weights.
ProjectresearchPythonOpen Jev replica on Gemma 4 E2B and Gemma 3 270M served on Modal, with eight demos and a TypeSafe eval head-to-head.
ProjectresearchPythonSmall experiment: reordering enum options changed a local Jev clone's answers on its old default, while hosted Jev's did not.
DemoresearchPython- JevK5★ 150
Apache-2.0 open-weight Jev alternative that answers yes/no, choice and score questions with per-option probabilities in one pass.
LibrarybetaPython Open System One decision model (LoRA on Qwen3.5-4B) that reads typed answers from logits in one pass, with its benchmark harness.
ModelbetaPython