cygnet-recipe
Recipe that answers JevBench typed decisions with frozen Gemma-4-12B-it on stock vLLM by reading one option-letter token's probability.
Overview
Cygnet is a recipe for producing Jev-style typed decisions (choice, noul, score) from an unmodified open model. It serves frozen google/gemma-4-12B-it with stock vLLM 0.30.0 and adds a small standard-library shim that presents options as letters, reads the model's own probability for each letter at a single answer position, and applies one calibration temperature. Cost is input tokens only, with one output token per decision.
It is aimed at researchers and teams who want a self-hosted alternative and was packaged for JevBench. The README reports Cygnet ranked first overall of 106 systems on JevBench v1.5.4 (a statistical tie with a fine-tuned Gemma model), while Jev 1.13.0 leads the benchmark's Capability view; the authors note how each figure was obtained and which are their own public-set runs. Hardware measured is a single 48 GB GPU, and inputs are limited to 16384 tokens.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which lettered option is correct for this request? | — |
| Yes / no | Is the statement true (yes/no as letters)? | yes · no |
| Score | Which ordered level applies? | — |
How it works
- Serve gemma-4-12B-it with vLLM
- Start the shim, which maps typed questions to lettered options
- Read each letter's probability at one answer position
- Apply a calibration temperature and return Jev-shaped answers
Strengths
- No fine-tuning; frozen open weights
- One output token per decision keeps cost to input tokens
- Strong reported JevBench ranking
- Self-hosted
Caveats
- Needs a 48 GB-class GPU; other cards unmeasured
- 16384-token context limit
- Benchmark lead is a statistical tie
Getting started
vllm serve google/gemma-4-12B-it --served-model-name cygnet --max-model-len 16384, then start the shim on the same machine.At a glance
- Type
- Model
- Status
- research
- Language
- Python
- Runtime
- vLLM + HTTP shim
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Typed decisions from a frozen Gemma-4-12B-it by reading one option-letter token over stock vLLM, packaged for JevBench.
MrJev/awesome-jev
Frequently asked questions
Is any training involved?
No, the model is frozen; only a single calibration temperature is applied.
How does it compare with Jev?
The README reports Cygnet first overall on JevBench v1.5.4, with Jev 1.13.0 third overall and first on the Capability view.
What hardware is needed?
It was measured on one RTX A6000 and one L40S (48 GB each).
Related projects
- TypeAR★ 946
TypeLLM: Jev-inspired type-safe decoding for existing autoregressive LLMs on SGLang, with schema-guaranteed typed outputs.
ProjectbetaPython Reads Jev-style choice, yes/no and score probabilities from masked diffusion language models with shared Yes/No weights.
ProjectresearchPythonOpen Jev-class decision models on Qwen3.5 with candidate-logit readout; the 4B model reports 80.1% on JevBench's 231 items.
ProjectresearchPythonOpen Jev replica on Gemma 4 E2B and Gemma 3 270M served on Modal, with eight demos and a TypeSafe eval head-to-head.
ProjectresearchPythonSmall experiment: reordering enum options changed a local Jev clone's answers on its old default, while hosted Jev's did not.
DemoresearchPython- JevK5★ 150
Apache-2.0 open-weight Jev alternative that answers yes/no, choice and score questions with per-option probabilities in one pass.
LibrarybetaPython