Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

open-jev-fast

Drop-in faster backend for Open-Jev-27B using custom CUDA kernels, a prefix tree and CUDA graphs: about 6x lower JevBench latency.

Modelresearch
View on GitHub ↗Live demo ↗

Overview

OpenJev-Fast is a faster local inference backend for Open-Jev, the open typed-decision model. It is a drop-in replacement for Open-Jev's own server with the same request format and the same Open-Jev-27B-v1.1 model on Qwen3.8-27B in bf16 with no quantization. It replaces the forward pass with hand-written CUDA kernels (including Gated DeltaNet and tree-attention kernels), a prefix tree that computes shared prompt text once, tuned matrix multiplies and CUDA Graphs.

On one NVIDIA B300, the README reports an example request (3 questions, 7 candidates, 539 tokens) dropping from 256.0 ms on the default path and 103.7 ms with FLA kernels to 17.3 ms, and mean JevBench latency from 258 ms to 42.0 ms, with accuracy 197/231 against 198/231 for the original server because one near-tie flipped. It requires an Open-Jev installation and weights and is aimed at teams serving Open-Jev at scale.

What the decision model decides

TypeQuestionAnswers
ChoiceRouting: which of 3 queues (from the example request)—
Yes / noShould the refund be reviewed?yes · no
ScoreUrgency (3 levels)—

How it works

  1. Install Open-Jev and download Open-Jev-27B-v1.1.
  2. Start open-jev-fast in place of python -m jev.server.
  3. Send the same requests; shared prompt text is computed once.
  4. Receive candidate probabilities with lower latency.

Strengths

  • About 6x lower JevBench latency per the README.
  • Same model and request format; no quantization.
  • Shared-prefix computation for multi-question requests.

Caveats

  • Needs high-end NVIDIA GPUs (tested on B300).
  • Depends on Open-Jev installation and weights.
  • Tiny accuracy difference from numeric changes.

At a glance

Type
Model
Status
research
Language
Python
Runtime
CUDA, local HTTP server
Has code
yes

Use cases and integrations

How curated lists describe it

  • Faster inference backend for Open-Jev-27B: fused CUDA kernels, a prefix tree and CUDA graphs. Needs an Open-Jev install and its weights.

    MrJev/awesome-jev

Frequently asked questions

Is it a new model?

No, it runs the same Open-Jev-27B-v1.1 model faster.

How much faster is it?

The README reports 42.0 ms mean JevBench latency versus 258 ms with FLA kernels.

Does accuracy change?

197/231 versus 198/231, due to one near-tie prediction flipping.

Related projects

  • go-system-one

    Native Go runtime serving Gemma 4 12B as a Jev-like decision model with hand-tuned PTX kernels and no CGo or CUDA toolkit.

    ProjectbetaGo
  • hearim

    Go gateway exposing the Jev /v1/systemone contract over ordinary LLM backends by reading single-token label logprobs.

    ServicebetaGo
  • llm2jev

    Turns any chat model on SGLang, vLLM, transformers or MLX into a Jev-compatible service with one prefill and one label token per question.

    ProjectbetaPython
  • verdict

    Self-hosted Jev-compatible System One server that sits in front of llama-server and reads option probabilities from any GGUF model.

    ServicebetaPython
  • jevinf

    Inference engine for Jev-style decision models using segmented forwards with prefix reuse, serving the Jev wire contract.

    ProjectresearchPython
  • dfip

    Research preview: Python client and pinned adapters for running System One decision models on your own Modal GPUs.

    LibraryresearchPython