Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

minicpm5-jev

Serves MiniCPM5-2B-8bit on Apple Silicon as a Jev-compatible /v1/systemone service and reproduces its JevBench score.

ModelprototypeMIT
View on GitHub ↗

Overview

minicpm5-jev wraps the mlx-community 8-bit build of MiniCPM5-2B as a Jev-compatible /v1/systemone decision service using the llm2jev adapter, then runs the official jevbench harness against it. The goal is an honest, reproducible number for a small local model rather than a new architecture.

The headline result reported in the README is 0.580 accuracy (95% CI 0.516-0.642) on the 231 public JevBench items on an Apple M1 Max, compared with 0.866 for TypeSafe's hosted Jev 1.13.0 on the same items. Scripts fetch the pinned model revision, serve it on localhost:8080 and run the full benchmark, regenerating RESULTS.md. The example request shows a customer-intent choice returning per-option probabilities and a confidence value. It suits Mac users who want a local Jev-shaped endpoint and anyone checking how far an untuned small model gets.

What the decision model decides

TypeQuestionAnswers
ChoiceWhat does the customer want?track_order · report_damage · cancel_order

How it works

  1. git clone --recurse-submodules and uv sync.
  2. bash fetch_model.sh downloads the pinned 8-bit model (~2.5 GB).
  3. bash serve.sh exposes /v1/systemone on port 8080.
  4. bash bench.sh runs all 231 JevBench items and writes RESULTS.md.

Strengths

  • Local, Jev-compatible endpoint on a Mac with no API key.
  • Reproducible benchmark run with confidence intervals.
  • MIT licensed with clearly credited upstream components.

Caveats

  • Accuracy is far below hosted Jev (0.580 vs 0.866 per the README).
  • Apple Silicon only (MLX backend).
  • No fine-tuning; it is a serving and measurement wrapper.

Getting started

git clone --recurse-submodules https://github.com/u007/minicpm5-jev && cd minicpm5-jev && uv sync && bash fetch_model.sh && bash serve.sh

At a glance

Type
Model
Status
prototype
Language
Python
Runtime
MLX on Apple Silicon, HTTP server
Has code
yes
License
MIT

Use cases and integrations

How curated lists describe it

Frequently asked questions

What does it serve?

MiniCPM5-2B-8bit behind the /v1/systemone typed-decision contract on localhost.

How accurate is it?

The README reports 0.580 on the 231 public JevBench items, against 0.866 for hosted Jev 1.13.0.

What hardware do I need?

An Apple Silicon Mac, uv, and about 2.5 GB of disk for the weights.

Related projects

  • systemone-lite

    Toy local Jev-shaped decision API on a 0.5B model that answers yes/no, choice and score questions by scoring option tokens.

    ServiceprototypePython
  • jiwo

    Small open decision models (0.8B and 4B, from Qwen3.5) with a server that speaks Jev's /v1/systemone format in one forward pass.

    ServicebetaPython
  • reflex★ 166

    Small open decision model on a frozen Qwen3.5-4B serving /v1/systemone in about 200 ms, benchmarked against Jev on JevBench.

    ServiceresearchPython
  • Carve-Jev

    Open Apache-2.0 Jev-like model family on Qwen3.8-27B that reads one-token candidate preferences and serves a TypeSafe-compatible API.

    ServiceresearchPython
  • minicpm5-2b-jev

    Open MiniCPM5-2B decision model serving /v1/systemone: typed choice, noul and score probabilities in one forward pass.

    ModelresearchPython
  • local-jev-api

    Docker recipes and ARM64 images for serving ten native decision checkpoints, such as Cloudflare Clef, behind /v1/systemone on DGX Spark.

    ServicebetaPython