Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

laya.cpp

Native C++ Laya inference on ggml with CUDA, Vulkan and Core ML, plus a Jev-compatible HTTP server with automatic batching.

Servicebeta
View on GitHub ↗

Overview

laya.cpp runs Laya typed decisions natively in C++ on ggml, with CUDA, Vulkan and Apple Core ML backends. Tokenisation, inference and JSON output need no Python. It supports Laya's English, multilingual and typed-decisions models and includes a Jev-compatible HTTP server that batches requests automatically. Binary releases provide Windows and Linux x64 CUDA and Vulkan executables and a macOS arm64 Core ML executable.

The README reports paired throughput comparisons against matching-precision Python on 250 fixed questions across all three models and batch sizes 1 to 8: optimised CUDA FP32 at 1.35 to 2.48 times Python on an RTX PRO 6000, and Vulkan results that range from slower to faster depending on precision and GPU. All answer checks pass with exact categories and numeric error at most 0.0001. Building requires C++20, CMake, ICU and nlohmann-json plus the chosen backend's toolchain. It suits deployments wanting Laya without a Python runtime.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich option fits the state?—
ScoreWhere does the state fall on the scale?—
Yes / noIs this statement true of the state?—

How it works

  1. Download a prebuilt binary or build with CMake for your GPU backend
  2. Load a Laya model
  3. Serve the Jev-compatible HTTP API
  4. Requests are batched automatically and answered with probabilities

Strengths

  • No Python at runtime
  • Multiple GPU backends including Vulkan and Core ML
  • Answer parity with Python checked
  • Prebuilt binaries

Caveats

  • Vulkan can be slower than Python in some modes
  • Build needs several native dependencies
  • Core ML build requires full Xcode

Getting started

Download a release binary, or git submodule update --init --recursive and build with cmake for CUDA, Vulkan or Core ML.

At a glance

Type
Service
Status
beta
Language
C++
Runtime
native binary with HTTP server (ggml)
Has code
yes
Has tests
yes

Use cases and integrations

How curated lists describe it

  • Native C++ inference for Laya on ggml with CUDA, Vulkan, and Core ML backends, plus a Jev-compatible HTTP server with automatic request batching and prebuilt binaries

    AnotiaWang/awesome-decision-models

Frequently asked questions

Is it faster than Python Laya?

On CUDA, 1.35 to 2.71 times in the README's tests; Vulkan varies.

Are answers identical?

Checks pass with exact categories and numeric error at most 0.0001.

Can I use a Jev client?

Yes, the HTTP server is Jev-compatible.

Related projects

  • decidealot

    Docker service exposing local Laya, Von, a CLM head or hosted Jev through one TypeSafe-compatible HTTP and MCP endpoint.

    MCP serverbetaPython
  • edgejev

    Converts open Jev reproductions (Laya, Kev, NanoJev, PlayJev) to INT8 ONNX for offline CPU decisions at about 15.6 ms per question.

    ServicebetaPython
  • go-system-one

    Native Go runtime serving Gemma 4 12B as a Jev-like decision model with hand-tuned PTX kernels and no CGo or CUDA toolkit.

    ProjectbetaGo
  • gojev

    Go harness for System One models: hosted Jev, open Kev in-process via llama.cpp, and Cloudflare Clef, with typed helpers and a CLI.

    ServicebetaGo
  • hearim

    Go gateway exposing the Jev /v1/systemone contract over ordinary LLM backends by reading single-token label logprobs.

    ServicebetaGo
  • intern-decision-mlx

    Runs Intern-Decision-0.8B on Apple Silicon behind a local /v1/systemone endpoint that accepts screenshots for agent decisions.

    ServicebetaPython