System One Playground
Apple Silicon workbench running Laya, Decider, Kev and Jev side by side with accuracy, calibration, latency and memory benchmarks.
Overview
System One Playground is a local workbench for Apple Silicon Macs that runs several decision models side by side: Laya, Decider, Kev and, optionally, TypeSafe's Jev through a proxy. Every engine speaks TypeSafe's /v1/systemone API, so requests written for Jev work against all of them. A tabbed playground lets users build requests with a visual question builder or raw JSON and compare answers across engines.
The project is for developers choosing among System One models who want measured numbers rather than claims. It includes public-dataset benchmarks for accuracy and calibration, input-truncation diagnostics, latency, CPU and memory measurements and load tests. Local engines run on the Apple GPU via MPS or MLX in a sandbox with outbound network denied, which the tests verify. A one-command quickstart installs isolated environments per engine, downloads about 6 GB of small models and opens the UI.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which team should handle this ticket? | billing · technical |
| Score | How urgent is it? | low · medium · high |
| Yes / no | Is the customer threatening to leave? | — |
How it works
- Run make quickstart to install engines, models and datasets.
- Start one or more engines and compose a request in the playground.
- Each engine returns probabilities through the same /v1/systemone API.
- Run benchmarks to compare accuracy, calibration, latency and memory.
Strengths
- Same API across engines makes comparisons direct.
- Everything local except the optional Jev engine.
- Sandboxed local engines with network access denied.
Caveats
- Requires an Apple Silicon Mac, uv and Node 20+.
- All models together need about 37 GB of weights.
- Jev comparison needs a TypeSafe key.
Getting started
git clone https://github.com/goodboybeau/system-one-playground && cd system-one-playground && make quickstartAt a glance
- Type
- Benchmark
- Status
- beta
- Language
- Python
- Runtime
- local web app on macOS (MPS/MLX)
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Local Apple Silicon workbench comparing Laya, Decider, Kev, Jev, and other engines side by side, with public-dataset benchmarks for accuracy and calibration, input-truncation diagnostics, latency, memory, and load tests
AnotiaWang/awesome-decision-models
Frequently asked questions
What does it compare?
Decision models such as Laya, Decider, Kev and Jev on accuracy, calibration, latency, CPU and memory using public datasets.
Does it need a cloud service?
No, local engines run on the Apple GPU; only the optional Jev engine calls TypeSafe.
How do I start?
Clone the repo and run make quickstart; make doctor reports missing requirements.
Related projects
Local benchmark comparing Jev with Laya-MLX on classifying GitHub issues for beginner-friendliness, CV fit and difficulty.
BenchmarkresearchPythonSpeed and accuracy benchmark of hosted Jev 1.13 versus local Laya 421M on 1,470 synthetic items across eight decision tasks.
BenchmarkresearchPythonBenchmarks Jev against local open decision models on 480 general Persian questions, literary questions and a classical poetry pilot.
BenchmarkresearchPythonTutorial to fine-tune Qwen3.5-4B into a Jev-like classifier with Unsloth on RunPod, serve via vLLM and benchmark against Jev.
BenchmarkresearchPythonDocker recipes and ARM64 images for serving ten native decision checkpoints, such as Cloudflare Clef, behind /v1/systemone on DGX Spark.
ServicebetaPythonToy local Jev-shaped decision API on a 0.5B model that answers yes/no, choice and score questions by scoring option tokens.
ServiceprototypePython