systemANE
Two-tier macOS decision engine: a MiniLM encoder on the Neural Engine decides, escalating unsure cases to Apple's on-device model.
Overview
systemANE is a proof of concept that runs typed decisions on the Apple Neural Engine. Tier 1 is a 22.6M-parameter MiniLM-L6 sentence encoder compiled to Core ML with no fine-tuning; it encodes the input once and answers each typed question with a dot product against calibrated anchors. When tier 1 is not confident, tier 2, Apple's on-device Foundation Model served by the fm CLI in macOS 27, answers instead.
The author's framing is that tier 1 is not smart but is cheap enough to run on everything and, once calibrated, honest about when it should not decide. The README reports 1.4 ms per decision with 94.6% of graph ops on the ANE, verified with Core ML's compute planner, and on CLINC150 reports 93.3% accuracy, ECE 0.041 and 0.994 AUROC for out-of-scope detection. It also measures fp16 drift and serving throughput. It suits Mac developers who want free, local, very low-latency classification with an escalation path.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which intent or category does this input belong to (or is it out of scope)?(inferred) | — |
| Score | How urgent is this ticket? | — |
| Yes / no | Is tier 1 confident enough to decide, or should it escalate to tier 2?(inferred) | — |
How it works
- Encode the input once with MiniLM on the Neural Engine.
- Score each typed question by dot product against anchors.
- Compare the margin with calibrated thresholds.
- If below threshold, ask Apple's Foundation Model via fm serve.
Strengths
- Very low latency and no per-call cost; runs fully on-device.
- Calibrated confidence decides when to escalate.
- ANE placement and fp16 drift are measured, not assumed.
Caveats
- Proof of concept; requires macOS 27 for tier 2.
- Tier 1 is untuned and relies on tier 2 for hard cases.
- No licence stated in the README.
Getting started
python3.12 -m venv .venv && .venv/bin/pip install -r requirements.txt, then run the repo's serve.py.At a glance
- Type
- Project
- Status
- prototype
- Language
- Python
- Runtime
- macOS (Core ML / Apple Neural Engine)
- Has code
- yes
Use cases and integrations
How curated lists describe it
Using Apple's Neural Engine as a fast and free local "System One" Decision Engine (macOS 27) - "Honey, we have Jev at home".
hellogumbo/awesome-jev
Frequently asked questions
What runs on the Neural Engine?
A MiniLM-L6 sentence encoder; the README reports 157 of 166 ops on the ANE.
What happens when tier 1 is unsure?
The input escalates to Apple's on-device Foundation Model, which is slower but more accurate.
Does it need a TypeSafe key?
No, both tiers run locally on the Mac.
Related projects
Japanese System One decision model on ModernBERT-ja (314.6M) returning typed choice, score and bool distributions with no generation.
ProjectbetaPython- cbjev★ 4
Self-hosted encoder model giving Jev-compatible choice, score and noul answers in one pass, pitched as a better-calibrated Laya successor.
ProjectresearchPython 11 MB CPU-only model that answers choice, noul and score questions with fitted calibration and no LLM at inference.
ProjectresearchPythonOEV: 184M-parameter decision engine that scores options in one pass, reporting 0.7705 on typed-decisions and 22 ms per question.
ProjectresearchPythonLocal System One-compatible server that reads option probabilities from a small MLX LLM, with measured calibration.
ProjectresearchPythonFastino's 340M Apache-2.0 decision classifier that scores caller-defined labels in one forward pass for routing, intent and scoring.
ModelproductionPython