Reflex
GGUF-native Rust and CUDA engine tuned for cold starts, with a system1 command that scores candidate answers in one forward pass.
Overview
Reflex from lateos-ai is a GGUF-native inference engine in Rust with CUDA kernels compiled ahead of time and shipped in the binary, so nothing is JIT-compiled at launch. It targets cold-start latency, the time from process launch to first result, which matters on serverless GPU platforms that bill per second for bursty single-shot work like a classification or tool call.
Its system1 subcommand scores a list of candidate continuations against a prompt in one forward pass, which is the Jev-style use: pick among fixed options rather than generate. The README reports about 483 ms p50 cold start for reflex system1 on a Tesla T4 with Qwen3-0.6B-Q4_K_M (about 0.6 s wall clock), and compares against llama.cpp and a vLLM worker on Runpod. Against TypeSafe's always-warm Jev API, it reports Reflex is 1.1-2.0x slower per call. It is single-tenant by design: one request at a time, no batching, and HTTP only through a separate sidecar.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which candidate continuation best completes the prompt? | — |
How it works
- Build the binary with nvcc available
- Launch reflex system1 with a GGUF model, prompt and --candidate options
- Score all candidates in one forward pass
- Return scored candidates; process exits
Strengths
- About 483 ms p50 cold start on a T4 per the README
- No JIT at startup
- Suited to per-invocation serverless billing
Caveats
- Needs an NVIDIA GPU and CUDA toolkit
- Single request at a time, no batching
- Slower than warm hosted Jev per call per the README
Getting started
git clone https://github.com/lateos-ai/reflex.git && cd reflex && cargo build --release && cargo run --release --bin reflex -- system1 model.gguf "prompt" --candidate " A" --candidate " B"At a glance
- Type
- Model
- Status
- beta
- Language
- Rust
- Runtime
- CLI, CUDA
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
GGUF-native Rust and CUDA engine built for cold-start latency, with a `system1` command that scores your candidates from a local model.
MrJev/awesome-jev
Frequently asked questions
What is the system1 command?
It scores candidate answers to a prompt in one forward pass instead of generating text.
Is it faster than Jev?
Not per call; the README says it is 1.1-2.0x slower than warm hosted Jev because it starts a process each time.
Why focus on cold start?
Serverless GPU platforms bill wall-clock time per invocation, so startup time is the cost for bursty work.
Related projects
Native Go runtime serving Gemma 4 12B as a Jev-like decision model with hand-tuned PTX kernels and no CGo or CUDA toolkit.
ProjectbetaGoGo gateway exposing the Jev /v1/systemone contract over ordinary LLM backends by reading single-token label logprobs.
ServicebetaGoTurns any chat model on SGLang, vLLM, transformers or MLX into a Jev-compatible service with one prefill and one label token per question.
ProjectbetaPythonSelf-hosted Jev-compatible System One server that sits in front of llama-server and reads option probabilities from any GGUF model.
ServicebetaPythonOpenAI-compatible local server pairing a small llama.cpp model for generation with Jev for the closed decisions.
ServiceprototypePythonReads candidate logits from existing MLX-LM/VLM models on Apple Silicon to map state and a request to an allowed action.
ProjectprototypePython