SOLO
Input layout optimizer that reorders rows and fields so a prefix-caching decision model judges every record in a big table faster.
Overview
SOLO (System One Layout Optimizer) runs a natural-language decision over every record in a dataset, such as asking 'Does the request ask for a monetary refund?' of each support ticket, and arranges rows and fields so a prefix-caching backend can reuse more input computation, then returns decisions in the original order. The backend still receives the complete record on every request. Inputs can be pandas DataFrames, NumPy arrays, JSON or JSONL.
The included backend serves AutoTrust JEV-9B, an independent open reproduction of Jev's decision behaviour built on Qwen3.5-9B, through vLLM with prefix caching, returning binary, categorical or scored decisions with probabilities; a VllmJevBackend also targets the open vllm-jev framework and Open-Jev-2B. The optimizer works with your own backend too. On one RTX 4090 the README reports up to 11.7x throughput over original order with long shared context and 3.28x over NDV sorting with correlated fields, on reproducible synthetic workloads. An offline example runs the optimizer with no GPU or network.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Does the request ask for a monetary refund? (per row) | — |
| Choice | Categorical decisions per record | — |
How it works
- Load a table or JSON records
- Reorder rows and fields to maximise shared prefixes
- Send each full record with the question to the prefix-caching backend
- Return decisions and probabilities in original order
Strengths
- Up to 11.7x throughput on synthetic workloads per the README
- Works with any prefix-caching backend
- Offline optimizer demo
Caveats
- Gains depend on shared context in the data
- GPU needed for the JEV-9B backend
Getting started
python -m pip install ".[pandas,hub]", then DecisionEngine().scan(df, "question") against a running JEV-9B service.At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Client library + vLLM backend
- Has code
- yes
Use cases and integrations
How curated lists describe it
Per-row semantic judgements over large relational tables on a decision model (Jev).
hellogumbo/awesome-jevReorders rows and fields so a prefix-caching backend reuses more work when a decision model judges every record in a large table; ships a vLLM backend for the open AutoTrust JEV-9B model and works with your own.
MrJev/awesome-jev
Frequently asked questions
What does SOLO optimise?
The order of rows and fields so a prefix-caching decision model reuses more computation.
Which model does it use?
AutoTrust JEV-9B via vLLM by default, or your own backend.
Does it truncate records?
No; the backend receives the complete record each time.
Related projects
Native Go runtime serving Gemma 4 12B as a Jev-like decision model with hand-tuned PTX kernels and no CGo or CUDA toolkit.
ProjectbetaGoGo gateway exposing the Jev /v1/systemone contract over ordinary LLM backends by reading single-token label logprobs.
ServicebetaGoTurns any chat model on SGLang, vLLM, transformers or MLX into a Jev-compatible service with one prefill and one label token per question.
ProjectbetaPythonSelf-hosted Jev-compatible System One server that sits in front of llama-server and reads option probabilities from any GGUF model.
ServicebetaPythonOpenAI-compatible local server pairing a small llama.cpp model for generation with Jev for the closed decisions.
ServiceprototypePythonPython batching proxy that asks the same Jev questions about many states and streams answers back as Apache Arrow tables.
ProxyprototypePython