parallelConstraintDecoding
Java (llama.cpp via FFM) and Python engines that fill boolean/enum JSON schemas in two forward passes with per-field confidence.
Overview
parallelConstraintDecoding fills a whole JSON schema of booleans and enums in two forward passes instead of generating it token by token, with calibrated confidence per field and schema-valid output by construction. The Java implementation drives llama.cpp directly through the JDK Foreign Function and Memory API without JNI, loads any GGUF including local Ollama models, and ships a web app that races it against grammar-constrained token-by-token generation and charts benchmarks; it was demonstrated at Devoxx.
A Python port of the original MLX engine is kept for comparison; the README clarifies that the Hugging Face repo named Qwen-2.5-1B-RLCD contains no weights or fine-tune, just stock Qwen2.5-1.5B-Instruct with custom decoding. Presets include fraud/AML triage, code-security triage, support triage, a 255-choice tariff router, spam and phishing triage, and conference talk routing. On an M1 Ultra, 28-field presets take about 266-277 ms cold and about 130 ms warm in Java, against about 2.4 s for token-by-token JSON.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which category for each enum field in the schema (e.g. fraud risk, incident type)? | — |
| Yes / no | Boolean fields such as requires_review | true · false |
How it works
- Load a GGUF model in the Java engine.
- Prefill the context, then evaluate all fields in parallel.
- Read per-field probabilities and confidence.
- Optionally keep schema KV cells warm between calls.
Strengths
- Large speedups over token-by-token JSON.
- Works with any GGUF, including Ollama models.
- Web app for side-by-side races and custom scenarios.
Caveats
- Only for boolean and enum fields.
- Python engine is Apple Silicon only.
At a glance
- Type
- Project
- Status
- research
- Language
- Java
- Runtime
- llama.cpp (FFM API), web app
- Has code
- yes
Use cases and integrations
How curated lists describe it
Java and llama.cpp experiments in parallel constrained decoding.
cobanov/awesome-jev
Frequently asked questions
How many forward passes does it need?
Two, regardless of the number of fields, per the README.
Does it use JNI?
No, it uses the JDK Foreign Function and Memory API.
Is the RLCD model fine-tuned?
No, the README says the referenced repo runs stock Qwen2.5-1.5B-Instruct.
Related projects
Hugging Face Space for parallel constrained decoding on Apple Silicon: fills multi-field JSON schemas in parallel with field confidence.
ModelresearchPythonllama.cpp-based C++ server that turns any LLM into a System One decision model, reading answers from next-token logits.
ProjectbetaC++Python library that imitates Jev-style typed decisions (choice, score, yes/no) on local LLMs via Ollama, vLLM or llama.cpp.
ProjectprototypePythonMLX library that turns an existing local model into a decision engine by reading candidate logits, with no fine-tuning, on Apple Silicon.
ProjectprototypePythonTurns any local LLM into a Jev-style engine via masked-logit softmax, or proxies OpenAI-compatible APIs or Jev; includes a chat crush radar.
ServiceprototypePythonLocal server (xks) implementing Jev's /v1/systemone contract on an open-weight LLM, accelerated with Xeon Phi cards.
ProjectresearchRust