my-jev
Tutorial to fine-tune Qwen3.5-4B into a Jev-like classifier with Unsloth on RunPod, serve via vLLM and benchmark against Jev.
Overview
my-jev is a step-by-step project for building your own Jev-like System One classifier. The model takes a state, a question and a list of options and returns one letter plus a probability for every option. Qwen3.5-4B is fine-tuned with LoRA via Unsloth on a 37,840-example dataset on a RunPod GPU, published to Hugging Face as Prompt48/my-jev-4b, served with vLLM and compared with TypeSafe Jev on accuracy, calibration, latency and cost.
The README reports 85.8% on its held-out test versus 80.6% for Jev, but 70.8% overall versus Jev's 75.5% once third-party phishing, tool-risk and ticket-routing sets are included, with ECE 0.212 versus 0.111. Phishing accuracy dropped after fine-tuning because the training data had no security tasks. Training took 4 h 40 min on an A40 for about $3 total. It is useful for developers learning the full train-serve-benchmark loop.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which option answers the question for this state? | — |
How it works
- Set up a 24 GB+ RunPod pod and build the dataset.
- Fine-tune with LoRA (loss on the answer letter only) and score before and after.
- Push merged weights and adapter to Hugging Face.
- Serve with vLLM and benchmark against Jev.
Strengths
- Complete, low-cost recipe (about $3 per the README).
- Honest comparison including where the fine-tune is worse.
- Optional GGUF builds for Ollama or llama.cpp.
Caveats
- Worse calibration than Jev in the reported results.
- Generalisation drops on task types missing from training data.
- Needs a GPU pod and Hugging Face write token.
Getting started
On a RunPod pod: clone the repo, cp .env.example .env, bash setup_pod.sh, then bash train.sh --max_train 200 as a smoke test.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Unsloth, vLLM on GPU
- Has code
- yes
Use cases and integrations
How curated lists describe it
Build your own Jev-like System One classifier: fine-tune Qwen3.5-4B with Unsloth on RunPod, serve with vLLM, benchmark vs TypeSafe Jev.
hellogumbo/awesome-jev
Frequently asked questions
How does my-jev-4b compare with Jev?
Per the README, better on its own held-out set (85.8% vs 80.6%) but worse overall (70.8% vs 75.5%) and less calibrated.
How much does training cost?
The README reports about 4 h 40 min on one A40, roughly $3 for the pod session.
Do I need a TypeSafe key?
Only to run the comparison benchmark against Jev.
Related projects
Apple Silicon workbench running Laya, Decider, Kev and Jev side by side with accuracy, calibration, latency and memory benchmarks.
BenchmarkbetaPythonLocal benchmark comparing Jev with Laya-MLX on classifying GitHub issues for beginner-friendliness, CV fit and difficulty.
BenchmarkresearchPythonSpeed and accuracy benchmark of hosted Jev 1.13 versus local Laya 421M on 1,470 synthetic items across eight decision tasks.
BenchmarkresearchPythonBenchmarks Jev against local open decision models on 480 general Persian questions, literary questions and a classical poetry pilot.
BenchmarkresearchPythonDocker recipes and ARM64 images for serving ten native decision checkpoints, such as Cloudflare Clef, behind /v1/systemone on DGX Spark.
ServicebetaPythonToy local Jev-shaped decision API on a 0.5B model that answers yes/no, choice and score questions by scoring option tokens.
ServiceprototypePython