Jev-Persian-Benchmark
Benchmarks Jev against local open decision models on 480 general Persian questions, literary questions and a classical poetry pilot.
Overview
Jev Persian Benchmark measures decision models on Persian-language questions: 480 authored general questions, 560 literary questions and a 24-excerpt classical poetry pilot with controls. Related general questions are batched while poetry questions run individually; raw responses are saved and scored locally without a model judge.
On the general set, with identical inputs, the README reports Jev at 99.6% choice accuracy, 99.4% noul accuracy and 95% of scores within half a rubric level, with the lowest choice Brier score. Local alternatives were run on an Apple M4: Jeff (Qwen3.5 0.8B and 2B on MLX) at 85.8-93.3% choice accuracy, Mizan v3 3B at 97.1% choice and best score accuracy, and Laya multilingual at 58.75%. Diagnostics cover matched pairs, sensitivity to English instructions and repeatability. Jev scored 24/24 on both poetry meaning recognition and situation matching.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which option correctly answers this Persian question? | — |
| Yes / no | Is this Persian statement true? | yes · no |
| Score | Which rubric level fits? | — |
How it works
- Author Persian questions with answer keys
- Run each model with identical inputs
- Score answers locally (exact choice, noul threshold, score within 0.5)
- Compute Brier and MAE, plus diagnostics
Strengths
- Covers a non-English language with local alternatives
- No model judge; local scoring
- Detailed reproduction guides per model
Caveats
- Authored questions may be easier than real-world text
- Small poetry pilot (24 excerpts)
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Benchmarks for \*\*Jev\*\* on \*\*480 authored general Persian questions\*\* and a \*\*24-excerpt classical Persian poetry pilot\*\* (48 main questions plus 48 controls). Related general questions are batched; poetry questions run individually. Raw responses are saved and answers are scored locally, without a runtime model judge. The general benchmark's historical Laya comparison is retained below.
logicrw/awesome-jev-projectsA Quick Typesafe.
wh000wh000/awesome-jev-live
Frequently asked questions
How did Jev do on Persian?
About 99.5% on choice and noul questions in the general set, per the README.
Which local model came closest?
Mizan v3 3B, with 97.1% choice accuracy and the best score accuracy.
Is a model used to grade answers?
No. Answers are scored locally against keys.
Related projects
Apple Silicon workbench running Laya, Decider, Kev and Jev side by side with accuracy, calibration, latency and memory benchmarks.
BenchmarkbetaPythonLocal benchmark comparing Jev with Laya-MLX on classifying GitHub issues for beginner-friendliness, CV fit and difficulty.
BenchmarkresearchPythonSpeed and accuracy benchmark of hosted Jev 1.13 versus local Laya 421M on 1,470 synthetic items across eight decision tasks.
BenchmarkresearchPythonTutorial to fine-tune Qwen3.5-4B into a Jev-like classifier with Unsloth on RunPod, serve via vLLM and benchmark against Jev.
BenchmarkresearchPythonDocker recipes and ARM64 images for serving ten native decision checkpoints, such as Cloudflare Clef, behind /v1/systemone on DGX Spark.
ServicebetaPythonToy local Jev-shaped decision API on a 0.5B model that answers yes/no, choice and score questions by scoring option tokens.
ServiceprototypePython