jev-report
Independent 52-page Chinese report on Jev that checks vendor claims and reproduces 50 Chinese tests, with traceable data.
This repository is mostly documentation or write-up rather than runnable code.
Overview
This independent Chinese-language report, a free 52-page PDF with 10 figures and 143 numbers traceable to their sources, examines Jev: who measured the vendor's headline figures, how, and whether they hold up when tested in Chinese. Every number in the text maps to a record in a data table graded by source tier (vendor docs, vendor evals, independent tests, community, this report's own tests, news) and marked verified only if the author read the figure in the original.
Its findings include that the vendor's 444.6x speed figure is a peak while independent tests measured 5-25x; that 'zero hallucination' means zero type errors, not zero wrong choices; and that a widely quoted 67.8% score is graded against the average of two frontier models, not humans. Its own 50 Chinese samples yielded 86 of 90 correct judgments (95.6%) with ECE 0.070, and at a 0.80 threshold all auto-passed answers were correct while the review band contained every error. It also offers a one-page brief to give a coding agent and five questions to ask about any fast cheap model. Code is MIT and text CC BY-NC-SA 4.0.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Chinese test items: yes/no judgments on self-written material(inferred) | yes · no |
| Choice | Chinese test items: choice judgments on self-written material(inferred) | — |
How it works
- Trace each vendor figure to its source and method
- Lock gold labels for 50 Chinese samples before testing
- Run each sample four times against Jev
- Compute accuracy, ECE and threshold behaviour
- Publish the PDF, data table and raw responses
Strengths
- Every number traceable to a source tier
- Reproduction pack costs about half a cent
- Practical questions and agent hand-off page
Caveats
- Single annotator and self-written samples
- Latency measured from Manila
- Mostly a written report
Getting started
cd data/中文实测, export TYPESAFE_API_KEY, then python3 run.py && python3 analyze.pyAt a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Report plus reproduction scripts
- Has code
- yes
- License
- MIT (code), CC BY-NC-SA 4.0 (text)
Use cases and integrations
How curated lists describe it
发明 RLHF 的人，这次做了个不会说话的模型：Jev 独立研究报告。52 页 PDF + 50 条中文实测复现包 + 143 条可回溯数据表.
hellogumbo/awesome-jev
Frequently asked questions
How accurate was Jev in Chinese?
86 of 90 judgments correct (95.6%) on 50 self-written samples, with ECE 0.070.
Are the vendor's speed claims accurate?
The report finds the 444.6x figure is a peak; independent tests measured 5-25x.
Can I reproduce it?
Yes. The scripts make 200 calls in about four minutes for about $0.005.
Related projects
Pre-registered audits of AI judges that measure how many decisions a judge can take alone at a given error rate, with calibration.
BenchmarkbetaPythonBenchmark of Jev on 159 community-reconstructed questions from Indonesia's SNBT 2025 university entrance exam, all responses published.
BenchmarkresearchPythonpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonLocal SvelteKit workbench and CLI for organizing, editing, running and comparing Jev experiments defined as JSON.
CLIbetaTypeScriptAgent skill and CLI that chains Jev question design: find decision calls, check fit, draft, lint, calibrate thresholds and plan a cascade.
Agent skillprototypePython