jev-acento
Pre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.
Overview
jev-acento is an independent, reproducible audit of how Jev handles Spanish. In run 20260921-es-v1 (19,200 calls over 3,200 paired items on version-pinned jev-1.13.0, costing USD 0.58 with no errors) it compares English state, Spanish state, and Spanish state with Spanish instructions on XNLI, PAWS-X, MASSIVE and Belebele. All figures are generated from results.json by make reproduce.
Findings: a Spanish state lowers accuracy on every dataset, by 3.0 to 6.4 percentage points, and roughly doubles ECE on XNLI (0.057 to 0.101) and PAWS-X (0.033 to 0.078). Operationally, automating at a maximum probability of 0.9 or more covers 72.2% of XNLI in English but 63.4% in Spanish, with lower accuracy on the automated items. Writing instructions in Spanish makes no detectable difference on three of four datasets, so the advice is to keep instructions and criteria in English. Spanish text also costs 17-38% more input tokens. A CLI lets anyone run the same paired comparison on their own labelled data.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Does a Spanish state or Spanish instructions change Jev's accuracy and calibration? | — |
How it works
- Pre-register comparisons and thresholds
- Run paired English/Spanish variants on four datasets
- Compute accuracy, ECE and coverage at probability cut-offs with confidence intervals
- Publish results reproducibly; CLI reruns on user data
Strengths
- Pre-registered, paired design with confidence intervals
- Fully reproducible from committed results
- Actionable advice: keep instructions in English
- Very cheap to rerun (USD 0.58 for the full run)
Caveats
- Only Spanish and four datasets
- Specific to jev-1.13.0
At a glance
- Type
- CLI
- Status
- research
- Language
- Python
- Runtime
- CLI
- Has code
- yes
Use cases and integrations
How curated lists describe it
Language evaluation: pre-registered paired audit of Jev on Spanish over 3,200 human-labelled items, finding that a Spanish `state` costs 3.0-6.4 pp of accuracy and roughly doubles ECE on XNLI and PAWS-X while writing `instructions` in Spanish changes nothing, and shipping a CLI to rerun the same comparison on your own labelled data.
yibie/awesome-jev
Frequently asked questions
Is Jev worse in Spanish?
The audit finds 3.0-6.4 points lower accuracy with a Spanish state and worse calibration on two datasets.
Should I write instructions in Spanish?
No measurable benefit was found; the README recommends keeping instructions and criteria in English.
Can I test my own language or data?
Yes, the CLI reruns the same paired comparison on your labelled data.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPythonPre-registered independent evaluation of Jev on intent classification and confidence-gated cascades, with a transparent erratum.
LibraryresearchPython