taiwan-sovereignty-decision-models
Study finding commercial decision models including Jev give different yes/no probabilities on Taiwan statements by Chinese script.
Overview
Same Statement, Different Script is an independent research paper with data and code. The author wrote 28 paired statements about Taiwan's status, half aligned with Taiwan's self-governance and half with the position of the People's Republic of China, and asked four decision models (OpenAI Decisions API, TypeSafe Jev, Cloudflare Clef and Clef-flash) whether each is true, in Traditional Chinese, Simplified Chinese and English, three times each, with Claude Haiku as a general LLM control. That came to more than 18,000 queries.
The paper reports that writing the same sentence in Simplified characters shifted all four decision models toward the PRC position, while control facts about other countries did not shift and the chat-model control barely moved. Without a persona, Jev, Clef and Clef-flash never gave more than 0.11 to "Taiwan is a country". The author argues that such buried judgments matter for moderation and routing, and advises testing in all three scripts. A full rerun cost under US$1 in October 2026.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this statement about Taiwan true? | — |
How it works
- Write paired statements from both political positions.
- Render each in Traditional Chinese, Simplified Chinese and English.
- Ask each model for the probability the statement is true, three times.
- Compare probabilities across scripts against control statements and an LLM control.
Strengths
- Reproducible with a single script and small cost.
- Controls separate script effects from general noise.
- Paper available in English and Traditional Chinese.
Caveats
- Covers one topic and 28 statements.
- Models change over time; results describe the versions tested.
Getting started
pip install -r requirements.txt, add provider keys to .env, then python run.py luna jev clef clef-flash haiku.At a glance
- Type
- Service
- Status
- research
- Language
- Python
- Runtime
- scripts
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Script-dependent answers about Taiwan in commercial decision models (OpenAI Decisions API, TypeSafe Jev, Cloudflare Clef): data, code, paper.
hellogumbo/awesome-jev
Frequently asked questions
What did the study find?
All four decision models leaned toward the PRC position when the same statement was written in Simplified rather than Traditional Chinese.
Which models were tested?
OpenAI's Decisions API, TypeSafe Jev, Cloudflare Clef and Clef-flash, with Claude Haiku as a control.
Can I rerun it?
Yes, with provider keys in .env; the README says a full run cost under US$1.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonAudit of 300 web pages for AI citability with nine Jev decisions each, checked against an LLM judge for agreement, cost and latency.
ProjectresearchPython