Jev vs GPT-4.1 on a synthetic survey
Pre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
Overview
This independent study asks whether a model that returns a native probability vector makes a better synthetic survey respondent than one asked to state probabilities in words. Jev (jev-1.13.0) and GPT-4.1 each played the same 300 respondents from the public Twin-2K-500 benchmark, covering 108 questions and 16 behavioural-economics tasks, for 24,596 paired answered cells per arm, under criteria fixed in advance.
When yes/no items were asked as a Noul (a single probability of yes), Jev led GPT-4.1's verbalised probabilities on all six measures at about a thirty-fourth of the cost and came within two points of GPT-4.1's hard-answer accuracy. Asked as a Choice between the two options, which was the original setup, Jev trailed on the distribution gap (0.1985 versus 0.1789). The headline conclusion is that how you ask moved results more than which model you used. The author paid for all usage and reports no affiliation with TypeSafe or OpenAI. The write-up is split into a general survey track and a Jev-specific track, with a visual explainer page and MIT-licensed code and raw runs.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Would this respondent answer yes to this survey item? | — |
| Choice | Which of the two options would this respondent pick? | — |
How it works
- Give the model facts about a real respondent
- Ask each survey item as Noul or Choice (Jev) or verbalised probability (GPT-4.1)
- Compare predicted distributions with real answers
- Score on pre-specified measures and cost
Strengths
- Pre-specified criteria and large paired sample
- Shows a clear question-format effect
- Raw runs and code published
Caveats
- Results depend heavily on question format
- Single benchmark of synthetic respondents
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- CLI
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Survey research: runs Jev and GPT-4.1 as the same 300 synthetic respondents over 24,596 paired Twin-2K-500 cells under criteria fixed in advance, finding that asking a yes/no item as `Noul` rather than `Choice` moves the result more than the gap between the two models, at a thirty-fourth of the cost, written up at [jjd-lab.github.io](https://jjd-lab.github.io/jev-synthetic-survey/).
yibie/awesome-jevNew to synthetic survey respondents? \[Start here\](#new-to-this-start-here). For the raw runs, the scored reports and the code, see \[where to go\](#where-to-go).
logicrw/awesome-jev-projects
Frequently asked questions
Did Jev beat GPT-4.1?
When asked as Noul it led GPT-4.1's verbalised probabilities on all six measures; as a Choice it trailed on the distribution gap.
What was the main lesson?
Question format (Noul versus Choice) mattered more than the model.
Was the study funded by either company?
No, the author reports paying for all usage with no affiliation.
Related projects
Calibration experiments showing Jev's Choice probabilities on fair dice and coins stay high while accuracy stays at chance.
LibraryresearchPythonpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.
CLIresearchPython