askjev
Curiosity project, not a benchmark: over a million questions from 312 sources put to Jev, mapped by topic and written up as experiments.
Overview
askjev by Brian Zhang is a portrait of Jev rather than a benchmark. Starting from TypeSafe's quip that nobody asks how Jev is doing, the author asked it about its wellbeing and then about everything else: favourite films, what "several" means, trolley problems, what year it thinks it is, and whether it caves under social pressure.
The project covers 1,043,973 questions from 312 sources, filed on a tree of 1,737 topics and drawn as a searchable map where each question shows Jev's probabilities next to real people's answers where they exist. An atlas of about 198 experiments writes each finding up as a short case study with caveats, and a fourteen-chapter long read ties it together. Jev is the only model called and also runs the project's own pipeline steps. The repository holds code and method; answers and site data are private and shared with TypeSafe. It is not affiliated with TypeSafe.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Closed questions of every kind (yes/no, pick one, rate) put to Jev to characterize it | — |
How it works
- Gather questions from 312 sources
- Ask Jev each one as a typed question
- File questions into a topic tree using Jev
- Compare with human answers and write up experiments
Strengths
- Huge, traceable question set
- Compares Jev with real people's answers
Caveats
- Explicitly not a benchmark
- Answers and site data are private
At a glance
- Type
- Benchmark
- Status
- research
- Runtime
- data pipeline + web site
- Has code
- yes
Use cases and integrations
How curated lists describe it
Everyone asks what Jev is; I asked how it's doing, then a million other things. A curious portrait of TypeSafe's Jev: every question a star, 192 weird experiments, zero benchmarks.
hellogumbo/awesome-jev
Frequently asked questions
Is this a benchmark?
No, it explores what Jev is like rather than how good it is.
How many questions?
1,043,973 of 1,109,409 answered, from 312 sources.
Can I see the data?
The repo holds code and method; answers and site data are private.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPythonPre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.
CLIresearchPython