Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

Jev Does Not Play Dice

Calibration experiments showing Jev's Choice probabilities on fair dice and coins stay high while accuracy stays at chance.

LibraryresearchMIT
View on GitHub ↗Live demo ↗

Overview

Jev Does Not Play Dice contains the code, recorded outputs and analysis scripts for experiments testing whether Jev's reported probabilities reflect known uncertainty. Inputs have exactly known true probabilities: hidden fair dice, coins and multi-way spinners, plus synthetic forecast documents that state a probability.

The README reports that Choice on a hidden fair die gave a mean probability of 82.9 percent with 19.0 percent accuracy over 400 rolls, and 92.0 percent on a fair coin with 52.0 percent accuracy. Noul was closer to reference values (19.2 percent for a die face) but over-reported low probabilities. In forecast documents, stated probabilities of 45 and 55 percent became 6.6 and 95.9 percent via Choice. The author stresses accuracy was at chance as expected; the finding is that probabilities read as model certainty, not event likelihood, and that results are specific to these prompts. Offline analysis needs only Python; API runs use a Vercel AI Gateway key.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich face did the hidden fair die land on?1 · 2 · 3 · 4 · 5 · 6
Yes / noDid the hidden die land on this face?—
ChoiceWill the forecast event happen, given a stated probability?—

How it works

  1. Generate inputs with known true probabilities
  2. Ask Jev Choice and Noul questions
  3. Record raw responses
  4. Compare reported probabilities with reference values offline

Strengths

  • Clean test with exactly known probabilities
  • Raw responses and offline recomputation
  • Separates Choice and Noul behaviour

Caveats

  • Results specific to these prompts and jev-1.13.0
  • Does not show Jev probabilities are generally unusable

Getting started

`python3 scripts/analyze.py` to analyse recorded results with no API calls.

At a glance

Type
Library
Status
research
Language
Python
Runtime
Analysis scripts plus Node request runner
Has code
yes
License
MIT

Use cases and integrations

How curated lists describe it

  • Code, recorded outputs, and analysis scripts for probability-output experiments with Jev: fair random draws, Noul (Yes/No) questions, and forecast documents.

    logicrw/awesome-jev-projects

  • **The probabilities reported by Jev reflect “how certain the model is,” not…

    wh000wh000/awesome-jev-live

  • Calibration check on inputs with known true probabilities (hidden fair dice, coins, four-way spinners) and on synthetic forecast documents; Choice put 82.9% on face 1 across all 400 die rolls at 19.0% accuracy and turned a stated 30% shortage risk into 5.3%, while Noul returned 26.7%; raw responses and offline recomputation in the repo, run in launch week against jev-1.13.0. [Write-up](https://kantahayashiai.github.io/posts/jev-does-not-play-dice/).

    cobanov/awesome-jev

  • Independent calibration evaluation of Jev on fair random draws with known probabilities and synthetic forecast documents, with recorded outputs and offline recomputation.

    hellogumbo/awesome-jev

Frequently asked questions

What was found?

Choice reported high confidence on fair random outcomes while accuracy stayed at chance; Noul was closer but still over-reported low probabilities.

Does this mean Jev's probabilities are useless?

The author says no; results are specific to these conditions and show probabilities reflect model certainty.

Can I reproduce it without a key?

Yes, the offline analysis of recorded responses needs no API calls.

Related projects

  • Jev vs GPT-4.1 on a synthetic survey

    Pre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.

    ProjectresearchPython
  • decision-model-audit

    Pre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.

    ProjectresearchPython
  • jev-architecture-research

    Black-box reverse-engineering archive studying how the Jev decision model behaves, with evidence matrices and calibration analysis.

    ProjectresearchPython
  • jev-calibration-audit

    Independent API-only audit of what Jev's probabilities mean: abstain removal, Korean, coherence, option order, interference and caching.

    ProjectresearchPython
  • Jev-decision-control

    Research framework (DART) testing whether Jev's probabilities are a reliable signal to answer, escalate or abstain.

    ProjectresearchPython
  • jev-exploration

    Evidence ledger auditing Jev claims, recomputing published calibration and running an 800-item difficulty-gradient study.

    AppresearchPython