Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-vs-decisions-rpg

RPG experiment pitting Jev against OpenAI's Decisions API as NPCs, comparing wins, per-move accuracy, calibration and cost.

Serviceresearch
View on GitHub ↗

Overview

This experiment uses Jev (jev-1.13.0) and OpenAI's Decisions API (gpt-6-luna, public beta) as NPC brains in a small turn-based RPG where two knights choose simultaneously among attack, power attack, defend, potion and flee. All dice, damage and win logic live in Python; the APIs only choose actions, and numeric state such as HP and MP is converted into word buckets (healthy, wounded, badly hurt, near death) before reaching either model.

The Decisions API won 82 of 100 head-to-head matches to Jev's 8. Against a four-rule if-statement NPC, Jev lost all 50 games while the Decisions API won 1, lost 46 and drew 3. Per-move accuracy over 100 positions in five option orders was 51.0% for Jev versus 59.4% (random baseline 24.9%, McNemar p = 0.007). Jev was much better calibrated (ECE 0.048 versus 0.161) and slightly faster (168 ms versus 199 ms median), at similar total cost. Play styles differed sharply: Jev defended on 67.7% of moves while the Decisions API attacked on 82.8%. Neither gave away items in a prompt-injection test. The article and results are in Japanese.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich action should this knight take this turn?attack · power attack · defend · potion · flee

How it works

  1. Engine describes state in word buckets
  2. Both APIs choose an action each turn
  3. Engine resolves dice, damage and outcomes
  4. Compare wins, per-move accuracy, calibration, latency and cost

Strengths

  • Paired, statistically tested comparison
  • Shows Jev's calibration advantage alongside its accuracy deficit
  • Prompt-injection bonus test

Caveats

  • Jev lost heavily in head-to-head play and to a simple rule NPC
  • Single game design; results in Japanese

At a glance

Type
Service
Status
research
Language
Python
Runtime
CLI
Has code
yes

Use cases and integrations

How curated lists describe it

Frequently asked questions

Which model won?

OpenAI's Decisions API won 82 of 100 matches against Jev's 8.

Where did Jev do better?

Calibration (ECE 0.048 versus 0.161) and slightly lower latency.

Did either model beat a simple rule-based NPC?

Barely: Jev lost all 50 games and the Decisions API won 1 of 50.

Related projects

  • System1Doom

    Harness where Jev or Laya pick DOOM tactics and targets from legal options while a deterministic controller presses buttons.

    ModelresearchPython
  • jevcheck

    pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.

    ProjectbetaPython
  • jevoss

    Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.

    ServicebetaPython
  • can-jev-play

    Experiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.

    Projectresearch
  • Jev × LexGLUE

    Zero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.

    ProjectresearchPython
  • Jev vs GPT-4.1 on a synthetic survey

    Pre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.

    ProjectresearchPython