Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

privatemode-decisions-benchmark

Benchmark comparing Privatemode (GLM-5.3-Flash), TypeSafe Jev and Laya on speed, accuracy, calibration and cost.

Benchmarkresearch
View on GitHub ↗

Overview

This benchmark compares three System One implementations on labelled public data with identical state, option names and instructions: Privatemode Decisions (GLM-5.3-Flash in an attested enclave, one masked forward pass), TypeSafe's hosted Jev, and Laya (a 421M ModernBERT-large encoder with a decision head, run locally).

On JevBench's 231 public items, Privatemode scored 0.894 with one output token against Jev's published 0.866, while GLM-5.3-Flash with structured output and thinking scored 0.983 at around 518 output tokens. On MMLU-Pro, one-pass Privatemode scored 61.9% against Jev's published 82.9%, with the gap in calculation questions. Across 29 datasets, normalised accuracy was 0.638 for Privatemode, 0.560 for Jev and 0.414 for Laya, with Privatemode ahead of Jev 16-8-3 (Wilcoxon p = 0.001), lower excess ECE, about 150 ms versus 251 ms latency, but higher cost per 1,000 decisions. Each row states what it was measured with.

What the decision model decides

TypeQuestionAnswers
ChoiceDataset label prediction for each benchmark example—

How it works

  1. Prepare 29 labelled datasets and JevBench items.
  2. Run each system with identical prompts and options.
  3. Measure accuracy, calibration, latency and cost.
  4. Report paired comparisons and significance.

Strengths

  • Controlled, paired comparison.
  • Reports where one-pass models fall short (MMLU-Pro).
  • Transparent measurement notes.

Caveats

  • Authored by the Privatemode team.
  • Some Jev figures are published rather than re-run.

At a glance

Type
Benchmark
Status
research
Language
Python
Runtime
CLI
Has code
yes

Use cases and integrations

How curated lists describe it

Frequently asked questions

Which systems are compared?

Privatemode Decisions, TypeSafe Jev and Laya.

Where does Jev win?

On MMLU-Pro, especially calculation questions, and on cost per decision.

Is it independent?

It is published by the Privatemode developers.

Related projects

  • jevcheck

    pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.

    ProjectbetaPython
  • jevoss

    Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.

    ServicebetaPython
  • can-jev-play

    Experiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.

    Projectresearch
  • Jev × LexGLUE

    Zero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.

    ProjectresearchPython
  • Jev vs GPT-4.1 on a synthetic survey

    Pre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.

    ProjectresearchPython
  • jev-acento

    Pre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.

    CLIresearchPython