Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-fanout-bench

Reproducible 2,976-request measurement of whether batching questions in one Jev call bills the state once and keeps answers stable.

Benchmarkresearch
View on GitHub ↗Live demo ↗

Overview

jev-fanout-bench tests one documented claim: Jev ingests the state once and evaluates every question in parallel. If true on the bill, N questions in one request should cost the state once, and N separate requests should cost it N times. TypeSafe's own cookbook measured this once as 12.2 times cheaper; this repository measures it across three state sizes (about 25, 680 and 3,000 tokens) and N of 2, 4 and 8, plus a one-word control state.

It checks whether billing is exactly linear by deriving the fixed per-request cost independently, how large savings are in percent and dollars per thousand items, and whether answers change by asking each question both batched and alone and repeating every request to measure noise. The 2,976 requests to jev-1.13-20260917 via OpenRouter cost $0.156 in total. List summaries report about 261 fixed input tokens per request, charges matching the published rate, answer differences comparable to repeat-request noise, and median input-token savings of 76 to 86 percent at eight questions. Aggregates are also published as a Hugging Face dataset.

What the decision model decides

TypeQuestionAnswers
OtherDoes batching N questions in one request bill the state once without changing answers?(inferred)—

How it works

  1. Build states of three sizes plus a control
  2. Ask N questions batched and separately, repeating each request
  3. Derive fixed per-request cost and savings
  4. Compare batched-vs-single differences with repeat noise

Strengths

  • Directly tests billing and answer stability
  • Noise floor from repeated identical requests
  • Cheap to reproduce

Caveats

  • Measured through OpenRouter on one model snapshot
  • Ticket-style states only

Getting started

See results/summary.md; aggregates are on Hugging Face as lisonallen/jev-ai-benchmark.

At a glance

Type
Benchmark
Status
research
Language
Python
Runtime
Benchmark scripts
Has code
yes

Use cases and integrations

How curated lists describe it

  • Model billing: compares batched with one-question-per-call requests across 2,976 calls to `jev-1.13-20260917` through OpenRouter's TypeSafe-compatible `/systemone` endpoint, reporting about 261 fixed input tokens per request, zero spread in the implied per-request cost across question counts, batched-vs-single answer differences comparable to repeat-request noise, and median input-token savings of 76–86% at eight questions.

    yibie/awesome-jev

  • Compares batched and separate Jev calls in 2,976 requests through OpenRouter, reporting approximately 261 fixed input tokens per request, charges matching the published token rate, and answer differences comparable to repeat-request noise.

    AnotiaWang/awesome-decision-models

  • Reproducible comparison of batched and one-question-per-call Jev requests whose 2,976-request OpenRouter run reports approximately 261 fixed input tokens per request, observed charges matching the published token rate, and batched-vs-single answer differences comparable to repeat-request noise.

    cobanov/awesome-jev

Frequently asked questions

Does batching save money?

Yes; summaries report 76 to 86 percent median input-token savings at eight questions.

Do answers change when batched?

Differences were comparable to the noise between identical repeated requests.

What did the study cost?

$0.156 for 2,976 requests.

Related projects

  • stuntdouble

    Zero-dependency Node proxy that shadows Jev with local decision models and reports whether you can safely swap, per question and policy.

    ProxybetaJavaScript
  • jev-measured

    Reproducible measurements of Jev's real cost, latency, response shapes and accuracy, plus 13 decision models on the same 633 items.

    Serviceresearch
  • laya-jev-lab

    Independent measurements of Jev versus open Laya on typed decisions, plus a local-first cascade that matches Jev at 1.8x speed.

    ServiceresearchPython
  • Jev broke our WebMCP benchmark

    X post by a WebMCP benchmark author reporting Jev plus a small LLM solved 100% of tasks at about 112x lower model cost.

    Discussiondiscussion
  • Halv cut AI agent cost by 57.1% using Jev

    Halv blog post reporting a 57.1% agent cost reduction from putting Jev in the agent loop on SWE-rebench Astra pairs.

    Articlediscussion
  • jev-arena★ 122

    Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.

    ProjectbetaTypeScript