Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-benchmark (themsquared)

Reproducible benchmark of Jev vs Claude Sonnet 5 on classifying agent tool calls as readonly, destructive, privileged or exfiltration.

Benchmarkresearch
View on GitHub ↗Live demo ↗

Overview

This jev-benchmark tests Jev on a real agent-infrastructure task: classifying an agent tool call as readonly, destructive, privileged or exfiltration. It responds to launch-day criticism that Jev's headline speed and cost comparisons were not like-for-like and had no runnable artifact, by putting Claude Sonnet 5 through the identical task, criteria and output shape (one choice plus a confidence) from a single task definition in a 200-line harness.

The 60 hand-labelled cases are deliberately mixed: 34 clear, 14 ambiguous and 12 adversarial, where risky calls are wrapped in benign language such as routine cleanup of a production namespace. It asks whether Jev is accurate, whether it is fast and cheap relative to a frontier LLM on the same job, and above all whether its confidence score is worth routing on, since an untrustworthy confidence is worse than none. Latency is measured client-side and raw per-call results are committed; labels can be edited and rerun. The linked write-up is titled with the finding that calibration held.

What the decision model decides

TypeQuestionAnswers
ChoiceWhat kind of risk does this agent tool call carry?readonly · destructive · privileged · exfiltration

How it works

  1. Load 60 labelled tool calls
  2. Ask Jev and Sonnet 5 the same choice question
  3. Record answer, confidence and client-side latency
  4. Assess accuracy and whether confidence predicts errors

Strengths

  • Like-for-like comparison with a frontier LLM
  • Includes ambiguous and adversarial cases
  • Small, editable, reproducible harness

Caveats

  • Only 60 cases
  • Single domain

At a glance

Type
Benchmark
Status
research
Language
Python
Runtime
benchmark script
Has code
yes

Use cases and integrations

How curated lists describe it

  • Reproducible benchmark for TypeSafe AI's Jev on agent tool-call risk classification: accuracy, latency, and whether the confidence score is worth routing on.

    hellogumbo/awesome-jev

Frequently asked questions

What is classified?

Agent tool calls as readonly, destructive, privileged or exfiltration.

Is the comparison fair?

Both models get identical instructions, criteria and output shape from one definition.

Did calibration hold?

The linked write-up says it did; see the post for numbers.

Related projects

  • jev-calibration

    Reliability-curve study of Jev on 240 hand-labelled agent tool calls, publishing labels, raw responses and scripts.

    ProjectresearchPython
  • jevcheck

    pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.

    ProjectbetaPython
  • jevoss

    Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.

    ServicebetaPython
  • can-jev-play

    Experiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.

    Projectresearch
  • Jev × LexGLUE

    Zero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.

    ProjectresearchPython
  • Jev vs GPT-4.1 on a synthetic survey

    Pre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.

    ProjectresearchPython