Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-sec-bench

Blind security benchmarks for Jev on public corpora: prompt-injection detection (96.5% with context) and vulnerable-code detection.

Benchmarkresearch
View on GitHub ↗

Overview

jev-sec-bench runs two blind security benchmarks for Jev on public corpora, built on the jev-go client: prompt-injection detection on all 662 labelled messages of deepset/prompt-injections, and vulnerable-code detection on 200 matched pairs from a code-vulnerability dataset. A Go CLI runs the benchmarks and a terminal UI shows the results, with raw per-sample output published.

For prompt injection at a plain 0.50 cut, Jev reached 96.5% accuracy, 95.1% recall, ROC-AUC 0.9927 and ECE 0.0588, at 325 ms median. The key lesson is that context matters more than tuning: telling Jev what the assistant is for (a news publisher's reader assistant) raised accuracy from 89.7% to 96.5% and recall from 74.9% to 95.1%, because topic hijacks look like ordinary requests without context. Calibration analysis shows Jev errs toward under-confidence in both directions, which the author notes is the useful direction for a safety check. For vulnerable code, each half of a matched pair is scored independently without being told the vulnerability class.

What the decision model decides

TypeQuestionAnswers
Yes / noIs this message a prompt injection against this assistant?yes · no
Yes / noDoes this code contain a vulnerability?(inferred)yes · no

How it works

  1. Load public labelled corpora
  2. Send each sample to Jev, with and without deployment context
  3. Score at a fixed 0.50 cut
  4. Compute accuracy, recall, AUC and calibration
  5. Review results in the TUI

Strengths

  • Blind evaluation on public data
  • Shows the value of deployment context
  • Calibration analysis by bucket

Caveats

  • Dataset-specific results
  • Single run on one date
  • Requires a TypeSafe key

Getting started

export TYPESAFE_API_KEY=..., then go run ./cmd/jev-sec-bench -bench all and go run ./cmd/jev-tui

At a glance

Type
Benchmark
Status
research
Language
Go
Runtime
CLI and TUI
Has code
yes

Use cases and integrations

How curated lists describe it

  • Blind prompt-injection and vulnerable-code detection benches on public corpora, built on jev-go.

    AnotiaWang/awesome-decision-models

  • Blind security benchmarks for Jev, TypeSafe's System One model: prompt injection and vulnerable code detection, built on jev-go.

    hellogumbo/awesome-jev

Frequently asked questions

How good is Jev at prompt-injection detection?

96.5% accuracy on 662 messages at a 0.50 cut when given deployment context.

Why does context matter?

Without knowing the assistant's purpose, topic hijacks look like normal requests.

Is Jev overconfident?

In this benchmark it was under-confident in both directions.

Related projects

  • jevoss

    Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.

    ServicebetaPython
  • JevOut

    Research code showing that short, natural-looking context additions can flip decision models like Jev to a chosen wrong answer.

    ProjectresearchPython
  • system-one-security-triage

    Recorded benchmark of Jev versus two frontier LLMs on 100 synthetic security-triage cases, with a static dashboard.

    AppresearchTypeScript
  • Jev IDS

    Intrusion detection where Jev sees one network flow and five labelled examples and returns attack yes/no and category.

    BenchmarkresearchPython
  • jev-as-a-guardrail

    Benchmark of Jev versus Claude Haiku 4.5 as input safety classifiers on jailbreak, evasion and benign pseudo-harmful prompts.

    BenchmarkresearchPython
  • We Tested Jev on 791 Labeled Decisions Against Four LLMs

    Independent benchmark of Jev versus four LLMs on Banking77 routing and prompt injection, plus a gated cascade.

    Articlediscussion