system-one-security-triage
Recorded benchmark of Jev versus two frontier LLMs on 100 synthetic security-triage cases, with a static dashboard.
Overview
system-one-security-triage is a reproducible recorded comparison of three evaluators on identical security evidence: TypeSafe's Jev (jev-1.13.0) through the TypeScript SDK, an OpenAI model through Codex CLI and Claude Opus through Claude tooling. The frozen synthetic TypeScript/Node.js corpus has 100 cases across injection, broken access control and SSRF, each labelled vulnerable, safe or insufficient context. Each case was run five times per evaluator, giving 1,500 decisions.
Jev answers one five-way typed Choice and its selected option is scored the same way as the LLMs' structured answers, with the full distribution kept and a multiclass Brier score reported for Jev. Metrics include balanced accuracy, vulnerability recall, false-safe and false-positive rates and model latency. Earlier runs were superseded after label leaks were found, and the dashboard build rejects them by corpus hash. The static dashboard renders saved JSON only. The longer-term goal is a benchmark on sanitized real repositories with Semgrep discovery and cascade experiments.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | What is the disposition of this code evidence (vulnerable by family, safe, or insufficient context)?(inferred) | vulnerable (injection) · vulnerable (broken access control) · vulnerable (SSRF) · safe · insufficient context |
How it works
- Load label-neutral evidence for each of 100 synthetic cases.
- Send the same evidence to Jev as JSON and to the LLMs in a structured prompt.
- Record Jev's choice, all five probabilities and confidence.
- Score accuracy, recall and error rates; build the static dashboard from saved JSON.
Strengths
- Same evidence and scoring for all evaluators.
- Superseded runs are documented and rejected by hash.
- Records Jev's full probability distribution for calibration analysis.
Caveats
- Synthetic corpus of 100 cases, not real repositories.
- Recording a run needs TypeSafe and frontier-model access.
Getting started
npm install, export TYPESAFE_API_KEY, then npm run demo:record -- --models jev,terra,opus --repetitions 5 --run-id "$RUN_ID".At a glance
- Type
- App
- Status
- research
- Language
- TypeScript
- Runtime
- Node CLI + static dashboard
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Recorded comparison of Jev, Terra, and Opus on 100 synthetic security-triage cases, five passes each, with a static inspectable dashboard.
hellogumbo/awesome-jev
Frequently asked questions
What does Jev decide here?
For each case it picks one of five dispositions covering vulnerability family, safe and insufficient context.
How is Jev compared with the LLMs?
The selected choice is scored exactly like the LLMs' structured answers, with no threshold in between.
Does viewing the dashboard need keys?
No, it only renders saved JSON; recording new runs needs provider credentials.
Related projects
Intrusion detection where Jev sees one network flow and five labelled examples and returns attack yes/no and category.
BenchmarkresearchPythonBlind security benchmarks for Jev on public corpora: prompt-injection detection (96.5% with context) and vulnerable-code detection.
BenchmarkresearchGoIndependent 52-page Chinese report on Jev that checks vendor claims and reproduces 50 Chinese tests, with traceable data.
ProjectresearchPythonLearning project benchmarking zero-shot Jev and GPT models against scikit-learn classifiers and ModernBERT on five datasets.
ProjectresearchPythonSREGym blog post on Kubernetes incident diagnosis with no LLM agent: Jev picks a root cause from collected evidence.
Articleresearch- jev-arena★ 122
Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.
ProjectbetaTypeScript