Jev IDS
Intrusion detection where Jev sees one network flow and five labelled examples and returns attack yes/no and category.
Overview
Jev IDS is an intrusion-detection prototype: it shows Jev one network flow's metadata plus labelled examples and asks a Noul (is it an attack) and a five-way Choice (which category), with Python applying a 0.5 cut. It was tested on NSL-KDD against an LLM (Gemini 3.6 Flash on Vertex AI), a Random Forest and an Isolation Forest, across 2,000 flows, three seeds and 0 to 8 examples per category.
With five examples (k = 1), the README reports Jev at a median 0.31 s versus 1.65 s per flow for the LLM, $74 versus $1,651 per million flows at list prices, similar precision (0.953 vs 0.942), and 43 versus 764 false alarms on 874 benign flows compared with the Random Forest. The LLM was the better detector on F1 at every k (0.880 vs 0.856 at k = 1). A CLI runs each detector, computes metrics including recall on novel attacks, and compares runs. A list summary notes an earlier 300-flow pilot against GPT-5.6 Luna and stated limits such as list-price costs.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this network flow an attack? | — |
| Choice | Which attack category is it? | — |
How it works
- Load NSL-KDD flows and sample k labelled examples per category
- Ask Jev the attack Noul and category Choice
- Apply a 0.5 threshold in Python
- Compare against LLM and classical baselines on F1, novel-attack recall, latency and cost
Strengths
- Far fewer false alarms than a few-shot Random Forest
- Much cheaper and faster than the LLM
- Reproducible CLI and raw outputs
Caveats
- LLM had higher F1 at every k
- Benchmark dataset, not live traffic
Getting started
`uv run jev-ids run --dataset data/nsl-kdd/dataset.json --detector jev --split pilot --k 0,1,2,4,8 --seeds 0,1,2`.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- CLI (uv)
- Has code
- yes
Use cases and integrations
How curated lists describe it
Network security: an Intrusion Detection System prototype that takes the metadata of a network flow and returns a verdict on whether it is an attack plus its threat category with probabilities, and on the NSL-KDD benchmark was 4.8× faster and 3.8× cheaper than a state-of-the-art LLM (GPT-5.6 Luna) while raising 15× fewer false alarms than a Random Forest model.
yibie/awesome-jevIntrusion-detection prototype where one Noul (attack or not) and a five-way Choice (category) are asked per NSL-KDD network flow and Python applies a 0.5 cut; ships a 300-flow pilot comparison against GPT-5.6 Luna and a few-shot Random Forest with latency, cost, and novel-attack recall, and lists its limits (pilot split, gateway retries, list-price costs).
cobanov/awesome-jev
Frequently asked questions
What does Jev decide?
Whether a network flow is an attack and which category it belongs to.
Is it better than an LLM?
Faster, cheaper and similarly precise, but with lower F1 per the README.
Which dataset is used?
NSL-KDD.
Related projects
Recorded benchmark of Jev versus two frontier LLMs on 100 synthetic security-triage cases, with a static dashboard.
AppresearchTypeScriptBlind security benchmarks for Jev on public corpora: prompt-injection detection (96.5% with context) and vulnerable-code detection.
BenchmarkresearchGoSREGym blog post on Kubernetes incident diagnosis with no LLM agent: Jev picks a root cause from collected evidence.
Articleresearch- jev-arena★ 122
Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.
ProjectbetaTypeScript Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScript