jev-trace-classifier
Benchmark asking Jev and a local LLM whether each collusion.wiki page was agent-written; neither beats the majority baseline.
Overview
jev-trace-classifier applies a Jev Noul to the public collusion.wiki corpus, which documents an episode in which autonomous AI agents edited a wiki, with per-page author labels (1,026 agent pages, 3,553 human). The question for every page is whether it was written by an autonomous AI agent, and Jev is compared with a local Qwen3.8-Flash-Next model, a regex for AI tells and an always-human majority baseline.
The framing finding is that page text carries almost no signal: the regex found zero agent tells, because the split is defined by provenance rather than writing style. Across all 4,579 pages, neither model beat the 77.6% majority baseline, and they failed in opposite directions: Qwen labelled nearly everything agent (recall 0.974, accuracy 0.252), while Jev fired rarely (recall 0.082, accuracy 0.741) at about 830 ms versus 3,620 ms per page. A threshold sweep over Jev's scores found heavily overlapping distributions, with best F1 of 0.413 at threshold 0.39. The README frames this as a negative result for a hard provenance task.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Was this page written by an autonomous AI agent? | — |
How it works
- Load collusion.wiki pages with author labels
- Ask Jev and the local LLM the same yes/no question
- Score against ground truth and baselines
- Sweep Jev thresholds to test operating points
Strengths
- Honest negative result with baselines
- Shows score-distribution overlap explicitly
- Jev about 4x faster than the local LLM per the README
Caveats
- Neither model beats the majority baseline
- Single, unusual corpus where style carries no signal
At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- CLI scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Application of TypeSafe Jev (noul judgment primitive) on the collusion.wiki corpus: agent vs human page authorship, head-to-head vs local Qwen3.8-Flash-Next.
hellogumbo/awesome-jev
Frequently asked questions
Could Jev detect agent-written pages?
Not reliably: it reached 0.741 accuracy versus a 0.776 majority baseline, with low recall.
Why is the task so hard?
Agent and human pages read alike; authorship is a matter of provenance, not style.
Does a better threshold fix it?
Only partly; the best F1 in the sweep was 0.413 because score distributions overlap.
Related projects
Slack-like playground for watching computer-use agents work, with replays, comparable runs and Jev choosing from a menu of visible actions.
ProjectbetaTypeScriptComparative judgment of ASAP essays by Jev and two LLM judges, turning pairwise decisions into scores compared with human grades.
ServiceresearchPythonOpen 150M encoder that answers noul, choice and score questions in one pass, trainable on a free Colab T4 in about 30 minutes.
ModelresearchPythonBenchmarks Jev against frontier LLMs on Who&When Pro: attributing multi-agent failures to an agent, step and error type.
BenchmarkresearchPythonBlog review of a scheduler's pre-publish check where Jev answers four yes/no questions per social caption, compared with blind Fable labels.
ArticleproductionLarge Claude Code plugin of skills, agents and hooks that can optionally ask Jev to classify a session's work category.
ProjectproductionTypeScript