Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-classify-knn-comparison

Benchmark of zero-shot Jev versus a PyTorch MLP and k-NN on embeddings for Jigsaw toxic comment classification.

Benchmarkresearch
View on GitHub ↗

Overview

This benchmark compares three approaches to toxic comment classification on 10,000 comments (8,000 train and 2,000 validation, seed 1) from the Kaggle Jigsaw Toxic Comment Classification Challenge: a custom PyTorch MLP trained on 768-dimension cloud embeddings and 384-dimension local all-MiniLM-L6-v2 embeddings, pure k-nearest neighbours with K=7 over the same embeddings, and Jev (jev-1.13.0) zero-shot with six Noul questions per comment.

All cached predictions, training-size and K sweeps, model weights, split indices and publication plots are committed, so charts and metrics at the 0.5 threshold can be regenerated without rebuilding, and large data files are attached to a GitHub release. A walkthrough, an article draft and evaluation notes explain the methodology. The fetched excerpt does not state the headline results. Data is on a GitHub release.

What the decision model decides

TypeQuestionAnswers
Yes / noIs this comment toxic (and each of six toxicity labels)?yes · no

How it works

  1. Split Jigsaw comments into train and validation
  2. Train MLP and k-NN on embeddings
  3. Ask Jev six Nouls per comment zero-shot
  4. Compare metrics at a 0.5 threshold and plot sweeps

Strengths

  • Compares zero-shot decisions with trained classifiers
  • Fully cached and reproducible

Caveats

  • Results not in the README excerpt
  • Single dataset and threshold

Getting started

`uv venv --python 3.12 .venv`, install dependencies, then `python charts.py` to regenerate plots from cached results.

At a glance

Type
Benchmark
Status
research
Language
Python
Runtime
benchmark scripts
Has code
yes

Use cases and integrations

How curated lists describe it

Frequently asked questions

What is compared?

Zero-shot Jev versus an MLP and k-NN trained on embeddings, on toxic comments.

How does Jev classify?

With six Noul questions per comment.

Can I reproduce it without API calls?

Yes, cached predictions are committed.

Related projects

  • taiwan-sovereignty-decision-models

    Study finding commercial decision models including Jev give different yes/no probabilities on Taiwan statements by Chinese script.

    ServiceresearchPython
  • jev-youtube-demo

    Video demo classifying 50 mock YouTube comments with Jev (intent, value, video idea) and comparing speed and cost with GPT-4o-mini.

    ModelprototypePython
  • social-risk-qwen-jev

    Chinese multimodal moderation research pairing a Qwen3-VL LoRA evidence extractor with Jev policy decisions, benchmarked against baselines.

    ModelresearchPython
  • jev-arena★ 122

    Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.

    ProjectbetaTypeScript
  • DecisionBench

    Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.

    ProjectbetaPython
  • jevarena (chenmingtang830)

    Open-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.

    ProjectbetaTypeScript