social-risk-qwen-jev
Chinese multimodal moderation research pairing a Qwen3-VL LoRA evidence extractor with Jev policy decisions, benchmarked against baselines.
Overview
This research project, documented mainly in Chinese, studies risk detection for image-and-text content on Chinese social platforms, with misogyny as the positive class in its dataset. The pipeline goes from multimodal perception to structured evidence JSON to a Jev policy decision to risk bands: a Qwen3-VL-32B model with a language-side LoRA (vision encoder frozen) extracts six evidence fields from image and text, and Jev applies a fixed policy to the transcript and evidence, without seeing Qwen's final label or the image. Risk bands are not calibrated probabilities.
The README publishes a frozen test leaderboard of 13 methods on 170 items. TF-IDF with logistic regression ranked first (Macro-F1 90.36%), the pre-registered 'text OR multimodal+Jev' design scored 87.25%, LoRA evidence plus Jev 79.74%, and pure Jev on text alone 63.04%. Adding Qwen evidence raised Jev's Macro-F1 by about 17 points and cut missed cases from 35 to 15, though the authors note this cannot be attributed to images alone and that Jev was not an unconditional gain. Code and summaries are public under MIT; data, predictions and weights are not.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Under the fixed policy, is this post harmful (misogynistic)? | — |
| Score | Which risk band does this post fall into?(inferred) | — |
How it works
- Qwen3-VL LoRA reads image and text and emits six evidence fields
- Remove Qwen's final label
- Jev applies a fixed policy to transcript plus evidence
- Threshold (t=0.40) into risk bands; compare with baselines
Strengths
- Honest leaderboard including stronger simple baselines
- Ablation shows value of structured evidence for Jev
- Privacy-aware: no raw data published
Caveats
- Simple TF-IDF baselines beat the Jev pipelines on test
- Small test set (170)
- Not a production moderation system
Getting started
python -m pip install -e . for the public modules; datasets and weights are not included.At a glance
- Type
- Model
- Status
- research
- Language
- Python
- Runtime
- Research code (GPU for Qwen)
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Chinese moderation: pairs a Qwen3-VL LoRA with a Jev decision model to judge multimodal social-risk content.
yibie/awesome-jevChinese multimodal moderation: Qwen3-VL LoRA, TypeSafe Jev decisions, reproducible baselines and privacy-aware evaluation.
hellogumbo/awesome-jev
Frequently asked questions
Did Jev beat simple baselines?
No; TF-IDF with logistic regression had the best test Macro-F1.
Did Qwen evidence help Jev?
Yes, Macro-F1 rose from 63.04% to 79.74% at the same threshold on test.
Does Jev see the images?
No, only the transcript and Qwen's evidence fields.
Related projects
Study finding commercial decision models including Jev give different yes/no probabilities on Taiwan statements by Chinese script.
ServiceresearchPythonPerplexity's Apache-2.0 multimodal decision model on Qwen3.8-27B, reporting 61.56 on the Decision Index versus 57.9 for Jev.
ServiceproductionPythonSmall open vision-language decision models (2B, 4B, 9B) that answer typed questions about photos and records with a trained can't-tell.
ModelbetaPythonVideo demo classifying 50 mock YouTube comments with Jev (intent, value, video idea) and comparing speed and cost with GPT-4o-mini.
ModelprototypePythonBenchmark of zero-shot Jev versus a PyTorch MLP and k-NN on embeddings for Jigsaw toxic comment classification.
BenchmarkresearchPython10,000-sample multimodal benchmark for decision models that return full probability distributions over declared candidates.
BenchmarkresearchPython