jev-anotacao-sentencas
FGV Direito SP experiment comparing Jev with Gemini 3.8 Flash and GPT-5.6 Luna for annotating 12 variables in 120 TJSP court rulings.
Overview
This is an experiment by LabDados at FGV Direito SP, documented in Portuguese, that compares Jev with Gemini 3.8 Flash and GPT-5.6 Luna using structured output for annotating judicial rulings. The task is to annotate 12 variables in 120 civil rulings from the São Paulo state court (TJSP), collected with the juscraper tool. A single schema defines the questions for all three providers, and a gold standard was built through blind adjudication of disagreements.
The README reports accuracy of 98.8% for Gemini 3.8 Flash, 96.8% for GPT-5.6 Luna and 96.6% for Jev, with median latency of 2.81 s, 3.49 s and 0.32 s, and cost of US$6.56, US$1.64 and US$0.25 per thousand rulings respectively. A full report is published as HTML and PDF. Raw model outputs and adjudication decisions are included, but the full ruling texts are not redistributed because of Brazilian data-protection law; collection scripts can recreate them.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Annotation variable for a civil ruling (e.g. outcome category), one of 12 schema questions(inferred) | — |
How it works
- Collect 120 TJSP civil rulings
- Ask each provider the same 12 schema questions
- Build a gold standard via blind adjudication
- Compare accuracy, latency, cost and calibration
Strengths
- Real legal-domain benchmark with a careful gold standard
- Jev reported about 9x faster than the next model at a fraction of the cost
- Fully reproducible pipeline
Caveats
- Small sample (120 rulings)
- Jev slightly less accurate than Gemini 3.8 Flash
- Documentation in Portuguese
Getting started
`uv sync`, then run the scripts in order from `scripts/01_coletar.py` through annotation and reference building.At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- research scripts, Quarto report
- Has code
- yes
Use cases and integrations
How curated lists describe it
Jev (TypeSafe) vs. Gemini 3.8 Flash vs. GPT-5.6 Luna na anotação estruturada de sentenças do TJSP: qualidade, tempo e custo.
hellogumbo/awesome-jev
Frequently asked questions
How did Jev compare?
96.6% accuracy versus 98.8% for Gemini 3.8 Flash, but at 0.32 s median and US$0.25 per thousand rulings.
What was annotated?
12 variables in 120 TJSP civil rulings.
Are the rulings included?
No, texts are not redistributed for data-protection reasons; scripts can re-collect them.
Related projects
Grounded extraction where code proposes spans with exact offsets and Jev only classifies each one; a LangExtract alternative.
LibrarybetaPythonZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonJev applied to SemEval-2026 DimABSA dimensional sentiment tasks using only typed primitives, no generation or fine-tuning.
ProjectresearchPythonBenchmark of Jev versus OpenAI structured outputs extracting smoking history from 1,000 synthetic notes.
BenchmarkresearchPythonBenchmark and web UI comparing dozens of zero-shot extraction and classification systems, including Jev-like engines.
BenchmarkresearchPythonSide-by-side comparison of Jev and DeepSeek-flash on speed, tokens, cost and accuracy for extraction, classification and reranking.
Projectresearch