LegalForecast-MTD
Benchmark asking models to forecast federal motion-to-dismiss rulings from the judge's record, scored with claim-defendant Brier metrics.
Overview
LegalForecast-MTD tests whether frontier models can forecast federal motion-to-dismiss rulings from the same written record the judge received. It argues that existing legal benchmarks test junior-associate tasks with objective answers, while higher-value legal judgment is hard to benchmark; prediction offers objective ground truth (the motion was granted or denied for each claim and defendant) on a task clients pay senior lawyers for.
Models output calibrated probabilities, scored with claim-defendant micro-Brier metrics and clustered intervals. Eligibility depends on a model's reported training cutoff preceding the earliest scored decision, known outcome leakage is excluded and models run without web access, which the README says reduces but does not eliminate contamination risk. The first public beta, published October 13, 2026, ranks eighteen model configurations on 91 cases and 387 scored units, with a working paper. Per the listing, Jev is among the models asked to predict rulings. Methods, reproduction and publication rules are documented.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Will the judge grant the motion to dismiss as to this claim and defendant? | — |
How it works
- Assemble the written record for each motion
- Ask each eligible model for the probability of dismissal per claim and defendant
- Score probabilities with micro-Brier and clustered intervals
- Check cutoff eligibility and leakage
- Publish rankings and the working paper
Strengths
- Objective ground truth for a judgment-heavy task
- Calibration-focused scoring
- Explicit contamination controls and governance
Caveats
- Public beta with 91 cases
- Does not prove zero contamination
- Prediction is not the same as legal analysis
Getting started
Read docs/METHODS.md and docs/reproduce-or-audit.md, or see results at legalforecastbench.org.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- benchmark pipeline
- Has code
- yes
Use cases and integrations
How curated lists describe it
Legal forecasting: benchmark that asks Jev to predict federal motion-to-dismiss rulings from the judge's written record and scores the calibrated probabilities with claim-defendant micro-Brier metrics.
yibie/awesome-jevLegalForecast-MTD benchmark alpha and official evaluation workflows.
hellogumbo/awesome-jev
Frequently asked questions
What do models predict?
Whether a federal judge granted a motion to dismiss for each claim and defendant.
How is it scored?
With claim-defendant micro-Brier scores and clustered intervals.
How is contamination handled?
Models must have cutoffs before the earliest decision, known leakage is excluded and web access is off.
Related projects
Zero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPythonFGV Direito SP experiment comparing Jev with Gemini 3.8 Flash and GPT-5.6 Luna for annotating 12 variables in 120 TJSP court rulings.
ProjectresearchPythonGrounded extraction where code proposes spans with exact offsets and Jev only classifies each one; a LangExtract alternative.
LibrarybetaPythonCalibration experiments showing Jev's Choice probabilities on fair dice and coins stay high while accuracy stays at chance.
LibraryresearchPythonHypothesis-driven experiments characterising the Jev API, intended to inform a future football prediction product.
ServiceresearchPython