Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

Jev × LexGLUE

Zero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.

Projectresearch
View on GitHub ↗

Overview

Jev × LexGLUE is a reproducible zero-shot benchmark of Jev on all seven LexGLUE legal NLP tasks. It freezes official test examples and sends typed decisions through OpenRouter's systemone endpoint: single-label examples become one Choice, and multi-label examples become one request with an independent Noul per label, with no chat completions or output parsing. Every completed request is checkpointed and reports cover micro and macro F1, usage and cost.

On 23,607 test examples the README reports Jev at 69.9 mean micro-F1 for $4.02, against GPT-5.6 Luna via chat JSON at 71.3 for $16.45; tuning per-label thresholds on the validation split raised Jev to 74.2. Jev beat open Kev-4B on all seven tasks, Clef-flash scored 68.3, and fine-tuned BERT-base scored 77.4, leading by 13 to 29 points on EUR-LEX, LEDGAR and UNFAIR-ToS while both decision models beat it on CaseHOLD.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich label applies to this legal text (single-label tasks)?—
Yes / noDoes this label apply (one per label for multi-label tasks)?—

How it works

  1. Freeze LexGLUE test examples
  2. Send Choice or per-label Noul requests via OpenRouter
  3. Checkpoint each request
  4. Compute F1, usage and cost; compare baselines

Strengths

  • Full-test-set evaluation
  • Cheap relative to an LLM with similar accuracy
  • Validation-tuned threshold variant

Caveats

  • Trails fine-tuned BERT on several tasks
  • Zero-shot only

At a glance

Type
Project
Status
research
Language
Python
Runtime
Benchmark scripts via OpenRouter
Has code
yes

Use cases and integrations

How curated lists describe it

  • Reproducible zero-shot run of Jev 1.13 (`typesafe/jev-1.13-20260917`) on all seven LexGLUE tasks, 23,607 test examples: mean micro-F1 69.9 at $4.02, against 71.3 at $16.45 for GPT-5.6 Luna via chat JSON

    AnotiaWang/awesome-decision-models

Frequently asked questions

How did Jev score?

69.9 mean micro-F1 zero-shot, or 74.2 with validation-tuned thresholds.

How does it compare to an LLM?

Close to GPT-5.6 Luna's 71.3 at about a quarter of the cost.

Does fine-tuning still win?

Fine-tuned BERT scored 77.4 overall.

Related projects

  • Vals AI's independent evaluation of Jev

    Vals AI's independent benchmark of Jev: matched a frontier model on claim verification at a fraction of cost, but ranked last on LegalBench.

    Discussiondiscussion
  • jevcheck

    pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.

    ProjectbetaPython
  • jevoss

    Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.

    ServicebetaPython
  • can-jev-play

    Experiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.

    Projectresearch
  • Jev vs GPT-4.1 on a synthetic survey

    Pre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.

    ProjectresearchPython
  • jev-acento

    Pre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.

    CLIresearchPython