Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-cj-asap

Comparative judgment of ASAP essays by Jev and two LLM judges, turning pairwise decisions into scores compared with human grades.

ServiceresearchMIT
View on GitHub ↗

Overview

jev-cj-asap accompanies a manuscript in preparation comparing a decision-model API with generative LLM judges for comparative judgment of student essays from the ASAP dataset. From each of eight essay sets a seeded sample of 50 essays is drawn and every pair is judged in both orders, giving 2,450 ordered judgments per set and judge. Judgments become essay scores with the Bradley-Terry model and Elo, which are compared with human scores along with time, tokens and cost.

Arms include Jev 1.13 pairwise, Jev with the rubric asked as questions (a holistic question plus one per rubric facet in one request), Jev absolute scoring (one request per essay with one score question per rubric part), gpt-oss-20b on a local RTX 3090, and gpt-5.4-mini. Reported mean Spearman correlations with human scores are 0.728 for Jev pairwise (CI 0.659-0.776) at $2.01 and 5.2 minutes for all eight sets, 0.726 for the rubric composite at $2.78, 0.715 for the rubric overall, and 0.679 for Jev absolute scoring at $0.04 and 0.2 minutes. Code is MIT; judgment logs and results are CC BY 4.0.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich of these two essays is better?(inferred)essay A · essay B
ScoreWhere does this essay sit on each rubric part?—

How it works

  1. Sample 50 essays per ASAP set
  2. Judge every pair in both orders (or score absolutely)
  3. Fit Bradley-Terry and Elo scores
  4. Compare with human scores, cost and time

Strengths

  • Systematic comparison with LLM judges
  • Absolute scoring is extremely cheap
  • Logs and results openly licensed

Caveats

  • Manuscript still in preparation
  • Pairwise judging scales quadratically with essays

At a glance

Type
Service
Status
research
Language
Python
Runtime
research scripts
Has code
yes
License
MIT

Use cases and integrations

How curated lists describe it

Frequently asked questions

How well do Jev's judgments match human graders?

Mean Spearman 0.728 for pairwise comparative judgment across eight sets.

What is the cheapest option?

Absolute rubric scoring: $0.04 for all sets with ρ 0.679.

Which judges were compared?

Jev variants, gpt-oss-20b and gpt-5.4-mini.

Related projects

  • DecisionBench

    Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.

    ProjectbetaPython
  • jev-writer

    Analytics engine that rates published posts on pre-registered rubrics with Jev and tests ratings against real engagement.

    ProjectbetaTypeScript
  • jevarena (chenmingtang830)

    Open-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.

    ProjectbetaTypeScript
  • jevcheck

    pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.

    ProjectbetaPython
  • typesafe-eval

    Document evaluation CLI and CI gate using Jev or OpenAI Decisions, with pre-registered validation and documented reliability limits.

    CLIbetaPython
  • judging-with-typesafe

    Russian-language Letta agent skill that uses Jev as a fast external judge, with rules for treating mid-range probabilities as 'unsure'.

    ProjectprototypePython