Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-orderby-bench

Measures whether SQL ORDER BY over Jev probabilities gives a defensible ranking: inversions, ordinality, calibration, wording and ties.

Benchmarkresearch
View on GitHub ↗

Overview

jev-orderby-bench is an independent measurement of whether sorting rows by a Jev probability, as DuckDB and Postgres extensions for Jev allow, produces a defensible order. It tests jev-1.13.0 on pairwise inversion rate, Score ordinality against human grades, sensitivity to wording versus evidence, calibration and sort-key ties, under a pre-registered gate.

On 360 human-labelled 20 Newsgroups rows, Jev passed all six gate conditions, though Score inversion (0.143) sat just under the 0.15 threshold, probabilities were underconfident in most bins, and two-decimal rounding left 53 rows tied at 0.99, making a LIMIT 20 arbitrary. Sending the same rows through one extension's default 40-row batching failed the ranking gate that one row per request passed. On a harder Amazon ESCI product-relevance probe, it failed four of six conditions, with inversion around 0.25 and paraphrases moving answers by 0.164 on average. The main run cost about $0.013.

What the decision model decides

TypeQuestionAnswers
Yes / noIs this row relevant to the topic or query?yes · no
ScoreHow relevant is this product to the shopping query (4 levels)?—

How it works

  1. Pre-register gate conditions and thresholds
  2. Send human-labelled rows to Jev one per request and in batches
  3. Compute inversion rates, ordinality, calibration and paraphrase sensitivity
  4. Check sort-key ties
  5. Report pass or fail per condition

Strengths

  • Tests ranking properties that aggregate calibration hides
  • Shows how integration choices like batching change results
  • Cheap and reproducible

Caveats

  • Two datasets only
  • Specific to jev-1.13.0
  • Reference UDF is not meant for production use

At a glance

Type
Benchmark
Status
research
Language
Python
Runtime
Benchmark harness
Has code
yes

Use cases and integrations

How curated lists describe it

  • Model evaluation: measures whether a SQL ORDER BY over a Jev probability is defensible (pairwise inversion, Score ordinality against a human grade, calibration, wording invariants, sort-key ties) under a pre-registered gate that jev-1.13.0 passes on 20 Newsgroups topics and fails four of six conditions on Amazon ESCI product relevance, and shows a DuckDB extension's default 40-row batching fails the ranking gate that one row per request passes.

    yibie/awesome-jev

  • Measures whether ORDER BY over a Jev probability is defensible (inversion rate, Score ordinality against a human grade, calibration, wording invariants, sort-key ties) under a pre-registered gate; passes on 20 Newsgroups topics, fails four of six conditions on Amazon ESCI product relevance, and shows a DuckDB extension's default 40-row batching fails the ranking gate that one row per request passes.

    cobanov/awesome-jev

Frequently asked questions

Is ORDER BY over Jev probabilities safe?

It passed on topic classification but failed four of six conditions on hard product relevance.

Does batching matter?

Yes. Default 40-row batching failed the ranking gate that one row per request passed.

Why are there ties?

Probabilities come back at two decimals, so many rows share values like 0.99.

Related projects

  • jevcheck

    pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.

    ProjectbetaPython
  • jevoss

    Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.

    ServicebetaPython
  • can-jev-play

    Experiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.

    Projectresearch
  • Jev × LexGLUE

    Zero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.

    ProjectresearchPython
  • Jev vs GPT-4.1 on a synthetic survey

    Pre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.

    ProjectresearchPython
  • jev-acento

    Pre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.

    CLIresearchPython