Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-experiment

Learning experiment comparing Jev with an OpenAI model on GitHub issue ownership and on AG News, TREC and SNIPS classification.

Modelresearch
View on GitHub ↗

Overview

Jev vs an LLM is a small learning experiment in plain Python with a single dependency. It first fetched 200 real GitHub issues (50 per team) from the Flutter project and asked Jev and an OpenAI model who owns each issue, then extended to a benchmark suite: AG News (400 articles), TREC (500 questions) and SNIPS (350 requests), each with frozen samples, label descriptions and provenance.

The runner makes about 2,500 billable calls on a full run, saving every response so reruns reuse results, and invalid model responses count as incorrect. A comparison script needs no network and produces scores with paired bootstrap intervals, latency and cost summaries, and an equally weighted cross-task summary. A plain-language report explains results and limitations for a general audience.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich team owns this GitHub issue?—
ChoiceWhich topic, question type or intent fits this text?—

How it works

  1. Freeze reproducible samples from public datasets
  2. Run Jev and the OpenAI model with alternating order
  3. Save each response and resume on errors
  4. Compare scores with bootstrap intervals, latency and cost

Strengths

  • Minimal dependencies and no training
  • Paired uncertainty intervals
  • Plain-language findings report

Caveats

  • Small learning experiment
  • Full run costs about 2,500 API calls

Getting started

Set up the venv and keys, then `python benchmark_data.py`, `python run_benchmarks.py --workers 3` and `python compare.py`.

At a glance

Type
Model
Status
research
Language
Python
Runtime
Scripts (requests)
Has code
yes

Use cases and integrations

How curated lists describe it

Frequently asked questions

What is compared?

Jev and an OpenAI model on issue ownership and three public classification sets.

Can I rerun cheaply?

Yes, saved responses are reused, and compare.py runs offline.

Where are the findings?

In REPORT.md, a plain-language account.

Related projects

  • jev-eval

    Third-party check of Jev versus GPT-4o-mini and Claude Sonnet 4.5 routing 60 multilingual photo-shoot booking inquiries.

    ProjectresearchTypeScript
  • decision-1-vs-jev

    C# System One client and benchmark comparing Microsoft-Decision-1, Jev and open-jev on the same labelled support tickets.

    LibraryprototypeC#
  • Decision Models Showdown

    Web app that sends identical ticket-routing and alert-triage questions to OpenAI Decisions and Jev side by side.

    BenchmarkprototypePython
  • jev-1.13-mini-benchmark★ 1

    Small labelled support-triage benchmark of jev-1.13 on OpenRouter covering noul, choice, score, routing, consistency and cost.

    BenchmarkresearchPython
  • jev-ood-calibration

    Independent test of whether Jev's probabilities stay calibrated on 900 unseen rule-generated tickets plus three public benchmarks.

    Benchmarkresearch
  • jev-triage-benchmark

    Independent benchmark of Jev versus GPT-4.1-mini on 200 South African customer enquiries, showing how question wording changed results.

    BenchmarkresearchPython