Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

decision-1-vs-jev

C# System One client and benchmark comparing Microsoft-Decision-1, Jev and open-jev on the same labelled support tickets.

Libraryprototype
View on GitHub ↗

Overview

This repository ports Microsoft's quickstart for Microsoft-Decision-1, a System One model in Azure Foundry that uses the same /v1/systemone API and price as TypeSafe's Jev, to C#. One client class with three factories talks to Jev, Microsoft-Decision-1 and open-jev, a self-hosted Jev reproduction on Apple Silicon.

A benchmark runs 48 labelled support tickets in English, Spanish and Russian with three questions each: team (choice of four), severity (score 0 to 3) and churn (noul). The README reports Decision-1 at 100% team accuracy with good calibration and zero answer flips when options were reversed, versus 85.4% and seven flips for open-jev with a 4B model, which was sometimes wrong at p = 1.00. It also shows that extra questions in one request were nearly free on Decision-1 but cost a prefill each on open-jev. Jev can be run on the same file with a separate command.

What the decision model decides

TypeQuestionAnswers
Choiceteam: which team should handle this ticket?—
Scoreseverity: how severe is this ticket?0 · 1 · 2 · 3
Yes / nochurn: is the customer threatening to leave?yes · no

How it works

  1. Load labelled tickets
  2. Send each to a backend via the shared SystemOne client
  3. Compute accuracy, Brier, coverage at p ≥ 0.8, flips and latency
  4. Compare backends side by side

Strengths

  • One client for three compatible servers
  • Measures calibration and option-order robustness, not just accuracy
  • Hermetic unit tests

Caveats

  • Small 48-ticket dataset written by the author
  • Jev results require a separate run

Getting started

dotnet test runs six hermetic tests; see the Run it section for benchmark commands.

At a glance

Type
Library
Status
prototype
Language
C#
Runtime
.NET
Has code
yes
Has tests
yes

Use cases and integrations

How curated lists describe it

Frequently asked questions

Is Microsoft-Decision-1 compatible with Jev?

The README says it uses the same endpoint shape, request body, answer fields and price.

What did the benchmark find?

Decision-1 scored 100% on team routing with no order flips; open-jev scored 85.4% and was sometimes overconfident.

What language is the client?

C# (.NET).

Related projects

  • chat2jev

    Workbench that turns OpenAI chat requests into Jev state and questions, compares both outputs, and publishes question sets as proxy routes.

    Proxyprototype
  • system-one-adapter-rust

    Rust port of TypeSafe's system-one-adapter that answers System One questions with OpenAI or Anthropic LLMs for comparison with Jev.

    ProjectbetaRust
  • jev-eval

    Third-party check of Jev versus GPT-4o-mini and Claude Sonnet 4.5 routing 60 multilingual photo-shoot booking inquiries.

    ProjectresearchTypeScript
  • system-one-adapter-python★ 392

    TypeSafe's official drop-in replacement for the SDK's system_one call backed by OpenAI, Anthropic or Gemini, for comparing Jev with LLMs.

    LibraryproductionPython
  • jev-experiment

    Learning experiment comparing Jev with an OpenAI model on GitHub issue ownership and on AG News, TREC and SNIPS classification.

    ModelresearchPython
  • Decision Models Showdown

    Web app that sends identical ticket-routing and alert-triage questions to OpenAI Decisions and Jev side by side.

    BenchmarkprototypePython