Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

Jev DSPy Lab

Offline-reproducible benchmarks of Jev decisions in DSPy pipelines: calibration, selective risk, confidence gating and cost.

BenchmarkresearchMIT
View on GitHub ↗

Overview

Jev DSPy Lab is a measurement companion for dspy-typesafeify and other DSPy-style pipelines that use TypeSafe decisions. It notes that the existing proof of concept compares only three cases, needs both OpenAI and TypeSafe keys, disables caching, and does not report dataset-level calibration, selective risk or a fail-closed abstention policy, and it fills those gaps.

It records and replays TypeSafe calls so benchmarks run offline from fixtures without keys. Reports include calibration, selective risk under a confidence gate, latency, tokens, modelled cost, bootstrap intervals, request hashes and an exploratory threshold sweep, with a warning not to report the best sweep row as an independent result. A v0.3 workflow fits Platt scaling only after the confirmatory gate is declared, to avoid leakage.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich owner should this ticket be assigned to?—

How it works

  1. Load a ticket dataset and recorded TypeSafe responses
  2. Apply a declared confidence gate
  3. Compute calibration, coverage, selective risk, latency and cost
  4. Optionally refit calibration on a held-out split after the gate is fixed

Strengths

  • Runs offline without API keys
  • Leakage-aware calibration refit
  • Bootstrap confidence intervals

Caveats

  • Unofficial
  • Fixture dataset is small and ticket-focused

Getting started

`uv sync && uv run pytest -q`, then `uv run jev-dspy-benchmark --dataset fixtures/tickets.jsonl --responses fixtures/typesafe_responses.jsonl --output evidence/benchmark --field owner --threshold 0.7`.

At a glance

Type
Benchmark
Status
research
Language
Python
Runtime
CLI (uv)
Has code
yes
Has tests
yes
License
MIT

Use cases and integrations

How curated lists describe it

  • Unofficial DSPy companion that records and replays TypeSafe calls while measuring calibration, selective risk, confidence-gated abstention, latency, tokens, and modeled cost.

    AnotiaWang/awesome-decision-models

  • Reproducible calibration and selective-risk benchmarks for Jev/TypeSafe decisions in DSPy workflows

    logicrw/awesome-jev-projects

Frequently asked questions

What does it measure?

Calibration, selective risk, abstention under a confidence gate, latency, tokens and modelled cost.

Do I need API keys?

No, recorded responses let the benchmark run offline.

Is it a DSPy fork?

No, it is a companion measurement lab.

Related projects

  • jevcheck

    pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.

    ProjectbetaPython
  • jevoss

    Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.

    ServicebetaPython
  • workshop-jev

    Portuguese workshop: a LangGraph sales agent where each decision step can be made by Jev or a frontier LLM and measured side by side.

    ProjectprototypePython
  • can-jev-play

    Experiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.

    Projectresearch
  • Jev × LexGLUE

    Zero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.

    ProjectresearchPython
  • Jev vs GPT-4.1 on a synthetic survey

    Pre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.

    ProjectresearchPython