Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-harness

Bun harness that runs Jev over datasets via OpenRouter, streams JSONL answers, and checks calibration, stability and regressions.

Librarybeta
View on GitHub ↗

Overview

jev-harness by cyanheads runs Jev over datasets. An experiment is one TypeScript file declaring which record fields to send and which questions to ask; the run command sends every record through OpenRouter, streams one JSONL row per record and prints a summary. Offline commands then calibrate answers against known labels, measure stability across repeated runs, and compare runs before and after a change.

Question builders cover Choice (1 to 255 options), Score (2 to 10 ordered levels with a probability-weighted position) and Noul, and instructions and options can be structured JSON with examples. A dry-run mode prints the request without calling anything, and a sample ticket-routing experiment is included. Documentation covers how to write questions. It requires Bun 1.3+ and an OpenRouter key.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich team should this ticket go to? (sample experiment)(inferred)—

How it works

  1. Write an experiment file selecting fields and questions
  2. Run over a JSONL dataset via OpenRouter
  3. Stream answers to results/ and print a summary
  4. Run calibrate, stability or compare offline

Strengths

  • Dataset-scale runs with structured output
  • Built-in calibration and stability checks
  • Dry-run without a key

Caveats

  • OpenRouter only
  • Requires Bun

Getting started

`bun install`, set OPENROUTER_API_KEY, then `bun run jev run ticket-routing --input samples/tickets.jsonl --dry-run`.

At a glance

Type
Library
Status
beta
Language
TypeScript
Runtime
Bun CLI
Has code
yes

Use cases and integrations

How curated lists describe it

Frequently asked questions

What does it do?

Runs typed Jev questions over every record in a dataset and evaluates the answers.

Can I test without spending?

Yes, --dry-run prints the request without calling Jev.

Which provider is used?

OpenRouter.

Related projects

  • jev-arena★ 122

    Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.

    ProjectbetaTypeScript
  • claude-jev-funnel

    Claude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.

    PluginbetaPython
  • jevcheck

    pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.

    ProjectbetaPython
  • jevoss

    Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.

    ServicebetaPython
  • can-jev-play

    Experiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.

    Projectresearch
  • goodwatch-monorepo

    GoodWatch movie and TV recommendation monorepo, listed for an experiment comparing Jev question designs for attribute scoring.

    ProjectresearchTypeScript