WorkflowEvals
Official TypeSafe code to reproduce evals.typesafe.ai: four workflow evals comparing Jev with frontier LLMs on cost and accuracy.
Overview
WorkflowEvals is TypeSafe's code for reproducing the results on its evals site. It assumes each workflow and its code are correct and evaluates models within them against the largest models from well-known labs. Four workflows are included: invoice approval and fraud triage under three policies (150 cases), customer-service routing and actions under three policies (204), agent-run action selection under two review profiles (111), and security-alert triage and containment (240).
The default model is jev-1.13.0, and other providers (OpenAI, Anthropic, Fireworks, Groq, Cerebras) can be selected with exact model IDs. Inputs and labels download from a Hugging Face collection, runs are saved and resumable, and pricing data gives cost estimates. There is a dry-run mode and options for concurrency and reasoning effort. As vendor-published evals, results are best cross-checked with independent benchmarks.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Should this invoice be approved, or flagged for fraud, under the given policy?(inferred) | — |
| Choice | How should this customer-service request be routed and handled?(inferred) | — |
| Choice | What containment action should this security alert get?(inferred) | — |
How it works
- Install with uv and set provider keys.
- Run a workflow with a chosen model.
- Inputs download from Hugging Face; results are saved per run.
- Compare accuracy and cost across models.
Strengths
- Reproducible vendor evals.
- Supports many providers with the same workflows.
- Resumable runs and dry-run mode.
Caveats
- Published by the model vendor.
- Workflows are synthetic business scenarios.
Getting started
uv sync --locked && export TYPESAFE_API_KEY=... && uv run python run.py invoice_processing --limit 5At a glance
- Type
- Library
- Status
- production
- Language
- Python
- Runtime
- CLI
- Has code
- yes
- Stars
- 17
Use cases and integrations
How curated lists describe it
evals.typesafe.ai workflow code published
wh000wh000/awesome-jev-liveEvals.typesafe.ai workflow code published.
hellogumbo/awesome-jev
Frequently asked questions
What does it reproduce?
The results on evals.typesafe.ai.
Which models can I test?
Jev plus OpenAI, Anthropic, Fireworks, Groq and Cerebras models.
Where does the data come from?
A WorkflowEvals collection on Hugging Face.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonAudit of 300 web pages for AI citability with nine Jev decisions each, checked against an LLM judge for agreement, cost and latency.
ProjectresearchPython