Evals
TypeSafe's published eval methodology and per-model results for System One decision workflows.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
evals.typesafe.ai is TypeSafe's site for published evaluations. According to the list summaries, it describes the evaluation methodology and reports per-model results for System One workflows.
It is the vendor's own measurement, useful alongside independent benchmarks such as the Decision Index or community field tests when judging how Jev performs on a given task type. The site content was not reviewed here.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Vendor evaluation of System One models on workflow tasks(inferred) | — |
How it works
- Browse methodology
- Compare per-model results for relevant workflows
- Re-run comparable evaluations on your own data before relying on results
Strengths
- Official methodology and results
- Describes methodology alongside results
Caveats
- Vendor-published; compare with independent benchmarks
- List-only entry
Getting started
Visit https://evals.typesafe.ai.At a glance
- Type
- Article
- Status
- production
Use cases and integrations
How curated lists describe it
Published eval methodology and per-model results for System One workflows.
hellogumbo/awesome-jevTypeSafe's published evaluations.
MrJev/awesome-jev
Frequently asked questions
Who publishes these evals?
TypeSafe.
What do they cover?
Methodology and per-model results for System One workflows.
Are they independent?
No, they are vendor-published.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonAudit of 300 web pages for AI citability with nine Jev decisions each, checked against an LLM judge for agreement, cost and latency.
ProjectresearchPython