Evaluation grading with decision models

System One Mosaic Benchmark comparing Jev and open decision models across 137 English and 27 Japanese Choice, Noul and Score tasks.

Agent skill that tells you when you need an eval and how to build it, with an opt-in lane routing closed-label residue to Jev.

CLI that checks Jev questions against your labelled examples and grades each: act on it, only sort by it, or rewrite it.

Offline CLI that measures whether a classifier's confidence matches reality and turns mistake costs into a human hand-off threshold.

Openlayer library packing agent evals and guardrails (tool choice, grounding, injection, PHI) into one decision-model request.

Pre-registered audits of AI judges that measure how many decisions a judge can take alone at a given error rate, with calibration.

Python toolkit that evaluates LLM, RAG and agent outputs and guards tool calls with typed decision-model questions.

Graded relevance study over 9,831 labelled pairs: Jev reranking alone does not beat embeddings, but fused with them it is best.

Experiment using Jev through langchain-typesafe as an agent-eval judge, compared with GPT-5.6 and Claude Sonnet 4.6 against human labels.

Large rubric-classification benchmark of Jev vs Claude Haiku 4.5, Sonnet 5, OpenJev and many open models, with full price tracking.

Benchmark for agent regression after config edits: a Jev Choice decides which approved answers changed meaning versus wording.

Reproducible evaluation of Jev 1.13 as a reward model, LLM judge and process verifier across eight benchmark tracks.

Exploratory study of zero-shot ham, spam and phishing email classification with Jev, compared with trained TF-IDF baselines.

Curated list of 136 Jev projects checked against their repositories, with a Top 25 ranked by stars and a Jev-rated interest score.

Browsable directory of 50 runnable Jev judge evals and 1,300+ community builds, also served to agents via MCP and llms.txt.

Vercel Labs terminal CLI for AI tasks that can use Jev as the evaluation model for its evaluate command.

Claude Code and Codex skills powered by Jev, published with a calibration curve measured on 4,995 real agent decisions.

Free web tool that checks any URL for 35 tells of AI-generated design and returns a slop score in about 250 ms.

Working paper placing a text judge beside a live trading system as a read-only signal, comparing Jev and Claude as blind headline judges.

TypeSafe's published eval methodology and per-model results for System One decision workflows.

TypeSafe cookbook on why repeated Jev calls can differ, showing how an abstention rule raises repeat agreement.

Kiln blog post on optimizing only the harness around Jev for Chinese polyphonic-character reading, halving errors at a seventh of the cost.

LangChain post comparing Jev with LLM judges for agent evaluation on accuracy, repeatability, latency and cost.

Study comparing one direct Jev question per row with 12–14 Jev-scored dimensions and fitted weights on three classification tasks.

Every Vibe Check article: Mike Taylor runs a writing corpus through TypeSafe's Jev and reports how it judged it.

ibl.ai article noting CriticGPT and Prometheus 2 built judge models in 2024, cited as prior art for Jev's design.

Boring Tools Kit's technical SEO audit cost study, where Jev ranks striking-distance fixes and content gaps by calibrated probability.

Show HN thread for Knowledge Signal, an open-source prototype that grades study notes against a rubric with Jev.