Evaluation & Benchmarking

CLI that checks Jev questions against your labelled examples and grades each: act on it, only sort by it, or rewrite it.

Offline CLI that measures whether a classifier's confidence matches reality and turns mistake costs into a human hand-off threshold.

Openlayer library packing agent evals and guardrails (tool choice, grounding, injection, PHI) into one decision-model request.

CLI that fits per-question confidence thresholds to a target accuracy on your labelled data, reports escalation share and fails CI on drift.

Pre-registered audits of AI judges that measure how many decisions a judge can take alone at a given error rate, with calibration.

Project Zomboid performance patches whose benchmark harness computes numbers in code and, per a list, asks Jev only for the verdict.

Apple Silicon workbench running Laya, Decider, Kev and Jev side by side with accuracy, calibration, latency and memory benchmarks.

Python toolkit that evaluates LLM, RAG and agent outputs and guards tool calls with typed decision-model questions.

Web app that sends identical ticket-routing and alert-triage questions to OpenAI Decisions and Jev side by side.

Weekly go-to-market workflows built on Jev, such as lead triage and deal risk, each tested against Claude.

Benchmarks Jev against Luna, Haiku, Gemini and Astra at choosing legal moves in tic-tac-toe and Connect Four.

Pong where the ball moves one step per model decision, pitting Jev against chat LLMs to make decision latency visible.

Small Node TypeScript CLI for trying Jev on synthetic support text: one question, three in one request, and latency benchmarks.

Mobile-first Hugging Face space for exploring tiny decision models: feel the latency gap, try a toy router and vote on repros.

Small harness calling Jev via OpenRouter's Decisions API, with 17 doc-derived cases and a latency and cost benchmark against two LLMs.

Small labelled support-triage benchmark of jev-1.13 on OpenRouter covering noul, choice, score, routing, consistency and cost.

Curiosity project, not a benchmark: over a million questions from 312 sources put to Jev, mapped by topic and written up as experiments.

Pre-registered independent check of Jev's calibration and type safety on Banking77 and CLINC150, with a split verdict and sealed logs.

Research framework for biomedical decision models on Qwen3.5, studying domain pretraining, NLI training, calibration and abstention.

Benchmark comparing Jev, other decision models and LLMs on four judgments over 60 synthetic candidate-experience reviews.

Benchmark of Jev, OpenAI Decisions, Clef, Microsoft-Decision-1 and Claude on 2,321 labelled cases for latency, cost and calibration.

Independent benchmark of Jev, constrained-output LLMs and baselines on intent and spam tasks: accuracy, latency, cost, calibration.

Bilingual evidence map of where Jev holds up and where it breaks, built from recorded API-call receipts rather than a leaderboard.

Offline-reproducible benchmarks of Jev decisions in DSPy pipelines: calibration, selective risk, confidence gating and cost.

Intrusion detection where Jev sees one network flow and five labelled examples and returns attack yes/no and category.

Reproducible benchmark of Jev vs Claude Haiku 4.5 on 2,000 emails deciding whether an agent should click a link, with calibration audit.

Fourteen-dataset reranking study of Jev vs Cohere Rerank 4, zerank-2 and chat models, with raw responses and bootstrap intervals.

Graded relevance study over 9,831 labelled pairs: Jev reranking alone does not beat embeddings, but fused with them it is best.

Benchmark of Jev 1.13.0 against 11 classical classification pipelines on eight datasets under a published protocol.

Benchmarks Jev against frontier LLMs on Who&When Pro: attributing multi-agent failures to an agent, step and error type.