Evaluation & Benchmarking

Reproducible evaluation harness and field note for the Jev Ultrafast research browser, with QC-graded baseline and stress suites.

Short report comparing Jev and many other fast decision classifiers with an LLM on grading section relevance in technical documents.

Small eval that uses Jev to judge whether shadcn-ui/lint findings are correct and whether each message says what to change.

Small Jev measurement bench with an append-only call log, labelled claims and a stated negative result.

Harness that runs Jev on all 19,646 TMMLU+ Traditional Chinese questions as four-way Choices, reporting accuracy and calibration.

Benchmark release evaluating Theory of Mind in five typed decision models versus seven generative baselines on nine ToM datasets.

Benchmark of BM25, embeddings, rerankers and Jev for picking the right MCP tool among 525, plus an experimental Jev search engine.

RPG experiment pitting Jev against OpenAI's Decisions API as NPCs, comparing wins, per-move accuracy, calibration and cost.

Learning project benchmarking zero-shot Jev and GPT models against scikit-learn classifiers and ModernBERT on five datasets.

Benchmark harness pitting Jev against Claude Sonnet 5 and Haiku 4.5 on Banking77 intent sorting with a confidence gate.

Research code showing that short, natural-looking context additions can flip decision models like Jev to a chosen wrong answer.

Benchmark of Jev against low-cost LLMs on 130,000 classification decisions across five datasets for accuracy, cost and latency.

Experiment testing whether a local Qwen read via label log-probabilities can replace Jev on synthetic AIOps alert decisions.

Local Jev-style decision model on Qwen3.5-4B with four stress tests that show where one-pass decisions break.

LogitTrail (formerly Mini Jev): local workbench that reads typed Choice, Noul and Score decisions from a frozen LLM's logits.

Recorded benchmark of Jev versus two frontier LLMs on 100 synthetic security-triage cases, with a static dashboard.

Study finding commercial decision models including Jev give different yes/no probabilities on Taiwan statements by Chinese script.

Static charts comparing Jev with 110 other models on Thai standardized exams for accuracy, cost, speed and calibration.

Short evaluation finding that Jev plays chess poorly, often calling unbalanced positions draws.

Reproducible experiments and reports on what Jev can and cannot do, from skill routing to GPQA and housing prices.

TypeSafe's official drop-in replacement for the SDK's system_one call backed by OpenAI, Anthropic or Gemini, for comparing Jev with LLMs.

Official TypeSafe code to reproduce evals.typesafe.ai: four workflow evals comparing Jev with frontier LLMs on cost and accuracy.

Bun harness that runs Jev over datasets via OpenRouter, streams JSONL answers, and checks calibration, stability and regressions.

C# System One client and benchmark comparing Microsoft-Decision-1, Jev and open-jev on the same labelled support tickets.

Reproducible bake-off of Playwright CLI alone, CLI plus Jev, and Playwright MCP on fuzzy browser goals, with honest results.

Interactive teaching site where Jev races frontier LLMs in games, an arena, a sandbox and quizzes, with a keyless replay mode.

Next.js site with six live Jev demos, each compared side by side with Gemini or Claude on latency, cost and agreement.

Writing checker running separate named Jev checks such as hedging, filler, tone and passive voice, each with its own verdict.

Community tool that fetches a SaaS homepage as Markdown and asks Jev ten questions about first-visit clarity, with a leaderboard.

Reliability testing for typed decision models: contracts, calibration metrics, metamorphic fuzzing and CI pass/warn/fail gates.