Evaluation & Benchmarking

Agent observability platform that finds and fixes agent failures, with an optional Jev preclassifier for conversation checks.

Open-source Claude Cowork alternative built on OpenCode; per list notes, its eval testkit uses Jev as a verification judge.

Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.

Runnable Jev samples: README checks, AGENTS.md readiness, PR triage, MovieLens next-watch ranking and issue triage, with measured runs.

DuckDB extension adding decide_probability, decide_choice and decide_score SQL functions backed by Jev or local open decision models.

Windows Codex hook that audits a conversation's recorded execution with Jev and answers yes/no questions like 'Did the tests pass?'.

Claude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.

Claude Code Stop hook that checks an assistant's claims against what it actually read this session using Jev, blocking unsupported ones.

Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.

Hermes skill that turns an intention into a judgeable finish line and a run prompt, with Jev deciding fuzzy requirements at the gate.

Analytics engine that rates published posts on pre-registered rubrics with Jev and tests ratings against real engagement.

CLI that grades the coding-agent sessions on your disk (Claude Code, Codex, opencode, Gemini CLI) with Jev for cents.

Open-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.

pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.

Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.

Semantic test matchers for Vitest and Jest that ask Jev whether output satisfies a plain-English expectation and threshold the probability.

Local-first meeting transcription and conversation memory app whose live voice path runs its evaluations through Jev.

pytest plugin for semantic assertions on LLM output, asking Jev whether plain-English claims hold with a 0.8 probability bar.

Slack-like playground for watching computer-use agents work, with replays, comparable runs and Jev choosing from a menu of visible actions.

Zero-dependency Node proxy that shadows Jev with local decision models and reports whether you can safely swap, per question and policy.

Rust port of TypeSafe's system-one-adapter that answers System One questions with OpenAI or Anthropic LLMs for comparison with Jev.

Document evaluation CLI and CI gate using Jev or OpenAI Decisions, with pre-registered validation and documented reliability limits.

Local SvelteKit workbench and CLI for organizing, editing, running and comparing Jev experiments defined as JSON.

Desktop tool for testing chat endpoints conversationally, with rule checks and judged checks including TypeSafe's System One.

'Vitest for LLM outputs': CLI running YAML/JSON eval suites on Jev for RAG faithfulness, hallucination and agent safety in CI.

Python CLI that shows which OpenAI model Jev would route a query to, independently reviewed by a local Ollama model.

Jev-protocol-compatible random baseline server in Rust: answers noul, choice and score with fast pseudo-random numbers, as a mock or floor.

Learning lab for DeepEval with Jev as the judge: replayed evaluations, adjustable decision rules, release gates and drift alerts.

Russian-language Letta agent skill that uses Jev as a fast external judge, with rules for treating mid-range probabilities as 'unsure'.

Jev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.