Skip to content

Calibration and threshold setting with decision models

Measuring whether a model's probabilities can be trusted and picking per-question confidence thresholds, abstention rules and fallbacks.

236 public projects · page 1 of 8

  • Rust CLI that ranks which agent skills fit the next step from live session context with Jev, can abstain, and hooks into Claude Code.

    CLIbetaRust
  • Janus (formerly calibre): measures on your data the confidence threshold at which a small model like Jev should defer to a larger one.

    ProjectbetaPython
  • Claude Code PreToolUse layer where Jev answers YAML-defined questions about each tool call, with 19 hard rules and calibration runs.

    ProjectbetaTypeScript
  • Instrumented 2048 web lab where every move is a Jev Choice over four directions, with probabilities, latency and cost shown live.

    ProjectbetaTypeScript
  • OpenAI-compatible endpoint that answers an agent's command-approval call with APPROVE, DENY or ESCALATE from a Jev Choice, failing closed.

    ProxybetaPython
  • Agent skills for placing, building, evaluating and improving decision-model systems, with Jev as the default hosted example.

    Agent skillbeta
  • Claude Code plugin that sends judgment calls to Jev, acts at 0.8+ for yes or 0.2- for no, and gathers more evidence in between.

    Agent skillbetaJavaScript
  • Claude Code plugin where Jev decides and Claude writes: sort piles of text, calibrate thresholds, route prompts and gate Bash commands.

    Pluginbeta
  • Claude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.

    PluginbetaPython
  • Unofficial Claude Code plugin that checks 'done' claims and small judgment calls with Jev, logs receipts, and can warn on unverified stops.

    Pluginbeta
  • Rust CLI and library that asks Jev typed questions via OpenRouter and combines the probabilities with fuzzy-logic rules into decisions.

    CLIbetaRust
  • GitHub Action that triages issues with Jev, applying label, spam, needs-info and duplicate tags only above set thresholds.

    ProjectbetaTypeScript
  • Unofficial standard-library Go client for Jev with generic typed Choice values and local confidence thresholds.

    ProjectbetaGo
  • Rust CLI for Jev: ask yes/no, choice and rubric questions about text and branch on exit codes, JSON or MCP tools.

    MCP serverbetaRust
  • Nautilus Compass agent memory layer whose jev-trust middleware logs Jev decisions and measures calibration in your domain.

    ServicebetaPython
  • Portuguese-language agent skill for millisecond Jev decisions via OpenRouter, with validation, an MCP mode and calibration evals.

    Agent skillbetaJavaScript
  • Editorial QA tool where Jev judges per fact whether an edit preserved the original, with a confidence gate and human review fallback.

    ProjectbetaJava
  • TypeScript library adding policy, confidence gates, shadow mode, recipes and an eval CLI around Jev answers.

    CLIbetaTypeScript
  • Natural-language linter combining ast-grep selectors with Jev to catch misleading names, stale comments and tests that miss their claims.

    ProjectbetaTypeScript
  • Eval-first MCP server for Jev: prototype questions, map them over many items, and measure accuracy and thresholds on labelled data.

    MCP serverbetaTypeScript
  • Rust MCP server for Codex that hands classification, relevance, candidate selection and evaluation to Jev with accept/verify policies.

    MCP serverbetaRust
  • Optimises the instructions and criterion descriptions of a Jev Choice from labelled data with EvoPrompt or GEPA, scored by Brier.

    ProjectbetaPython
  • Toolkit for Jev decision contracts: compiled policies, egress controls, tracing, stability checks and certified decision SLOs.

    ProjectbetaTypeScript
  • Decision-forensics library that explains a Jev answer by ablating parts of the input and measuring how far the probability moves.

    ProjectbetaPython
  • pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.

    ProjectbetaPython
  • CLI that has Jev judge a video's subtitles segment by segment, then cuts a highlight reel and a filler-free full version.

    ProjectbetaPython
  • Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.

    ServicebetaPython
  • Python library that replaces hard-coded RAG thresholds with calibrated decisions for stopping, chunking, selection, abstention and caching.

    ServicebetaPython
  • SQLite and Node library adding Jev judgments to SQL for matching, evidence picking, ranking, review queues and semantic CI checks.

    ProjectbetaJavaScript
  • Record linkage for researchers: write the match rule in plain English and Jev judges each candidate pair with a probability.

    ProjectbetaPython

Related categories

Related coding agents