`calibre`：独立标定实验证明"阈值参数不跨数据集迁移"
Janus (formerly calibre): measures on your data the confidence threshold at which a small model like Jev should defer to a larger one.
Overview
This repository now carries the Janus project. Janus sends each decision to a small model or a larger one depending on how confident the small model is, and it measures where that line should sit on your data before routing anything. It ships no default threshold.
It measures from a labelled dataset (reporting how often each model is right) or from a log of decisions already taken (reporting how often the logged model agrees with a reference, explicitly without implying ground truth). Backends are optional extras, including TypeSafe Jev as primary (with advice to pin a resolved version rather than an alias) and DeepSeek or OpenAI-compatible models. The listing describes the earlier calibre work as an independent calibration experiment showing threshold parameters do not transfer across datasets. Example outputs are replayed from committed raw JSONL.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Primary model's answer to each item (e.g. Jev)(inferred) | — |
| Other | At what confidence threshold should decisions be escalated to the larger model? | — |
How it works
- Run the small and large models on a labelled set or a log
- Measure accuracy or agreement by confidence
- Pick a threshold for your data
- Route: small model above threshold, larger model below
Strengths
- No magic default threshold
- Honest about logs without ground truth
- Minimal dependencies (numpy plus extras)
Caveats
- Thresholds measured on one dataset may not transfer
- Requires labelled data or logs
Getting started
pip install "janus-decide[typesafe,deepseek]"At a glance
- Type
- Project
- Status
- beta
- Language
- Python
- Runtime
- CLI / library
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Independently measured Jev calibration on two labeled datasets.
wh000wh000/awesome-jev-live
Frequently asked questions
Why no default threshold?
Because thresholds do not transfer across datasets; Janus measures one on your data.
Can it work without labels?
Yes, from a decision log, reporting agreement with a reference model rather than accuracy.
Should I use jev-latest?
The README advises pinning a resolved version like jev-1.13.0.
Related projects
CLI that fits per-question confidence thresholds to a target accuracy on your labelled data, reports escalation share and fails CI on drift.
BenchmarkbetaPythonLocal-first routing engine for long-running agent tasks that uses a Jev decider to pick a model group per turn.
PluginproductionRust, TypeScriptOpen-source AI gateway for 100+ LLMs that can use Jev to classify requests for complexity routing and to compact tool results.
ProxyproductionPythonOpen-source workspace for running and reviewing AI coding agents across devices, with an optional Jev-based automatic model router.
ProjectproductionTypeScript- jev-router★ 593
Per-turn model routing for Claude Code and Codex: wraps the real CLI and lets Jev send simple turns to fast models and hard ones to strong.
ProxybetaJavaScript - jev-codex-router★ 274
Per-call router for Codex where Jev picks the model and thinking effort for every model call, including tool continuations.
ProjectbetaPython