Janus
Python tool that measures where to route decisions between Jev and a larger model on your own data, then routes by confidence.
Overview
Janus decides whether each decision goes to a small model such as Jev or to a larger fallback, based on how confident the small model is, and it measures where that line should sit on your data before routing anything. It ships no default threshold. It can measure from a labelled dataset (how often each model is right) or from a log of past decisions with confidences (how often the logged model agrees with a reference model), and refuses to describe log results as accuracy since there is no ground truth.
It is published on PyPI as janus-decide with backends as extras; the example compares typesafe:jev-latest against deepseek-v4-pro. Runs support small samples to check wiring and cost, a budget cap that refuses to start when a model has no known price, and resumption without re-paying. The lists report measurements on Banking77 and Web of Science, including 80.2% accuracy at $0.103 per 500 decisions on Banking77. It has a test workflow and an MIT licence.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which banking intent does this customer query express? | lost_or_stolen_card · card_arrival · ... |
| Other | Is the primary model's confidence high enough to keep its answer, or should this go to the fallback?(inferred) | keep · fallback |
How it works
- Prepare labelled JSONL and a label file, or a decision log
- Run janus measure with primary and fallback models
- Read the accuracy/cost table across confidence thresholds
- Choose a threshold and route low-confidence decisions to the fallback
Strengths
- Threshold is measured on your data, not assumed
- Budget cap and resumable runs control spend
- Honest reporting: logs yield agreement, not accuracy
- Lists report 80.2% at $0.103 per 500 decisions on Banking77
Caveats
- Needs a few hundred labelled rows for stable estimates
- Results depend on chosen fallback model and pricing
Getting started
`pip install "janus-decide[typesafe,deepseek]"`, then `janus measure --dataset mydata.jsonl --labels mylabels.json --primary typesafe:jev-latest --fallback deepseek:deepseek-v4-pro --out janus.json`.At a glance
- Type
- Benchmark
- Status
- beta
- Language
- Python
- Runtime
- CLI, library
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Framework for measuring when to employ Jev versus generative LLMs on proprietary datasets, routing queries based on measured benchmarks.
logicrw/awesome-jev-projectsMeasures when to use Jev versus other models and routes accordingly.
cobanov/awesome-jevCalibration and confidence-based routing measured on Banking77: 80.2% accuracy at $0.103 per 500 decisions.
hellogumbo/awesome-jevMeasures Jev's calibration and confidence-based routing on Banking77 and Web of Science.
MrJev/awesome-jev
Frequently asked questions
What does Janus decide?
Whether to keep Jev's answer or send the decision to a larger model, using a confidence threshold measured on your data.
Why no default threshold?
Because the right cut-off depends on your data and costs, so Janus measures it.
Can it run on logs without labels?
Yes, but it then reports agreement with a reference model, not accuracy.
Related projects
Research framework (DART) testing whether Jev's probabilities are a reliable signal to answer, escalate or abstain.
ProjectresearchPythonpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPython