The official `consistency_choice` cookbook formally answered "why do repeated-call results change"
TypeSafe cookbook on why repeated Jev calls can differ, showing how an abstention rule raises repeat agreement.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
This is an official TypeSafe documentation cookbook, consistency_choice, addressing why repeated Jev calls can return different results. According to the list summary, it ran the same review rubric of 8 Choice questions, repeating each condition 15 times. Raw agreement was 90.8%, with 2 of 8 questions flipping, but every flip was between a specific label and "uncertain", with no conflicting labels.
Adding an abstention rule, requiring the top probability to be at least 0.60 and otherwise routing "uncertain" to a human, raised agreement to 99.2% with 74.2% handled automatically and no extra model calls. Comparison figures are given for several LLMs. The cookbook stresses that these numbers measure repeatability, not correctness, that 0.60 is only an example, and that production thresholds should come from labelled samples and costs.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Review rubric questions (8 Choice questions, including an uncertain option) | — |
| Other | Is the top probability at least 0.60, or should the item abstain to a human? | — |
How it works
- Ask the same Choice questions repeatedly under each condition.
- Measure raw agreement across repeats.
- Apply an abstention rule on the top probability.
- Measure agreement and automation rate after abstention.
Strengths
- Official guidance on handling run-to-run variation.
- Abstention improves repeatability without extra calls.
Caveats
- Measures repeatability, not accuracy.
- The 0.60 threshold is an example, not a recommendation for production.
At a glance
- Type
- Article
- Status
- production
Use cases and integrations
How curated lists describe it
The same review rubric, **8 Choice × each condition repeated 15 times**. TypeSafe **raw agree 90.8%** (2 of 8 questions flipped, but **conflicts=0**—the flips occurred only between "specific label ↔ uncertain"). **After adding an abstention rule (using `probabilities` to require top probability ≥ 0.60; otherwise `uncertain` routes to a human), agreement rose to 99.2%**, automatic 74.2%, **with no increase in model calls**. Comparison: Haiku t=0 100%, Opus 4.8 reasoning 92.5%, gpt-5.4-mini t=0 99.2%. Official statement: this does not make the model certain; these figures measure **repeatability, not correctness**; 0.60 is an example strategy, and production thresholds should be determined using labeled samples and costs.
wh000wh000/awesome-jev-live
Frequently asked questions
Why do repeated Jev calls change?
Per the cookbook, flips occurred between a specific label and uncertain, not between conflicting labels.
How can I reduce variation?
Add an abstention rule using the probabilities, such as requiring a top probability of at least 0.60 and routing the rest to a human.
Does higher agreement mean higher accuracy?
No, the cookbook states these figures measure repeatability, not correctness.
Related projects
Library of 43 field-tested Jev question patterns with templates, thresholds and failure modes, plus 10 anti-patterns; CC0, EN/ZH.
ArticleproductionTypeSafe's documentation of where Jev 1.13 is known to fail: counting, dates, indirection, distractors, adversarial state and more.
ArticleproductionFlavio Copes' hands-on walkthrough of Jev and the System One idea, aimed at developers who have only seen LLM marketing.
ArticlediscussionIndependent multilingual guide site covering Choice, Score, Noul, Python SDK requests, thresholds, fallbacks and human escalation.
ArticlePythonShow HN thread for Knowledge Signal, an open-source prototype that grades study notes against a rubric with Jev.
DiscussiondiscussionAgent skills for placing, building, evaluating and improving decision-model systems, with Jev as the default hosted example.
Agent skillbeta