Jev cannot emit an invalid output, but where is the reliability curve?
Reddit thread arguing Jev's type-safety guarantee is real but its calibration claim lacks a published ECE or reliability curve.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
This Reddit post in r/ArtificialInteligence argues that two of Jev's claims deserve different levels of trust. The type-safety guarantee, that Jev cannot emit an answer outside the allowed options, is real and follows from how it works. The calibration claim, that its probabilities can be trusted as frequencies, is not yet backed by a published expected calibration error or reliability curve from TypeSafe.
The yibie list calls it the sharpest form of a question that recurs throughout the ecosystem, and several independent audits in this directory set out to answer it. Only the list summary was available.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Is Jev's calibration claim supported by published reliability data?(inferred) | — |
How it works
- Separate the type-safety guarantee from the calibration claim
- Ask for published ECE and reliability curves
- Look to independent audits for evidence
Strengths
- Clear framing of what is guaranteed versus claimed
Caveats
- Discussion only; no new measurements
At a glance
- Type
- Discussion
- Status
- discussion
- Has code
- no
Use cases and integrations
How curated lists describe it
Reddit: argues the type-safety guarantee is real and the calibration claim is not yet backed by a published ECE or reliability curve, the sharpest form of the question this list keeps running into.
yibie/awesome-jev
Frequently asked questions
Is Jev's output really always valid?
Yes, the post accepts that it cannot emit an answer outside the allowed set.
What is disputed?
Whether its probabilities are calibrated, absent a published reliability curve.
Where can I find evidence?
Independent audits such as jev-calibration and jev-calibration-audit.
Related projects
Article cross-checking 108 statements to argue that Jev's calibration is not unique, including the author's own corrections.
ArticlediscussionBlog post by Alex Molas with an HN thread arguing statistically that Jev's calibration claim cannot hold, with methods written out.
ArticlediscussionHacker News thread on RLCD, the training approach described as behind Jev's calibrated decisions.
DiscussiondiscussionAuto-updated bilingual leaderboard of Jev ecosystem projects, tutorials and discussions, discovered daily and sorted by GitHub stars.
ProjectproductionJanus (formerly calibre): measures on your data the confidence threshold at which a small model like Jev should defer to a larger one.
ProjectbetaPythonChatGPT-style web chat where every word of every reply is a Jev multiple-choice decision over a ~1,800-word vocabulary plus END.
ProjectprototypeTypeScript