Jev Can't Be Calibrated
Blog post by Alex Molas with an HN thread arguing statistically that Jev's calibration claim cannot hold, with methods written out.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
Jev Can't Be Calibrated is a blog post by Alex Molas, discussed on Hacker News (59 points and 60 comments per the yibie list). It makes a statistical argument that the calibration claim attached to Jev cannot hold in general, and writes out the methods behind the argument.
It is part of a broader debate in the ecosystem about what calibration means for a hosted decision model, alongside empirical audits that find calibration holding on some tasks and failing on others. The post itself was not fetched, so the specific argument should be read at the source.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Can a single model's probabilities be calibrated across all tasks and distributions?(inferred) | — |
How it works
- Read the statistical argument
- Compare with empirical calibration audits
- Calibrate thresholds on your own data
Strengths
- Clear theoretical counterpoint to marketing claims
Caveats
- Argument not summarised in the source list
- Opinion piece, not an empirical study
At a glance
- Type
- Article
- Status
- discussion
- Has code
- no
Use cases and integrations
How curated lists describe it
Blog + HN thread (59 points, 60 comments): a statistical argument that the calibration claim cannot hold, with the methods written out.
yibie/awesome-jev
Frequently asked questions
What does the post argue?
That Jev's calibration claim cannot hold in general, on statistical grounds.
Was it discussed?
Yes, on Hacker News with 59 points and 60 comments.
What should practitioners do?
Measure calibration on their own labelled data before relying on thresholds.
Related projects
Article cross-checking 108 statements to argue that Jev's calibration is not unique, including the author's own corrections.
ArticlediscussionReddit thread arguing Jev's type-safety guarantee is real but its calibration claim lacks a published ECE or reliability curve.
DiscussiondiscussionHacker News thread on RLCD, the training approach described as behind Jev's calibrated decisions.
DiscussiondiscussionAuto-updated bilingual leaderboard of Jev ecosystem projects, tutorials and discussions, discovered daily and sorted by GitHub stars.
ProjectproductionJanus (formerly calibre): measures on your data the confidence threshold at which a small model like Jev should defer to a larger one.
ProjectbetaPythonChatGPT-style web chat where every word of every reply is a Jev multiple-choice decision over a ~1,800-word vocabulary plus END.
ProjectprototypeTypeScript