typesafe-jev-calibrate-for-code-review
Write-up with measurements on calibrating Jev as a code reviewer: context effects, textual tells, thresholds and noise near thresholds.
Overview
This repository is an essay with reproducible measurements on what it took to make Jev useful for code review in a Unity game project. The author wrote two rulesets from their coding conventions, one for comments and one for code, built a small Python CLI, and tested on representative files.
Findings include: adding questions barely changes answers but adding state changes them a lot, so topics that do not share context should not be batched; one request per comment plus the whole file performed best; Jev does better reading explicit facts (for example an accessibility field from an analyzer) than inferring them; useful thresholds can sit well below the common 0.8 default; and noise is largest in the mid-range near thresholds, with standard deviations up to about 0.04 there versus near zero at the extremes. The tool drove real fixes in the codebase.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Does this comment merely restate the code? | yes · no |
| Yes / no | Does this declaration carry the keyword public or internal? | yes · no |
How it works
- Turn coding conventions into Jev rulesets.
- Send comment and code pairs with chosen context.
- Compare flags across request shapes and wordings.
- Tune thresholds per rule and measure run-to-run variance.
Strengths
- Concrete measurements on context, wording and noise.
- Practical advice for threshold setting.
- Reproducible numbers.
Caveats
- Based on a few files from one project.
- The CLI itself is a small personal tool.
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- CLI
- Has code
- yes
Use cases and integrations
How curated lists describe it
About calibrating Jev for code reviews
logicrw/awesome-jev-projectsAbout calibrating Jev for code reviews.
hellogumbo/awesome-jev
Frequently asked questions
What is the biggest lesson?
State, not question count, drives answers; batch only rules that share the same context.
What threshold should I use?
It depends on the rule; the author started at 0.6 and found useful separation even below 0.5 in some cases.
Where is the noise worst?
In the mid-range near typical thresholds, not at the extremes.
Related projects
Natural-language linter combining ast-grep selectors with Jev to catch misleading names, stale comments and tests that miss their claims.
ProjectbetaTypeScriptAutonomous PR review bot with typed Jev questions, deterministic approval gates and escalation of uncertain reviews to trusted owners.
ProjectprototypePython- sapho★ 1
Rust library and CLI for compound decisions: ask Jev small questions, combine answers with rules in a YAML/JSON decision graph.
CLIbetaRust Nautilus Compass agent memory layer whose jev-trust middleware logs Jev decisions and measures calibration in your domain.
ServicebetaPythonOptimises the instructions and criterion descriptions of a Jev Choice from labelled data with EvoPrompt or GEPA, scored by Brier.
ProjectbetaPythonDecision-forensics library that explains a Jev answer by ablating parts of the input and measuring how far the probability moves.
ProjectbetaPython