system-one-reviewer
Local PR reviewer: deterministic diff handling plus Jev or Laya judgments of severity, category and true-positive per hunk.
Overview
system-one-reviewer is a single-file Python pull-request reviewer that splits work between deterministic code and a System One model. Code gathers the diff, isolates change clusters, applies thresholds and composes the verdict; the model answers three typed questions per cluster: a severity Score, a true-positive Noul and a category Choice. Deletion-only clusters get a separate question set focused on removal risk. It prints findings with scores, a PR-level risk and a needs-human-review probability.
The default provider is TypeSafe's hosted Jev, with local Laya available but only tested with a fake so far. Measurement is built in: every run logs all scores to a local jsonl and a golden mode scores precision, recall and F1 against planted issues. The README reports F1 rising from 0.0 to 0.29 to 0.50 across three design changes and notes that structured before/after code state raised a planted bug's true-positive score. Output is advisory only.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Score | How severe is the issue in this change? | — |
| Yes / no | Is this a real issue (true positive)? | — |
| Choice | What category is the issue (e.g. bug-risk)? | — |
How it works
- Collect the diff for a commit range or the working tree.
- Split it into isolated change clusters with before/after code lists.
- Ask three typed questions per cluster in one model call.
- Apply thresholds to report findings and compose a PR verdict.
- Log all scores and optionally score against golden fixtures.
Strengths
- One batched call per cluster keeps latency and cost low enough to run on every diff, per the README.
- Deterministic plumbing makes behaviour reproducible.
- Built-in precision/recall harness.
Caveats
- Experimental and advisory only.
- Laya provider has no live run yet.
- Hosted provider requires a TypeSafe API key.
Getting started
Download the release script and SHA256SUMS, verify, install it on PATH, set TYPESAFE_API_KEY, then run system-one-reviewer --repo PATH --range main..HEAD.At a glance
- Type
- Project
- Status
- prototype
- Language
- Python
- Runtime
- CLI
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Local PR reviewer: deterministic code + TypeSafe Jev (System One) judgments, with a built-in precision/recall evaluation harness. Built in public.
hellogumbo/awesome-jev
Frequently asked questions
What does the reviewer ask the model?
For each change cluster it asks severity (Score), whether the issue is real (Noul) and its category (Choice).
How is it different from an LLM code reviewer?
There is no agent loop or prompting; code does the plumbing and the model returns probabilities that thresholds turn into findings.
Can it run without TypeSafe?
A local Laya provider exists but is only tested with a fake; published numbers use hosted Jev.
Related projects
'Vitest for LLM outputs': CLI running YAML/JSON eval suites on Jev for RAG faithfulness, hallucination and agent safety in CI.
CLIprototypeTypeScriptSecure, persistent VM sandboxes for AI agents, with a PR-review example that compares a general model with Jev on candidate findings.
ProjectproductionPythonUnix-style tools where Jev learns a repo's decision norms and judges each node of a code lead as a bug, lead or clean.
ProjectbetaJavaScriptOh My Pi skill for method-first bulk classification: decide when to use Jev, then pilot and freeze a rubric for judge_batch.
Agent skillbetaPython- jev-harness★ 30
Research proposal-review contract: an LLM proposes one action, Jev answers four narrow questions, code produces evidence, applies nothing.
LibraryresearchTypeScript Agent observability platform that finds and fixes agent failures, with an optional Jev preclassifier for conversation checks.
ProjectproductionTypeScript