judgekit
One YAML task to classify, route, score or verify text with rules, Jev, NanoJev or an OpenAI-compatible model, with cost records.
Overview
judgekit defines small, clearly bounded decisions as YAML tasks and returns a consistent decision JSON with a cost record. Four primitives are supported: classify (label), route (routing label), score (0 to 1) and verify (boolean). A task can start on free local keyword rules, which support classify and route, and switch to TypeSafe Jev's native API, an OpenAI-compatible endpoint or local NanoJev without changing the definition. Only the declared input field is sent to providers; gold labels and metadata stay local.
Failed or low-confidence model decisions can fall back to rules or be escalated. List sources report a confidence gate at 0.7 catching 3 of 3 misjudgments at 9% escalation on 130 items, 97.7% accuracy at about ¥0.105 per thousand decisions on a Chinese scenario, and a human-labelled spam set rising from 60.0% to 68.3% at a tuned threshold. A demo runs offline with rules in English or Chinese. It suits scripts and support workflows needing cheap, auditable decisions.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which department should handle this ticket? | refund · shipping |
| Yes / no | Does the text meet this clearly defined criterion? (verify) | — |
| Score | How well does this item match? (0-1 score) | — |
How it works
- Write a YAML task with primitive, criteria, labels and provider
- Run locally with rules or on Jev, NanoJev or an OpenAI-compatible model
- Low-confidence or failed decisions fall back to rules or escalate
- Output decision JSON with cost
Strengths
- Same task definition across backends
- Free rule-based start
- Only declared input leaves the machine
- Cost recorded per decision
Caveats
- Rule confidence is heuristic
- Published numbers come from small samples
- Score and verify need a model
Getting started
git clone https://github.com/lexingtonhibiki/judgekit && cd judgekit && python -m pip install -e . && python -m judgekit demoAt a glance
- Type
- CLI
- Status
- beta
- Language
- Python
- Runtime
- CLI and library
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Model evaluation: runs declarative YAML judgment tasks natively on Jev `Choice`/`Score`/`Noul` or any OpenAI-compatible backend (with a free rules fallback), gates low confidence at 0.7 (caught 3/3 misjudgments at 9% escalation, n=130), and publishes Chinese-scenario cost-accuracy numbers — 97.7% @ ¥0.105/1k decisions and 60.0% → 68.3% on a frozen 120-item human-labeled spam set at τ=0.10.
yibie/awesome-jev
Frequently asked questions
Can I run it without an API key?
Yes, the rules provider handles classify and route locally.
Which backends are supported?
Local rules, TypeSafe Jev, NanoJev and OpenAI-compatible endpoints.
What happens when the model is unsure?
A confidence gate can escalate or fall back to rules.
Related projects
GitHub Action that triages issues with Jev, applying label, spam, needs-info and duplicate tags only above set thresholds.
ProjectbetaTypeScriptProbot GitHub app that classifies, scores, checks completeness and dedupes issues and PRs with Jev, routing uncertain ones to humans.
ProjectbetaTypeScriptMultiplayer GitHub triage dashboard where Jev answers kind, severity, urgency, duplicate and next step per issue.
AppbetaCLI, MCP server and skill for issue triage where Jev predicts issue kind, affected module and whether information is sufficient.
MCP serverprototypePythonPython sandbox turning Jev issue-category and reproduction judgments into testable routing proposals, with offline replay.
ProjectprototypePythonPortuguese-language Go proof of concept that tests whether Jev's confidence is calibrated enough to auto-route support tickets.
DemoprototypeGo