jev-shadcn-lint-eval
Small eval that uses Jev to judge whether shadcn-ui/lint findings are correct and whether each message says what to change.
Overview
jev-shadcn-lint-eval is a second evaluation for the shadcn-ui/lint project (commit 53de86f). The upstream repo measures how agents style UI before and after lint feedback; this one asks TypeSafe's Jev whether each lint result is right and whether its message tells you what to change. It provides a design-system file with real color tokens and component variants for five fixtures, the six rules, the questions sent to Jev, and 131 rule test cases, 79 that should be reported and 52 that should not.
In one run on 2026-09-17, Jev caught 91% of real violations but flagged 16 of 52 clean cases, for 82% overall accuracy with 0.7 as the yes cutoff. Many errors involved cases a rule deliberately ignores, which rules.json does not yet describe. On message quality, Jev rated 115 messages from 0 to 2 with an average of 1.41. The run made 246 requests with no errors using 249,384 input tokens. Two earlier runs without component data scored 79% and 82%, so adding components did not clearly change accuracy.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Does this code break the rule? | yes · no |
| Score | Does this lint message say what to change? | 0 no help · 1 · 2 names exact variant, color or file |
How it works
- Build shadcn-ui/lint locally
- Extract the design system from the lint repo
- Run the linter on each rule test case
- Ask Jev whether each result is correct and the message helpful
- Save records and summary scores
Strengths
- Small, reproducible scripts
- Reports both catches and false flags
- Honest about unchanged accuracy across runs
Caveats
- Single run; results may vary
- Rules do not yet tell Jev what they ignore
- Agent-run states not yet sent to Jev
Getting started
With a built shadcn-ui/lint checkout, set TYPESAFE_API_KEY and run node run-rule-cases.mjs rule-cases.jsonl --repo <lint checkout>At a glance
- Type
- Project
- Status
- research
- Language
- JavaScript
- Runtime
- Node scripts
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
A small second eval for shadcn-ui/lint that uses TypeSafe's Jev to judge the linter's own output.
hellogumbo/awesome-jev
Frequently asked questions
What does Jev decide?
Whether code breaks a lint rule, and how actionable each lint message is on a 0 to 2 scale.
How accurate was it?
82% of 131 cases in one run, catching 91% of real violations but flagging 16 of 52 clean cases.
What comes next?
Telling Jev what each rule ignores and testing it on real agent runs.
Related projects
Node tool that crosschecks a folder of Markdown notes for contradictions and repeats, and ranks notes by any question, using Jev.
ProjectbetaJavaScriptCLI and agent skill that checks every line of a document against every rule with Jev and outputs flags for an AI agent to fix.
Agent skillbetaTypeScriptAgent observability platform that finds and fixes agent failures, with an optional Jev preclassifier for conversation checks.
ProjectproductionTypeScriptOpen-source Claude Cowork alternative built on OpenCode; per list notes, its eval testkit uses Jev as a verification judge.
ProjectproductionTypeScriptWindows Codex hook that audits a conversation's recorded execution with Jev and answers yes/no questions like 'Did the tests pass?'.
ProjectbetaPowerShellClaude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.
PluginbetaPython