typesafe-eval
Document evaluation CLI and CI gate using Jev or OpenAI Decisions, with pre-registered validation and documented reliability limits.
Overview
typesafe-eval evaluates documents such as RFCs, design docs and specs along several typed dimensions and can act as a CI quality gate. It uses the TypeSafe System One API (jev-1.13.0) or the OpenAI Decisions API. Presets include safety (deterministic offline rules for secrets and PII, with detailed handling of Japanese phone numbers and email roles), tech-spec (test plan presence and readiness), design-doc (section presence) and quality (clarity and tone, for regression against a baseline).
What sets it apart is its honesty about reliability. It separates tuning and held-out corpora, pre-registers evaluation rounds and pass criteria, and labels each check by measured reliability, including which checks are not reliable and which are based on very small samples (N of 2 to 3). It is MIT licensed with CI and online documentation.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Does this spec include a verification or test plan? | yes · no |
| Choice | How ready is this spec for engineering? | — |
| Yes / no | Does this design doc describe alternatives / rollback / open questions? | yes · no |
| Score | How clear is this document? | — |
How it works
- Choose a preset for the document type.
- Run deterministic safety rules offline.
- Ask typed questions via Jev or OpenAI Decisions.
- Fail CI when checks or baseline regressions exceed limits.
Strengths
- Pre-registered validation and documented failure modes.
- Safety preset runs offline with zero API calls.
- Supports two decision APIs.
Caveats
- Several checks are validated on very small samples.
- Some checks are explicitly marked unreliable.
- Quality preset cannot distinguish real revisions from originals.
At a glance
- Type
- CLI
- Status
- beta
- Language
- Python
- Runtime
- CLI, CI gate
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Fast, typed multi-dimensional document evaluation CLI powered by TypeSafe System One (Jev).
hellogumbo/awesome-jev
Frequently asked questions
Which models does it use?
Jev via TypeSafe and gpt-6-luna via the OpenAI Decisions API.
How reliable are the checks?
Each preset documents its measured reliability; some checks are reliable, others are not or have tiny samples.
Does the safety preset call an API?
No, it uses deterministic offline rules.
Related projects
Analytics engine that rates published posts on pre-registered rubrics with Jev and tests ratings against real engagement.
ProjectbetaTypeScriptRussian-language Letta agent skill that uses Jev as a fast external judge, with rules for treating mid-range probabilities as 'unsure'.
ProjectprototypePythonWriting checker running separate named Jev checks such as hedging, filler, tone and passive voice, each with its own verdict.
DemobetaTypeScriptCommunity tool that fetches a SaaS homepage as Markdown and asks Jev ten questions about first-visit clarity, with a leaderboard.
ServicebetaTypeScriptUpload a paper PDF and get a fast Jev-based judgment of understandability and completeness, with low-confidence answers flagged.
DemoprototypeTypeScriptComparative judgment of ASAP essays by Jev and two LLM judges, turning pairwise decisions into scores compared with human grades.
ServiceresearchPython