judging-with-typesafe
Russian-language Letta agent skill that uses Jev as a fast external judge, with rules for treating mid-range probabilities as 'unsure'.
Overview
judging-with-typesafe is a skill for Letta agents, documented in Russian, that uses TypeSafe's Jev as a fast, cheap external judge for texts and decisions. It wraps the API with noul, choice, score and batch helpers, and includes a document scanner that labels each paragraph as a supported claim, an enumeration, a question or an admission of openness.
The author stresses that the value is in the interpretation rules rather than the API calls. Values between 0.30 and 0.70 mean 'uncertain' and should go to a human rather than be forced into a verdict. A single question proved biased in their tests, with scientific-sounding nonsense scoring 0.74 to 0.82 while a textbook scored 0.48, so a four-question rubric is used instead. Thresholds must be calibrated per material using a contrasting pair of texts. The README reports run-to-run spread of 0.004 to 0.014 over eight runs, and says plainly that Jev cannot check facts: an invented source scored 0.80 against 0.87 for a real one.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this statement true / supported? (e.g. Are there more apples than pears?) | — |
| Choice | What kind of paragraph is this: supported claim, enumeration, question or admission of openness? | supported claim · enumeration · question · admission of openness |
How it works
- Install the skill into a Letta agent and set TYPESAFE_API_KEY
- Agent calls noul, choice, score or batch on a state
- For documents, scan_text.py judges each paragraph with a four-question rubric
- Values between 0.30 and 0.70 are marked uncertain for human review
- Thresholds are calibrated on a contrasting pair of texts
Strengths
- Reproducible verdicts with small run-to-run spread
- Explicit handling of uncertainty
- Honest limits documented
Caveats
- Cannot verify facts or sources
- Single questions can be biased; rubric needed
- Documentation in Russian
Getting started
letta install https://github.com/carlsonchik/judging-with-typesafe/tree/main/skills/judging-with-typesafeAt a glance
- Type
- Project
- Status
- prototype
- Language
- Python
- Runtime
- Letta agent skill
- Has code
- yes
Use cases and integrations
How curated lists describe it
Скилл для агентов Letta: суждения по критериям через TypeSafe System One (Jev).
hellogumbo/awesome-jev
Frequently asked questions
Can it fact-check?
No. The README shows an invented source scoring almost as high as a real one.
What does a 0.5 mean?
Values from 0.30 to 0.70 are treated as 'uncertain' and routed to a human.
Which agent framework does it target?
Letta.
Related projects
Analytics engine that rates published posts on pre-registered rubrics with Jev and tests ratings against real engagement.
ProjectbetaTypeScriptDocument evaluation CLI and CI gate using Jev or OpenAI Decisions, with pre-registered validation and documented reliability limits.
CLIbetaPythonWriting checker running separate named Jev checks such as hedging, filler, tone and passive voice, each with its own verdict.
DemobetaTypeScriptCommunity tool that fetches a SaaS homepage as Markdown and asks Jev ten questions about first-visit clarity, with a leaderboard.
ServicebetaTypeScriptUpload a paper PDF and get a fast Jev-based judgment of understandability and completeness, with low-confidence answers flagged.
DemoprototypeTypeScriptComparative judgment of ASAP essays by Jev and two LLM judges, turning pairwise decisions into scores compared with human grades.
ServiceresearchPython