jev-test
Concept prototype for marking NZQA scripts from rubric criteria alone with one Jev call per script and confidence-based triage.
Overview
This prototype explores AI-assisted marking for any assessment standard from its written criteria alone, with no labelled data and no model per standard. It targets the long tail of low-volume standards that would never justify a trained model. For each script, one Jev call returns guardrail Nouls (is it a genuine attempt, does it need safeguarding follow-up), a Score per performance criterion with a probability distribution, and an overall grade Score (Not Achieved, Achieved, and where applicable Merit and Excellence).
Calibrated confidence then triages each script to auto-accept, human review (low confidence or possibly off-topic) or escalation for safeguarding, and a threshold slider re-triages instantly without new API calls. Included standards are a paraphrased Year 10 literacy writing standard, an illustrative Level 3 agriculture and horticulture standard, and a builder for new standards. All learner responses are synthetic and reference grades are the author's. It is explicitly not endorsed by NZQA and is a concept only.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this a genuine attempt at the task? | — |
| Yes / no | Does this script need safeguarding follow-up? | — |
| Score | What overall grade does this script earn? | Not Achieved · Achieved · Merit · Excellence |
How it works
- Load a standard's task and criteria
- One Jev call per script returns guardrails, criterion scores and grade
- Triage by confidence: auto-accept, human review or escalate
- Adjust the threshold slider to re-triage without new calls
Strengths
- No labelled data or per-standard model needed
- Humans focus on uncertain or safeguarding cases
- Zero npm dependencies to install
Caveats
- Concept only, not endorsed by NZQA
- Synthetic responses and author-set reference grades
- Criteria wording paraphrased and must be verified
Getting started
git clone, cp .env.example .env (set TYPESAFE_API_KEY), `npm start`, then open http://localhost:3000.At a glance
- Type
- Project
- Status
- prototype
- Language
- JavaScript
- Runtime
- Node web app
- Has code
- yes
Use cases and integrations
How curated lists describe it
Prototype: AI-assisted NZQA marking from rubric criteria alone, using TypeSafe Jev for guardrails, criterion scores and confidence-based triage. NOT ENDORSED BY NZQA - CONCEPT ONLY
logicrw/awesome-jev-projects
Frequently asked questions
Is this used by NZQA?
No. It is a concept prototype and explicitly not endorsed by NZQA.
What does one Jev call return?
Guardrail yes/no answers, a score per criterion and an overall grade, each with probabilities.
Are the scripts real student work?
No, all learner responses are synthetic.
Related projects
Claude Code and Codex skills plus a study course for building with Jev, including Jevaluate, which rates how well projects use Jev.
ProjectbetaRead-aloud English tutor: streaming ASR transcribes, then Jev judges word by word whether each word was read correctly.
ProjectprototypeJavaScriptLearning lab for DeepEval with Jev as the judge: replayed evaluations, adjustable decision rules, release gates and drift alerts.
ProjectprototypePythonTypeSafe cookbook on why repeated Jev calls can differ, showing how an abstention rule raises repeat agreement.
ArticleproductionShow HN thread for Knowledge Signal, an open-source prototype that grades study notes against a rubric with Jev.
DiscussiondiscussionUnix-style tools where Jev learns a repo's decision norms and judges each node of a code lead as a bug, lead or clean.
ProjectbetaJavaScript