ASSAY-001
Pre-registered independent check of Jev's calibration and type safety on Banking77 and CLINC150, with a split verdict and sealed logs.
Overview
ASSAY-001 is an independent, pre-registered evaluation of two claims about Jev: that its probabilities are calibrated and that its outputs are type-safe. The protocol was frozen and hashed on 2026-09-17 before any query, the Banking77 and CLINC150 corpora were pinned and hashed, and every request and response was logged verbatim and sealed.
The verdict is split. On CLINC150 the chosen-option probabilities were calibrated (ECE 0.0204); on Banking77 they were not (ECE 0.0936, systematically overconfident). Across 8,576 responses there were zero type errors. The harness asks one Choice per item with null option descriptions, a deterministic scorer and positive controls are included, an independent blind re-score on a different base model matched every field, and amendments and an erratum are documented. The README stresses that results apply only to these artifacts and criteria.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which intent label fits this utterance? (Banking77 / CLINC150) | dataset intent labels |
How it works
- Freeze and hash the protocol and corpora
- Ask one Choice per item and log every response
- Score calibration (ECE) and type errors deterministically
- Run controls and an independent blind re-score
- Publish report, amendments and erratum
Strengths
- Pre-registration and hashes guard against p-hacking
- Full raw logs for reanalysis
- Independent re-score matched every field
Caveats
- Results limited to two intent datasets on one model version
- Re-running the model produces a new run, not this one
Getting started
python3 harness/controls.py, then re-run the scorer per the README (re-querying the model needs a TypeSafe key).At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- evaluation harness
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Independent pre-registered check of Jev calibration and type safety on Banking77 / CLINC150, with a split verdict and full logs, written up at [donttrustme.ai](https://donttrustme.ai/assay-001.html).
yibie/awesome-jevIndependent pre-registered check of Jev calibration and type safety on Banking77 / CLINC150. Split verdict, full logs. Write-up: [donttrustme.ai](https://donttrustme.ai/assay-001.html)
AnotiaWang/awesome-decision-models
Frequently asked questions
Is Jev calibrated?
On CLINC150 yes (ECE 0.0204); on Banking77 it was overconfident (ECE 0.0936).
Did it ever return off-schema answers?
No, zero type errors across 8,576 responses.
Can I verify the scoring?
Yes, raw logs are sealed and the scorer and controls can be re-run without a key.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPythonPre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.
CLIresearchPython