Awesome Jev Robustness
Curated list of 138 independent tests of how Jev's answers shift with wording, order, repetition or injection, and of its calibration.
This repository is mostly documentation or write-up rather than runnable code.
Overview
Awesome Jev Robustness gathers independent tests of Jev's robustness and calibration, grouped by what they measured: sensitivity to question or option wording, option order and count, repeated calls, input language, planted steering text, inputs that fit no option, and calibration. It lists 138 entries, almost all against jev-1.13.0, in a snapshot dated 2026-10-12, with model version and sample size for each.
It warns that most tests were run by single people on small budgets in the first weeks and should be read as evidence to inspect, not settled results. Highlighted examples include a large calibration audit, a 123,805-request pre-registered study where calibration held on support routing but collapsed on random 3-SAT, and a study where arithmetic accuracy depended on whether the right option was listed first. Plain accuracy benchmarks are kept in a separate data file.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | How stable and calibrated are Jev's typed answers under perturbation?(inferred) | — |
How it works
- Collect independent robustness tests
- Group by property measured
- Record model version and sample size
- Publish list and TSV data
Strengths
- Focused on reliability rather than accuracy
- Machine-readable data
- Honest about evidence quality
Caveats
- Mostly small, single-author studies
- Numbers are the authors' own
At a glance
- Type
- Article
- Status
- research
- Has code
- no
- Has tests
- no
Use cases and integrations
How curated lists describe it
Curated list: 109 independent tests of Jev's calibration, consistency, prompt injection, abstention and failure modes, grouped by what they measured, each with model version and sample size.
yibie/awesome-jevTests, calibration audits and failure-mode studies of Jev.
wh000wh000/awesome-jev-liveTests, calibration audits and failure-mode studies of Jev (TypeSafe System One): jaggedness, consistency, injection, abstention.
hellogumbo/awesome-jev
Frequently asked questions
What counts as robustness here?
How answers change with wording, order, repetition, language, injected text or no-fit inputs, plus calibration.
Are results conclusive?
No, the list says to read them as evidence to inspect.
Which model version?
Almost all against jev-1.13.0.
Related projects
Curated list of the open System One ecosystem: open models, independent benchmarks, calibration and constrained-decoding tools.
ArticleproductionCurated list of papers on the security, robustness and safety of Jev and RLCD decision models, from attacks to deployment.
ArticleresearchPortuguese-language agent skill for millisecond Jev decisions via OpenRouter, with validation, an MCP mode and calibration evals.
Agent skillbetaJavaScriptTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonResearch code showing that short, natural-looking context additions can flip decision models like Jev to a chosen wrong answer.
ProjectresearchPythonStdlib-Python guardrail and model router for LLM gateways on Jev, with an independent accuracy, calibration and latency evaluation.
ServiceresearchPython