jevoss
Test harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
Overview
JevOss measures how often a decision model is right, whether its probabilities match that accuracy, and which inputs change its answers. It targets failures an average score hides, such as following a hidden instruction or changing a decision when options are reordered. Any server implementing POST /v1/systemone can be tested, including the author's JevAlt, Kev and Laya.
The eval command reports accuracy and calibration, calibrate fits confidence on your own data, and compare diffs two recorded runs. Probes test permutation (does option order change the answer), injection (does a hidden instruction override the task), distractors (does irrelevant padding reduce accuracy), noul consistency (do yes/no and equivalent two-option questions agree) and determinism (do repeated requests return the same probabilities). A recipes folder holds ready requests for ticket routing, policy checks, security triage, tool permissions and a game NPC. A measured-problems report compares models; hosted Jev results use a different item set and are shown for context only.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Does changing option order, adding hidden instructions or padding change the model's decision? (robustness probes) | — |
| Choice | Which team should handle this ticket? (support_routing recipe)(inferred) | — |
How it works
- Start a decision-model server that speaks /v1/systemone
- Run jevoss eval on a suite for accuracy and calibration
- Run jevoss probe to test permutation, injection, distractors, noul consistency and determinism
- Optionally fit calibration on your data and compare runs
Strengths
- Works with any Jev-compatible server
- Probes expose specific failure modes, not just averages
- Recorded decisions published on Hugging Face
Caveats
- Hosted Jev comparisons use a different item set
- Suite scores are not evidence for your own application without testing on your inputs
- Requires a running model server
Getting started
pip install "jevoss[suites] @ git+https://github.com/mertkayacs/jevoss" && jevoss eval typed-decisions && jevoss probe typed-decisions --limit 100At a glance
- Type
- Service
- Status
- beta
- Language
- Python
- Runtime
- CLI
- Has code
- yes
Use cases and integrations
How curated lists describe it
Test any decision model that speaks the Jev API: accuracy, honest probabilities, and where it breaks (hidden instructions, shuffled options, padding, negations). Probes, calibration and 12 ready recipes.
hellogumbo/awesome-jev
Frequently asked questions
What does JevOss test?
Accuracy, calibration and robustness to option reordering, hidden instructions, padding, yes/no consistency and determinism.
Which models can I test?
Any server implementing POST /v1/systemone, such as JevAlt, Kev or Laya.
Does it need a TypeSafe key?
Not for local servers; testing hosted Jev would need access to it.
Related projects
Research code showing that short, natural-looking context additions can flip decision models like Jev to a chosen wrong answer.
ProjectresearchPythonPaper testing Jev, Laya, Decider and Nimble against classifiers and LLM judges on prompt injection and risk screening.
Paperresearchpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPython