jev-testbench
Small Jev measurement bench with an append-only call log, labelled claims and a stated negative result.
Overview
jev-testbench studies TypeSafe Jev as a typed probabilistic decision primitive and, separately, documents the cognitive debt that builds up when AI-assisted project complexity grows faster than the maintainer's understanding. It is an experimental harness rather than a demo or leaderboard. Every real API call appends one line to an append-only JSONL log, and every claim is labelled as an official vendor claim, design assumption, local measurement, derived calculation or limitation.
The evidence base is 42 core records plus 12 records from a preregistered replication, all resolving to jev-1.13.0. Experiments cover foundational mechanics such as batching and the effect of instructions versus criteria; for example, instruction-only wording recovered 80.4% of a gap and criteria-only 71.4% in one experiment. The headline result is stated plainly as no strong Jev-specific novel finding yet: nothing is both new relative to TypeSafe's published material and supported by the repository's own data. A section spells out what cannot be concluded. An offline test suite passes and v0.1.0 is an evidence freeze.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Measures how Choice, Score and Noul answers respond to batching, instructions, criteria and boundary cases | — |
How it works
- Design an experiment (some preregistered)
- Make real API calls, each logged as one JSONL record
- Label every claim by evidence type
- Report results including null and negative findings
Strengths
- Strict evidence discipline and labelled claims
- Negative and null results kept
- Offline test suite
Caveats
- Small sample (54 records total)
- Author states there is no strong novel finding yet
- Exploration is closed
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- CLI harness
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
\`jev-testbench\` is a lightweight Jev testbench built through \*\*Agentic Engineering\*\*.
logicrw/awesome-jev-projects
Frequently asked questions
What is the main finding?
A negative one: no Jev-specific finding that is both new and supported by the repository's own data.
How is evidence recorded?
Every API call appends one line to an append-only JSONL log, and claims are labelled by evidence type.
What is the cognitive-debt part?
A separate process record of what happens when AI-assisted complexity outgrows the maintainer's understanding.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPythonPre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.
CLIresearchPython