decguard
Reliability testing for typed decision models: contracts, calibration metrics, metamorphic fuzzing and CI pass/warn/fail gates.
Overview
DecGuard tests probabilistic decision models that answer a typed question with a probability per answer. You describe a decision once in a small Decision Contract, run a labelled dataset through any backend, and get a reproducible reliability report with PASS, WARN or FAIL gates for CI. Reports include accuracy, macro-F1, calibration (ECE, Brier, NLL), selective accuracy at a confidence threshold, latency and errors.
A fuzz command checks metamorphic properties: option order, label format, irrelevant or repeated context, whitespace, paraphrase, noul inversion and score monotonicity, seeded and minimised. It can also compare model versions and check production records offline. The systemone backend is validated end to end against a Kev-0.8B server and real Jev through OpenRouter. It is alpha (v0.1) with four runtime dependencies and needs no GPU or account.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Does this decision model meet the contract's accuracy, calibration and invariance gates? | PASS · WARN · FAIL |
How it works
- Write a decguard.yaml contract and dataset
- decguard validate checks everything offline
- decguard test computes metrics and gates; --all adds properties
- Use exit codes to gate CI
Strengths
- Calibration metrics and selective accuracy out of the box
- Metamorphic fuzzing catches option-order and paraphrase sensitivity
- Backend-agnostic, lightweight install
Caveats
- Alpha release
- Quality of gates depends on your labelled dataset
Getting started
pip install decguard, then decguard test examples/refund/decguard.yaml --all.At a glance
- Type
- Demo
- Status
- prototype
- Language
- Python
- Runtime
- CLI
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Reliability testing for probabilistic AI decisions in system-one-models.
hellogumbo/awesome-jev
Frequently asked questions
What does DecGuard measure?
Accuracy, calibration, selective coverage, latency, errors and metamorphic invariants of a typed decision.
Which backends work?
Any; the systemone backend was validated against Kev-0.8B and Jev via OpenRouter.
Can it run in CI?
Yes, gates return exit codes for PASS, WARN and FAIL.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonHarness that runs Jev on all 19,646 TMMLU+ Traditional Chinese questions as four-way Choices, reporting accuracy and calibration.
ProjectresearchPython- reflexbench★ 6
Open benchmark for System One decision models on Choice, Score and Noul, reporting accuracy, calibration and robustness separately.
BenchmarkproductionPython Pre-registered audits of AI judges that measure how many decisions a judge can take alone at a given error rate, with calibration.
BenchmarkbetaPythonClaude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.
PluginbetaPython