Jev Calibration Study (Nautilus Assay)
Nautilus Compass agent memory layer whose jev-trust middleware logs Jev decisions and measures calibration in your domain.
Overview
This entry points to nautilus-compass, an open-source memory and reliability layer for AI agents from the Nautilus Platform, available as a plugin for Claude Code, Cline, Cursor, Continue.dev, Zed and any MCP client. The README focuses on long-term memory, which it reports beats mem0 on LongMemEval-S while staying local, plus drift detection, cross-agent contracts, and a newer independent judging layer with a 1.7B LoRA judge and deployment-precision checks.
The Jev connection is a component: jev-trust, trust middleware for the Jev decision API that logs every call, measures calibration in the user's own domain from recorded outcomes (accuracy, Brier, top-label ECE), annotates answers with measured effective confidence, and signs the evidence with ed25519 for independent recomputation. The cobanov list also cites an independent calibration evaluation of hosted Jev on 240 seeded questions with all raw artifacts published, reporting accuracy 92.2%, Brier 0.048 and ECE 0.041 overall. The fetched README excerpt does not detail that study.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Logs Jev decisions and measures their calibration against recorded outcomes | — |
How it works
- Route Jev calls through jev-trust
- Log each typed decision
- Record real outcomes
- Compute accuracy, Brier and ECE; annotate effective confidence
- Sign evidence for independent recomputation
Strengths
- Domain-specific calibration measurement
- Signed, recomputable evidence
- Part of a broader agent reliability toolkit
Caveats
- Jev is a small part of a large project
- Calibration study details not in the README excerpt
Getting started
Install jev-trust from PyPI to wrap Jev calls; nautilus-compass itself is on PyPI and the MCP registry.At a glance
- Type
- Service
- Status
- beta
- Language
- Python
- Runtime
- middleware, MCP plugin
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Independent, reproducible calibration evaluation of hosted Jev (240 seeded questions, all raw artifacts published): accuracy 92.2%, Brier 0.048, ECE 0.041 overall.
cobanov/awesome-jevPython middleware around the Jev API that logs every typed decision, measures calibration in your own domain from outcomes you record (accuracy, Brier, top-label ECE), annotates each answer with its measured effective confidence, and signs the evidence (ed25519) for independent recomputation.
cobanov/awesome-jev
Frequently asked questions
How does this relate to Jev?
Its jev-trust middleware logs Jev calls and measures their calibration in your domain.
What calibration did the study find?
Per the cobanov list, 92.2% accuracy, Brier 0.048 and ECE 0.041 on 240 seeded questions.
Why sign the evidence?
So others can independently recompute the calibration measurements.
Related projects
Decision-forensics library that explains a Jev answer by ablating parts of the input and measuring how far the probability moves.
ProjectbetaPythonOffline Python toolkit that measures sensitivity, Fisher information and information loss in the probabilities of Jev-style agent systems.
ProjectresearchPythonPython trust middleware for the Jev API that logs decisions, measures calibration from your recorded outcomes and signs the evidence.
ServicePythonToolkit for Jev decision contracts: compiled policies, egress controls, tracing, stability checks and certified decision SLOs.
ProjectbetaTypeScriptRecords the full distribution of every Jev answer by state, question and model, so thresholds can move over history with no new calls.
LibrarybetaPythonMonitoring proxy and client that records Jev answer probabilities and alerts on version changes, drift, edge crowding or accuracy drops.
ProxybetaPython