Tiltmeter
Monitoring proxy and client that records Jev answer probabilities and alerts on version changes, drift, edge crowding or accuracy drops.
Overview
Tiltmeter watches Jev decisions in production. It records the probabilities for every question an app asks and raises four kinds of alert: the model behind jev-latest changed, the distribution of answers drifted from its baseline, too many answers sit next to the threshold your code acts on, or estimated accuracy fell. Accuracy is estimated from the probabilities alone, with no labels.
It runs either as a drop-in /v1/systemone proxy that any client can point at, or as an in-process wrapper for Pydantic AI's TypeSafe provider. It stores only probabilities, the model name and a fingerprint of each question, never the text sent to Jev. Alerts can go to a webhook such as Slack. The README lists clear limitations: the accuracy estimate assumes calibration holds, very slow drift can slip through, and data lives in one local SQLite file.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Has the answer distribution for this question drifted, crowded the threshold, or lost estimated accuracy? | version · drift · edge · accuracy |
How it works
- Point the client or TYPESAFE_BASE_URL at the Tiltmeter proxy, or wrap the Pydantic AI provider.
- Each Jev answer's probabilities are recorded per question fingerprint.
- Windows of answers are compared statistically (PSI with chi-square tests).
- Alerts fire once per issue to a webhook and clear when resolved.
Strengths
- Detects silent model version changes and input drift without labels.
- Never stores the text sent to Jev.
- Statistical gating keeps false alarms low; the README reports at most 0.3 percent in simulation.
- Passes requests through unchanged and never adds an API key.
Caveats
- Accuracy estimate relies on Jev staying calibrated.
- Single local SQLite store; not meant for a fleet.
- Demo and self-test use simulated, not real, Jev answers.
Getting started
pip install "git+https://github.com/abe75ch/tiltmeter" then run `tiltmeter demo`.At a glance
- Type
- Proxy
- Status
- beta
- Language
- Python
- Runtime
- HTTP proxy, Pydantic AI wrapper
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Monitoring: drop-in `/v1/systemone` proxy and Pydantic AI client that records every Jev answer's probabilities and alerts, without labels, when `jev-latest` switches versions, a question's answers drift (chi-square-tested PSI), answers crowd a decision threshold, or estimated accuracy falls.
yibie/awesome-jev
Frequently asked questions
What does Tiltmeter alert on?
A changed model version, drift in answer distributions, too many answers near your threshold, and falling estimated accuracy.
Does it see my data?
It stores probabilities, model names and question fingerprints only, not the state text.
Does it need labels?
No; accuracy is estimated from probabilities, assuming calibration holds.
Related projects
Python trust middleware for the Jev API that logs decisions, measures calibration from your recorded outcomes and signs the evidence.
ServicePythonRust CLI and library that asks Jev typed questions via OpenRouter and combines the probabilities with fuzzy-logic rules into decisions.
CLIbetaRustUnofficial standard-library Go client for Jev with generic typed Choice values and local confidence thresholds.
ProjectbetaGoRust CLI for Jev: ask yes/no, choice and rubric questions about text and branch on exit codes, JSON or MCP tools.
MCP serverbetaRustZero-dependency local gateway and live visualiser that proxies Jev calls, logs every decision to SQLite and draws them on a real-time map.
ProxybetaJavaScriptRecords the full distribution of every Jev answer by state, question and model, so thresholds can move over history with no new calls.
LibrarybetaPython