jevrag
Python library that replaces hard-coded RAG thresholds with calibrated decisions for stopping, chunking, selection, abstention and caching.
Overview
JevRAG is a common decision substrate for retrieval-augmented generation pipelines. It treats each threshold in a RAG pipeline as an explicit decision point with one interface, state to Decision to confidence to action, and a swappable decision backend. Five primitives are built: evidence sufficiency (iterative, decide when retrieval can stop), chunk boundary (one-shot, before retrieval), context selection (per passage), answer abstain (a post-generation grounding check) and cache trust.
Each primitive is evaluated on real datasets with a calibration-first harness, with results and caveats in a primitives document; for example the abstain check is benchmarked against a free alternative and reported as a tie. A CRC calibration layer sets per-decision thresholds, a second log-probability backend shows the abstraction generalises, and a shadow mode compares a challenger against production. Two end-to-end runs chain all five primitives. It is aimed at RAG engineers who want measurable, calibrated decision points instead of magic numbers.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is the retrieved evidence sufficient to answer, or should retrieval continue? | — |
| Yes / no | Should this passage be included in the context? | — |
| Yes / no | Is the generated answer grounded, or should the system abstain? | — |
| Yes / no | Can this cached result be trusted? | — |
How it works
- Build state for a decision point in the RAG pipeline
- Backend (Jev or log-prob) returns a decision with confidence
- CRC calibration sets the threshold per decision
- Action taken: stop retrieval, select passage, abstain or reuse cache
- Shadow mode compares a challenger without affecting production
Strengths
- Explicit, calibrated decision points instead of hard-coded thresholds
- Swappable backends including a zero-cost log-prob option
- Honest evaluation with disclosed bugs and ties
Caveats
- Some primitives tie with free alternatives
- Evaluation sizes are modest (hundreds of questions)
- Jev backend needs a key
Getting started
pip install jevrag (or uvx jevrag --help).At a glance
- Type
- Service
- Status
- beta
- Language
- Python
- Runtime
- library and CLI
- Has code
- yes
- Has tests
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Replaces hardcoded RAG thresholds with explicit calibrated decision points. Five primitives (retrieval stopping, chunk splitting, context selection, answer abstention, cache trust) behind one swappable state → Decision → confidence → action interface, each evaluated on real datasets with a calibration harness that reports honestly.
logicrw/awesome-jev-projectsA pluggable decision substrate for RAG pipelines;
wh000wh000/awesome-jev-live
Frequently asked questions
What decisions does JevRAG make?
When to stop retrieving, where to split chunks, which passages to include, whether to abstain and whether to trust a cache.
Must I use Jev?
No. The backend is swappable, and a log-probability backend is included.
How are thresholds chosen?
A CRC calibration layer sets a threshold per decision from evaluation data.
Related projects
Python harness stress-testing Jev's confidence, its accuracy as a RAGTruth hallucination judge, and its use as a RAG reranker.
DemoresearchPythonPermission-aware document library using Jev or Cloudflare Clef for hierarchical search, passage scoring and automatic filing.
LibrarybetaTypeScriptGraph-database-agnostic GraphRAG framework where swappable Laya or Jev decisions score edges, route intents and gate hallucinations.
ServicebetaPythonEditorial QA tool where Jev judges per fact whether an edit preserved the original, with a confidence gate and human review fallback.
ProjectbetaJavaMCP server giving coding agents typed judgment tools (verify, screen, rerank, gate, score) backed by Jev, with auto/review/escalate policy.
MCP serverbetaPythonTyped-decision accuracy layer for RAG: six checks for intent, relevance, clarification, sufficiency, grounding and caveats.
ServicebetaPython