Jev-decision-control
Research framework (DART) testing whether Jev's probabilities are a reliable signal to answer, escalate or abstain.
Overview
Decision-Native Control of Generative AI Inference is a research project studying the decision layer separately from generation. It asks whether a decision-native model such as Jev can provide a reliable control signal for deciding when an AI system should stop, escalate to a stronger model, or abstain, and whether that signal holds up under paraphrase and distribution shift.
The evaluation framework is called DART (Decision-native AI Reliability Testing). Initial experiments focus on calibration and reliability; planned work covers counterfactual stability, distribution shift, state manipulation, decision representation and computation-aware routing. Working hypotheses include that Jev's probability is informative but calibration varies across domains, that semantically equivalent rewording may change probabilities, and that reliability may degrade under shift. It is aimed at researchers rather than practitioners looking for a tool.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Should the system answer, escalate to a stronger model, or abstain?(inferred) | answer · escalate · abstain |
How it works
- Pose tasks to Jev and record decision probabilities
- Measure calibration against correctness
- Perturb inputs (paraphrase, domain shift) and re-measure stability
- Use the signal to gate downstream generation and assess compute trade-offs
Strengths
- Studies calibration and robustness rather than only accuracy
- Structured hypotheses and a named framework
Caveats
- Early research; later experiments still planned
- No results reported in the excerpt
At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Research notebooks and scripts
- Has code
- yes
Use cases and integrations
How curated lists describe it
Research framework for evaluating calibration, stability, robustness, and…
wh000wh000/awesome-jev-live
Frequently asked questions
What is the research question?
Whether Jev's probability output can reliably control when a system answers, escalates or abstains.
What is DART?
Decision-native AI Reliability Testing, the project's evaluation framework.
Are there results yet?
The README describes initial calibration experiments; results are not summarised in the excerpt.
Related projects
Python tool that measures where to route decisions between Jev and a larger model on your own data, then routes by confidence.
BenchmarkbetaPythonDecision sidecar and Copilot CLI pre-turn hook that mimics Jev choice, noul and score using Azure Foundry Cohere Rerank.
HookprototypePythonPre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPythonBlack-box reverse-engineering archive studying how the Jev decision model behaves, with evidence matrices and calibration analysis.
ProjectresearchPythonIndependent API-only audit of what Jev's probabilities mean: abstain removal, Korean, coherence, option order, interference and caching.
ProjectresearchPythonResearch code showing confidence cascades let a 0.8B prefill-only decision model approach a 4B model's accuracy at a fraction of the cost.
ProjectresearchPython