Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-triage

Test-failure triage for coding agents: local rules first, then one batched Jev call labels failures RETRY, FIX_CODE or FIX_ENV.

ProjectbetaMIT
View on GitHub ↗

Overview

jev-triage gives a coding agent a third option when a test fails, besides re-running blindly or pasting thousands of lines of stack trace into an LLM: classify the failure in one call and say what class of action it needs. RETRY means flaky or transient, so re-run unchanged; FIX_CODE means a real defect in the code under test; FIX_ENV means a missing dependency, unset environment variable or unreachable service. It works with pytest, vitest, jest and go test, as a CLI, a Python library and a Claude Code PostToolUse hook.

Runner output is parsed offline into structured failures, about a dozen unambiguous symptoms are resolved by deterministic rules at zero cost, and remaining failures go to Jev in one batched request. Below the confidence threshold the verdict is UNKNOWN and escalated. On 16 logs captured from real pytest and vitest runs, the README reports 16 of 16 correct across three repeat runs, 44% resolved locally, and about 48 times cheaper than an LLM, with median Jev latency about 1.1 s. It openly notes the small, self-constructed corpus and that an earlier version scored 76.9%.

What the decision model decides

TypeQuestionAnswers
ChoiceIs this test failure a RETRY, FIX_CODE or FIX_ENV case?RETRY · FIX_CODE · FIX_ENV

How it works

  1. Parse runner output into structured failures
  2. Resolve unambiguous symptoms with local rules
  3. Send remaining failures to Jev in one batched call
  4. Emit verdict with confidence; below threshold report UNKNOWN and escalate

Strengths

  • Zero-cost local rules handle obvious cases
  • One model call per triage regardless of failure count
  • Claude Code hook integration
  • Honest reporting of corpus limits and earlier lower score

Caveats

  • 16-log evaluation constructed by the author
  • Measured latency 0.8-2.5 s, above launch-material figures

At a glance

Type
Project
Status
beta
Language
Python
Runtime
CLI / library / Claude Code hook
Has code
yes
Has tests
yes
License
MIT

Use cases and integrations

How curated lists describe it

Frequently asked questions

What are the three verdicts?

RETRY for flaky failures, FIX_CODE for real defects and FIX_ENV for setup problems.

How accurate is it?

The README reports 16/16 on real captured logs but stresses this is a small, self-built corpus.

How fast is it?

Median Jev latency was about 1.1 s in the author's tests; rule-resolved cases are instant.

Related projects

  • jevbrief

    Python library and CLI that turns noisy sources like CI logs, diffs and web pages into small, traceable Jev briefings.

    BenchmarkbetaPython
  • jev-harness

    Deterministic-first 'System 1.5' gate for coding agents: triages test failures, breaks doom loops and governs effort, with optional Jev.

    CLIbetaPython
  • jevable

    Bash-native CLI that filters any stream by meaning with Jev, like grep for semantics, so an agent wakes only for lines that matter.

    ProjectbetaTypeScript
  • oh-my-claudecode

    Multi-agent orchestration layer for Claude Code with opt-in Jev hooks for fast judgment points, decision caching and egress controls.

    PluginproductionTypeScript
  • compact-adviser

    Agent plugin that asks Jev whether the session is at a safe boundary to /compact and can trigger it on Pi and Claude Code.

    PluginbetaTypeScript
  • Jev_steer_or_queue

    Coding-agent hook where Jev decides whether a mid-turn message should steer the running task, be queued, or interrupt it.

    PluginbetaPython