jev\_playground
Measurement-first lab deciding which classifier technique, Jev, rules or other models, earns a place in a coding-agent fleet.
Overview
jev_playground by jyeswak is where a team running a fleet of coding agents decides which technique should answer typed questions inside the agent loop: which file to read, whether a tool result is an injection, whether a command is risky, whether a recalled memory belongs in context, whether a review finding is new. Jev is one technique among several, and winners ship through one model-neutral classifier CLI.
The method is strict: blind labels first, a constant and the cheapest regex or keyword rule set the bar, the bar is committed before any call, all backends are calibrated the same way, surfaces must fire live and stay silent on planted negatives, and each keeps a benefit-minus-cost receipt. Findings so far: cheap rules often beat a typed model (tool-call risk, phishing, review novelty); typed models win where the label is in the text but lexical cues fail (FiQA passage selection 76.2% top-1 versus 39.3% for BM25); on Banking77, Clef-Flash scored .962, Kev-4B .810 and Jev .787; and matching calibration erased an apparent Clef-over-Jev win.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this tool result a prompt injection? | yes · no |
| Yes / no | Is this command risky? | yes · no |
| Yes / no | Does this recalled memory belong in context? | yes · no |
| Choice | Which file should the agent read next? | — |
How it works
- Label a question's dataset blind
- Score a constant and the cheapest rule first
- Commit the bar, then test calibrated model backends
- Ship winners through the classifier CLI and verify live
- Keep or remove each surface based on measured value
Strengths
- Rigorous, model-neutral evaluation process
- Honest reporting that cheap rules often win
- Measurement ledger published
Caveats
- Specific to the team's agent fleet and data
- Complex repository with many experiments
At a glance
- Type
- App
- Status
- research
- Runtime
- CLI and experiments
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Jev answers typed questions about a state with calibrated numbers. This repo is where we find out which of those numbers deserve to drive code, and where a regex or a constant does the job better.
logicrw/awesome-jev-projectsMeasure what Jev can actually do before you build on it.
wh000wh000/awesome-jev-live
Frequently asked questions
Does Jev always win?
No. Keyword and regex rules beat it on several tasks; it wins where labels are in the text but lexical cues break.
Which models are compared?
Jev, Clef, Clef-Flash, Kev-4B, rules, constants and others.
How are techniques kept or dropped?
Each live surface has a benefit-minus-cost receipt; a negative one turns it off.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonPortuguese workshop: a LangGraph sales agent where each decision step can be made by Jev or a frontier LLM and measured side by side.
ProjectprototypePythonBenchmark testing whether Jev or open Kev can replace an LLM judge in agent guardrail callbacks on 7,925 public examples.
ProjectresearchPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPython