jev-secret-detection
Benchmark of how well Jev spots real secret credentials in file snippets with a single Noul per case, plus latency analysis.
Overview
jev-secret-detection measures how well TypeSafe's Jev identifies real secret credentials in file snippets. Each case is sent as one Noul question asking whether the content contains a real secret credential someone could use, and the probability is compared with the expected label. There is deliberately no regex matching or provider verification, so the score reflects Jev alone.
The default set has 100 cases, with smaller edge-case (placeholders and redactions paired with real secrets) and config-shaped batches. The report shows per-case results and overall accuracy at 0.5, AUC, recall, precision, mean probabilities, Brier score, the share in a 0.3-0.7 review band, per-category accuracy, and latency. The README notes server time of about 75-90 ms at p50, matching TypeSafe's documentation, while round trips depend on network distance (about 290 ms warm from a 185 ms RTT location); sending 100 requests at once raised latency and triggered overload retries, so it sends 16 at a time.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Does this content contain a real secret credential that someone reading it could use? | yes · no |
How it works
- Load labelled snippet cases
- Send each as a Noul with file path and content
- Compare probabilities with labels
- Report accuracy, AUC, calibration, review band and latency
Strengths
- Isolates the model's own ability
- Edge and config-shaped test sets
- Detailed latency breakdown
Caveats
- Small hand-built test sets
- Pinned to one model version
- Not a production scanner
Getting started
echo "TYPESAFE_API_KEY=..." > .env, then uv run python main.pyAt a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark script
- Has code
- yes
Use cases and integrations
How curated lists describe it
Secret-in-diff detector with repeatable Jev verdicts.
AnotiaWang/awesome-decision-modelsBenchmark and tool evaluating how well TypeSafe Jev identifies real secret credentials in file snippets.
logicrw/awesome-jev-projectsMeasures how well TypeSafe.
wh000wh000/awesome-jev-liveMeasures how well TypeSafe's RLCD-Jev model spots real secret credentials in file snippets.
hellogumbo/awesome-jev
Frequently asked questions
What does Jev decide?
Whether a file snippet contains a real, usable secret credential.
Does it use regexes?
No, deliberately; the goal is to score Jev alone.
How fast is Jev?
Server time p50 of about 75-90 ms; round trip depends on network distance.
Related projects
Example of using Jev as a webhook guardrail inside the agentgateway proxy to inspect model requests and responses.
ProxyproductionRustClaude Code PreToolUse layer where Jev answers YAML-defined questions about each tool call, with 19 hard rules and calibration runs.
ProjectbetaTypeScriptAdvisor for coding agents: git and Stop hooks flag test tampering, secrets and skipped hooks, with optional Jev review findings.
HookbetaCommit-msg hook where one Jev request checks the message against the staged diff and flags filler, contradictions and secrets.
HookbetaPythonSafety gate plugin for Claude Code, Codex CLI and Antigravity: local sensitive-file checks, then Jev judges blast radius.
PluginbetaTypeScriptClaude Code mod that answers auto mode's tool-call permission question with eight Jev yes/no checks behind a shell-aware blocklist.
ProjectbetaTypeScript