Testing Jev on public and private data: classifier or filter?
Blog post testing Jev in 16,000 calls against two OpenAI models, covering where it wins, where it breaks and how to set thresholds.
This summary is based on how curated lists describe the project; its README was not available to review.
Overview
This blog post by Aman Kumar reports an evaluation of Jev on public and private data, framed around whether Jev is best used as a classifier or as a filter. According to the list summary, it covers about 16,000 calls comparing Jev with gpt-5.4-mini and gpt-5.6-luna, describes where Jev wins and where it breaks, and proposes a procedure for choosing thresholds.
No article text is captured in the directory, so specific accuracy figures are not repeated here. It is relevant for teams deciding whether to trust Jev's top answer directly or to use its probabilities as a gate in front of a stronger model.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Task-specific classification questions on public and private datasets(inferred) | — |
How it works
- Run the same tasks through Jev and two OpenAI models.
- Compare accuracy across datasets and failure cases.
- Derive a threshold procedure for using Jev as a filter.
Strengths
- Large number of calls across public and private data, per the summary.
- Practical threshold guidance.
Caveats
- Only a one-line summary is captured here.
- Private-data results may not be reproducible.
At a glance
- Type
- Article
- Status
- research
Use cases and integrations
How curated lists describe it
16,000 calls vs gpt-5.4-mini and gpt-5.6-luna; where it wins, where it breaks, and a threshold procedure
AnotiaWang/awesome-decision-models
Frequently asked questions
What was compared?
Jev against gpt-5.4-mini and gpt-5.6-luna over roughly 16,000 calls, per the list summary.
What is the main question?
Whether Jev works better as a standalone classifier or as a filter in front of other models.
Does it give threshold advice?
Yes, the summary says it includes a threshold procedure.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPythonPre-registered paired audit of Jev on Spanish across 3,200 labelled items, plus a CLI to rerun it on your own data.
CLIresearchPython