snap-vs-jev-swe
Reproducible benchmark of SubQ Snap, TypeSafe Jev and three open decision models on software-engineering decisions.
Overview
snap-vs-jev-swe benchmarks fast decision models on software-engineering tasks through the same /v1/systemone contract: SubQ Snap (subq-snap-preview), TypeSafe Jev 1.13.0, and three open models (Clef-flash 9B, H2O-Lightning-4B and Intern-Decision-4B). The author discloses taking part in SubQ's creator challenge, which has prizes, and that Snap launched publicly on 12 October 2026 and may change.
On choosing which of 10 candidate files must change for 200 SWE-bench Verified issues, Snap and Jev were close with one choice (78.5% vs 79.0%), and in per-file yes/no mode Snap led 85.5% to 81.8% but not significantly (p=0.12), with a clearer AUROC difference. Snap could also take all files in one ~71k-token request (87.5%), beyond Jev's ~32k limit. On seven SE classification datasets (3,522 cases) Snap led on six, significantly on three; vulnerability detection was near chance for both, and rewording a question moved accuracy more than the model gap. Reversing options changed 5.0% of Snap's answers and 7.1% of Jev's. Estimated costs are reported, and results regenerate offline from committed data.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which of these 10 files must change to fix this issue? | — |
| Yes / no | Must this file change to fix the issue? | — |
| Choice | Software-engineering classification labels (e.g. clone detection, comment type) | — |
How it works
- Build cases from SWE-bench Verified and seven SE datasets
- Run each model through the same /v1/systemone contract
- Score accuracy, AUROC, option-order stability and cost
- Report paired significance tests and run-to-run variation
Strengths
- Same contract for hosted and open models
- Significance tests and order-robustness checks
- Results reproducible without downloads
Caveats
- Author participates in a SubQ prize challenge
- Single rerun; preview models may change
Getting started
export SUBQ_API_KEY and TYPESAFE_API_KEY, then uv run snapbench run --model typesafe-jev --cases data/cases/swe-per-file.jsonl; make evaluate regenerates tables.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- Benchmark CLI
- Has code
- yes
- License
- MIT (code), CC BY 4.0 (results)
Use cases and integrations
How curated lists describe it
Reproducible benchmark of fast AI decision models.
wh000wh000/awesome-jev-live
Frequently asked questions
Which model did best?
Snap led on most tasks, but several differences with Jev were not statistically significant.
Why could Snap take bigger inputs?
Jev has a roughly 32k-token limit; Snap accepted much longer requests.
Is there a conflict of interest?
The author discloses participating in SubQ's prize challenge.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePythonLlamaIndex notebook comparing Jev with open decoders, encoders and specialised tools on document-pipeline Choice tasks.
ProjectresearchPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonReproducible measurements of Jev's real cost, latency, response shapes and accuracy, plus 13 decision models on the same 633 items.
ServiceresearchShort report comparing Jev and many other fast decision classifiers with an LLM on grading section relevance in technical documents.
Projectresearch