subq-vs-jev
Reproducible test of SubQ Snap and Jev on 3,840 questions about a fictional store's event log from 2K to 360K tokens of context.
Overview
subq-vs-jev compares two decision models, SubQ Snap and TypeSafe Jev, on the same 3,840 questions about a fictional store's event log, with logs from 2K to 360K tokens. Six kinds of questions (find, cross-check, count) are generated from a fixed seed, and a test re-derives every answer from the log text alone. Byte-identical requests go to both APIs one at a time, alternating order, with raw responses saved and a bilingual results page.
Up to 30K tokens, accuracy was close (93.9% Snap, 94.7% Jev) and Snap was slightly better calibrated (ECE 0.028 vs 0.038). Jev rejects requests above about 32.7K tokens, so on longer logs it read 29K chunks with answers combined; at 360K, chunked Jev reached 77.1% versus Snap's single-call 58.8%. Snap returned byte-identical output on repeat 90% of the time versus 50.8% for Jev, option shuffling changed 3 versus 5 of 80 answers, and Jev was faster at 8K (0.31 s vs 0.75 s). The author notes Snap was a preview that changed between runs, latency included a home connection in Spain, and costs use input prices only.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Questions about a store event log (find, cross-check, count) | — |
How it works
- Generate logs and questions from a fixed seed
- Verify ground truth from log text
- Send identical requests to both models, chunking for Jev past its limit
- Grade accuracy, calibration, determinism, order robustness and latency
Strengths
- Ground truth independent of the generator
- Covers very long contexts
- Standard-library-only Python
Caveats
- SubQ Snap was a changing preview
- 20 logs per size
- Latency includes home network
Getting started
python3 -m unittest tests/test_truth.py, export SUBQ_API_KEY and JEV_API_KEY, then python3 bench/run.py && python3 bench/analyze.py.At a glance
- Type
- Model
- Status
- research
- Language
- Python
- Runtime
- Scripts + static results site
- Has code
- yes
- Has tests
- yes
Use cases and integrations
How curated lists describe it
Reproducible test of two decision models, SubQ Snap and Jev, from 2K to 360K…
wh000wh000/awesome-jev-live
Frequently asked questions
Which model was more accurate?
Close up to 30K tokens; at 360K, chunked Jev 77.1% beat Snap's 58.8%.
Why chunk Jev?
Jev rejects requests above about 32.7K tokens.
Which was more deterministic?
Snap, with 90% byte-identical repeats versus 50.8% for Jev.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePythonLlamaIndex notebook comparing Jev with open decoders, encoders and specialised tools on document-pipeline Choice tasks.
ProjectresearchPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonReproducible measurements of Jev's real cost, latency, response shapes and accuracy, plus 13 decision models on the same 633 items.
ServiceresearchShort report comparing Jev and many other fast decision classifiers with an LLM on grading section relevance in technical documents.
Projectresearch