jev vs. open alternatives
LlamaIndex notebook comparing Jev with open decoders, encoders and specialised tools on document-pipeline Choice tasks.
Overview
This run-llama repository compares Jev, TypeSafe's hosted decision model, with open alternatives on document-pipeline chores, using liteparse (pdfium plus tesseract) for all PDF work. Tasks are built from generated, labelled data: language detection over Wikipedia excerpts rendered to PDF in 12 languages (Choice of 12), orientation detection on rotated scans using four-way OCR text (Choice of 4), RVL-CDIP document classification over 16 classes, bundle splitting with a two-way Choice per page boundary, and parse-tier triage deciding from liteparse complexity signals and an OCR excerpt whether a degraded scan needs a heavier parser.
Alternatives include Qwen3.5-4B served on Modal with direct logit readout, local open encoders Laya and jeff, and specialised tools such as lingua for language and tesseract OSD for orientation. A quick start builds the corpora, deploys the Modal model and opens the notebook, and small CLI tools expose liteparse functions for page info, text, rendering and orientation. Results are in the notebook rather than the README.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which of 12 languages is this document in? | — |
| Choice | Which rotation (0/90/180/270) is this scan in? | 0 · 90 · 180 · 270 |
| Choice | Which of 16 RVL-CDIP classes is this document? | — |
| Choice | Does a new document start at this page boundary? | — |
How it works
- Build labelled PDF corpora and OCR with liteparse
- Ask Jev, an open decoder, open encoders and specialised tools
- Compare per task in the notebook
Strengths
- Compares hosted Jev with several open options
- Realistic document-pipeline tasks
- Reusable liteparse CLIs
Caveats
- Results not summarised in the README
- Requires Modal for the open decoder
Getting started
uv sync, add TYPESAFE_API_KEY to .env, uv run build-corpus all, uv run modal deploy modal_app.py, then open jev_vs_open.ipynb.At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Jupyter + Modal
- Has code
- yes
Use cases and integrations
How curated lists describe it
Document pipelines: compares Jev against open models and specialised tools on five chores — language detection, orientation, RVL-CDIP classification, bundle splitting and parse-tier triage — asked as `Choice(2)` up to `Choice(16)`.
yibie/awesome-jev
Frequently asked questions
Which tasks are compared?
Language detection, orientation detection, RVL-CDIP classification, bundle splitting and parse-tier triage.
Which open alternatives are tested?
Qwen3.5-4B with direct logits, Laya and jeff encoders, plus lingua and tesseract OSD.
Where are the results?
In the jev_vs_open notebook.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonJev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.
ProjectprototypePythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonReproducible measurements of Jev's real cost, latency, response shapes and accuracy, plus 13 decision models on the same 633 items.
ServiceresearchShort report comparing Jev and many other fast decision classifiers with an LLM on grading section relevance in technical documents.
ProjectresearchBenchmark release evaluating Theory of Mind in five typed decision models versus seven generative baselines on nine ToM datasets.
ProjectresearchPython