Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

zero-shot-ie-bench

Benchmark and web UI comparing dozens of zero-shot extraction and classification systems, including Jev-like engines.

Benchmarkresearch
View on GitHub ↗

Overview

zero-shot-ie-bench demos, benchmarks and cross-compares a large set of zero-shot information-extraction and classification systems in one repository with a web UI. The README lists 76 systems (73 benchmarked) across 35 families; the list summary cites 60 and 32 at an earlier point. Families include extractor encoders (GLiNER 2.5, GLiFormer), relation extractors, zero-shot classifiers (GLiClass), cross-encoder rerankers, and typed-decision engines such as LAYA, Von, Kev and AgentJev.

Systems run locally, through the Ollaya daemon, or hosted via OpenRouter's decision and rerank endpoints, Cloudflare, Fastino's API and TypeSafe. It measures accuracy, latency and cost, includes a nine-language suite, and records sizes and licences for each system, including licence inconsistencies. Each system's licence is recorded, including cases where a model card and repository disagree, which is useful when choosing something for commercial use.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich label or entity type applies? (compared across systems)—
ExtractWhich spans are entities or relations of the requested types?—

How it works

  1. Install or connect each system.
  2. Run shared extraction and classification tasks.
  3. Measure accuracy, latency and cost.
  4. Compare in charts and the web UI.

Strengths

  • Very broad comparison including open Jev-like engines.
  • Local, Ollaya and hosted options.
  • Licence and size details per system.

Caveats

  • System counts change as it grows.
  • Results depend on the chosen task suite.

At a glance

Type
Benchmark
Status
research
Language
Python
Runtime
benchmark harness + web UI
Has code
yes

Use cases and integrations

How curated lists describe it

  • 60 zero-shot information-extraction & classification systems (57 benchmarked) across 32 families — extractor encoders, zero-shot classifiers, cross-encoder rerankers, typed-decision engines — local, Ollaya-served & hosted (OpenRouter, Cloudflare, TypeSafe); measured accuracy/latency/cost benchmarks, 9-language suite, web UI.

    hellogumbo/awesome-jev

Frequently asked questions

How many systems are compared?

76 listed and 73 benchmarked in the current README.

Are Jev-like models included?

Yes, including LAYA, Von, Kev and AgentJev, plus hosted decision endpoints.

What is measured?

Accuracy, latency and cost, including a nine-language suite.

Related projects

  • DecisionBench

    Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.

    ProjectbetaPython
  • OpenJev (GPT-AGI)

    Jev-compatible open System One engine with a Claude Code-style terminal REPL showing live probabilities, plus an experiment harness.

    ProjectprototypePython
  • jev vs. open alternatives

    LlamaIndex notebook comparing Jev with open decoders, encoders and specialised tools on document-pipeline Choice tasks.

    ProjectresearchPython
  • jev-anotacao-sentencas

    FGV Direito SP experiment comparing Jev with Gemini 3.8 Flash and GPT-5.6 Luna for annotating 12 variables in 120 TJSP court rulings.

    ProjectresearchPython
  • jev-dimabsa

    Jev applied to SemEval-2026 DimABSA dimensional sentiment tasks using only typed primitives, no generation or fine-tuning.

    ProjectresearchPython
  • jev-eval

    Benchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.

    ProjectresearchPython