system-one-adapter-python
TypeSafe's official drop-in replacement for the SDK's system_one call backed by OpenAI, Anthropic or Gemini, for comparing Jev with LLMs.
Overview
System One Adapter is TypeSafe's official open-source drop-in replacement for typesafe_sdk's system_one evaluation API, backed by LLM APIs instead of TypeSafe. Its purpose is comparing TypeSafe against an LLM on cost, speed and intelligence using the same Noul, Choice and Score questions. Provider SDKs are optional extras for OpenAI-compatible providers, native Anthropic and native Gemini.
The client is configured with how the LLM should answer: native structured outputs or prompted JSON, probabilities or discrete answers, and optional probability normalisation. Each call names a provider and model, or passes a provider instance such as a custom OpenAI-compatible endpoint. OpenAI uses the Responses API with strict JSON Schema, Gemini the Interactions API, and Anthropic has a configurable output token limit, with clear errors when limits are hit. Responses subclass the SDK's SystemOneResponse, adding usage across retries, and errors expose attempt history; retries follow the SDK's RetryPolicy. Other projects, such as pytest-jev, use it to benchmark LLM judges on the same questions.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | The book review is positive. | — |
| Choice | Any choice question, answered by an LLM in place of Jev | — |
How it works
- Install with the provider extra
- Configure answer mode (structured, probabilities, normalisation)
- Call system_one with provider and model
- Compare the response with Jev's on cost, speed and accuracy
Strengths
- Same API and response types as the official SDK
- Fair LLM-versus-Jev comparisons
- Supports OpenAI, Anthropic, Gemini and compatible endpoints
Caveats
- LLM probabilities may not be calibrated like Jev's
- Provider costs and latency apply
Getting started
pip install 'system-one-adapter[openai]', then SystemOneAdapterClient(...).system_one(state, questions, provider="openai", model="gpt-4o-mini").At a glance
- Type
- Library
- Status
- production
- Language
- Python
- Runtime
- Python library
- Has code
- yes
- Stars
- 392
Use cases and integrations
How curated lists describe it
Python ecosystem: TypeSafe AI's official open-source drop-in adapter for running and benchmarking Jev System One decision evaluations across OpenAI- and Anthropic-compatible LLM APIs.
yibie/awesome-jevOfficial drop-in `TypeSafeClient` replacement backed by LLM APIs, for comparing Jev against chat models on the same questions. `pip install system-one-adapter`.
AnotiaWang/awesome-decision-modelsDrop-in TypeSafeClient replacement backed by LLM APIs
wh000wh000/awesome-jev-liveDrop-in adapter for comparing the System One interface with LLM providers.
cobanov/awesome-jevOfficial drop-in TypeSafeClient replacement backed by OpenAI, Anthropic, and compatible LLM APIs, for comparing Jev against chat models.
hellogumbo/awesome-jev
Frequently asked questions
What is it for?
Running the same System One questions on LLMs to compare them with Jev.
Is it official?
Yes, it is published by TypeSafe AI.
Which providers are supported?
OpenAI-compatible, Anthropic and Gemini.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonRust port of TypeSafe's system-one-adapter that answers System One questions with OpenAI or Anthropic LLMs for comparison with Jev.
ProjectbetaRustBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPython