jev-align (Sutro)
Sutro's active-learning CLI that finds rows Jev is least sure about, asks you to label them, and uses GEPA to improve the definition.
Overview
jev-align from Sutro is an experimental Python CLI for building AI Functions with Jev from human feedback. Each round it evaluates the configured CSV, Parquet or JSONL dataset with Jev and measures uncertainty, selects ambiguous rows plus a random audit sample for you to label, runs GEPA prompt optimisation on your accumulated labels and optional rationales, then shows the score, certainty change and a diff of the proposed definition so you can accept, reject, rewind or resume later. A higher training score never accepts a proposal automatically.
Task types are binary, multiclass, multilabel and ordered-rubric score. Advanced settings cover 5-20 annotations per round, an optional 20% held-out set and the GEPA metric-call budget (300 by default), and it uses the first 1,000 rows by default. Functions can be used in applications and shared through ai-functions.dev. It needs a TypeSafe key (or Vercel or Cloudflare access) and an LLM key for GEPA, and is Apache-2.0 licensed.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Binary AI Function: is this row True or False under the definition? | True · False |
| Choice | Multiclass AI Function: which fixed label applies? | — |
| Score | Which level of the ordered rubric does this row reach? | — |
How it works
- Point the CLI at a dataset and define a function
- Jev evaluates rows and measures uncertainty
- Label ambiguous and audit rows
- GEPA proposes a revised definition from labels
- Review score change and diff; accept, reject or rewind
Strengths
- Labels go where the model is least sure
- Human approval required for every definition change
- Supports common data formats and task types
Caveats
- Experimental
- Needs both a Jev provider and an LLM key
Getting started
`uv tool install jev-align`, set TYPESAFE_API_KEY and OPENAI_API_KEY (or another LLM key), then run `jeva`.At a glance
- Type
- CLI
- Status
- beta
- Language
- Python
- Runtime
- CLI
- Has code
- yes
- License
- Apache-2.0
Use cases and integrations
How curated lists describe it
Dataset engineering: evaluates CSV, Parquet, and JSONL rows with Jev `Choice`, `Score`, or `Boolean` decisions, sends ambiguous and audit samples to a human, and uses accepted human labels to optimize the saved definition with GEPA.
yibie/awesome-jevUnofficial active-learning CLI that evaluates CSV, Parquet, and JSONL rows with Jev, asks people to label uncertain and audit samples, and uses GEPA to propose improved definitions
AnotiaWang/awesome-decision-models1. Evaluates the configured dataset and measures uncertainty. 2. Selects ambiguous rows plus a random audit sample for you to label. 3. Uses your accumulated labels and optional rationales to run GEPA. 4. Shows the score, certainty change, and proposed definition diff. 5. Lets you accept, reject, rewind, or resume later.
logicrw/awesome-jev-projectsBuild calibrated AI Functions from human feedback using Jev and GEPA.
wh000wh000/awesome-jev-liveActive-learning CLI where Jev evaluates rows and surfaces uncertain or audit samples for human labeling, then GEPA proposes revised decision definitions that the user can accept or reject.
cobanov/awesome-jevActive-learning CLI that uses human labels and GEPA to improve Jev decision definitions.
hellogumbo/awesome-jevCLI from Sutro that finds the examples a Jev function is least sure about, asks you to label them, and uses GEPA to improve the question.
MrJev/awesome-jev
Frequently asked questions
What does jev-align improve?
The definition (question and criteria) of a Jev-based AI Function, using your labels and GEPA.
Will it change my definition automatically?
No, you review and accept or reject each proposal.
What data formats are supported?
CSV, Parquet and JSONL.
Related projects
dbt package whose macros run Jev Choice, Noul and Score over SQL values in DuckDB and ClickHouse models.
ProjectbetaSQL (Jinja)DuckDB extension that asks Jev about every row in SQL and returns answers as ENUM, DOUBLE or STRUCT columns.
PluginbetaC++SQLite and Node library adding Jev judgments to SQL for matching, evidence picking, ranking, review queues and semantic CI checks.
ProjectbetaJavaScriptRecord linkage for researchers: write the match rule in plain English and Jev judges each candidate pair with a probability.
ProjectbetaPythonPolars library for parallel LLM inference that also asks Jev per-row Noul, Choice and Score questions through a native Rust layer.
LibrarybetaPythonDecision-first CSV cleaning: code proposes repairs, Jev scores action, applicability and risk, a fixed policy applies or abstains.
ProjectprototypePython