jev-prompt-optimization
Optimises the instructions and criterion descriptions of a Jev Choice from labelled data with EvoPrompt or GEPA, scored by Brier.
Overview
Jev Prompt Optimization is an open-source tool that automatically tunes a Jev Choice question from labelled data. Given a Choice definition and a dataset with gold labels, it searches the instructions and each criterion's description while keeping label names fixed. Candidates are scored by the normalised Brier score of Jev's probability distributions, and a GPT model acts as the mutation operator in either EvoPrompt (a genetic algorithm over a population of candidates) or GEPA (which uses natural-language feedback on why predictions were right or wrong).
A local web UI lets you enter the Choice definition, load JSON, JSONL or CSV data with text and label fields (with automatic stratified train/validation split), choose an optimiser and a metric-call budget, and compare before and after. A CLI covers the same workflow. Runs call external APIs, so the README advises reviewing the dataset and budget first. Documentation is in English and Japanese.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | User-defined classification (for example, billing or technical) | billing · technical |
How it works
- Enter a Choice definition and load a labelled dataset
- Jev scores candidate definitions on the training split
- Compute normalised Brier score
- EvoPrompt or GEPA proposes improved instructions and criteria
- Export the optimised definition
Strengths
- Optimises against calibration, not just accuracy
- Label names stay fixed, so code integration does not change
- Web UI and CLI
Caveats
- Needs both a TypeSafe key and an optimiser LLM key
- Optimisation spends API budget
- Risk of overfitting small datasets
Getting started
git clone the repo, cp .env.example .env (set TYPESAFE_API_KEY and OPTIMIZER_API_KEY), uv sync, uv run jevopt web, then run the frontend with npm run dev.At a glance
- Type
- Project
- Status
- beta
- Language
- Python
- Runtime
- CLI and local web UI
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
automatically optimizing the instructions and decision criteria of TypeSafe Jev Choice from labeled data
logicrw/awesome-jev-projects
Frequently asked questions
What gets optimised?
Only the instructions and each criterion's description; label names stay fixed.
What metric is used?
Normalised Brier score of Jev's probability distributions.
Which optimisers are available?
EvoPrompt (genetic algorithm) and GEPA (feedback-driven).
Related projects
Research demo where Jev answers plain questions about text and a small learned head adds new questions from human feedback.
ProjectresearchPythonTool that recomputes Jev's confidence identities on outputs published in public repositories to spot answers Jev never actually produced.
ServiceresearchPythonClaude Code and Codex skills powered by Jev, published with a calibration curve measured on 4,995 real agent decisions.
ProjectAgent skills for placing, building, evaluating and improving decision-model systems, with Jev as the default hosted example.
Agent skillbetaClaude Code plugin and CLI for the Jev funnel: judge items in bulk, settle confident ones in code, send only uncertain ones onward.
PluginbetaPythonUnofficial Claude Code plugin that checks 'done' claims and small judgment calls with Jev, logs receipts, and can warn on unverified stops.
Pluginbeta