padflow-jev-evals
Public benchmark of PadFlow's land-development SaaS decisions: schemas, anonymised seed rows and a runner for calibrated decision models.
Overview
padflow-jev-evals publishes the typed decisions PadFlow, a SaaS platform for land-development companies, makes inside its software: routing an incoming email or file to a project and document type, coding a synced QuickBooks transaction to a budget line and draw, and classifying a value read from an imported schedule or budget. Each schema has an input, a decision enum and a confidence; PadFlow auto-posts above a threshold and queues the rest for a human.
The runner scores any OpenAI-compatible model on accuracy, calibration (accuracy of examples marked above the auto-post threshold, default 0.90), cost per decision and p50/p95 latency. The README argues calibration matters most: a model that knows which 60% it is sure of beats a more accurate but uniformly confident one. Current baselines use only 5-7 hand-written seed rows per decision, because no human-labelled production rows existed at export time, so results are very noisy.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which project, and what kind of document is it? | — |
| Choice | Which budget line and draw does this transaction belong to? | — |
| Choice | What is this imported value? | — |
How it works
- Load schemas and labelled rows.
- Run a model on each decision via run_baseline.py.
- Measure accuracy, above-threshold accuracy, cost and latency.
- Use the auto-post threshold to compare models.
Strengths
- Real workload decisions from a production SaaS.
- Focus on calibration at an operating threshold.
- Reproducible runner.
Caveats
- Only 5-7 seed rows per decision so far.
- No labelled production set yet.
Getting started
Run scripts/run_baseline.py against a model to fill the results table.At a glance
- Type
- Benchmark
- Status
- research
- Language
- Python
- Runtime
- CLI
- Has code
- yes
Use cases and integrations
How curated lists describe it
Typed-decision benchmark from PadFlow (land development SaaS): schemas, anonymized labeled rows, and a runner for confidence-calibrated models like TypeSafe Jev.
hellogumbo/awesome-jev
Frequently asked questions
What decisions are covered?
Document routing, transaction coding and classifying imported values.
Why emphasise calibration?
PadFlow auto-posts only above a threshold, so the accuracy of confident answers matters most.
How big is the dataset?
Currently 5-7 hand-written seed rows per decision.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPythonLlamaIndex notebook comparing Jev with open decoders, encoders and specialised tools on document-pipeline Choice tasks.
ProjectresearchPython