System One tasks
The four System One primitives and the four workflows in the benchmark, each with every model ranked on exactly the questions that belong to it.
Primitives
The question types a System One model answers. A model is ranked on every question of that type across both suites.
A question with a fixed set of named options and a description for each; the model returns a probability per option and the answer is the argmax.
Leads: Jev 0.743
An ordered scale (none / low / medium / high, 1–5, …). The model returns a distribution over levels and the answer is the expected level. The primitive relevance classifiers cannot do.
Leads: Jev 0.663
A statement about the state, answered with a probability that it holds. Named after the Jev primitive; the threshold for "yes" is 0.5 unless the caller sets one.
Leads: LAYA typed-decisions 0.857
Return the text spans that match each label (person, organization, date, money, …). Jev and the pure decision models never return spans; only the GLiNER family does.
Leads: GLiNER2 multi 1.000
Workflows
The domains the questions come from. Each workflow exists on at least one suite; the ranking uses only its questions.
Support tickets: which team, is a refund requested, how frustrated, churn risk, urgency, does a human need to step in. The domain TypeSafe uses in its own docs.
Leads: Jev 0.790
Alerts and incident write-ups: severity, active exploitation, credential compromise, true positive, disposition, urgency.
Leads: Jev 0.756
Traces of an autonomous agent run: outcome, risk, urgency, what the observability system should do, whether a human should review it.
Leads: LAYA typed-decisions 0.648
Invoices against purchase orders: does it match, is it a duplicate, discrepancy severity, disposition, urgency.
Leads: Jev 0.756