laya-jev-lab
Independent measurements of Jev versus open Laya on typed decisions, plus a local-first cascade that matches Jev at 1.8x speed.
Overview
laya-jev-lab publishes independent, reproducible comparisons of typed-decision models: Jev, TypeSafe's closed API, and Laya, Convai's open weights. Everything was run on an Apple M4 Max with MLX, and every number comes from a script in the repository with raw output saved. It asks which primitives are actually reliable, whether confidence can be trusted to gate automation, and whether a local-first cascade beats calling the API directly; its short answers are 'only some, only sometimes, and yes if traffic is mostly clear-cut'.
On 40 Chinese support-ticket classifications, Jev scored 78% at 588 ms and Laya 57% at 7.6 ms; Jev was perfect on clear single-intent inputs, and both struggled on boundary inputs with no correct label. A cascade that tries Laya first and escalates below a confidence threshold matched Jev's 78% at a 0.60 threshold, solving 45% of traffic locally at 1.8 times Jev's speed; higher thresholds added latency without gaining accuracy. The sample is small, as the tables make clear.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which category does this support ticket belong to? | — |
| Yes / no | Is the local model's confidence high enough to accept, or escalate to Jev? | — |
How it works
- Run the same tickets through Jev and local Laya
- Measure accuracy, confidence and latency by difficulty tier
- Sweep cascade thresholds: accept Laya above threshold, else call Jev
- Report accuracy, escalation rate and speed
Strengths
- Independent, script-backed numbers
- Practical cascade recipe with threshold sweep
- Breaks results down by input difficulty
Caveats
- Only 40 cases
- Chinese support tickets only
- Apple Silicon setup
At a glance
- Type
- Service
- Status
- research
- Language
- Python
- Runtime
- scripts with MLX and Jev API
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Independent measurements of typed-decision models: Jev (TypeSafe API) vs Laya (open weights), and a local-first cascade that matches Jev's accuracy at 1.8x the speed
logicrw/awesome-jev-projects
Frequently asked questions
How did Jev compare with Laya?
78% versus 57% accuracy on 40 tickets, but 588 ms versus 7.6 ms latency.
Does the cascade help?
At a 0.60 threshold it matched Jev's accuracy while handling 45% of traffic locally, 1.8 times faster.
Are the results reproducible?
Yes, every number comes from a script in the repo with raw outputs saved.
Related projects
pytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonTest harness for any Jev-API decision model: accuracy, calibration and robustness probes for reordering, injection and padding.
ServicebetaPythonZero-dependency Node proxy that shadows Jev with local decision models and reports whether you can safely swap, per question and policy.
ProxybetaJavaScriptExperiments testing whether Jev can spot positive-expected-value dice bets from payout tables, and whether recent wins or losses sway it.
ProjectresearchZero-shot run of Jev 1.13 on all seven LexGLUE legal tasks (23,607 examples) versus GPT-5.6 Luna, open models and fine-tuned BERT.
ProjectresearchPythonPre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPython