JevForge
Open stack to synthesise decision data, train and calibrate a Qwen3.5-0.8B scorer, evaluate on Mind2Web and serve Jev-style.
Overview
JevForge is an end-to-end toolkit that turns structured decisions into one reproducible pipeline: given a state, a question and a candidate set, it builds training records, trains and calibrates a scorer, evaluates fixed splits, and serves a full probability distribution through a Jev-compatible API. Choice, Noul and ordered Score questions share one decision path. Its first proving ground is the interaction loop of Jev-style web agents: choosing the next click, navigation target, form control, route or escalation from the elements currently on a page. Model weights and a Mind2Web-derived dataset are published on Hugging Face, with an interactive demo, design notes and preliminary RLCD results.
On fixed, website-disjoint test and out-of-distribution splits, the README reports JevForge 0.8B at 0.579 test and 0.637 OOD Choice top-1, against 0.543 and 0.610 for Jev-1.13 shown as a separate reference arm and 0.235 and 0.340 for the raw backbone. On Noul, Jev-1.13 is better on the test split (accuracy 0.910, Brier 0.092 versus 0.826 and 0.128) while JevForge 0.8B is better on OOD. Listings note the metrics are project-local. Documentation is in English and Chinese.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which page element should the agent act on next? | — |
| Yes / no | Yes/no web-interaction judgments (e.g. should this escalate?)(inferred) | — |
| Score | Ordered score questions on the same decision path | — |
How it works
- Synthesise auditable decision records from Mind2Web
- Train and calibrate a Qwen3.5-0.8B scorer
- Evaluate on fixed test and OOD splits
- Serve via a Jev-compatible endpoint
Strengths
- Fully reproducible pipeline with public artifacts
- Choice accuracy above the Jev-1.13 reference on these splits per the README
- Jev-compatible serving
Caveats
- Metrics are project-local
- Jev-1.13 still better on test-split Noul
- Focused on web interaction tasks
At a glance
- Type
- Service
- Status
- research
- Language
- Python
- Runtime
- Training pipeline + local server
- Has code
- yes
Use cases and integrations
How curated lists describe it
Open research: an end-to-end stack for auditable data construction, Qwen3.5-0.8B training, fixed Mind2Web and OOD evaluation, local serving, and a preliminary RLCD baseline.
yibie/awesome-jevAuditable Qwen3.5-0.8B pipeline covering data synthesis, training, calibration, fixed evaluation, Jev-compatible serving, and public model and dataset artifacts; its metrics are project-local.
cobanov/awesome-jevEnd-to-end Jev-style structured-decision stack for auditable data construction, Qwen3.5-0.8B training, fixed Mind2Web and OOD evaluation, preliminary RLCD, local serving, and interactive replay.
hellogumbo/awesome-jevEnd-to-end toolkit for the other direction: synthesise decision data, train a calibrated candidate scorer on it, evaluate it, and serve it behind a Jev-compatible endpoint.
MrJev/awesome-jev
Frequently asked questions
How does it compare with Jev?
On the project's splits, JevForge 0.8B leads on Choice top-1 while Jev-1.13 leads on test-split Noul.
Can existing Jev clients use it?
Yes, it serves a Jev-compatible API.
Are the weights public?
Yes, the model and dataset are on Hugging Face.
Related projects
- cbjev★ 4
Self-hosted encoder model giving Jev-compatible choice, score and noul answers in one pass, pitched as a better-calibrated Laya successor.
ProjectresearchPython Joke decision model trained to answer wrongly with total confidence: 1.9% accuracy at 0.957 mean confidence, via a Jev-style API.
ProjectresearchPythonPre-registered field test of Jev, GLiNER2.5-Decide, Laya and other decision models against small LLMs on fresh September 2026 data.
ProjectresearchPython11 MB CPU-only model that answers choice, noul and score questions with fitted calibration and no LLM at inference.
ProjectresearchPythonTiny pre-registered experiment testing whether a confidence-scoring reward (RLCD-like) yields confidences safe to automate on at 0.9.
ProjectresearchPythonResearch code for an image-and-text System One model that reads option-letter logits from a multimodal LLM, with training and calibration.
ProjectresearchPython