jev-research-eval
Reproducible evaluation harness and field note for the Jev Ultrafast research browser, with QC-graded baseline and stress suites.
Overview
jev-research-eval is a reproducible evaluation harness for a Jev Ultrafast research-browser session. It is not a fork of Jev Ultrafast; it drives a pinned upstream checkout of browser-use/jev-ultrafast and regenerates reports from locked fixtures. It contains 11 baseline research cases, human multi-hop stress cases and quantitative operations stress cases, with quality-control grades locked per case.
Scripts run the suites live through an agent wrapper that records full history, merge automatic results with grades, and generate an HTML field note and two interactive notebooks with per-step trace timelines. The field note and notebooks can be regenerated offline without a browser or API keys. A design folder documents visual tokens, and a methodology document explains the approach. The harness is MIT-licensed and the README notes it is not investment advice.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which browser action should the research agent take next (in the evaluated upstream agent)?(inferred) | — |
How it works
- Pin an upstream jev-ultrafast checkout
- Run baseline and stress cases through the agent wrapper
- Grade outputs and lock QC results
- Generate the field note and notebooks
Strengths
- Pinned upstream commit for reproducibility
- Offline regeneration of reports
- Trace-level drill-down
Caveats
- Evaluates one upstream agent at one commit
- Small case counts
- Not investment advice
Getting started
python scripts/generate_report_v4.py --input fixtures/qc_rescored.json --output /tmp/jev_note_regen.htmlAt a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Evaluation scripts
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Research automation: reproducible eval harness plus field note for Jev Ultrafast research-browser tasks, with QC'd cases, a suite runner, and a report generator.
yibie/awesome-jevReproducible Jev Ultrafast research-browser eval harness + field note (QC’d cases, suite runner, report generator). Not investment advice.
hellogumbo/awesome-jev
Frequently asked questions
Is this a Jev fork?
No. It drives a pinned upstream jev-ultrafast checkout.
Can I view results without keys?
Yes. The field note and notebooks regenerate offline.
What is evaluated?
Baseline research tasks plus human and quantitative stress suites.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPythonAudit of 300 web pages for AI citability with nine Jev decisions each, checked against an LLM judge for agreement, cost and latency.
ProjectresearchPython