Jev-api-experiments
Hypothesis-driven experiments characterising the Jev API, intended to inform a future football prediction product.
Overview
Jev API Experiments is an empirical research repository for understanding TypeSafe's Jev API through controlled experiments. It follows a strict loop of hypothesis, experiment design, controlled test, raw observations, quantitative analysis and conclusion, labelling each hypothesis as supported, rejected, inconclusive or untested, and explicitly refuses to treat marketing claims, blog posts or demos as facts until tested.
The author's long-term goal is to decide whether and how Jev should appear in a future FootyQuant v2 system for football prediction and betting-market analysis, but the README is clear that this repository is research only, not a betting product, application or recommendation engine. Experiments 001-006 (phase 1, API understanding) have been run using early access and a small promotional credit. It pins jev-1.13.0 rather than the moving alias, writes raw output to results files, and has pytest tests.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Other | Tests hypotheses about how the Jev API behaves | — |
How it works
- State a hypothesis
- Design and run a controlled experiment against pinned jev-1.13.0
- Save raw results
- Analyse and label the hypothesis status
Strengths
- Disciplined, evidence-first approach
- Pinned model version for reproducibility
- Raw evidence saved
Caveats
- Early phase (API understanding only)
- No findings summarised in the README excerpt
Getting started
Create a venv, `pip install -r requirements.txt`, set TYPESAFE_API_KEY in .env, then `python experiments/001-api-sanity/experiment.py`.At a glance
- Type
- Service
- Status
- research
- Language
- Python
- Runtime
- research scripts
- Has code
- yes
- Has tests
- yes
- Stars
- 0
Use cases and integrations
How curated lists describe it
Empirical experiments and API research for TypeSafe's Jev System One model trained using RLCD
wh000wh000/awesome-jev-live
Frequently asked questions
Is this a betting tool?
No. The README says it is research only.
Which model version is tested?
jev-1.13.0, pinned instead of jev-latest.
What has been done so far?
Experiments 001-006 covering basic API understanding.
Related projects
Pre-specified study running Jev and GPT-4.1 as 300 synthetic survey respondents; question format mattered more than model.
ProjectresearchPythonExperiment testing how well Jev predicts financial markets; the author reports it does not do well, like most LLMs.
ProjectresearchCalibration experiments showing Jev's Choice probabilities on fair dice and coins stay high while accuracy stays at chance.
LibraryresearchPythonBenchmark asking models to forecast federal motion-to-dismiss rulings from the judge's record, scored with claim-defendant Brier metrics.
BenchmarkresearchPython- jev-arena★ 122
Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.
ProjectbetaTypeScript Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPython