jev-geo-audit
Audit of 300 web pages for AI citability with nine Jev decisions each, checked against an LLM judge for agreement, cost and latency.
Overview
Jev GEO Audit checks 300 public pages from 10 sites for generative engine optimisation, meaning whether an AI assistant would quote the page. For each page Jev answers one page-type question and eight yes/no checks, and the probabilities become a score and a fix list. An LLM judge (Claude Opus 5) answers the same nine questions on a seeded sample of 100 pages to measure agreement.
The README reports Jev cost $0.0228 for all 300 pages versus $1.21 for the judge's 100, about 158 times cheaper per page, with 61 ms median latency per page (four pages per request) versus about 2.8 seconds. On the 900 sampled decisions, the two agreed 85.8 percent of the time, rising to 98.2 percent when Jev's confidence was at least 0.90 (324 decisions) and 78.8 percent below that. Three scripts fetch, audit and evaluate, and a static page shows scores and fix lists.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | What type of page is this? | docs · blog · landing · changelog · comparison · pricing · other |
| Yes / no | Does the page pass each of eight citability checks? | — |
How it works
- fetch.py collects pages
- audit.py asks Jev nine questions per page and builds a score and fix list
- eval.py asks an LLM judge the same questions on 100 pages
- Publish agreement, cost and latency on a static page
Strengths
- Much cheaper and faster than an LLM judge in this run
- High agreement on confident answers
- All run data committed
Caveats
- Agreement with an LLM judge is not ground truth
- Ten sites only
Getting started
Run fetch.py, audit.py and eval.py, then open site/index.html.At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- Scripts with static site
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
300 public pages audited for AI citability with Jev decisions, checked against an LLM judge: agreement, cost and latency, measured
logicrw/awesome-jev-projects
Frequently asked questions
What does Jev decide?
Each page's type and whether it passes eight citability checks.
How did it compare with an LLM judge?
85.8 percent agreement overall, 98.2 percent when Jev's confidence was 0.90 or higher.
What did it cost?
$0.0228 for 300 pages, versus $1.21 for 100 pages with the judge.
Related projects
Evaluation suite for decision models with 23,900 pinned rows across nine task families and three primitives, plus a leaderboard.
ProjectbetaPythonAnalytics engine that rates published posts on pre-registered rubrics with Jev and tests ratings against real engagement.
ProjectbetaTypeScriptOpen-source BYOK arena where Jev faces another model as a judge on your question, with blind voting before costs are revealed.
ProjectbetaTypeScriptpytest-style tool that pins fixtures and expected Jev answers, then fails a model upgrade when answers flip or confidence drops.
ProjectbetaPythonBenchmark harness comparing Jev with OpenRouter models and open Jev-shaped models on accuracy, calibration, latency and cost.
ProjectresearchPythonWrite-up testing whether Jev beats small fast LLM tiers on public and private classification, as a classifier or a filter.
ProjectresearchPython