candidate-experience-benchmark
Benchmark comparing Jev, other decision models and LLMs on four judgments over 60 synthetic candidate-experience reviews.
Overview
Candidate Experience Feedback Benchmark asks how well TypeSafe Jev and general-purpose language models classify feedback candidates write about their hiring experience. Each method makes four judgments on the same 60 fictional reviews, complaints and potential testimonials, and saved results include output validity, agreement with provisional references, prompt variations, request times, token counts and costs where recorded.
An explorer lets you pick decision models, LLMs or both and inspect any saved run with its source report. A coverage audit accounts for 1,004 selectable run entries, including repeats and partial runs, every one now recording individual answers or explicit invalid, failed or unsent outcomes. The README is careful that repeats reuse the same 60 reviews and are not independent test sets, and references are provisional.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Four classification judgments per candidate review (e.g. type of feedback)(inferred) | — |
How it works
- Prepare 60 synthetic reviews and reference labels
- Run Jev, other decision models and LLMs with prompt variants
- Record validity, agreement, time, tokens and cost
- Publish runs in an explorer with an evidence report
Strengths
- Complete per-answer records
- Cost and token tracking
- Honest about non-independent repeats
Caveats
- Synthetic data and provisional references
- Small set of 60 reviews
Getting started
Explore results at https://adambkovacs.github.io/candidate-experience-benchmark/At a glance
- Type
- Benchmark
- Status
- research
- Runtime
- benchmark harness + static explorer
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Compare TypeSafe Jev and LLMs on 60 synthetic candidate-experience reviews…
wh000wh000/awesome-jev-liveCompare TypeSafe Jev and LLMs on 60 synthetic candidate-experience reviews: four classification tasks, prompt variants, costs, tokens, and reproducible evidence.
hellogumbo/awesome-jev
Frequently asked questions
What is classified?
Candidate feedback about hiring experiences, with four judgments each.
Which models are compared?
Jev, other decision models and general-purpose LLMs.
Are the reviews real?
No, they are fictional.
Related projects
- jev-arena★ 122
Side-by-side tool comparing Jev and DeepSeek on labelling the same comments for relevance, sentiment and intent, with replay and reports.
ProjectbetaTypeScript Jev applied to SemEval-2026 DimABSA dimensional sentiment tasks using only typed primitives, no generation or fine-tuning.
ProjectresearchPythonJapanese demo judging the same utterance with Jev and an LLM to compare emotion-shift outputs and speed side by side.
ServiceprototypeTypeScriptChinese 'emotion lie detector' demo comparing Jev's Choice, Score and Noul judgments on a message with a general LLM.
CLIprototypePythonBenchmark of Jev 1.13.0 against 11 classical classification pipelines on eight datasets under a published protocol.
BenchmarkresearchPythonSmall reproducible benchmark of Jev against LLMs, fine-tuned BERT, Laya and zero-shot NLI on SST-2, AG News and Banking77.
BenchmarkresearchPython