Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

transcript-scorecard

Proof of concept that scores replayed support-call transcripts live against a weighted scorecard using OpenAI Decisions or Jev.

Appprototype
View on GitHub ↗

Overview

Transcript Scorecard evaluates support calls for a fictional company, ACME, against a weighted employee scorecard. A live replay reveals a transcript turn by turn (by default one turn every 0.5 seconds) and scores it in batches every five turns; if an evaluation is still running, new turns are folded into the next request. The weighted score updates as the call progresses.

The server has adapters for OpenAI Decisions and TypeSafe Jev, so the two providers can be compared on the same calls. Final evaluations, evidence, confidence and score history are stored in SQLite, and the UI highlights server-measured round-trip time per request. It is built with React, Vite, Effect and Turborepo, and keeps API keys on the server only. It suits teams exploring quality-assurance scoring for contact centers.

What the decision model decides

TypeQuestionAnswers
ScoreHow well does the agent meet each scorecard criterion so far?(inferred)—

How it works

  1. Replay a transcript turn by turn.
  2. Every five turns, send the revealed prefix to the selected provider.
  3. Store per-criterion scores, evidence and confidence.
  4. Recompute the weighted score and persist history in SQLite.

Strengths

  • Side-by-side provider comparison with measured latency.
  • Incremental scoring as the call unfolds.
  • API keys never reach the browser.

Caveats

  • Proof of concept with demo data.
  • Requires OpenAI and/or Jev API keys.

Getting started

pnpm install --frozen-lockfile, cp .env.example .env (add OPENAI_API_KEY and JEV_API_KEY), pnpm dev

At a glance

Type
App
Status
prototype
Language
TypeScript
Runtime
Node API + React/Vite web app
Has code
yes

Use cases and integrations

How curated lists describe it

  • ACME live support-call scoring demo with TypeSafe AI, Effect, SQLite, React, Vite, and Turborepo.

    hellogumbo/awesome-jev

Frequently asked questions

What does it score?

Support-call transcripts against weighted criteria in an employee scorecard.

Which providers are supported?

OpenAI Decisions and TypeSafe Jev, through server-side adapters.

Is it production software?

No, it is a runnable proof of concept.

Related projects

  • dasheng

    Read-aloud English tutor: streaming ASR transcribes, then Jev judges word by word whether each word was read correctly.

    ProjectprototypeJavaScript
  • HACKSPAIN-2026

    Hackathon crisis agent that filters incoming calls and messages, plans with an LLM plus a solver, and uses Jev for in-call perception.

    ProjectprototypePython
  • celesto

    Secure, persistent VM sandboxes for AI agents, with a PR-review example that compares a general model with Jev on candidate findings.

    ProjectproductionPython
  • citation-verifier

    Checks whether each cited paper supports its citing sentence: code fetches papers, Claude proves a quote, Jev scores it, a human decides.

    Serviceproduction
  • JevPromptCoach

    Claude Code plugin that logs your prompts locally and later has Jev score how well you prompt the coding agent over time.

    PluginbetaTypeScript
  • jsort

    sort where the key is a description: Jev judges pairs of texts and a Bradley-Terry fit turns them into a scored ranking.

    ProjectbetaPython