Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

Relay

Slack-like playground for watching computer-use agents work, with replays, comparable runs and Jev choosing from a menu of visible actions.

Projectbeta
View on GitHub ↗Live demo ↗

Overview

Relay is a hosted and runnable playground for computer-use agents built around a mock Slack-like workspace. You pick a task, model and interface (API, accessibility tree, page JSON or pixels), press Run, and watch a live browser replay with an inspectable trace. Runs are isolated and comparable, and a results page exposes 306 model trials plus a matched 96-run interface study and a 48-run follow-up with costs and replays. A free tier covers cheap models, and you can bring a Ramp Router or TypeSafe key.

Jev runs without a server in the client: connect Jev via TypeSafe and it selects from a recorded menu of visible actions, with the side panel showing its returned probability distribution. Because current Jev is text-only, it works with text interfaces such as the accessibility tree rather than pixels. The README frames the interface study results (API 21/24 passed, accessibility 18/24, page JSON 16/24, pixels 3/24) as describing the fixed harness, not a reliable capability ranking.

What the decision model decides

TypeQuestionAnswers
ChoiceWhich visible action should the agent take next?—

How it works

  1. Choose a task, model and interface
  2. For Jev, build a menu of visible actions from a text interface
  3. Jev returns a probability per action and the top action is executed
  4. Record the trace and replay for inspection and comparison

Strengths

  • Every trial links to its trace and recorded UI
  • Compares interfaces under one harness
  • Shows Jev's full probability distribution per step

Caveats

  • Jev is text-only, so pixel interfaces do not apply
  • Study results describe the harness, not general capability
  • Hosted free tier has daily caps

Getting started

Open relay.kevinliu.studio, or run locally per the README, and connect a TypeSafe key for Jev.

At a glance

Type
Project
Status
beta
Language
TypeScript
Runtime
Web app
Has code
yes

Use cases and integrations

How curated lists describe it

  • Watch computer-use agents work. Slack-like tasks, live browser replay, BYOK models, Jev decisions, and inspectable evaluation traces.

    hellogumbo/awesome-jev

Frequently asked questions

What does Jev decide in Relay?

Which visible action to take next, chosen from a recorded menu, with its distribution displayed.

Can Jev use the pixel interface?

No; the README notes current Jev is text-only.

Do I need a key?

Not for the free tier's cheap models or the mock workspace; Jev needs a TypeSafe key.

Related projects

  • jev-agent-failure-benchmark

    Benchmarks Jev against frontier LLMs on Who&When Pro: attributing multi-agent failures to an agent, step and error type.

    BenchmarkresearchPython
  • MM-JevBench

    10,000-sample multimodal benchmark for decision models that return full probability distributions over declared candidates.

    BenchmarkresearchPython
  • Jev broke our WebMCP benchmark

    X post by a WebMCP benchmark author reporting Jev plus a small LLM solved 100% of tasks at about 112x lower model cost.

    Discussiondiscussion
  • mobile-jev

    Standalone Android agent on Mobilerun where Jev picks every operation and target, with a live React studio, CLI and traces.

    CLIprototypeTypeScript
  • Open Jev (intikhab49)

    Open 150M encoder that answers noul, choice and score questions in one pass, trainable on a free Colab T4 in about 30 minutes.

    ModelresearchPython
  • pi-Jev-browser

    Browser and macOS desktop agent for Pi where Jev picks each action from a structured text observation in a bounded loop.

    BenchmarkbetaTypeScript