Skip to content
Metatext Decide — typed decisions for your agents, with a probability on every answerGet early access
Metatext
Sign Up

jev-position-test

Small experiment: reordering enum options changed a local Jev clone's answers on its old default, while hosted Jev's did not.

Demoresearch
View on GitHub ↗

Overview

jev-position-test asks the same six questions with the same three options to a local Jev clone (jevmlx) and to hosted Jev, changing only the order in which options are listed. With jevmlx's default slots scoring at the time, answers changed with order for 5 of 6 messages, and 14 of 18 answers were simply the second-listed option; adding prior correction made it worse (6 of 6). Label scoring removed order effects but rated an empty message high risk every time. Hosted jev-1.13.0 never changed its answer in two passes, and a rerun on 3 October confirmed this, though confidence still moved somewhat with order.

After the first runs, jevmlx switched its default to label scoring, citing its own measurement. The README is clear about limits: six messages, no ground truth, one clone, one model size, one enum width, and no claim about why hosted Jev is stable. It offers a ten-minute recipe to test your own wrapper by reordering options.

What the decision model decides

TypeQuestionAnswers
ChoiceWhat risk level does this message carry (three options in varying order)?(inferred)—

How it works

  1. Fix six messages and three options with descriptions
  2. Permute option order
  3. Query the clone under several scoring settings and hosted Jev
  4. Count messages whose answer changes with order

Strengths

  • Quick, reproducible test anyone can run on their wrapper
  • Raw outputs included
  • Led to a default change in the clone

Caveats

  • Only six messages and no ground truth
  • Does not explain the cause
  • Hosted alias does not pin a version

Getting started

python3.12 -m venv .venv && source .venv/bin/activate && pip install the pinned jevmlx commit, then python3 clone_jevmlx_position.py

At a glance

Type
Demo
Status
research
Language
Python
Runtime
Scripts
Has code
yes

Use cases and integrations

How curated lists describe it

  • Reorder the enum options: a Jev clone changes its answer, Jev doesn't. n=6, raw data included.

    hellogumbo/awesome-jev

Frequently asked questions

What was found?

A local clone's answers depended on option order under its old default; hosted Jev's did not.

Did the clone fix it?

It switched its default to label scoring after these runs.

Is this a benchmark?

No. The README stresses six messages, no ground truth and one model size.

Related projects

  • TypeAR★ 946

    TypeLLM: Jev-inspired type-safe decoding for existing autoregressive LLMs on SGLang, with schema-guaranteed typed outputs.

    ProjectbetaPython
  • jev-dllm

    Reads Jev-style choice, yes/no and score probabilities from masked diffusion language models with shared Yes/No weights.

    ProjectresearchPython
  • metask-jev

    Open Jev-class decision models on Qwen3.5 with candidate-logit readout; the 4B model reports 80.1% on JevBench's 231 items.

    ProjectresearchPython
  • system-one-open

    Open Jev replica on Gemma 4 E2B and Gemma 3 270M served on Modal, with eight demos and a TypeSafe eval head-to-head.

    ProjectresearchPython
  • JevK5★ 150

    Apache-2.0 open-weight Jev alternative that answers yes/no, choice and score questions with per-option probabilities in one pass.

    LibrarybetaPython
  • lev

    Open System One decision model (LoRA on Qwen3.5-4B) that reads typed answers from logits in one pass, with its benchmark harness.

    ModelbetaPython