Prosodia
Audio-native Jev-style model: a frozen Whisper encoder encodes speech once and typed questions return emotion and prosody distributions.
Overview
Prosodia is a research reproduction of the System One shape with speech as the state. Audio is encoded once by a frozen Whisper encoder (the decoder never runs), and several typed questions branch over that shared encoding in parallel, returning Choice, Score or Noul distributions about emotion, sentiment and acoustic properties. There is no transcript and no generated text. Question text and option sets are given at request time, and a pointer readout lets any number of candidates work without retraining.
The author's motivation is that other audio entries in Jev lists route ASR to text to Jev, so the audio itself never reaches the model. The README is candid about results: a 12-arm ablation on MELD found ECE unable to rank arms, the WavLM control lost to Whisper, and prosody effects did not survive base-rate controls. It also reports that the independent-inner-product readout satisfies IIA exactly, unlike Jev, and proposes an IIA probe as a conformance test for reimplementations.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | What emotion is expressed in this utterance?(inferred) | — |
| Choice | What is the pitch direction of this utterance? | — |
| Score | Sentiment or acoustic property levels supplied at request time(inferred) | — |
How it works
- Encode the audio clip once with a frozen Whisper encoder
- Fold each typed question into the batch dimension so questions cannot attend to each other
- Score options with a pointer readout against option embeddings
- Return a probability distribution per question
Strengths
- No ASR step, so no transcription latency or error propagation
- Dynamic option sets at request time
- Trained with cross-entropy plus Brier, a proper scoring rule
Caveats
- Research prototype; results are single-seed and the README calls single numbers approximate
- Satisfies IIA exactly, which the author says Jev does not, so it is not yet in Jev's functional class
- Demo Space is private
At a glance
- Type
- Service
- Status
- research
- Language
- Python
- Runtime
- PyTorch; Hugging Face Space (Docker)
- Has code
- yes
Use cases and integrations
How curated lists describe it
Audio-native Jev-style decision model: a frozen Whisper encoder turns speech into one shared state and typed questions branch over it in parallel, returning Choice/Score/Noul distributions over emotion, sentiment and acoustic properties with no ASR step and no generated text; question text and option sets are supplied at request time, and a hosted demo takes mic input or an uploaded file.
cobanov/awesome-jev
Frequently asked questions
What does Prosodia decide?
It answers typed questions about speech, such as emotion or pitch direction, as probability distributions directly from audio features.
How is it different from ASR plus Jev?
It skips transcription; the audio encoding itself is the state.
Is it as calibrated as Jev?
Unclear. It is trained with a proper scoring rule, but TypeSafe's RLCD method is unpublished and the README warns ECE could not rank its variants.
Related projects
Mangaba AI's local, calibrated typed-decision model for Portuguese text, trained to optimize the Brier score; proprietary since v5.
ProjectproductionPythonStandalone Jev-class emotion model: one pass over a message sequence returns calibrated PAD scores, an emotion family and yes/no judgments.
ProjectresearchConverts open Jev reproductions (Laya, Kev, NanoJev, PlayJev) to INT8 ONNX for offline CPU decisions at about 15.6 ms per question.
ServicebetaPythonValence: macro news sentiment engine scoring global headlines' market impact with a contrastive System-One model on serverless GPU.
ProjectprototypePythonSmall reproducible benchmark of Jev against LLMs, fine-tuned BERT, Laya and zero-shot NLI on SST-2, AG News and Banking77.
BenchmarkresearchPython- TypeAR★ 946
TypeLLM: Jev-inspired type-safe decoding for existing autoregressive LLMs on SGLang, with schema-guaranteed typed outputs.
ProjectbetaPython