MetaCog
Inference-time loop where a System One judge like Jev scores a model's thought paths and picks which reasoning branch to keep.
Overview
MetaCog wraps any language model, local or via API, in an inference-time metacognition loop. The model writes one answer and a separate judge scores it. If the judge is confident, that answer is returned. If not, the model writes two to six more thought paths in parallel, sized by uncertainty, the judge scores every path and every distinct final answer, and the best is returned. There is no training and no weights are touched. The judge can be hosted Jev, the open Reflex model or a local log-probability judge via llama.cpp.
The README reports that on 180 paired GPQA Diamond and AIME problems across three hosted 'thinker' models, accuracy rose from 76.7% for the thinker alone to 86.7% with MetaCog (p = 0.02), at 0.89 times the wall time and 1.39 times the thinker tokens; pooled over 856 rows on four benchmarks, an earlier configuration went from 75.0% to 79.4%. A results document includes a ledger of every idea tried. It works with any OpenAI-compatible thinker, and a companion coding agent, Heuristic, builds it in.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Yes / no | Is this answer correct (is the judge confident)?(inferred) | — |
| Choice | Which thought path or final answer is best?(inferred) | — |
How it works
- Thinker writes a greedy answer
- Judge scores it; if confident, return it
- Otherwise generate 2-6 more paths in parallel
- Judge scores each path and distinct answer
- Return the best
Strengths
- No training required
- Measured accuracy gains on hard benchmarks
- Works with any OpenAI-compatible model
- Local judge option
Caveats
- Uses more thinker tokens
- Gains measured on specific benchmarks
- Hosted judge needs a key
Getting started
pip install -e ., then MetaCog(OpenAICompatThinker(...), SystemOneJudge.jev(), Config()).run(problem).At a glance
- Type
- Project
- Status
- research
- Language
- Python
- Runtime
- library and CLI
- Has code
- yes
Use cases and integrations
How curated lists describe it
Wraps a language model in a metacognition loop where a System One judge, Jev or the open Reflex model, picks which branch of reasoning continues.
MrJev/awesome-jev
Frequently asked questions
What does the judge decide?
Whether the first answer is trustworthy and which of several thought paths is best.
How much does it help?
From 76.7% to 86.7% on 180 GPQA Diamond and AIME problems, per the README.
Must I use Jev?
No, the open Reflex model or a local log-probability judge also work.
Related projects
Checks whether each cited paper supports its citing sentence: code fetches papers, Claude proves a quote, Jev scores it, a human decides.
ServiceproductionAgent observability platform that finds and fixes agent failures, with an optional Jev preclassifier for conversation checks.
ProjectproductionTypeScriptSemantic test matchers for Vitest and Jest that ask Jev whether output satisfies a plain-English expectation and threshold the probability.
ProjectbetaTypeScriptpytest plugin for semantic assertions on LLM output, asking Jev whether plain-English claims hold with a 0.8 probability bar.
PluginbetaPython'Vitest for LLM outputs': CLI running YAML/JSON eval suites on Jev for RAG faithfulness, hallucination and agent safety in CI.
CLIprototypeTypeScriptLearning lab for DeepEval with Jev as the judge: replayed evaluations, adjustable decision rules, release gates and drift alerts.
ProjectprototypePython