FlightBench
Fixed-wing landing simulator and benchmark where Jev maps four Choice decisions to aircraft controls against humans, baselines and LLMs.
Overview
FlightBench is a landing simulator and benchmark for decision models, LLMs and classical controllers built on JSBSim. Jev receives the aircraft state and answers four typed Choice questions that become actuator commands, flying in real time. You can try landing yourself, watch recorded flights, or compare against a baseline controller and Codex-driven LLMs.
The README reports that Jev landed in real time at about 2.4 calls per second with a score of 56 for about $0.0122, while Codex-driven GPT-6 models, even with the simulation paused for each step, either crashed or ran off the runway at much higher latency and cost. It is candid that Jev's performance is not strong: across runs it reports 14% successful landings, 74% runway excursions and 9% nose-first contacts. The author's human record is 70.1 out of 100, and support for open Jev-like models is planned.
What the decision model decides
| Type | Question | Answers |
|---|---|---|
| Choice | Which command should each of the four actuators take this step? | — |
How it works
- Simulator sends aircraft state
- Jev answers four Choice questions
- Choices map to actuator commands
- Score the landing and compare controllers
Strengths
- Real-time control at low cost
- Head-to-head with LLMs and humans
- Honest reporting of failure rates
Caveats
- Low successful landing rate reported
- Simulation only
Getting started
uv sync, uv run uvicorn flightbench.server:app --port 8000, then cd frontend && npm ci && npm run dev; uv sync --extra jev and TYPESAFE_API_KEY for Jev.At a glance
- Type
- Demo
- Status
- prototype
- Language
- Python
- Runtime
- JSBSim + web frontend
- Has code
- yes
- License
- MIT
Use cases and integrations
How curated lists describe it
Flight simulation: a fixed-wing landing simulator and benchmark where you can compete with Jev in landing a plane, mapping four Jev `Choice` decisions to aircraft controls in JSBSim and benchmarking landings against human-pilot, baseline, and LLM controllers, with planned support for open-source Jev-like decision models.
yibie/awesome-jevA fixed-wing landing simulator and benchmark where you can compete with Jev in landing a plane, mapping four Jev `Choice` decisions to aircraft controls and benchmarking landings against human-pilot, baseline, and LLM controllers.
AnotiaWang/awesome-decision-modelsFixed-wing aircraft landing simulator and benchmark where Jev Choice outputs actuator commands.
logicrw/awesome-jev-projects
Frequently asked questions
Can Jev land the plane?
Sometimes; the README reports 14% successful landings across runs.
How does it compare with LLMs?
Jev flew in real time at far lower cost; tested Codex models did not land successfully.
Can I fly it myself?
Yes, human-pilot mode needs no API key.
Related projects
MuJoCo pilot where Jev, Claude Haiku and reactive rules choose pick-and-place actions; Jev matched the rules exactly.
ProjectresearchPythonMuJoCo Franka arm stacking cubes: code proposes and simulates small moves, Jev picks moves, grasp, release and completion, vs Claude.
AppprototypePythonBrowser 2D driving sim where Jev answers lane, speed, hazard and pedestrian-yield questions every 200 ms, with an LLM comparison mode.
ProjectprototypeJavaScriptThree.js box steered by plain-language instructions: an LLM plans steps, Jev decides move, turn and jump in real time.
ProjectprototypeTypeScriptThree.js driving simulator with a fast reflex layer, a TTC safety brake and an optional Jev API strategy layer.
ProjectprototypeJavaScriptThree.js driving simulator where Jev picks among locally sampled steering-and-speed paths while code handles geometry and safety.
ProjectprototypeTypeScript