RocketWM: A framework for building controllable World Models for aerospace flight

Controllable World Models (WMs) of aerospace flight are currently only feasible for those with proprietary data: flight logs that pair video with commands belong to the launch provider, which leaves competitors' vehicles, retired vehicles, and historical launches without a simulator. Public launch broadcasts are openly available but show only what happened, and their on-screen telemetry describes the vehicle's state rather than its commands. A video WM trained on such footage therefore learns appearance and trajectory statistics but not the effect of control commands: its action channel improves reconstruction during training, yet generation remains unresponsive to actions at inference. Methods that turn observation-only video into action-conditioned training data are thus lacking, a gap this work aims to fill. We present RocketWM, a method that generates physics-grounded controls from launch vehicle footage alone. We extract the observed flight dynamics from the on-screen telemetry, initialise an aerospace physics engine with them, and render the resulting trajectories in a game engine whose camera is matched to the broadcast. The resulting physics twin reproduces the recorded flight within the telemetry's own tolerance. Varying the twin's commands yields paired counterfactual episodes with rendered ground-truth video, and mapping its best-fitting commands back onto the real footage produces an action-conditioned launch dataset. We fine-tune two WMs on this dataset to predict several seconds ahead and compare them against conventional telemetry-based conditioning on two flight phases, booster and ship landing burns. The resulting WMs respond to control commands, and their generated trajectories closely follow the real footage. Although demonstrated on aerospace launch vehicles, our framework is designed to be applicable in any domain where video and state are observed but commands are not.