acceptodds
Under review as a conference paper at ICLR 2027

RocketWM: A framework for building controllable World Models for aerospace flight

Abstract

Controllable World Models (WMs) of aerospace flight are currently only feasible for those with proprietary data: flight logs that pair video with commands belong to the launch provider, which leaves competitors' vehicles, retired vehicles, and historical launches without a simulator. Public launch broadcasts are openly available but show only what happened, and their on-screen telemetry describes the vehicle's state rather than its commands. A video WM trained on such footage therefore learns appearance and trajectory statistics but not the effect of control commands: its action channel improves reconstruction during training, yet generation remains unresponsive to actions at inference. Methods that turn observation-only video into action-conditioned training data are thus lacking, a gap this work aims to fill. We present RocketWM, a method that generates physics-grounded controls from launch vehicle footage alone. We extract the observed flight dynamics from the on-screen telemetry, initialise an aerospace physics engine with them, and render the resulting trajectories in a game engine whose camera is matched to the broadcast. The resulting physics twin reproduces the recorded flight within the telemetry's own tolerance. Varying the twin's commands yields paired counterfactual episodes with rendered ground-truth video, and mapping its best-fitting commands back onto the real footage produces an action-conditioned launch dataset. We fine-tune two WMs on this dataset to predict several seconds ahead and compare them against conventional telemetry-based conditioning on two flight phases, booster and ship landing burns. The resulting WMs respond to control commands, and their generated trajectories closely follow the real footage. Although demonstrated on aerospace launch vehicles, our framework is designed to be applicable in any domain where video and state are observed but commands are not.

Primary area:
generative models
Keywords:
World Model, Physical AI, Aerospace, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Gn52OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RocketWM addresses the construction of action-conditioned video world models from public launch broadcasts without access to proprietary control logs. The framework models telemetry delay and quantization, reconstructs flight trajectories, and uses physical dynamics and additional mass constraints to estimate recoverable controls. These estimates label real footage, while a physics twin generates rendered counterfactual episodes for training and evaluation.

The paper also introduces CommandBench, which compares generated responses to paired command interventions with rendered references and several negative controls. The strongest video results use Cosmos 3 Edge: the mixed real/render model follows ignition interventions on 56/64 and 56/61 held-out synthetic upper-stage pairs, versus zero successes for the real-only model. Separate real-video continuation tests show that mixing real footage substantially improves visual fidelity. Quantitative control from real conditioning frames remains unvalidated, and the mixed booster model does not follow the tested interventions.

Critical strengths
  1. The framework explicitly addresses display delays, quantization, thrust–mass ambiguity, external mass constraints, and uncertainty in recovered quantities. Forward re-flight checks and deliberately perturbed controls provide meaningful consistency tests, while unobservable or unvalidated quantities are identified rather than uniformly treated as reliable labels (§3; Appendices B–C). This reconstruction and data-construction work has value for domains where observations are available but control logs are inaccessible.

  2. CommandBench evaluates matched interventions and includes same-command/different-seed, null-action, and state-swap controls. Counting ties and missing readouts in the denominator makes the reported success rates more informative. The real-only and render-only comparisons show that rendered intervention data produces measurable ignition sensitivity, while Tables 2–3 expose the accompanying trade-off with real-video fidelity. This is a consequential methodological contribution to evaluating action-conditioned video models.

Critical weaknesses
  1. The primary readout measures warm-pixel onset or burn duty. It does not establish whether changing a command produces the corresponding change in the generated vehicle’s trajectory, velocity, or attitude. Indeed, the plume-compositing positive control in Appendix F.2 achieves perfect diagnostic scores by adding the rendered plume difference to the model’s factual generation. This is a useful detector check, but it also illustrates that the metric can succeed without validating the vehicle’s dynamical response. The real-conditioned tests cannot close this gap because their judge has not passed validation (Appendix F.5; Figures 10–11). Consequently, the evidence for controllable flight prediction is narrower than the headline framing suggests, even within the evaluated vehicle family.

  2. The successful upper-stage experiments use four engine-command channels whose source is the directly observed engine indicator; upper-stage throttle is reserved, and recovered thrust direction is not the command tested in the main benchmark (Appendix C.3; Table 10). Comparing H with H-real establishes the benefit of adding rendered episodes, but does not separate that benefit from the contribution of the newly inferred control information. The twin clearly supplies useful states and counterfactual trajectories; the unresolved issue is which part of the proposed physical reconstruction is necessary for the downstream gain. This attribution is further complicated by the coexistence of the older search-based landing environment and newer accepted-twin factories, alongside the v4.2/v5.0 dataset distinction. The contribution of command recovery to learned controllability therefore remains insufficiently demonstrated.

Questions for authors
  1. On the existing held-out counterfactual pairs, what evidence shows that SWAP changes the generated vehicle’s motion consistently with the commanded intervention? Are there paired measurements or blinded assessments of position, descent rate, or attitude that distinguish the model’s response from the plume-compositing control in Appendix F.2?

  2. Which inputs or tested interventions in the successful upper-stage experiments require physical inversion rather than direct engine-indicator readings? Is there a matched comparison that separates the benefit of flight-specific physical reconstruction or recovered control labels from the benefit of adding rendered ignition interventions, while holding the training recipe and data budget fixed?

  3. Please map the Table 2 training and evaluation sets to their exact twin implementation, label registry, and acceptance results. In particular, how does Appendix B.5’s statement that the landing-pair environment comes from the search-based twin relate to the accepted-twin description in Appendix F.1 and the newer factory in Appendix D.1? This determines which reconstruction evidence supports the counterfactual data used for the headline results.

2 comments

Thank you for a detailed review that points where our evidence stopped. Each point was addressed by an experiment or an in-place correction. The results are in a new Section 6.3 of the revised PDF (Figure 3), and we highlight changes with blue colour. M0 is the render-only model, H the mixed one, H-real the real-only one.

W1 / Q1. The warm-pixel readout does not establish a change in trajectory, velocity or attitude, and the plume-compositing control passes it. Are there paired measurements or blinded assessments of position, descent rate or attitude that distinguish the model from that control?

You are right: a measure the control can pass cannot show that the vehicle moves. We added a new experiment that measures the hull instead of the plume (Section 6.3, "The vehicle, not only the plume", Figure 3, Appendix G.6, Table 19). Attitude. Yes. The hull turns in the commanded direction on 64 of 64 pairs for M0 and 58 of 64 for H, on none for H-real. The turn grows with the commanded shift, including three sizes never seen in training (rank correlation 0.96 and 0.87, 123 pairs). On the twin's own renders the measure tracks the true attitude change at 0.95. We read direction and ordering, not magnitude. The plume-compositing control. It fails this measure: 9 of 64 pairs, and does not change across shift sizes. Position and descent rate. Not measured. The cameras follow the vehicle, so neither can be read, while Section 8 states this limitation. Blinded assessment: not of vehicle motion. The onset judge is now validated on real conditioning frames (0.913 against two blind raters, Appendix F), and with it we measure no command effect on real footage.

W2 / Q2. Which inputs require physical inversion, and is there a matched comparison that separates the reconstruction or the recovered labels from the benefit of adding rendered ignition interventions?

You are right that the submission did not separate them. Result R2 now claims only the commanded ignition.

Inputs that require the inversion: none of the tested upper-stage commands. The inversion supplies the labels on the real footage and the acceptance test of Table 1 (Sections 3 and 4). Ascent throttle, the one tested command only the inversion supplies, is not followed in a new test (2 of 10 pairs on Sections 6.3 and 8). Matched comparison. Three new experiments at the same recipe and budget (Section 6.3, Appendix G.6, Table 19). With the commands shuffled across the same renders, following disappears (0 of 64 pairs for M0, 1 of 64 for H): the gain comes from the pairing of command and video. With 40 test pairs rebuilt on the inversion, following is intact (M0 40 of 40, H 30 of 40). For the labels themselves, H-disp is the mixed model H-b (the mixture retrained on the booster dataset) with the real clips' commands as the broadcast displays them instead of the inversion's. It follows the upper-stage ignition exactly as H-b does (63 of 64 pairs). Its video matches the counterfactual on fewer pairs (48 against 57 of 64) and its hull turns on fewer (55 against 62 of 64), while on the booster it follows on more (40 against 33 of 40). So the recovered labels do not change whether the upper-stage ignition is followed. What they change is how closely the rest of the video follows. Each is one training run, and the two training sets differ slightly in how often real engine-on windows are drawn.

Q3. Please map the Table 2 sets to their twin implementation, label registry and acceptance results. How does B.5 relate to F.1 and D.1?

Appendix B.5 is right. F.1 and D.1 were wrong, and we thank you for the catch. A new Appendix D.2 states the two layers: the inversion reconstructs, labels and is what Table 1 accepts, the search-based twin flies the pairs. Its Table 14 maps every row of Table 2 to its labels, registry, renders, test pairs and the check each has passed.

Two new results support the headline. The inversion's Flight 13 landing burn is accepted once scored to engine cut-off (0.94 m/s rms, Appendix B.6), and the 40 pairs rebuilt on it give the same following. The Flight 8 booster environment does not pass its re-flight check, so booster results are read as ignition timing only.

Thank you for the response and additional experiments. The new attitude results substantially address my concern that the model might only change the plume. The benefit of the recovered labels remains less clear, but I appreciate the clarification and narrower claims. Overall, I will increase my rating from 2 to 3.

Reviewer_kr4mOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies controllable world models for rocket flight, where the obstacle is data: public broadcasts show state but never commands, so a video world model trained on them is unresponsive to actions at inference. RocketWM recovers the missing commands from physics: a six-degree-of-freedom twin is built from the on-screen telemetry, validated by re-flying it against the broadcast, and rendered with broadcast-matched cameras, with limits of identifiability stated. Re-flying it under altered commands yields counterfactual pairs that both train the model and form CommandBench, the held-out set on which control is measured. Post-trained on it, the world model follows commanded ignitions on held-out pairs where footage-only training follows on none.

Critical strengths
  1. The paper's real contribution is a recipe rather than a rocket model: where video and state are observed but commands are not, invert the governing equations to recover the commands, then re-fly the same model to synthesise the counterfactuals that observation-only data can never supply. That converts an unlearnable problem into a supervised one, and the recipe is explicitly domain-agnostic.

  2. This work builds CommandBench around counterfactual pairs with second-seed, no-action and state-swap controls, so "responds to commands" becomes a measurable claim rather than an impression.

Critical weaknesses
  1. The physics twin is both the labeler and the grader: recovered commands are never checked against true commands (unavailable by construction), counterfactual pairs are rendered by the same twin, and command following is scored on those renders. The result therefore reads as simulator distillation rather than control recovered from footage, which is still a contribution, but narrower than the claims.

  2. Only 104 of 5,061 real clips contain an engine-command change, so the command signal comes almost entirely from rendered counterfactuals. The claim of controls generated "from launch vehicle footage alone" is therefore overstated: the footage constrains the twin, but the control signal itself is simulated.

Questions for authors
  1. All command-following pairs are twin renders. Among the 104 real clips that contain a command change, could you hold some out and test whether generated ignition timing moves with the actual command, so control is shown on real footage and not only inside the simulator?

  2. Your post-training budget is five orders of magnitude below published recipes. Would the real-only variant be able to correctly follow commands if trained to a comparable scale?

1 comment

Thank you for reading the paper as a recipe, and for two questions that sparked our interest and ran additional experiments to answer. The results are in a new Section 6.3 of the revised PDF (Figure 3), while the changes are highlighted in blue.

W1. The physics twin is both the labeler and the grader. The result reads as simulator distillation rather than control recovered from footage.

You are correct, and this is a fair description of where the control signal comes from. Section 8 now states it: "the commands a model learns come from those renders, and the footage constrains the twin that produces them." Your comment initiated two new experiments (Section 6.3). With the commands shuffled across the same rendered videos, following disappears (0 of 64 pairs for the render-only model, 1 of 64 for the mixed one): the model learns the link between command and video, not the look of the renders. With 40 test pairs rebuilt on our physics-based reconstruction instead of the twin that rendered the benchmark, following stays (40 of 40 render-only, 30 of 40 mixed). Both reconstructions share the vehicle tables, so this is not a test across independent simulators (Appendix D.2).

W2. Only 104 of 5,061 real clips contain an engine-command change. "From launch vehicle footage alone" is overstated. You are correct that this was overstated. We updated the abstract and the conclusion to describe the contribution accurately: the footage constrains the twin, and the twin supplies the interventions.

Q1. Could you hold out some of the 104 real clips and test whether generated ignition timing moves with the actual command?

A good test that we had not run. We added it as a new experiment (Section 6.3; Appendix G.6, Table 21). On 24 held-out real clips we generate under the recorded command and under the same command shifted by 1.5 s, and ask which is closer to the real footage. On real footage the generated ignition does not move with the command: the recorded command is closer on 8 of 19 booster clips (10 of 19 for a re-seed) and 3 of 5 upper-stage clips (3 of 5 for a re-seed). Section 8 now states that control is shown on rendered counterfactuals and not on real footage.

Q2. Your post-training budget is five orders of magnitude below published recipes. Would the real-only variant follow if trained to a comparable scale?

We have done exactly that, run further experiments with five (30k) and ten (60k) the original post-training budget. What we could test is in a new experiment (Section 6.3; Appendix G.6, Table 19 and Figure 13):

  • At 30k and 60k iterations, five and ten times the submitted budget, the real-only variant still does not follow (33 of 64 pairs at both, chance level).
  • This rules out a too-small submitted budget as the explanation, but not a benefit from far more training (Section 8).
  • More iterations may not suffice: the real footage holds few engine events, “flight-specific artefacts” e.g. orbital flight phases, (black of space), booster returns inside clouds, payload bay views. In contrast, these artefacts do not exist in the rendered scenarios. Longer training repeats them without adding variety (Appendix C.4).
  • Better sampling and curation of these events is future work.
Reviewer_b4bsOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to build controllable video world models for aerospace flight using public videos without recorded control commands. It proposes RocketWM, a framework that reconstructs flight trajectories, recovers control commands using physics-based inverse dynamics, and generates counterfactual videos for training action-conditioned world models. Experiments on Starship flight data show that the proposed approach improves command following on upper-stage landing tasks while maintaining realistic video generation, although the improvements are less consistent on booster landing tasks.

Critical strengths

S1. The paper studies an interesting and relatively novel problem. Building controllable world models from public rocket videos without access to control commands is challenging, and I think this is a meaningful research direction.

S2. The authors have clearly put considerable effort and resources into this work. I appreciate the work involved in building the physics simulator, collecting and processing the data, and training the world models.

S3. The experiments are reasonably solid, especially considering the difficulty of conducting experiments in this setting. I appreciate the effort the authors put into validating their approach.

Critical weaknesses

W1. The results are not fully consistent across different flight tasks. While the proposed method works well on upper-stage landing burns, the real-and-render mixed model fails to follow commands on the booster landing task, even though the render-only model performs well. I am not sure what causes this difference or how much it limits the effectiveness of the proposed approach.

W2. The evaluation mainly focuses on whether the generated videos follow ignition commands, particularly the timing of visible engine ignition. I think this is a reasonable evaluation given the difficulty of the setting, but I am not sure how much it demonstrates that the world model has learned the underlying flight dynamics beyond these visible effects.

W3. One important contribution of the paper is recovering unknown control commands, such as throttle and thrust direction, using physics-based inverse dynamics. However, the main world model experiments focus on engine ignition, which can already be observed from the broadcast's engine indicator. I am not sure how much of the improvement comes from the proposed command recovery method and how much comes from the simulated counterfactual training data.

Questions for authors

Q1. Why does the real-and-render mixed model perform well on upper-stage landing burns but fail to follow commands on the booster landing task, while the render-only model succeeds? What do you think causes this difference?

Q2. CommandBench mainly evaluates visible engine ignition. How much do these results tell us about whether the model has learned the underlying flight dynamics, rather than just changing the visual appearance of the engine plume?

Q3. Since the engine ignition commands used in the main experiments can already be observed from the broadcast, how much does the physics-based command recovery contribute to the final world model results? Could you clarify the respective roles of command recovery and simulated counterfactual training data?

2 comments

Thank you for three questions that each became an experiment. The results are in a new Section 6.3 of the revised PDF (Figure 3), while we highlight changes in blue.

W1 / Q1. The mixed model fails to follow commands on the booster landing task, while the render-only model succeeds. What causes this, and how much does it limit the approach?

You are right that the submission left this unexplained. We ran three new experiments (Section 6.3, booster paragraph, Appendix G.6, Table 19 and Figure 13). Cause. The share of booster renders and the training budget. We tested on 40 held-out booster pairs (the 10 of Table 2 and 30 drawn the same way). With render shares matched, the mixed model follows the booster ignition on 33 of 40 at the submitted budget, but re-seeding the same command also moves 13. Trained five times longer (30k iterations), it follows on all 40, and re-seeding moves only 1. The real booster labels are not the cause: withholding them lowers the result to 24 of 40 (Table 19). Limit. At the submitted budget the mixture does not reliably control the booster; at five times that budget it does. The booster environment does not pass our re-flight check, so this is a claim about ignition timing only (Appendix D.2).

W2 / Q2. CommandBench mainly evaluates visible engine ignition. Has the model learned flight dynamics, or does it only change the appearance of the plume?

A fair concern: our measure looked only at the exhaust. We added a new experiment that measures the orientation of the vehicle body instead (Section 6.3, "The vehicle, not only the plume"; Figure 3, right). The body turns in the commanded direction on 64 of 64 pairs for the render-only model and 58 of 64 for the mixed model, on none for the real-only model, and on 9 of 64 when the plume is simply pasted onto the video. The turn grows with the size of the commanded shift, including three sizes never seen in training. So the vehicle's attitude responds, not only the plume. Position and descent rate cannot be read because the cameras follow the vehicle, and we honestly state this limitation on Section 8.

W3 / Q3. Ignition can already be observed from the broadcast's engine indicator. How much comes from the command recovery and how much from the counterfactual data?

You are right that the submission did not separate the two. Three new experiments do (Section 6.3; Appendix G.6, Table 19). Counterfactual data. With the commands shuffled across the same rendered videos, following disappears (0 of 64 pairs for the render-only model, 1 of 64 for the mixed one). The model learns the link between command and video, which only the counterfactual pairs provide. For ignition, this data carries the effect, and on the Results, R2 now claims only that. Command recovery. It supplies the labels on the real footage and the acceptance test of Table 1. With 40 test pairs rebuilt on the recovered flight, following is intact (40 of 40 render-only, 30 of 40 mixed; Appendix D.2). We also compared the labels directly: a mixed model whose real clips carry the commands as the broadcast displays them, same recipe and budget, follows the upper-stage ignition just as well (63 of 64 pairs for both). Its video matches the counterfactual on fewer pairs (48 against 57 of 64), and on the booster it follows on more (40 against 33 of 40). So the recovered labels do not decide whether the upper-stage ignition is followed. Throttle, the one tested command only the recovery supplies, is not followed (2 of 10 pairs at 6k, 4 of 10 at 30k). Sections 6.3 and 8 say so.

Thank you for the detailed rebuttal and the additional experiments. I appreciate the effort to address all three of my questions.

The new booster experiments help explain the earlier inconsistency, and the vehicle orientation results provide stronger evidence of command following beyond visual ignition effects. I also appreciate the clarification of the respective contributions of command recovery and counterfactual training, along with the discussion of the remaining limitations.

Overall, I think the rebuttal has substantially addressed my main concerns. While some limitations remain, I am more positive about the paper and will increase my rating from 3 to 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.