- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RocketWM addresses the construction of action-conditioned video world models from public launch broadcasts without access to proprietary control logs. The framework models telemetry delay and quantization, reconstructs flight trajectories, and uses physical dynamics and additional mass constraints to estimate recoverable controls. These estimates label real footage, while a physics twin generates rendered counterfactual episodes for training and evaluation.
The paper also introduces CommandBench, which compares generated responses to paired command interventions with rendered references and several negative controls. The strongest video results use Cosmos 3 Edge: the mixed real/render model follows ignition interventions on 56/64 and 56/61 held-out synthetic upper-stage pairs, versus zero successes for the real-only model. Separate real-video continuation tests show that mixing real footage substantially improves visual fidelity. Quantitative control from real conditioning frames remains unvalidated, and the mixed booster model does not follow the tested interventions.
Critical strengths
-
The framework explicitly addresses display delays, quantization, thrust–mass ambiguity, external mass constraints, and uncertainty in recovered quantities. Forward re-flight checks and deliberately perturbed controls provide meaningful consistency tests, while unobservable or unvalidated quantities are identified rather than uniformly treated as reliable labels (§3; Appendices B–C). This reconstruction and data-construction work has value for domains where observations are available but control logs are inaccessible.
-
CommandBench evaluates matched interventions and includes same-command/different-seed, null-action, and state-swap controls. Counting ties and missing readouts in the denominator makes the reported success rates more informative. The real-only and render-only comparisons show that rendered intervention data produces measurable ignition sensitivity, while Tables 2–3 expose the accompanying trade-off with real-video fidelity. This is a consequential methodological contribution to evaluating action-conditioned video models.
Critical weaknesses
-
The primary readout measures warm-pixel onset or burn duty. It does not establish whether changing a command produces the corresponding change in the generated vehicle’s trajectory, velocity, or attitude. Indeed, the plume-compositing positive control in Appendix F.2 achieves perfect diagnostic scores by adding the rendered plume difference to the model’s factual generation. This is a useful detector check, but it also illustrates that the metric can succeed without validating the vehicle’s dynamical response. The real-conditioned tests cannot close this gap because their judge has not passed validation (Appendix F.5; Figures 10–11). Consequently, the evidence for controllable flight prediction is narrower than the headline framing suggests, even within the evaluated vehicle family.
-
The successful upper-stage experiments use four engine-command channels whose source is the directly observed engine indicator; upper-stage throttle is reserved, and recovered thrust direction is not the command tested in the main benchmark (Appendix C.3; Table 10). Comparing H with H-real establishes the benefit of adding rendered episodes, but does not separate that benefit from the contribution of the newly inferred control information. The twin clearly supplies useful states and counterfactual trajectories; the unresolved issue is which part of the proposed physical reconstruction is necessary for the downstream gain. This attribution is further complicated by the coexistence of the older search-based landing environment and newer accepted-twin factories, alongside the v4.2/v5.0 dataset distinction. The contribution of command recovery to learned controllability therefore remains insufficiently demonstrated.
Questions for authors
-
On the existing held-out counterfactual pairs, what evidence shows that SWAP changes the generated vehicle’s motion consistently with the commanded intervention? Are there paired measurements or blinded assessments of position, descent rate, or attitude that distinguish the model’s response from the plume-compositing control in Appendix F.2?
-
Which inputs or tested interventions in the successful upper-stage experiments require physical inversion rather than direct engine-indicator readings? Is there a matched comparison that separates the benefit of flight-specific physical reconstruction or recovered control labels from the benefit of adding rendered ignition interventions, while holding the training recipe and data budget fixed?
-
Please map the Table 2 training and evaluation sets to their exact twin implementation, label registry, and acceptance results. In particular, how does Appendix B.5’s statement that the landing-pair environment comes from the search-based twin relate to the accepted-twin description in Appendix F.1 and the newer factory in Appendix D.1? This determines which reconstruction evidence supports the counterfactual data used for the headline results.
2 comments
Thank you for the response and additional experiments. The new attitude results substantially address my concern that the model might only change the plume. The benefit of the recovered labels remains less clear, but I appreciate the clarification and narrower claims. Overall, I will increase my rating from 2 to 3.
Thank you for a detailed review that points where our evidence stopped. Each point was addressed by an experiment or an in-place correction. The results are in a new Section 6.3 of the revised PDF (Figure 3), and we highlight changes with blue colour. M0 is the render-only model, H the mixed one, H-real the real-only one.
You are right: a measure the control can pass cannot show that the vehicle moves. We added a new experiment that measures the hull instead of the plume (Section 6.3, "The vehicle, not only the plume", Figure 3, Appendix G.6, Table 19). Attitude. Yes. The hull turns in the commanded direction on 64 of 64 pairs for M0 and 58 of 64 for H, on none for H-real. The turn grows with the commanded shift, including three sizes never seen in training (rank correlation 0.96 and 0.87, 123 pairs). On the twin's own renders the measure tracks the true attitude change at 0.95. We read direction and ordering, not magnitude. The plume-compositing control. It fails this measure: 9 of 64 pairs, and does not change across shift sizes. Position and descent rate. Not measured. The cameras follow the vehicle, so neither can be read, while Section 8 states this limitation. Blinded assessment: not of vehicle motion. The onset judge is now validated on real conditioning frames (0.913 against two blind raters, Appendix F), and with it we measure no command effect on real footage.
You are right that the submission did not separate them. Result R2 now claims only the commanded ignition.
Inputs that require the inversion: none of the tested upper-stage commands. The inversion supplies the labels on the real footage and the acceptance test of Table 1 (Sections 3 and 4). Ascent throttle, the one tested command only the inversion supplies, is not followed in a new test (2 of 10 pairs on Sections 6.3 and 8). Matched comparison. Three new experiments at the same recipe and budget (Section 6.3, Appendix G.6, Table 19). With the commands shuffled across the same renders, following disappears (0 of 64 pairs for M0, 1 of 64 for H): the gain comes from the pairing of command and video. With 40 test pairs rebuilt on the inversion, following is intact (M0 40 of 40, H 30 of 40). For the labels themselves, H-disp is the mixed model H-b (the mixture retrained on the booster dataset) with the real clips' commands as the broadcast displays them instead of the inversion's. It follows the upper-stage ignition exactly as H-b does (63 of 64 pairs). Its video matches the counterfactual on fewer pairs (48 against 57 of 64) and its hull turns on fewer (55 against 62 of 64), while on the booster it follows on more (40 against 33 of 40). So the recovered labels do not change whether the upper-stage ignition is followed. What they change is how closely the rest of the video follows. Each is one training run, and the two training sets differ slightly in how often real engine-on windows are drawn.
Appendix B.5 is right. F.1 and D.1 were wrong, and we thank you for the catch. A new Appendix D.2 states the two layers: the inversion reconstructs, labels and is what Table 1 accepts, the search-based twin flies the pairs. Its Table 14 maps every row of Table 2 to its labels, registry, renders, test pairs and the check each has passed.
Two new results support the headline. The inversion's Flight 13 landing burn is accepted once scored to engine cut-off (0.94 m/s rms, Appendix B.6), and the 40 pairs rebuilt on it give the same following. The Flight 8 booster environment does not pass its re-flight check, so booster results are read as ignition timing only.