- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a design choice in history-conditioned dynamics models used for planning. During an imagined rollout, the model can either keep the context inferred from observed history fixed ("Frozen": the GRUCell receives m₀ as hidden input at every step) or carry its own recurrent state forward ("Updated": m_{h+1} = c_h). Using small (~40k-parameter) GRU models trained on Neural-Fly real-flight logs and on five simulated quadrotor datasets, the authors show four main results:
- Without LayerNorm, Updated has lower logged-action prediction error but worse MPPI tracking at nonzero winds (e.g., 0.323 vs 0.210 m RMSE at 6.1 m/s).
- Replacing logged future actions with MPPI candidate actions, at fixed histories and states, reverses the prediction ranking.
- Training recipe (Dense/Repeated/Re-simulated/Mixed with candidate actions) and optimization budget can flip which design tracks better. Long-budget Mixed training helps Updated but hurts Frozen, and this is confirmed on five new datasets.
- Memory-input LayerNorm penalizes Updated more than Frozen in prediction.
Statistics use paired intervals across training datasets and Holm-corrected tests for fixed-log contrasts, and all arms and episodes are retained.
Critical strengths
- Exceptionally careful and transparent experimental methodology. Matched parameter counts and initialization, paired comparisons across independently generated training datasets, retention of every arm and episode, failure-aware metrics, explicit separation of prospective confirmation (Table 2, five new datasets) from result-informed follow-ups, and a released package of 450 model exports. This level of rigor is rare in model-based control papers and makes the reported effects credible within their scope.
- A concrete, actionable cautionary result. The action-source test (Sec. 5) cleanly shows that logged-action prediction error can rank models opposite to how they rank on the planner's own queries, holding history, initial state and noise fixed. The practical recommendation in Sec. 8 is sensible and broadly relevant to anyone selecting world models by validation loss: evaluate on planner candidate actions and in closed loop.
Critical weaknesses
- The central insight is largely known, and the new findings are narrow and conditional. The fact that models selected by one-step or logged prediction can control worse is the objective-mismatch phenomenon (Lambert et al., 2020, cited). The fact that planners query a learned model on action sequences outside the training distribution, where errors are exploited, is a well-known issue in model-based RL. The specific new contribution is that a recurrent rollout state ("Updated") is more fragile to planner-candidate actions than a fixed-context Markov transition ("Frozen"). But this is then shown to reverse under a different training recipe and budget (Table 1: Mixed-L Updated 0.112 m vs Frozen 0.263 m). The conclusion is therefore "it depends on actions, recipe, and budget", without a mechanism (the authors state that they "do not identify representation geometry, gating or optimization as the cause"). An ICLR reader gets a careful case study but limited understanding of why. The unexplained finding that candidate exposure hurts Frozen at the long budget is a good example: it is reported but not analyzed.
- Scope is too narrow to support general claims about world-model memory design. All controlled results use one simulator family, ~40k-parameter GRU/RNN decoders, a 12-channel proprioceptive state, and one MPPI configuration with 64 candidates. There are no latent world models from observations (RSSM/Dreamer-style, transformer or SSM world models), even though the introduction and title frame the question as one about world models' memory. The learned controllers are also weak in absolute terms. All learned variants lose to nominal-physics MPPI at zero wind, all are far from true-mean-wind MPPI (0.067 m vs the best learned 0.112 m at 6.1 m/s), and planning takes 24 ms against a 20 ms control period. Real-flight data provide only prediction evidence. It is therefore unclear whether the reversal persists for models that are actually competitive, or for the architectures where this design choice is commonly made.
Questions for authors
- Can you offer a mechanistic account of why Updated degrades on candidate actions? For example: does the hidden state c_h leave the training-time state distribution under perturbed action sequences (measured via norms or density of c_h), and does this explain the step-1 angular-rate gap you observe?
- Why does Mixed (candidate-action) training hurt Frozen at the long budget while helping Updated? Is this an optimization or capacity effect, given that Frozen's per-step function has to fit a more diverse target distribution?
- Do the qualitative conclusions hold for (a) a larger model, (b) a latent world model trained from observations (e.g., an RSSM with deterministic state carry vs. fixed context), or (c) a different planner such as CEM, or MPPI with more candidates? Even one such replication would substantially increase generality.
- Many follow-up studies were motivated by inspected OOD results. Which findings, other than the Table 2 confirmation, would you consider confirmatory rather than exploratory?
Friendly Reminder
Thank you again for recognizing the controlled action-source test and the transparency of our methodology. In response to your request for evidence beyond the original planner, MPPI with 256 candidates and CEM both preserve the opposing training effects at both primary winds, with all eight paired effect intervals excluding zero (Section 8 and Appendix S).
We also tested larger models, memory-input LayerNorm, and matched MLP baselines, and added budget-controlled data collection (Sections 7–8; Appendices S–T). The architecture tests clarify the limits of the effect: larger capacity and LayerNorm remove the resolved Frozen deterioration. New fixed-weight diagnostics locate opposing training responses in velocity prediction at specific later queries and test their sensitivity to predicted-state feedback (Appendix W, Figure 5). We distinguish these reviewer-informed extensions from the prospective confirmation in Table 2 and do not claim a universal memory rule or an established hidden-state mechanism.
As the discussion draws to a close, we would appreciate your assessment of how this evidence affects your concerns about contribution and scope, and which concern, if any, remains decisive. Thank you for your time and detailed feedback.