acceptodds
Under review as a conference paper at ICLR 2027

When Memory Updates Help Prediction but Hurt Control: A Flight Dynamics Study

Abstract

Robots use recent observations to predict how they will respond to future actions. During planning, however, imagined actions supply no new observations. Should a dynamics model keep the memory inferred from observed history fixed, or update it along its own predicted trajectory? We study this question in quadrotor simulation using matched, separately trained Frozen and Updated memory designs under a common planner. With the original logged-training recipe and no layer normalization (LN), Updated improves logged prediction, including physical position, yet produces worse closed-loop tracking. Replacing logged actions with planner candidates reverses the mean prediction ranking even when histories and initial physical states are held fixed. We further show that training recipe and optimization budget can reverse which design controls better. At the long budget, Mixed training with candidate actions improves Updated’s tracking but worsens Frozen’s relative to Re-simulated training with logged actions. A prospective test on five new datasets confirms these opposing effects at both extrapolation winds. These findings show that memory design, training recipe and optimization bud- get jointly shape the link between prediction and control. They motivate assessing robot dynamics models on logged prediction, planner queries and closed-loop control before choosing how memory should evolve during planning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, Model-Based Control, Recurrent Dynamics Models, Memory Representation, Planning with Learned Models, Quadrotor Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_qFKJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a design choice in history-conditioned dynamics models used for planning. During an imagined rollout, the model can either keep the context inferred from observed history fixed ("Frozen": the GRUCell receives m₀ as hidden input at every step) or carry its own recurrent state forward ("Updated": m_{h+1} = c_h). Using small (~40k-parameter) GRU models trained on Neural-Fly real-flight logs and on five simulated quadrotor datasets, the authors show four main results:

  1. Without LayerNorm, Updated has lower logged-action prediction error but worse MPPI tracking at nonzero winds (e.g., 0.323 vs 0.210 m RMSE at 6.1 m/s).
  2. Replacing logged future actions with MPPI candidate actions, at fixed histories and states, reverses the prediction ranking.
  3. Training recipe (Dense/Repeated/Re-simulated/Mixed with candidate actions) and optimization budget can flip which design tracks better. Long-budget Mixed training helps Updated but hurts Frozen, and this is confirmed on five new datasets.
  4. Memory-input LayerNorm penalizes Updated more than Frozen in prediction.

Statistics use paired intervals across training datasets and Holm-corrected tests for fixed-log contrasts, and all arms and episodes are retained.

Critical strengths
  1. Exceptionally careful and transparent experimental methodology. Matched parameter counts and initialization, paired comparisons across independently generated training datasets, retention of every arm and episode, failure-aware metrics, explicit separation of prospective confirmation (Table 2, five new datasets) from result-informed follow-ups, and a released package of 450 model exports. This level of rigor is rare in model-based control papers and makes the reported effects credible within their scope.
  2. A concrete, actionable cautionary result. The action-source test (Sec. 5) cleanly shows that logged-action prediction error can rank models opposite to how they rank on the planner's own queries, holding history, initial state and noise fixed. The practical recommendation in Sec. 8 is sensible and broadly relevant to anyone selecting world models by validation loss: evaluate on planner candidate actions and in closed loop.
Critical weaknesses
  1. The central insight is largely known, and the new findings are narrow and conditional. The fact that models selected by one-step or logged prediction can control worse is the objective-mismatch phenomenon (Lambert et al., 2020, cited). The fact that planners query a learned model on action sequences outside the training distribution, where errors are exploited, is a well-known issue in model-based RL. The specific new contribution is that a recurrent rollout state ("Updated") is more fragile to planner-candidate actions than a fixed-context Markov transition ("Frozen"). But this is then shown to reverse under a different training recipe and budget (Table 1: Mixed-L Updated 0.112 m vs Frozen 0.263 m). The conclusion is therefore "it depends on actions, recipe, and budget", without a mechanism (the authors state that they "do not identify representation geometry, gating or optimization as the cause"). An ICLR reader gets a careful case study but limited understanding of why. The unexplained finding that candidate exposure hurts Frozen at the long budget is a good example: it is reported but not analyzed.
  2. Scope is too narrow to support general claims about world-model memory design. All controlled results use one simulator family, ~40k-parameter GRU/RNN decoders, a 12-channel proprioceptive state, and one MPPI configuration with 64 candidates. There are no latent world models from observations (RSSM/Dreamer-style, transformer or SSM world models), even though the introduction and title frame the question as one about world models' memory. The learned controllers are also weak in absolute terms. All learned variants lose to nominal-physics MPPI at zero wind, all are far from true-mean-wind MPPI (0.067 m vs the best learned 0.112 m at 6.1 m/s), and planning takes 24 ms against a 20 ms control period. Real-flight data provide only prediction evidence. It is therefore unclear whether the reversal persists for models that are actually competitive, or for the architectures where this design choice is commonly made.
Questions for authors
  1. Can you offer a mechanistic account of why Updated degrades on candidate actions? For example: does the hidden state c_h leave the training-time state distribution under perturbed action sequences (measured via norms or density of c_h), and does this explain the step-1 angular-rate gap you observe?
  2. Why does Mixed (candidate-action) training hurt Frozen at the long budget while helping Updated? Is this an optimization or capacity effect, given that Frozen's per-step function has to fit a more diverse target distribution?
  3. Do the qualitative conclusions hold for (a) a larger model, (b) a latent world model trained from observations (e.g., an RSSM with deterministic state carry vs. fixed context), or (c) a different planner such as CEM, or MPPI with more candidates? Even one such replication would substantially increase generality.
  4. Many follow-up studies were motivated by inspected OOD results. Which findings, other than the Table 2 confirmation, would you consider confirmatory rather than exploratory?
1 comment
AuthorsOpenReview ↗

Friendly Reminder

Thank you again for recognizing the controlled action-source test and the transparency of our methodology. In response to your request for evidence beyond the original planner, MPPI with 256 candidates and CEM both preserve the opposing training effects at both primary winds, with all eight paired effect intervals excluding zero (Section 8 and Appendix S).

We also tested larger models, memory-input LayerNorm, and matched MLP baselines, and added budget-controlled data collection (Sections 7–8; Appendices S–T). The architecture tests clarify the limits of the effect: larger capacity and LayerNorm remove the resolved Frozen deterioration. New fixed-weight diagnostics locate opposing training responses in velocity prediction at specific later queries and test their sensitivity to predicted-state feedback (Appendix W, Figure 5). We distinguish these reviewer-informed extensions from the prospective confirmation in Table 2 and do not claim a universal memory rule or an established hidden-state mechanism.

As the discussion draws to a close, we would appreciate your assessment of how this evidence affects your concerns about contribution and scope, and which concern, if any, remains decisive. Thank you for your time and detailed feedback.

Reviewer_apBNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper does not propose a particularly novel method. Instead, it investigates whether models perform better when planning future actions by keeping their original memory fixed or by updating that memory as they predict future states. The paper compares two main approaches: Frozen, which updates the predicted future state while keeping memory fixed, and Updated, which updates both the predicted future state and memory.

It presents five main experiments in sequence. The first examines how the two approaches affect future-state prediction accuracy. The second investigates whether more accurate future-state predictions lead to better performance when the models are used with a controller. The third examines whether models trained only on the original data maintain their prediction accuracy when predicting future states resulting from controller-generated actions. The fourth investigates how including controller-generated actions in training affects the state prediction accuracy of the two approaches. The fifth evaluates their performance when used with a controller after this additional training.

The final conclusion is that, among the models trained in this study, the Updated approach trained jointly on controller-generated actions and the original recorded actions achieves the best performance in both state prediction accuracy and control.

Critical strengths
  1. Extensive ablation studies: The paper examines a broad range of architectural choices and training settings.
  2. Good support for reproducibility: The authors repeat key experiments on newly generated datasets and provide code and model artifacts to facilitate reproduction.
Critical weaknesses
  1. The experimental presentation lacks a clear focus. Experiment 1 offers little practical value for the intended control application, since it does not reflect the conditions under which the models will be used. The evaluation could instead begin with Experiment 3 to directly assess state prediction generalization across action distributions. The presentation feels more like a sequence of accumulated experiments than a well-organized investigation.

  2. The findings offer limited insight. The main finding seems consistent with an intuitive expectation: additional training on controller-generated actions improves generalization and consequently improves state prediction and control performance. In contrast, Frozen’s improved state prediction accuracy accompanied by worse control performance is less intuitive and arguably more interesting. The paper does not explain why this occurs.

Questions for authors
  1. Please address the concerns raised in the Weaknesses section.

  2. Does adding memory-input LayerNorm change the ranking of Frozen and Updated under Mixed training? Specifically, does Mixed-trained Updated still outperform Mixed-trained Frozen in both state prediction accuracy and closed-loop control performance, or could either ranking reverse?

  3. Do the conclusions obtained with a fixed controller generalize to other controllers? Would the same conclusions hold if the controller were also trained?

1 comment

Thank you for the substantive follow-up and for updating your assessment. Your distinction between showing when the effect appears and explaining why it occurs directly motivated the additional component-wise and physical-state-feedback diagnostics in Appendix W. We appreciate your careful consideration of this evidence and the revised presentation. Your feedback helped us make the contribution more specific and distinguish supported findings from unresolved mechanisms. We remain available for any final questions.

Reviewer_Pu2COpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper tries to answer the central question of whether a dynamics model in a planner will perform better under two designs. In the first design, what the authors refer to as "Frozen", they use the actual observation from the environment for each inference, i.e. no imaginary state updates are used as intermediate inputs during planning. In the second design, what the authors refer to as "Updated", they use imaginary states in rollout as inputs. Neither design is hypothesized to outperform the other. Logical arguments can be made that either design can perform well. The authors then try a rather comprehensive set of experiment parameters, training regimens, and optimization budgets for both prediction and control on a quadrotor simulation domain. The results vary depending on the aforementioned circumstances.

Critical strengths
  1. The experiments are thorough, and plenty of details are provided in the substantial appendix. Data, model weights, and scripts that help reproduce the experiments are provided. My only complaint here would be that it's too thorough. Careful selection of relevant results to a clear finding/conclusion would have been highly appreciated. But I also understand that transparency and exceptional effort to reproducibility should not be criticized. Many of the less relevant results can be left in the appendix, while the main text should focus on interesting findings and patterns that can be generalized for the model-based learning community.
Critical weaknesses
  1. The paper reads like a laundry list of results, with no clear direction or narrative. The experiment results are reported as is: Frozen performs better than Updated in some contexts, while the reverse is true in others. What are the potential patterns or mechanisms behind these results? It makes for a very difficult read when no argument is made, other than "your mileage may vary". Table 1 lists all eight settings for training and their results, with nothing bolded for emphasis. I'm not sure what I'm supposed to be looking at. I understand that those were the results, and commend the authors for reporting them faithfully, but am not sure what I am supposed to learn from them.

  2. The writing style relies heavily on jargon (many of which aren't even widely accepted word choices) or is otherwise unclear. Take the key contributions in the introduction for example. I'm not sure what "we establish a paired reversal" means. It would be much clearer to simply state that "Updated" makes more accurate predictions, while "Frozen" performs better on control tasks. The section, subsection, and subfigure titles also are quite opaque. Instead of "One observed history, two memory designs" in Fig 1A, why not just a title like "Dynamics model with Frozen and Updated memory schemes"? The same applies for subsection 4.1 and Fig 2A. It feels like these titles belong on slides rather than paper sections. For another example, it would be much simpler and clearer to state in Section 8 that "the results imply that further investigation could be benefitted from a practical evaluation procedure", and that "the reader should treat observations fresh..." etc. with a subject in each sentence.

Questions for authors
  1. In Section 4.3, the authors state that "at zero wind, every learned variant loses to nominal-physics MPPI in every dataset". How meaningful are the results, with regards to freezing or updating memory, when the GRU-based dynamics model fails to outperform the baseline?
1 comment
AuthorsOpenReview ↗

Friendly Reminder

Thank you again for recognizing the thorough experiments and reproducibility materials. We took your concern about the narrative seriously and reorganized the introduction and Sections 5–7 around one specific finding: at the original long training budget, the same candidate-action training change helps Updated but hurts Frozen, with independent confirmation on five new training datasets (Table 2).

The new budget-controlled collection experiment then tests a practical intervention: collecting controller-visited data improves both designs beyond continued optimization (Section 7, Table 3). We also addressed your baseline question: on matched evaluation episodes at 6.1/8.5 m/s wind, iterative Updated achieves 0.062/0.124 m tracking RMSE versus 0.310/0.453 m for nominal-physics MPPI; the zero-wind comparison is reported separately in Appendix T.

Could you let us know whether the revised narrative and these results address your concerns about the paper's informativeness and practical relevance? If a central concern remains, a brief indication would help us respond directly before the discussion closes. Thank you for your consideration.

Reviewer_xAXYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper asks whether the memory of the observed flight history should stay fixed or keep updating while the planner imagines the future, since the latter might be prone to performance-degrading hallucinations.
  • It is studied in simulation with small recurrent models and one planner. They report several findings on which combination is better in which case.
Critical strengths
  • A very relevant question for model-based control: what to do with the memory of the past while the planner imagines the future: how to not be biased by you own incomplete imaginations
  • A controlled simulation is a good tool to answer this question, and the compared models are carefully matched in size and inputs.
  • The statistics are careful and honest. All planned comparisons are kept, also the null results, and the main follow-up is tested on new datasets.
  • A useful practical lesson: a model that predicts logged data best can still be the worse model inside the controller.
  • All code, the trained models and the numerical records are shared.
Critical weaknesses
  • Main weakness: the control results are simulation-only with a simple quadrotor model and slow references. The real flight logs are only used for prediction.
  • Only a limited set of possibilities is studied: there might be other influences that are important but not studied. Especially with the reversal, the reader does not feel confident he got to the bottom of this
  • An iterative baseline that retrains on data from the closed-loop planner could not be found. Mixed training is a one-shot version of it.
  • The core effect looks like the known mismatch between logged training actions and planner actions.
  • The title says memory updates hurt control, but with long training on planner actions, the updating model controls best.
  • The difference is already there at the first rollout step: does this not mean it comes from training, not from the memory update itself?
  • The paper is not easy to read: heavy hedging, five decimals, internal names like "round 9" and "donor episodes", and ... to put is mildly ... quite a few tables.
  • The statistics rest on five simulated datasets from one physics model, and several follow-ups were designed after seeing the results.
Questions for authors
  • Internal names like Round 9, 10A and the code labels recorded and log mean very little to the reader.
  • The numbers in the introduction have five decimals.
  • Explain what the normalized error means.
  • The speed of the reference trajectories could not be found in the main text. In other words: how hard was the tracking task?
  • A non-recurrent baseline with a fixed context could not be found. It would show how much the recurrent memory matters at all.
1 comment
AuthorsOpenReview ↗

Friendly Reminder

Thank you again for recognizing the relevance of the question, the controlled comparisons, and the care taken with statistics and reproducibility. Your suggestions led to two concrete additions: budget-controlled iterative data aggregation and two parameter-matched MLP baselines (Section 7, Table 3, and Appendix T).

Both one-shot and iterative collection improve both memory designs relative to equal-step continued training at both primary winds. The comparison also distinguishes the benefit of collecting controller-visited data from the smaller additional benefit of iteration. We have clarified the first-step difference between separately trained models, the reference-trajectory speeds, and the conditions under which the title's prediction/control reversal occurs.

As the discussion draws to a close, we would appreciate your assessment of whether these additions address your main concerns, and whether any material issue remains. Thank you for helping us make the study clearer and more useful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.