- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how an underoptimized dynamics predictor affects representation learning during joint training. It characterizes the difference between representation gradients obtained using the current predictor and a predictor optimally fitted to the same representation. A controlled sequential-VAE study on a double pendulum examines the resulting changes in representations and prediction quality. The authors then propose Asymmetric Coupling Warm-up (ACW), which temporarily weakens dynamics-to-representation gradients, and evaluate it in three additional model–task settings.
Critical strengths
-
The controlled refitting experiment is useful. Freezing the learned representations and fitting fresh predictors under matched procedures helps distinguish changes in representation quality from differences in the original predictors’ optimization.
-
The intervention is simple and applicable to different model families. Results on Koopman-AE [1], PLDM [2], and STORM [3] provide preliminary evidence that regulating early dynamics feedback can affect downstream performance across different learning setups.
[1] Mori–Zwanzig Latent Space Koopman Closure for Nonlinear Autoencoder. Priyam Gupta et al. (2025)
[2] Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models. Uladzislau Sobal et al. (2025)
[3] STORM: Efficient Stochastic Transformer Based World Models for Reinforcement Learning. Weipu Zhang et al. (2023)
Critical weaknesses
-
Prior work, including DreamerV2 [1] and Tang et al. [2], already recognizes that predictor optimization and asymmetric gradient updates affect representation learning. Although the paper acknowledges these connections, it does not clearly establish what substantive understanding its analysis adds. The central theoretical characterization follows from an optimal-value decomposition and an application of Danskin’s theorem [3]. This provides a useful description of the gradient discrepancy, but does not identify conditions under which that discrepancy harms representation learning. My concern is therefore the additional explanatory value of the analysis, rather than the correctness of the identity.
-
The comparisons do not sufficiently isolate the value of ACW’s design. That is, relevant alternatives include keeping the dynamics-to-encoder gradient at a small, fixed weight throughout training; gradually increasing the weight of the entire dynamics loss, which scales both encoder and predictor gradients; and, where applicable, pretraining the encoder and decoder on reconstruction before starting joint dynamics training. These comparisons would clarify whether ACW’s particular timing and asymmetric gradient scaling are necessary. Section 4 already tests additional predictor-only updates through Catch-up, but Section 5 does not compare ACW against this alternative under comparable computational budgets.
-
several improvements are accompanied by substantial variability. For example, STORM’s greedy return changes from $1205.3\pm156.1 \quad\text{to}\quad 1471.5\pm314.0. $ The paper highlights a 22.1% improvement, but does not report uncertainty on that improvement or the individual paired performance differences. Since the runs are paired by seed, those differences are particularly relevant.
-
Generalization across tasks within each model family remains untested. Koopman-AE is evaluated only on cylinder flow, PLDM only on Two-Rooms, and STORM only on MsPacman. Because the model and task change together across experiments, the results establish improvements on three selected model–task combinations but do not demonstrate robustness across datasets for any individual model. Repeating each experiment with three seeds addresses training variability, not this limitation in task coverage.
-
The writing and organization make the central experiments difficult to follow. Section 4 introduces several training procedures and diagnostics without a sufficiently clear explanation of their relationship. In particular, the extra predictor updates used during Catch-up training are easy to confuse with the separate optimization used to construct diagnostic references for both Joint and Catch-up checkpoints. Important notation also requires inference or consultation of the appendix: the Adam update vectors () and () are introduced only implicitly, () is not explicitly identified as conditional mutual information when introduced, and the meaning of wrapped () is explained in the appendix. Since interpreting these experiments is central to assessing the contribution, the main text needs a clearer experimental overview and explicit definitions at first use.
[1] Mastering Atari with Discrete World Models. Danijar Hafner et al. (2021)
[2] Understanding Self-Predictive Learning for Reinforcement Learning. Yunhao Tang et al. (2023)
[3] The Theory of Max-Min and Its Application to Weapons Allocation Problems. John M. Danskin. (1967)
Questions for authors
See weaknesses.
2 comments
A Gentle Reminder About Our Response
Dear Reviewer EnkT,
Thank you again for your time and review. As the discussion period closes on November 18 (AoE), we wanted to kindly check whether you have had a chance to read our response and revised manuscript.
Following your comments, we clarified our contribution relative to prior work and added several experiments. In particular, we (1) added gradient analyses and controlled experiments to examine the effect of ACW; (2) compared ACW with alternative coupling designs (ablation) and Catch-up, including computational cost; and (3) expanded STORM to four Atari games with six seeds each, and PLDM to six seeds. We also clarified the experimental procedures and notation in the manuscript.
We would appreciate it if you could take another look at our response and consider whether the revisions address your concerns. We would also be glad to clarify anything that remains unclear.
We thank the reviewer for the constructive feedback. We have revised the manuscript and added experiments to address the concerns below.
W1: Mechanistic Novelty and Explanatory Value
We acknowledge prior insights from DreamerV2 and Tang et al. on predictor optimization and gradient asymmetry; our analysis uses standard mathematical tools. We distinguish the inductive bias specified by the dynamics model class and fitting objective from additional representation feedback caused by predictor suboptimality. This matters because the latter introduces optimization-state-dependent influence beyond the intended bias, with unknown consequences. Controlled Joint/Catch-up experiments and matched transition refits show that this influence changes representation formation and persists as differences in the dynamical quality supported by the final representation. We do not presume it to be universally harmful; rather, we isolate and empirically investigate its effects.
W2: ACW Design Ablations and Computational Cost
To examine whether ACW's gains depend on its specific coupling design, we added three ablations in the double-pendulum sequential VAE (Section 5.4): Fixed coupling (γ=0.2 throughout), Symmetric warm-up (scaling both representation- and predictor-side dynamics gradients), and Pretraining (optimizing non-transition objectives before joint training). All use three seeds and matched fresh transition refits.
Matched-refit latent NMSE ↓ (three-seed mean; Table 3)
Fixed coupling underperforms Joint, while the other two improve upon Joint but remain below ACW. These results support temporarily attenuating representation feedback while maintaining predictor optimization.
To assess schedule sensitivity, we also varied the attenuation onset and full-coupling restoration across nine configurations in the same setting. All improved upon Joint at four horizons (Figure 6; Appendix D.3).
We further compared ACW with Catch-up in performance and computational cost (Section 5.4; Appendix D.3). Under 10,000 outer updates, ACW achieves lower matched-refit NMSE at three horizons and comparable performance at H=100. For seed 1101, Catch-up requires 580,744 transition updates / 25,813 s, compared with ACW's 10,000 / 793 s (recorded training-loop times, excluding refitting/evaluation).
W3/W4: Robustness Across Seeds and Tasks
To address W3's concern about variability and paired improvements, we expanded PLDM and STORM to six paired seeds. The tables below report individual differences (ACW − Native) and their mean ± SD. To address W4's concern about within-model task generalization, we further evaluated STORM on Qbert, Alien, and BankHeist alongside MsPacman, testing multiple tasks within the same model family (Sections 5.2–5.3; Appendix C.2–C.3).
PLDM: Per-seed paired differences
STORM: Per-seed paired return differences
The results show improvements in five of six PLDM seeds on both metrics and positive mean gains across all four STORM games, while explicitly revealing seed variability. Full Native/ACW scores and condition SDs are provided in Tables 10 and 12 (Appendix C.2–C.3), with task-level summaries in Table 2. Each STORM seed result averages 20 evaluation episodes.
W5: Experimental Clarity and Notation
We have revised the relevant passages to clarify the controlled experiments and their notation: