- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DiPA-WM addresses representation drift when training diffusion policies within continually updated world models. It first trains a world model and Gaussian policy using offline data and early online experience, then freezes the context encoder while adapting the dynamics module and training a diffusion policy in imagination. Slower context encoder updates are also experimented with, showing that relatively low update rate can achieve comparable performance to frozen encoder.
Critical strengths
- The separation of representation stability and dynamics adaptation is practical and well-motivated.
- Encoder-update ablations cover full-rate updates, slower updates, and freezing, with consistent results supporting the design.
Critical weaknesses
-
Limited qualitative evidence. Tables and training curves do not clearly illustrate which failures DiPA-WM avoids or how encoder-update rates affect behavior. Matched-scenario trajectories, representative policy failures, and examples of large world-model prediction errors would make the design choices more intuitive.
-
The benefit of switching to diffusion and its sensitivity to representation drift are not fully isolated. Gaussian and diffusion policies operate in different stages, preventing a direct comparison. Starting from the same Stage1 world-model checkpoint and replay buffer, the authors could compare continued Gaussian-policy training against switching to diffusion under matched budgets, while varying encoder-update rates. This would clarify both the performance benefit and whether drift is particularly problematic for diffusion policies.
-
The DiWA comparison requires clearer explanations. DiWA already provides a frozen-world-model baseline. However, it is unclear whether world-model configurations, warm-up, policy initialization, and subsequent budgets are matched. Please clarify these conditions.
Questions for authors
- Can the authors provide qualitative comparisons showing policy failures and world-model prediction gaps across methods and encoder-update rates?
- Under matched initialization and budgets, how does continued Gaussian-policy training compare with switching to diffusion, including sensitivity to encoder updates?
- Which model configurations, warm-up procedures, initializations, and training budgets are shared between DiWA and DiPA-WM experiments?
2 comments
Thank you for your rebuttal. However, I still find the answer to Q1 largely descriptive instead of being informative. By "qualitative results", I mean visualizations of policy failure and rollout trajectories. The rebuttal does not fully address that. Therefore, I'm retaining my original assessment.
We thank the reviewer for their insightful questions and feedback.
Q1) We note that during inference in CARLA, the Gaussian policy of stage 1 frequently applies brakes when approaching an intersection, regardless of the traffic light state or whether a vehicle is visible. The diffusion policy, which is further trained on online data, does not suffer from this symptom and can generate a smoother action chunk. For state-based tasks, we provide quantitative world model prediction gaps for MuJoCo in the appendix.
Q2) We have run additional experiments to validate the comparison between using the gaussian policy and switching to diffusion in stage 2. We continue to train the Gaussian policy for an additional 1M environment steps after stage 1, and note that performance is mostly saturated. Switching to the diffusion policy allows for further improvement beyond what is offered by a Gaussian policy. We have added these results to the main paper.
Q3) We match the configurations of DiPA-WM and DiWA where possible. We use the model architecture for both methods, and pretrain the world model equally on the same offline dataset. However, DiWA initializes its diffusion policy on the offline data, then optimizes it imagination. DiPA-WM initializes its diffusion policy on the replay buffer of the Gaussian policy. The total training budget of the diffusion policy for DiWA and DiPA-WM (stage 2) are matched, at 1 million steps.