- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The most significant contribution of this paper is that it identifies the issue of unreliable sampling in existing ACWMs. This problem can cause the model to generate visually appealing frames during rollouts while failing to faithfully simulate the real world, preventing it from serving as a solid foundation for simulation, planning, and policy evaluation.
To address this issue, the authors first conducted denoising-stage branching experiments, revealing that early stages strongly affect prediction fidelity and task-critical interaction dynamics, whereas later-stage stochasticity mainly changes visual details. Based on this observation, the authors introduce an RL post-training framework for ACWMs following the Flow-GRPO paradigm. Specifically, 1) they employ a video-similarity reward combined with a dynamics reward to evaluate the quality of rollouts. 2) They apply transition-scale correction to rectify the gradient norm imbalance in the early stages. 3) They use variance-initialized timestep sampling to allocate higher update frequencies to early stages, thereby producing more variable credit signals. 4) they introduce a downside penalty to further penalize low-fidelity outcomes while preserving favorable training signals.
Finally, the authors demonstrate the effectiveness of their proposed method and its individual components through experiments across various ACWMs.
Critical strengths
The primary strength of this paper lies in proposing a targeted post-training paradigm for ACWMs based on insights from denoising-stage branching experiments, which exploits the property that early denoising stages dictate the generated structure and interaction dynamics. Specifically, the proposed Transition-scale correction and Variance-initialized timestep sampling are both explicitly designed around this characteristic.
Critical weaknesses
1.The denoising-stage branching experiments in this paper are similar to prior findings [1] and do not offer a novel insight. Furthermore, the authors merely adapt existing RL methods for video diffusion models to the ACWM setting. For instance, techniques like Variance-initialized timestep sampling have already been explored in TempFlowGRPO and GRPO-Guard; the paper lacks a clear discussion distinguishing its approach from these existing methods.
2.Both this work and PersistWorld utilize RL post-training to enhance the rollout fidelity of ACWMs. However, the paper lacks a detailed comparison to clarify the key differences and distinctions against PersistWorld.
[1] Wang B, Vastola J J. Diffusion models generate images like painters: an analytical theory of outline first, details later[J]. arXiv preprint arXiv:2303.02490, 2023.
Questions for authors
-
For ACWMs in embodied domain, a crucial aspect is the physical consistency of the rollouts. However, the reward design in this paper overlooks this factor and relies solely on visual-based rewards. Are there more suitable reward models specifically tailored to physical consistency for ACWMs?
-
Why are the metrics for the "Full method" in Table 4a inconsistent with those reported for "DreamTamer (Ours)" in Table 1?