- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method that uses an information bottleneck to extract action relevant features from a pretrained video diffusion model. The method uses spatiotemporal aggregation followed by a Gaussian bottleneck to compress one-step video features into world tokens, which condition a separate action diffusion transformer. The model is jointly trained with video and action flow matching losses and KL regularization. The authors evaluate on LIBERO, LIBERO-Plus, RoboTwin, and real world manipulation tasks, reporting improved in-domain performance and improved robustness to visual perturbations.
Critical strengths
- The paper addresses a well-motivated problem. Features learned for video reconstruction may preserve visual details that are unnecessary for action prediction and potentially harmful under distribution shifts. Introducing an information constraint at the video action interface is a reasonable approach, and extracting features in a single video model forward pass avoids the overhead of full video generation.
- The ablations are well designed and informative. The authors separately evaluate the spatiotemporal aggregation module and the information bottleneck objective, showing that the two are most effective when combined. The experiments varying bottleneck capacity and regularization strength further support the proposed trade-off between information compression and action prediction.
- The experimental evaluation is fairly extensive, covering multiple simulation benchmarks and real world manipulation. The results on LIBERO-Plus are encouraging, particularly under camera and sensor-noise perturbations. I also appreciate the intervention experiments in Appendix A.4.1, which provide additional evidence that the resulting policy is less sensitive to visual changes.
Critical weaknesses
- The contribution of video supervision and the learned representations could be better isolated. While the ablations convincingly demonstrate the benefit of the bottleneck, the action flow matching loss already provides direct supervision to the world tokens and video backbone. It remains unclear how much the video prediction loss contributes during joint training. A useful ablation would be to train with and without this loss, starting from the same video-pretrained checkpoint and keeping the action loss and KL regularization unchanged. The paper could also examine the world tokens directly under the visual perturbations used in Appendix A.4.1. Comparing their sensitivity with the original video features would provide more direct evidence that the bottleneck suppresses irrelevant visual information, rather than only showing reduced variation in the predicted actions.
- The related work discussion overlooks much of the latent action and motion-centric representation literature. Works such as LAPA [1], Moto [2], UniVLA [3], villa-X [4], ViPRA [5], and AdaWorld [6] have explored learning compact dynamics-related representations from video for downstream control or world modeling. These works address closely related questions about learning control-relevant representations from video, and the paper would benefit from better positioning its contribution relative to them.
- The robustness improvements are somewhat uneven. Most of the improvement on LIBERO-Plus comes from camera and sensor noise perturbations, while gains under background and layout shifts are comparatively modest, despite their relevance to the paper's motivation. Performance also drops substantially on RoboTwin clean2random. These results do not undermine the demonstrated improvements, but suggest that the robustness benefits may be more limited than the general claims imply.
Questions for authors
- Have the authors examined how the learned world tokens change under the controlled visual perturbations in Appendix A.4.1? Comparing their sensitivity with the original video features could provide more direct evidence for the information filtering mechanism.
- How much does the video flow matching objective contribute during joint training? Would performance remain similar if the Video DiT were updated only through the action loss and KL objective, starting from the same pretrained checkpoint?
- How does performance change when the Video DiT is frozen during joint training? This would help establish whether adapting the video backbone is necessary or whether its pretrained representations are already sufficient.
References
[1] Ye et al. Latent Action Pretraining from Videos. ICLR 2025.
[2] Chen et al. Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos. ICCV 2025.
[3] Bu et al. Learning to Act Anywhere with Task-centric Latent Actions (UniVLA). RSS 2025.
[4] Chen et al. villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models. ICLR 2026.
[5] Routray et al. ViPRA: Video Prediction for Robot Actions. ICLR 2026.
[6] Gao et al. AdaWorld: Learning Adaptable World Models with Latent Actions. ICML 2025.
Discussion with the authors
Dear reviewers,
the authors have provided responses for your reviews. Can you please let them if you comments have been addressed?
Thank you, Your AC