- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses a specific problem in latent actions learned from observational changes with a future-prediction objective: they are required to encode both information on robot movements and their variable outcomes on the environment. The proposed approach uses skeleton-based state representations to learn latents that only encode robot motion, uses them to supervise the inference of "kinematic" representations from raw observational changes and separates them from latents capturing environmental changes. The paper shows that simple action-latent projector can be learned to obtain action-conditioned world models that better reconstruct future images compared to methods in prior works (end-effector error reduced by 19.5% and LPIPS by 11.1% over DreamDojo). Further experiments show the latent actions can enable human to robot imitation and the action-conditioned world model can be used as a crude policy evaluator.
Critical strengths
- The method is intuitive, easy to understand and well presented. The stage-wise training and frozen skeleton-derived latent swapping and supervision ensures model is forced to separate out robot motion from residual effects.
- The use of skeleton ensures method can scale: skeletons have been shown to be a useful embodiment invariant representation in prior works.
- The comparison for measuring reconstruction cover multiple prior world modeling works including both latent-action based and non latent action based, and includes multiple metrics.
Critical weaknesses
In summary, the evaluations in the paper do not seem to fully justify the methodological contributions, and the gaps are detailed below:
-
Gaps in current reconstruction evaluations:
a. The complete approach is only marginally better than "no residual latent" ablation. Without error bars in Table 2, it is not clear if modeling residuals separately is useful. Since a primary motivation is addressing impact of environment outcomes and a primarily contribution of the work beyond simply combining skeleton-based world models with latent action world models is separately modeling the residual latent.
b. The paper additionally uses AgiBot. The compared methods are trained with different datasets (eg. Ctrl-World trains only on DROID), so the quantitative and qualitative comparisons are not completely fair.
-
Gaps in application evaluations beyond reconstruction:
a. The cross-embodiment transfer experiments show only crude translation (up vs down motion) and do not demonstrate transfer of fine-grained motions involving object interactions or gripper motions. Results showing relatively similar start states in Agibot and Droid can result in similar outcomes would demonstrate the transfer. This is important as a primary motivation for using latent actions is to benefit from cross-embodiment datasets.
b. The policy evaluation numbers are low and the paper only compares to DROID. It is important to study the effect of ablations on downstream applications to understand how the modeling choices transfer beyond better reconstruction.
c. It is not clear if the world model improvements can result in better robot control: Prior work tests how the world models can be used for policy improvement [2] or action selection [3].
-
Applications beyond world models: Latent actions learned with the objectives similar to those studied in the paper have also been used for training robot policies. [1]
[1] Latent Action Pretraining from Videos. Ye et al. ICLR 2025
[2] VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model Guo et al. ICML 2026.
[3] Cross-Embodiment Robot Foundation World Models with Latent Actions. Huang et al. ICML 2026.
Questions for authors
- Baseline details are missing. How are explicit action world models (Ctrl-World) adapted to train on multi-embodiment datasets for the Agibot results in appendix? How different is it from the explicit action baseline in [3]?
- Why does modeling residuals separately not translate to improvements in visual quality in Table 2? Can it help capture the necessary variation in outcomes?
- As an extension study, can the model be used to sample multiple possible futures all with similar robot motion but different environmental effects?
[3] Cross-Embodiment Robot Foundation World Models with Latent Actions. Huang et al. ICML 2026.