acceptodds
Withdrawn from ICLR 2027

Skeleton-Grounded Latent Action Model for Robot World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robot world models simulate how environments evolve in response to robot actions. Latent action models enable learning dynamics from videos by encoding visual transitions as proxy actions. However, reconstruction-driven latent actions mix robot motion with object responses and other scene changes, complicating their correspondence with robot controls. We propose **SkeLAM**, a skeleton-guided latent action model for controllable robot world modeling. SkeLAM decomposes latent actions into a kinematic component grounded by robot skeleton supervision and a residual component retaining complementary visual dynamics. Cross-modal swap reconstruction encourages RGB-derived and skeleton-derived kinematic representations to be interchangeable during video reconstruction. The world model incorporates both components through separate conditioning pathways, while branch dropout reduces its dependence on future-derived residual information. A lightweight adapter maps robot actions to the kinematic space, and subsequent world-model fine-tuning adapts generation to these predicted conditions. At inference, the model generates future observations from the current observation and robot actions. On DROID, SkeLAM reduces end-effector ADE by 19.5% and LPIPS by 11.1% compared with DreamDojo. Motion probing supports kinematic specialization, while qualitative results demonstrate zero-shot human-to-robot motion transfer in generated rollouts. Policy evaluation shows agreement between predicted success rates and those measured in real robot executions.

Primary area:
applications to robotics, autonomy, planning
Keywords:
latent action model, video world model, robot learning, embodied ai

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_8ZZiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses a specific problem in latent actions learned from observational changes with a future-prediction objective: they are required to encode both information on robot movements and their variable outcomes on the environment. The proposed approach uses skeleton-based state representations to learn latents that only encode robot motion, uses them to supervise the inference of "kinematic" representations from raw observational changes and separates them from latents capturing environmental changes. The paper shows that simple action-latent projector can be learned to obtain action-conditioned world models that better reconstruct future images compared to methods in prior works (end-effector error reduced by 19.5% and LPIPS by 11.1% over DreamDojo). Further experiments show the latent actions can enable human to robot imitation and the action-conditioned world model can be used as a crude policy evaluator.

Critical strengths
  1. The method is intuitive, easy to understand and well presented. The stage-wise training and frozen skeleton-derived latent swapping and supervision ensures model is forced to separate out robot motion from residual effects.
  2. The use of skeleton ensures method can scale: skeletons have been shown to be a useful embodiment invariant representation in prior works.
  3. The comparison for measuring reconstruction cover multiple prior world modeling works including both latent-action based and non latent action based, and includes multiple metrics.
Critical weaknesses

In summary, the evaluations in the paper do not seem to fully justify the methodological contributions, and the gaps are detailed below:

  1. Gaps in current reconstruction evaluations:

    a. The complete approach is only marginally better than "no residual latent" ablation. Without error bars in Table 2, it is not clear if modeling residuals separately is useful. Since a primary motivation is addressing impact of environment outcomes and a primarily contribution of the work beyond simply combining skeleton-based world models with latent action world models is separately modeling the residual latent.

    b. The paper additionally uses AgiBot. The compared methods are trained with different datasets (eg. Ctrl-World trains only on DROID), so the quantitative and qualitative comparisons are not completely fair.

  2. Gaps in application evaluations beyond reconstruction:

    a. The cross-embodiment transfer experiments show only crude translation (up vs down motion) and do not demonstrate transfer of fine-grained motions involving object interactions or gripper motions. Results showing relatively similar start states in Agibot and Droid can result in similar outcomes would demonstrate the transfer. This is important as a primary motivation for using latent actions is to benefit from cross-embodiment datasets.

    b. The policy evaluation numbers are low and the paper only compares to DROID. It is important to study the effect of ablations on downstream applications to understand how the modeling choices transfer beyond better reconstruction.

    c. It is not clear if the world model improvements can result in better robot control: Prior work tests how the world models can be used for policy improvement [2] or action selection [3].

  3. Applications beyond world models: Latent actions learned with the objectives similar to those studied in the paper have also been used for training robot policies. [1]

[1] Latent Action Pretraining from Videos. Ye et al. ICLR 2025

[2] VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model Guo et al. ICML 2026.

[3] Cross-Embodiment Robot Foundation World Models with Latent Actions. Huang et al. ICML 2026.

Questions for authors
  1. Baseline details are missing. How are explicit action world models (Ctrl-World) adapted to train on multi-embodiment datasets for the Agibot results in appendix? How different is it from the explicit action baseline in [3]?
  2. Why does modeling residuals separately not translate to improvements in visual quality in Table 2? Can it help capture the necessary variation in outcomes?
  3. As an extension study, can the model be used to sample multiple possible futures all with similar robot motion but different environmental effects?

[3] Cross-Embodiment Robot Foundation World Models with Latent Actions. Huang et al. ICML 2026.

Reviewer_tb2TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a world model that predicts robot videos from robot actions. It leverages “latent actions” learned from videos to specify the robot action. These latents usually mix robot motion with other changes in the scene, which makes them hard to link to real robot actions. To fix this issue, the authors split the latent into two parts: a kinematic part, which a robot skeleton image supervises during training and a residual part, which holds the rest of the visual change. A “swap” loss makes the kinematic latent from RGB interchangeable with the one from the skeleton.

Authors evaluate their approach on world modelling (video generation condition on action), zero-shot action transfer and policy evaluation. On DROID, the method beats DreamDojo on end-effector error and on LPIPS. The paper also shows two human-to-robot transfer examples where kinematic of human video is used to control a robot and a small policy-evaluation study.

Critical strengths
  • The problem is real, and the idea is simple and makes sense.
  • Skeletons are needed only during training, not at test time.
  • The paper compares against many baselines.
  • The results on DROID are strong, and each part of the method helps there.
Critical weaknesses
  • It's unclear from the main paper how is the Cosmos-2B model adapted to take into account the new residual and kinetics latent.
  • The baselines use different models, data and training budgets. The paper should compare against the same Cosmos-2B model, trained on the same data for the same number of steps, but conditioned directly on robot actions. Without this, we can’t tell whether the gains come from the method.
  • Results don’t hold on AgiBot. On the second dataset (appendix only, not discussed), SkeLAM is beaten by other action-conditioned models on end-effector error. The human-to-robot transfer is not limited. It shows only two examples and no numbers.
Questions for authors
  • Were the baselines retrained on your DROID split?
  • How is Cosmos-2B adapted to the new action encoder?
  • Why do we see a different behavior on AgiBoT?
Reviewer_vqr5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SkeLAM, a skeleton-supervised latent action model for controllable robot world modeling. Its main contribution is a kinematic--residual decomposition that provides a dedicated motion representation for robot-action adaptation while retaining complementary visual dynamics for training. Skeleton supervision grounds the kinematic component in robot motion, and cross-modal swap reconstruction encourages RGB-derived and skeleton-derived representations to provide compatible motion conditioning at the reconstruction level. Separate conditioning pathways, branch dropout, and action-conditioned fine-tuning enable deployment using the current image and robot actions, without future-derived residual information.

The approach is evaluated on DROID and AgiBot through video-prediction metrics, component ablations, linear motion probes, and latent interventions. On DROID, it reports 19.5% lower end-effector ADE and 11.1% lower LPIPS than DreamDojo. Representation analyses provide evidence of kinematic specialization and cross-modal compatibility. Additional experiments examine qualitative human-to-robot motion transfer in generated videos and task-outcome agreement under open-loop replay of recorded RoboArena actions.

Critical strengths
  1. The combination of skeleton-grounded kinematic and residual latents with cross-modal swap reconstruction is a meaningful methodological contribution. The design gives robot-action adaptation a dedicated motion target while retaining complementary information for visual reconstruction. Grounding, alignment, and swap reconstruction provide complementary constraints on how the representation is learned and used.
  2. The authors implement the complete pipeline, from latent representation learning to world-model training and robot-action adaptation. The ablations, linear probes, and latent interventions provide useful evidence beyond aggregate video-quality scores, connecting the representation design to its intended function.
Critical weaknesses
  1. The reported OSCAR results are inconsistent. Appendix D.2 gives an ADE of 40.36 and states that OSCAR outperforms SkeLAM, whereas Tables 1 and 17 report 43.74, which places it below SkeLAM's 41.15. This changes the relative ranking and potentially the claim of leading on seven of eight metrics. The final results and associated conclusions need to be reconciled.
  2. The improvements do not hold consistently on AgiBot, and the paper provides little explanation. Compared with DreamDojo, SkeLAM has worse ADE (21.10 versus 19.55) and substantially lower direction F1 (0.2824 versus 0.5720), alongside weaker magnitude metrics. Since AgiBot is included in training, these results warrant analysis of where the proposed representation or adaptation procedure fails. Without such evidence, the scope and reliability of the method's benefits remain unclear.
Questions for authors

1.Which OSCAR ADE is correct, 40.36 in Appendix D.2 or 43.74 in Tables 1 and 17? Please reconcile the values and update the corresponding ranking statements. 2.Why does SkeLAM improve on DROID but underperform DreamDojo on AgiBot, particularly in direction prediction? 3.A controlled comparison on another robot dataset would strengthen the evidence that the benefits extend beyond DROID.

Reviewer_JYswOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SkeLAM, a latent action model that decouples skeleton-supervised kinematic components from residual components. Through cross-modal swap reconstruction, the model enables the interchangeability of motion representations derived from RGB video and those derived from skeleton data. The authors train a world model using these two components—incorporating a "branch dropout" technique—and subsequently adapt it for robotic actions via an "action-to-kinematics" mapping. The research motivation is sound and ok, and the proposed training pipeline is logically coherent. However, certain aspects of the experimental setup make the primary results difficult to interpret.

Critical strengths
  1. Using skeleton supervision to provide a dedicated objective for action adaptation is a sound design choice. Compared to simply incorporating an auxiliary motion loss, the approach of employing cross-modal swap reconstruction is more targeted.

  2. Ablation studies for DROID demonstrate that removing either skeleton supervision or swap reconstruction leads to a significant decline in model performance. This strongly confirms that these components are essential within the experimental setup described in the paper.

Critical weaknesses
  1. DROID evaluation uses realized future state changes rather than issued commands. This does not establish performance when the executed motion differs from the intended action?
  2. Are the baseline comparisons fair? Checkpoints, fine-tuning budgets, and action-input equivalence are insufficiently documented. It is unclear how much improvement comes from SkeLAM itself.
  3. How reliable is the policy evaluation? Policy ranking uses only seven policies, and the episode-level evaluation contains just nine actual successes. SkeLAM identifies only three of those successes.
Questions for authors

What information would be available at deployment to construct the DROID action inputs? Can the main comparison be repeated with issued robot commands, or can the paper explicitly restrict its conclusions to executed-trajectory conditioning?

Which checkpoints and adaptation procedures were used for DreamDojo, CD-LAM, and OSCAR? Were their control inputs equivalent to those used by SkeLAM?

Which OSCAR ADE is correct, and what explains the substantially weaker AgiBot motion results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.