- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces CLASS, a structured latent action learning framework designed for cross-embodiment motion generation. Through supervised contrastive learning, the method maps identical actions into a unified latent space, while feeding embodiment or scene information into the action decoder as distinct conditioning inputs. The cluster centers for each action category serve as fixed control signals, enabling cross-embodiment motion control and object manipulation across various environments. The authors validate the approach using DMC, Loco-MuJoCo, and Franka robotic arm simulations, demonstrating capabilities such as cross-embodiment control, generalization to unseen embodiment-action combinations, efficient adaptation to new embodiments with minimal parameter updates, and action interpolation.
Critical strengths
(1)The research problem holds practical significance, and the methodological design is clear. This paper focuses on decoupling action semantics from physical embodiment, allowing a single latent action to directly drive different physical forms. Compared to approaches that merely learn shared representations, CLASS further validates the executability of cluster centers within a closed-loop control framework, offering tangible practical value. (2)The experiments are quite comprehensive. The authors validate their method across a variety of embodied motion and robotic manipulation tasks, including experiments on cross-embodiment transfer, generalization to unseen combinations, parameter adaptation, and motion interpolation. Ablation studies further demonstrate that contrastive learning and embodiment-conditioned embeddings play a significant role in performance on motion tasks.
Critical weaknesses
(1) The methodological innovation is relatively limited, and there is a lack of thorough comparison with the most relevant existing approaches. CLASS primarily integrates supervised contrastive learning, embodiment embeddings, and category cluster centers that have already been extensively studied. While the authors provide comparisons with LAPO and conditional behavioral cloning, they fail to benchmark against more directly relevant methods for cross-embodiment skill representation or those utilizing supervised category control. Consequently, it is difficult to determine whether the observed performance gains stem from the underlying action learning itself, or simply from the combination of category supervision and a conditional decoder.
(2) Evidence for disentanglement and cross-embodiment generalization remains insufficient. Although the paper emphasizes the separation of body-related and scene-related information, the latent representations can still predict body categories with 80.4% accuracy and recover partial information about object locations (R² = 0.62). Furthermore, performance in cross-embodiment control is significantly lower than in same-embodiment control; adaptation to a new embodiment reaches only about 51%–53% of the performance achieved with full training, and manipulation transfer in the LIBERO benchmark also performs poorly. These results indicate that while the method possesses some degree of generalization capability, it is not sufficient to support the claim of robust disentanglement between body and scene.
Questions for authors
Q1. Regarding the novelty of the proposed method and baseline comparisons: Could you include comparisons with existing cross-embodiment skill representation methods (such as XSkill, UniSkill, or UniVLA)? If these methods are difficult to reproduce under the same experimental setup, I suggest including a baseline that employs supervised class prototypes and a shared body-conditioned decoder. This would help verify whether the performance gains truly stem from latent action learning, rather than from the class supervision itself.
Q2. Regarding the disentanglement of body and scene information: The latent actions can still predict body categories with 80.4% accuracy and recover some object position information (R² = 0.62). Could you further analyze the source of this residual information and quantify the actual degree of disentanglement—for instance, by examining metrics such as the distances between latent representations across different bodies, or the correlations between body classification accuracy and control performance?
Q3. Regarding the practical utility of latent action representations: In the context of object manipulation, the authors consistently employ category-mean substitution (p=1) when training the decoder. Notably, even after removing the contrastive loss, the model maintains an average success rate of 82.7% while preserving the full categorical clustering structure. This raises a fundamental question: has the model effectively degenerated into behavior cloning conditioned on category prototypes, rather than learning transferable latent actions derived from state transitions? It is suggested that the authors conduct a comparative analysis, evaluating fixed learnable category embeddings, category-mean substitution, and the original latent action inputs, to determine whether the IDM provides any meaningful information beyond simple category labels in this setting.
Q4. Regarding cross-environment generalization: There is a clear asymmetry in transfer across benchmarks, with a significant drop in manipulation success rates within the LIBERO framework. Could you provide further analysis into the reasons for these failures, shifts in observation distributions, differences in robot dynamics, or inconsistencies in the underlying controllers?
8 comments
Response to Reviewer JZsk (2/6): W2 Evidence for disentanglement and cross-embodiment generalization
#3-2 (W2) Evidence for disentanglement and cross-embodiment generalization
The numbers you cite are as reported. Because body and object-position information remains in the per-step , our wording "robust disentanglement of body and scene" could be read as applying to as well. We have revised this wording. At the same time, what the paper shows is a division of roles: the latent-action centroid carries the meaning of the motion, and separate conditions carry the body and scene information, so that one centroid drives the same action across bodies and scenes. The additional experiment shows that grasping in LIBERO transfers for box-shaped objects close to the training object and drops for objects of a different shape. Adaptation to a new body reaches about half of full training by adapting about 32 parameters with the latent space and decoder frozen, and humanoid and cheetah reach a similar level. The remaining half is a current limitation.
The evidence is the following four results.
- Driving across bodies (Table 3): Within the same benchmark, driving with another body's centroid gives 345.6 on cheetah, 329.6 on walker, 327.2 on UnitreeH1, and 363.7 on Cassie, which is 0.79–0.88 of the 411.1–419.2 obtained with the body's own centroid. With the reward is 9.5–13.6. The drop across benchmarks is answered in #3-5.
- Held-out combinations (Table 15): When humanoid walk and walker stand are excluded from training and driven with the same-motion centroid of another body, the rewards are 222.2 and 222.6. One-hot BC gives 128.3 and 144.3, and gives 66.8 and 42.4. Over five training seeds the rewards are 223.3 and 225.7 (Table R10 in #2-3).
- Driving across scenes (Table 2): A fixed centroid per action drives at 85.5% success over object positions on the whole table. Success drops to 31.6% with and to 50.6% when the model is retrained with , so both the centroid and are used for driving. The for predicting the object position from the fixed centroid is 0.41, the same as from the action label alone, so the fixed centroid carries no within-action position information and carries the position.
- LIBERO (Table R7): On the same robot with a different controller, grasping of the three box-shaped objects close to the training object reaches 53.3–80.0%, 68.0% on average, on par with the 65.0% pick success in the training environment. For the can, whose shape differs, it is 4.7%; the difference is +63.3 with a 95% confidence interval of +47.4 to +79.3. Shape differences are not yet fully absorbed.
Existing methods that pass latent actions to a per-body decoder share only the recognition of a motion; with this division of roles, CLASS shares the driving signal itself. A centroid computed on one body therefore drives the same action on another body, on a body–motion combination absent from training, on a body added from little data, and at object positions absent from training, and motions can be switched by interpolating between centroids.
The 51–53% of full training on the new body is 53–59% for humanoid and cheetah in the same way (Table R2), with the latent space and decoder frozen, which shows that the motion representation learned on the trained bodies is reused unchanged on the new body. Ant, by contrast, stays at 37% after adaptation, and bodies whose shape differs greatly are a current limitation. In the revision we define "separated" in the main text as supplying body and scene information to the decoder as conditions separate from the latent action, word the claim as "the driving centroid can be used across bodies and scenes," and add Tables R7 and R2 to the appendix.
Response to Reviewer JZsk (3/6): Q2 Where the residual information comes from, and quantifying the degree of disentanglement
#3-3 (Q2) Where the residual information comes from, and quantifying the degree of disentanglement
We analyzed the source of the residual information and the degree of disentanglement with the two metrics you suggest, the distances between latent representations across bodies and the relation between body-classification accuracy and control performance. Body information remains because the observed transitions the IDM takes as input differ by body, and object-position information remains because the IDM input includes the end-effector state. As for the degree, in the space the body difference is less than one tenth of the motion difference. Removing more body information, however, lowers cross-body driving, so CLASS keeps part of it by design. Object-position information is not present in the fixed centroid used for driving. What can be said to be independent of body and scene is the driving centroid, not the individual , and we state this distinction in the revised main text.
We answer for body information and object position in turn.
Body information
- Why body information remains (Table 4): A linear classifier predicts the body from with 80.4% accuracy, and the margin above the 75% obtained by always answering walker is the remaining body information. Contrastive learning gathers the same motion into one cluster but does not remove the body differences within the cluster.
- Distances between bodies (Table R11): We measured them on the same DMC model as Table 4. Centroids of the same motion on different bodies are 0.34 within-cluster mean radii apart. Centroids of different motions are 3.4–4.7 radii apart on the same body and 3.5–4.7 on different bodies. Without contrastive learning, the same-motion distance is 0.76 and the cross-motion distances are 0.03–1.1, so the order is reversed.
- Body-classification accuracy and control performance (Table R5): We varied the adversarial body-classification weight over 0, 0.2, 0.5, and 1.0. Body accuracy decreases as 94.7%, 80.4%, 79.3%, and 77.8%, approaching the 75% floor at weight 1.0. Same-body driving on cheetah gives 421.3, 419.2, 411.2, and 389.8, keeping 93% even at weight 1.0. Cross-body driving gives 337.3, 345.6, 318.0, and 245.4, highest at weight 0.2 and down to 71% at weight 1.0.
In the space the body difference is less than one tenth of the motion difference, and the remaining body information corresponds to the offset of 0.34 within-cluster radii. Body-classification accuracy and cross-body driving reward decrease together, so the remaining body information is used for cross-body driving, and the CLASS weight of 0.2 is the setting with the highest cross-body driving. In this construction, it is difficult to keep cross-body driving without any remaining body information.
Response to Reviewer JZsk (1/6): W1 & Q1 Novelty, and comparison with the closest methods or a class-prototype baseline
Thank you for the careful and detailed review. We have added experiments and analyses in response to your comments and questions. What CLASS shares across bodies is the driving signal itself, not only the recognition of a motion. The class-prototype baseline you suggest corresponds to the BC of Tables 15 and 16. Here is where the revised PDF (changes in red) addresses your review; the comments below give our point-by-point reply. Table numbers are those of the submission; added tables and figures are R1–R11 in Appendix M.
Reply numbers such as #1-1 or #2-2 refer to our replies; the first digit is the reviewer (1 = QbHd, 2 = yft1, 3 = JZsk).
#3-1 (W1 & Q1) Novelty, and comparison with the closest methods or a class-prototype baseline
We agree that supervised contrastive learning, body embeddings, and category cluster centers have each been used in prior work. In existing cross-embodiment skill representations, including XSkill, UniSkill, and UniVLA, what is shared across bodies is the recognition of a motion, and the mapping from latent action to commands is held by a per-body decoder or policy, so no point in the latent space means "run" for every body. By contrast, CLASS gathers latent actions inferred from observed transitions into one cluster per motion and gives body information to a single decoder as a separate condition, so the center of each cluster becomes one driving signal executable on every body. The novelty of CLASS is not the combination of components but what it shares across bodies: the driving signal, rather than only the recognition of a motion. This is what enables driving held-out body–motion combinations, adding a body by adapting about 32 parameters, and switching motions by interpolating between centroids.
These three methods assume human videos or large-scale video pretraining, so a comparison in the same setting as our state-vector experiments is difficult. We therefore use the second option you suggest, a baseline with supervised class prototypes and a shared body-conditioned decoder. The performance difference comes from grouping the inferred latent actions into one cluster per motion, not from category supervision with a conditional decoder.
The evidence is as follows.
The comparable reward on trained combinations shows that a class prototype with a shared decoder suffices when a prototype exists for the combination. The gap on held-out combinations arises because the prototype has no entry for a combination absent from training, whereas the CLASS centroid is computed from the transitions of the other bodies. Removing contrastive learning lowers the cross-body driving reward by about 88% (Table 5), so what produces the difference is grouping the inferred latent actions into one cluster per motion.
In the revision we will move Table 15 to the main text and state in the captions of Tables 15 and 16 that the comparison is against class-prototype BC.