acceptodds
Under review as a conference paper at ICLR 2027

CLASS: Clustered Latent Actions Separated from Body and Scene for Action Generation

Abstract

Action representations shared across bodies would let skills learned on one robot be reused on another. Existing latent-action learning, however, entangles motion meaning, what is done, with body-specific dynamics, on which body it is done, in one latent variable, so cross-body reuse of the representation remains limited. We propose Clustered Latent Actions Separated from Body and Scene (CLASS), a framework for action generation that draws latent actions into one cluster per motion category across bodies through supervised contrastive learning and conditions the action decoder on body information as a domain embedding , or on the object position as a scene variable in manipulation. The center of each cluster is then a drivable signal shared across bodies and scenes. On four DMC and Loco-MuJoCo bodies, per-step latent actions and centroids drive locomotion within and across bodies and benchmarks. Other bodies' centroids drive held-out body-motion combinations, and adapting about 32 parameters adds a new body. Cross-body cumulative reward drops about 80% without , and about 88% with contrastive learning removed. With a continuity regularizer, cosine interpolation between centroids lowers the fall rate at mid-rollout switches from about 98% to 13 to 19%. In object manipulation in simulation, fixed per-action centroids drive reach, pick, place, and push across object positions, and their four-action mean success rate of 85.5% exceeds the 79.0% of per-step re-estimation. The separation thus lets a centroid computed on one body drive the same action on another body or scene, and an action can be specified by a category name, one unlabeled demonstration, or interpolation between centroids.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
latent action, action representation, action abstraction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JZskOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CLASS, a structured latent action learning framework designed for cross-embodiment motion generation. Through supervised contrastive learning, the method maps identical actions into a unified latent space, while feeding embodiment or scene information into the action decoder as distinct conditioning inputs. The cluster centers for each action category serve as fixed control signals, enabling cross-embodiment motion control and object manipulation across various environments. The authors validate the approach using DMC, Loco-MuJoCo, and Franka robotic arm simulations, demonstrating capabilities such as cross-embodiment control, generalization to unseen embodiment-action combinations, efficient adaptation to new embodiments with minimal parameter updates, and action interpolation.

Critical strengths

(1)The research problem holds practical significance, and the methodological design is clear. This paper focuses on decoupling action semantics from physical embodiment, allowing a single latent action to directly drive different physical forms. Compared to approaches that merely learn shared representations, CLASS further validates the executability of cluster centers within a closed-loop control framework, offering tangible practical value. (2)The experiments are quite comprehensive. The authors validate their method across a variety of embodied motion and robotic manipulation tasks, including experiments on cross-embodiment transfer, generalization to unseen combinations, parameter adaptation, and motion interpolation. Ablation studies further demonstrate that contrastive learning and embodiment-conditioned embeddings play a significant role in performance on motion tasks.

Critical weaknesses

(1) The methodological innovation is relatively limited, and there is a lack of thorough comparison with the most relevant existing approaches. CLASS primarily integrates supervised contrastive learning, embodiment embeddings, and category cluster centers that have already been extensively studied. While the authors provide comparisons with LAPO and conditional behavioral cloning, they fail to benchmark against more directly relevant methods for cross-embodiment skill representation or those utilizing supervised category control. Consequently, it is difficult to determine whether the observed performance gains stem from the underlying action learning itself, or simply from the combination of category supervision and a conditional decoder.

(2) Evidence for disentanglement and cross-embodiment generalization remains insufficient. Although the paper emphasizes the separation of body-related and scene-related information, the latent representations can still predict body categories with 80.4% accuracy and recover partial information about object locations (R² = 0.62). Furthermore, performance in cross-embodiment control is significantly lower than in same-embodiment control; adaptation to a new embodiment reaches only about 51%–53% of the performance achieved with full training, and manipulation transfer in the LIBERO benchmark also performs poorly. These results indicate that while the method possesses some degree of generalization capability, it is not sufficient to support the claim of robust disentanglement between body and scene.

Questions for authors

Q1. Regarding the novelty of the proposed method and baseline comparisons: Could you include comparisons with existing cross-embodiment skill representation methods (such as XSkill, UniSkill, or UniVLA)? If these methods are difficult to reproduce under the same experimental setup, I suggest including a baseline that employs supervised class prototypes and a shared body-conditioned decoder. This would help verify whether the performance gains truly stem from latent action learning, rather than from the class supervision itself.

Q2. Regarding the disentanglement of body and scene information: The latent actions can still predict body categories with 80.4% accuracy and recover some object position information (R² = 0.62). Could you further analyze the source of this residual information and quantify the actual degree of disentanglement—for instance, by examining metrics such as the distances between latent representations across different bodies, or the correlations between body classification accuracy and control performance?

Q3. Regarding the practical utility of latent action representations: In the context of object manipulation, the authors consistently employ category-mean substitution (p=1) when training the decoder. Notably, even after removing the contrastive loss, the model maintains an average success rate of 82.7% while preserving the full categorical clustering structure. This raises a fundamental question: has the model effectively degenerated into behavior cloning conditioned on category prototypes, rather than learning transferable latent actions derived from state transitions? It is suggested that the authors conduct a comparative analysis, evaluating fixed learnable category embeddings, category-mean substitution, and the original latent action inputs, to determine whether the IDM provides any meaningful information beyond simple category labels in this setting.

Q4. Regarding cross-environment generalization: There is a clear asymmetry in transfer across benchmarks, with a significant drop in manipulation success rates within the LIBERO framework. Could you provide further analysis into the reasons for these failures, shifts in observation distributions, differences in robot dynamics, or inconsistencies in the underlying controllers?

8 comments
AuthorsOpenReview ↗

Response to Reviewer JZsk (1/6): W1 & Q1 Novelty, and comparison with the closest methods or a class-prototype baseline

Thank you for the careful and detailed review. We have added experiments and analyses in response to your comments and questions. What CLASS shares across bodies is the driving signal itself, not only the recognition of a motion. The class-prototype baseline you suggest corresponds to the BC of Tables 15 and 16. Here is where the revised PDF (changes in red) addresses your review; the comments below give our point-by-point reply. Table numbers are those of the submission; added tables and figures are R1–R11 in Appendix M.

  • W1 and Q1 (novelty and baselines): Sec. 2, Sec. 4.4 with Table 15, App. G with Table 16.
  • W2 (disentanglement and cross-embodiment generalization): Sec. 1, Sec. 4.2, Limitations, Tables R2 and R7.
  • Q2 (where the residual information comes from; degree of disentanglement): Sec. 4.2 and App. B with Table 4, Tables R5, R6, and R11.
  • Q3 (degeneration into category-prototype BC): App. C with Tables 8 and R4.
  • Q4 (asymmetry and LIBERO): Sec. 4.3 and App. D with Table 9, App. L with Table R7.

Reply numbers such as #1-1 or #2-2 refer to our replies; the first digit is the reviewer (1 = QbHd, 2 = yft1, 3 = JZsk).

#3-1 (W1 & Q1) Novelty, and comparison with the closest methods or a class-prototype baseline

We agree that supervised contrastive learning, body embeddings, and category cluster centers have each been used in prior work. In existing cross-embodiment skill representations, including XSkill, UniSkill, and UniVLA, what is shared across bodies is the recognition of a motion, and the mapping from latent action to commands is held by a per-body decoder or policy, so no point in the latent space means "run" for every body. By contrast, CLASS gathers latent actions inferred from observed transitions into one cluster per motion and gives body information to a single decoder as a separate condition, so the center of each cluster becomes one driving signal executable on every body. The novelty of CLASS is not the combination of components but what it shares across bodies: the driving signal, rather than only the recognition of a motion. This is what enables driving held-out body–motion combinations, adding a body by adapting about 32 parameters, and switching motions by interpolating between centroids.

These three methods assume human videos or large-scale video pretraining, so a comparison in the same setting as our state-vector experiments is difficult. We therefore use the second option you suggest, a baseline with supervised class prototypes and a shared body-conditioned decoder. The performance difference comes from grouping the inferred latent actions into one cluster per motion, not from category supervision with a conditional decoder.

The evidence is as follows.

  1. Class-prototype baseline (Tables 15 and 16): This is a BC that feeds a one-hot motion category or a learned motion embedding, together with the body embedding, to the same decoder as CLASS, with the same data and optimization. Since the weights of the first layer receiving the one-hot vector act as a learnable embedding per category, both one-hot BC and learned-embedding BC are baselines with supervised class prototypes and a shared body-conditioned decoder.
  2. Trained combinations (Table 16): The reward is comparable to CLASS. On walker walk, CLASS reaches 390.2, one-hot BC 367.8, and learned-embedding BC 396.1; on walker run, 395.4, 342.7, and 365.3. Over the four tasks, one-hot BC reaches 337.9–367.8 and learned-embedding BC 353.8–396.1.
  3. Held-out combinations (Table 15): Driving humanoid walk with the walker walk centroid gives 222.2 for CLASS, 128.3 for one-hot BC, and 110.1 for learned-embedding BC. Driving walker stand with the humanoid stand centroid gives 222.6, 144.3, and 92.9. z=0z=0 gives 42.4–66.8. Over five training seeds, CLASS reaches 223.3 and 225.7, one-hot BC 133.7 and 145.1, and learned-embedding BC 101.7 and 92.7 (Table R10 in #2-3).

The comparable reward on trained combinations shows that a class prototype with a shared decoder suffices when a prototype exists for the combination. The gap on held-out combinations arises because the prototype has no entry for a combination absent from training, whereas the CLASS centroid is computed from the transitions of the other bodies. Removing contrastive learning lowers the cross-body driving reward by about 88% (Table 5), so what produces the difference is grouping the inferred latent actions into one cluster per motion.

In the revision we will move Table 15 to the main text and state in the captions of Tables 15 and 16 that the comparison is against class-prototype BC.

AuthorsOpenReview ↗

Response to Reviewer JZsk (2/6): W2 Evidence for disentanglement and cross-embodiment generalization

#3-2 (W2) Evidence for disentanglement and cross-embodiment generalization

The numbers you cite are as reported. Because body and object-position information remains in the per-step ztz_t, our wording "robust disentanglement of body and scene" could be read as applying to ztz_t as well. We have revised this wording. At the same time, what the paper shows is a division of roles: the latent-action centroid carries the meaning of the motion, and separate conditions carry the body and scene information, so that one centroid drives the same action across bodies and scenes. The additional experiment shows that grasping in LIBERO transfers for box-shaped objects close to the training object and drops for objects of a different shape. Adaptation to a new body reaches about half of full training by adapting about 32 parameters with the latent space and decoder frozen, and humanoid and cheetah reach a similar level. The remaining half is a current limitation.

The evidence is the following four results.

  1. Driving across bodies (Table 3): Within the same benchmark, driving with another body's centroid gives 345.6 on cheetah, 329.6 on walker, 327.2 on UnitreeH1, and 363.7 on Cassie, which is 0.79–0.88 of the 411.1–419.2 obtained with the body's own centroid. With z=0z=0 the reward is 9.5–13.6. The drop across benchmarks is answered in #3-5.
  2. Held-out combinations (Table 15): When humanoid walk and walker stand are excluded from training and driven with the same-motion centroid of another body, the rewards are 222.2 and 222.6. One-hot BC gives 128.3 and 144.3, and z=0z=0 gives 66.8 and 42.4. Over five training seeds the rewards are 223.3 and 225.7 (Table R10 in #2-3).
  3. Driving across scenes (Table 2): A fixed centroid per action drives at 85.5% success over object positions on the whole table. Success drops to 31.6% with z=0z=0 and to 50.6% when the model is retrained with c=0c=0, so both the centroid and cc are used for driving. The R2R^2 for predicting the object position from the fixed centroid is 0.41, the same as from the action label alone, so the fixed centroid carries no within-action position information and cc carries the position.
  4. LIBERO (Table R7): On the same robot with a different controller, grasping of the three box-shaped objects close to the training object reaches 53.3–80.0%, 68.0% on average, on par with the 65.0% pick success in the training environment. For the can, whose shape differs, it is 4.7%; the difference is +63.3 with a 95% confidence interval of +47.4 to +79.3. Shape differences are not yet fully absorbed.

Existing methods that pass latent actions to a per-body decoder share only the recognition of a motion; with this division of roles, CLASS shares the driving signal itself. A centroid computed on one body therefore drives the same action on another body, on a body–motion combination absent from training, on a body added from little data, and at object positions absent from training, and motions can be switched by interpolating between centroids.

The 51–53% of full training on the new body is 53–59% for humanoid and cheetah in the same way (Table R2), with the latent space and decoder frozen, which shows that the motion representation learned on the trained bodies is reused unchanged on the new body. Ant, by contrast, stays at 37% after adaptation, and bodies whose shape differs greatly are a current limitation. In the revision we define "separated" in the main text as supplying body and scene information to the decoder as conditions separate from the latent action, word the claim as "the driving centroid can be used across bodies and scenes," and add Tables R7 and R2 to the appendix.

AuthorsOpenReview ↗

Response to Reviewer JZsk (3/6): Q2 Where the residual information comes from, and quantifying the degree of disentanglement

#3-3 (Q2) Where the residual information comes from, and quantifying the degree of disentanglement

We analyzed the source of the residual information and the degree of disentanglement with the two metrics you suggest, the distances between latent representations across bodies and the relation between body-classification accuracy and control performance. Body information remains because the observed transitions the IDM takes as input differ by body, and object-position information remains because the IDM input includes the end-effector state. As for the degree, in the zz space the body difference is less than one tenth of the motion difference. Removing more body information, however, lowers cross-body driving, so CLASS keeps part of it by design. Object-position information is not present in the fixed centroid used for driving. What can be said to be independent of body and scene is the driving centroid, not the individual ztz_t, and we state this distinction in the revised main text.

We answer for body information and object position in turn.

Body information

  1. Why body information remains (Table 4): A linear classifier predicts the body from ztz_t with 80.4% accuracy, and the margin above the 75% obtained by always answering walker is the remaining body information. Contrastive learning gathers the same motion into one cluster but does not remove the body differences within the cluster.
  2. Distances between bodies (Table R11): We measured them on the same DMC model as Table 4. Centroids of the same motion on different bodies are 0.34 within-cluster mean radii apart. Centroids of different motions are 3.4–4.7 radii apart on the same body and 3.5–4.7 on different bodies. Without contrastive learning, the same-motion distance is 0.76 and the cross-motion distances are 0.03–1.1, so the order is reversed.
  3. Body-classification accuracy and control performance (Table R5): We varied the adversarial body-classification weight over 0, 0.2, 0.5, and 1.0. Body accuracy decreases as 94.7%, 80.4%, 79.3%, and 77.8%, approaching the 75% floor at weight 1.0. Same-body driving on cheetah gives 421.3, 419.2, 411.2, and 389.8, keeping 93% even at weight 1.0. Cross-body driving gives 337.3, 345.6, 318.0, and 245.4, highest at weight 0.2 and down to 71% at weight 1.0.

In the zz space the body difference is less than one tenth of the motion difference, and the remaining body information corresponds to the offset of 0.34 within-cluster radii. Body-classification accuracy and cross-body driving reward decrease together, so the remaining body information is used for cross-body driving, and the CLASS weight of 0.2 is the setting with the highest cross-body driving. In this construction, it is difficult to keep cross-body driving without any remaining body information.

Reviewer_yft1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CLASS, a framework for cross-embodiment control with latent actions. CLASS infers latent actions from state transitions and uses supervised contrastive learning to cluster actions of the same category across different bodies. A shared action decoder is conditioned on a separate body embedding (or scene variable), so that a fixed category centroid can serve as a command for locomotion or manipulation. The evaluation covers four locomotion bodies in DMC and Loco-MuJoCo, including held-out body-motion combinations, adaptation to a new body, and motion switching, as well as four simulated Franka manipulation tasks. The reported results indicate that centroid-based control is effective within these simulators, including cross-body transfer and manipulation across object positions.

Critical strengths

The centroid is tested as an actual control input. Cross-body locomotion rewards remain substantial when a centroid computed on another body is used (Tables 1 and 3). The contrastive-loss and conditioning ablations sharply reduce performance, and the held-out body-motion comparisons in Table 15 offer evidence beyond visualizing latent clusters. Together, these make the central control claim credible within the tested settings.

The paper examines several ways to use the representation. Fixed centroids work across object positions, a demonstration can supply a driving signal, and interpolation substantially reduces falls at motion switches when the model is trained with the continuity regularizer. These experiments show what the latent variable can do, not only how well it classifies actions.

Critical weaknesses

The evidence supports partial, not complete, separation and transfer. Body identity can still be predicted from zz with 80.4% accuracy (versus a 75% majority baseline), and object position is recoverable with R2=0.62R^2=0.62 (Table 4). Cross-benchmark transfer is markedly asymmetric (Table 9), and LIBERO grasping transfers poorly (Table 25). These results limit the breadth of the claim that the action semantics are body- and scene-independent.

The manipulation results do not clearly establish an advantage over category conditioning. In this setting, the decoder receives a same-category mean during training, so the fixed centroid is functionally close to a learned category embedding. CLASS reaches 85.5% mean success versus 79.7% for one-hot conditioned BC, with substantial seed variation on picking (Table 2). The added value of inferring a latent action, specifically in the scene-transfer setting, remains uncertain.

Questions for authors

Table 2 and Table 14: Could you report paired per-seed differences between CLASS and one-hot conditioned BC, particularly for picking, and evaluate the one-hot baseline on the object-position bands excluded from training (Table 14)? This would clarify whether the manipulation benefit is robust.

Tables 9 and 15: Are the cross-benchmark and held-out body-motion results consistent across training seeds? Also, how do you interpret the weaker Loco-MuJoCo-to-DMC transfer in light of the residual body information measured in zz?

5 comments
AuthorsOpenReview ↗

Response to Reviewer yft1 (1/3): W1 Separation and transfer are partial

Thank you for the careful review. We have added experiments and analyses in response to your points and questions. In short, the revision defines "separated", states that the per-step ztz_t retains body and scene information, and keeps the claim that one centroid per action is shared across bodies and scenes. Here is where the revised PDF (changes in red) addresses your review; the comments below give our point-by-point reply. Table numbers are those of the submission; added tables and figures are R1–R11 in Appendix M. In addition, regarding the ethics flag, all experiments in this work are conducted in simulation, and no human-subject data or personal information is used.

  • W1 (partial separation and transfer): Sec. 1, Sec. 4.2 with Tables 4, R5, R6, and R11, App. L with Table R7, Limitations.
  • W2 and Q1 (advantage in manipulation; per-seed differences; one-hot BC on the excluded bands): Sec. 4.2 with Tables 2 and R3, Sec. 4.4 and App. F with Tables 14 and R8.
  • Q2 (seed consistency and asymmetry): Sec. 4.3 and App. D with Table 9, Tables R9 and R10.

Reply numbers such as #1-1 or #2-2 refer to our replies; the first digit is the reviewer (1 = QbHd, 2 = yft1, 3 = JZsk).

#2-1 (W1) Separation and transfer are partial

Of the four results cited, the two about ztz_t concern our wording: "action semantics independent of body and scene" could be read as applying to the per-step ztz_t as well, which retains body and object-position information, and we have revised this wording. The other two do not narrow the claim: the asymmetry in Table 9 comes from the reference used for the ratio, and grasping in LIBERO transfers for box-shaped objects close to the training object. What the results show is that one centroid per action is a driving signal shared across bodies and scenes. CLASS does not remove body and scene information from ztz_t; it supplies them to the decoder as a separate condition. In the revision we have defined "separated" in this sense in the main text and worded the claim as "the driving centroid is shared across bodies and scenes."

The evidence is the following three results.

  1. Driving across bodies: in Table 3, driving another body in the same benchmark gives 0.79–0.88 of the same-body reward.
  2. Driving across scenes: in Table 2, a fixed centroid per action drives at 85.5% success over object positions on the whole table; the fixed centroid, being one vector per action, carries no object-position information, and cc carries the position (Table R6).
  3. Driving across benchmarks: in Table 9, all eight pairs exceed z=0z=0 and random zz; the asymmetry between directions comes from the reference used for the ratio, and we answer it in #2-3.

For each point raised, what holds and what we state as a limitation in the revision are as follows.

  1. Body information: ztz_t retains body information to the extent that body classification reaches 80.4% against the 75% of always answering walker. Raising the adversarial body-classification weight reduces it, but the cross-body driving reward falls with it (Table R5). CLASS favors cross-body driving, and we state the remaining body information as a limitation of this trade-off.
  2. Object position: the fixed centroid used for driving carries no position information and cc carries it, so centroid driving holds across object positions. The individual ztz_t retains position information: the R2R^2 = 0.62 of Table 4 becomes 0.85 per action (Table R6). We state the partial separation of ztz_t as a limitation.
  3. LIBERO: with the same Franka Panda and a different controller, grasping transfers at 68.0% for box-shaped objects close to the training object (Table R7). It drops for the can, whose shape differs, so we list diversifying object shapes as future work.

As the evidence for the three points above, Tables R5, R6, and R7 will be added to the appendix of the revision.

AuthorsOpenReview ↗

Response to Reviewer yft1 (2/3): W2 & Q1 Advantage over category conditioning in manipulation; per-seed differences and one-hot BC on the excluded bands

#2-2 (W2 & Q1) Advantage over category conditioning in manipulation; per-seed differences and one-hot BC on the excluded bands

We agree that, when driving with a fixed centroid at trained object positions, CLASS is functionally close to BC conditioned on a learned category embedding, because the decoder input is replaced by the category mean during training. Under this condition the success rates are also close, 85.5% vs 79.7%. At the same time, the added value of inferring a latent action lies in transfer to object positions absent from training and in specifying an action from an unlabeled demonstration. This lets CLASS drive at object positions never used in training with higher success than one-hot BC and specify an action from one unlabeled demonstration; we confirmed that both hold across training seeds.

We answer the two questions in turn: (1) the paired per-seed differences and (2) the one-hot baseline on the object-position bands excluded from training.

  1. Paired per-seed differences (Table R3): For the models of Table 2, subtracting one-hot BC from CLASS within the same training seed gives +5.8 for the four-action mean (95% CI +1.0 to +10.6) and +27.2 for pick (95% CI +3.1 to +51.3), with CLASS higher in all five seeds for both; push is −6.4-6.4 (95% CI −17.5-17.5 to +4.7). Per-seed values are in Table R3 (cont.).
  2. One-hot BC on the excluded bands (Table R8): Trained on the same band-excluded data as Table 14, one-hot BC reaches 76.5% on the trained positions and 73.2% on the excluded positions; CLASS reaches 78.5% and 86.3% and does not drop on the excluded positions. The difference is +2.0 (95% CI −2.1-2.1 to +6.1) on the trained positions and +13.2 (95% CI +1.6 to +24.7) on the excluded positions, and +37.0 for pick, with CLASS higher in all five seeds. Per-seed values are in Table R8 (cont.).

Because the centroid is obtained from observed transitions rather than from action labels, CLASS keeps its success at object positions absent from training. CLASS can also specify an action from one unlabeled demonstration. In Table 2, driving with the mean zz inferred from the demonstration in place of the centroid reaches 85.3%. One-hot BC, whose input is the action label, cannot make this specification.

In the revision we will state the manipulation claim as "success equal to or higher than category-conditioned BC, higher success at object positions absent from training, and action specification from unlabeled demonstrations." Tables R3 and R8 will go to the appendix; Table 2 will gain columns with the paired differences and their 95% confidence intervals, and Table 14 will gain rows for one-hot BC trained on band-excluded data.

Table R3 Per-seed differences between CLASS and one-hot BC. Difference = CLASS −- one-hot BC.

actionCLASSone-hot BCmean difference95% CI of difference
reach100.0 ± 0.0098.4 ± 3.20+1.6[−2.8, +6.0]
pick65.0 ± 13.8137.8 ± 27.58+27.2[+3.1, +51.3]
place91.6 ± 7.1190.8 ± 6.49+0.8[−13.0, +14.7]
push85.2 ± 3.8791.6 ± 6.12−6.4[−17.5, +4.7]
4-action mean85.5 ± 2.6179.7 ± 5.74+5.8[+1.0, +10.6]

Table R8 CLASS and one-hot BC trained on band-excluded data.

method, object positionreachpickplacepushMean
CLASS, training bands (Table 14)100 ± 047 ± 2479 ± 988 ± 678.5 ± 7.1
CLASS, excluded bands (Table 14)100 ± 080 ± 967 ± 1698 ± 386.3 ± 4.8
one-hot BC, training bands (new)100 ± 044 ± 1676 ± 686 ± 576.5 ± 6.1
one-hot BC, excluded bands (new)100 ± 043 ± 1065 ± 985 ± 573.2 ± 5.6
AuthorsOpenReview ↗

Response to Reviewer yft1 (3/3): Q2 Consistency across seeds, and why Loco-MuJoCo→DMC transfer looks weaker

#2-3 (Q2) Consistency across seeds, and why Loco-MuJoCo to DMC transfer looks weaker

For the two questions, we re-evaluated Tables 9 and 15 over five training seeds (Tables R9 and R10). The additional experiments show that cross-benchmark driving and the driving of held-out combinations hold across seeds. The apparently weaker Loco-MuJoCo-to-DMC transfer comes from the reference value used for the ratio in Table 9, not from body information remaining in zz.

We answer each in turn.

  1. Seed consistency: for the eight pairs of Table 9, the five-seed means of CLASS are 153–261 with standard deviations of 5–47 across seeds, above z=0z=0 (53–71) and random zz (41–75) even allowing for this spread. For Table 15, the five-seed means of CLASS are 223–226, above one-hot BC (134–145) and learned-embedding BC (93–102).
  2. Loco-MuJoCo-to-DMC transfer: the ratio in Table 9 is relative to the reward of the target body driven by its own centroid. The rewards obtained with another body's centroid are of the same order in both directions, 152–260 toward Loco-MuJoCo and 158–199 toward DMC, but in the joint model the Loco-MuJoCo bodies reach only 188–280 with their own centroids against 388–395 for the DMC bodies, so only the ratios with Loco-MuJoCo targets come out high. With the per-body models of Table 3 as the reference, the ratios are 0.37–0.63 toward Loco-MuJoCo and 0.38–0.48 toward DMC, and the directional difference largely disappears.

The body information remaining in zz keeps cross-body driving at 0.79–0.88 of the same-body reward (Table 3), but it is unrelated to the directional difference between benchmarks.

In the revision we will add a column to Table 9 with the ratio relative to the per-body models, note in the captions of Tables 9 and 15 that they report one training seed, and add the five-seed results as an appendix table.

Table R9 The eight cross-benchmark pairs, evaluated over five training seeds.

source centroid → target bodypaper value (seed 0)mean ± std over 5 seedsz=0z=0random z
cheetah_run → UnitreeH1259.8 ± 16.0261.46 ± 17.4258.61 ± 1.7344.53 ± 3.14
cheetah_run → Cassie225.5 ± 58.7226.94 ± 31.5356.35 ± 5.1649.74 ± 4.27
walker_walk → UnitreeH1151.8 ± 17.1153.46 ± 17.7459.46 ± 8.9244.36 ± 7.31
walker_walk → Cassie164.2 ± 18.5163.72 ± 15.7262.38 ± 9.4347.31 ± 8.59
UnitreeH1_walk → cheetah177.0 ± 34.8181.23 ± 24.7267.78 ± 12.3675.43 ± 13.62
UnitreeH1_walk → walker190.0 ± 59.5196.18 ± 37.5371.37 ± 9.2558.33 ± 9.41
Cassie_run → cheetah158.4 ± 17.8157.0 ± 4.863.2 ± 3.141.2 ± 3.5
Cassie_run → walker198.6 ± 134.4199.25 ± 47.3753.45 ± 6.8247.64 ± 10.25

Table R10 Combinations excluded from training, evaluated over five training seeds.

combination (source centroid)CLASS in the paper (seed 0)CLASSone-hot BClearned-embedding BCz=0z=0
humanoid walk (walker's walk centroid)222.2 ± 25.73223.32 ± 13.63133.74 ± 22.56101.67 ± 15.6173.71 ± 12.75
walker stand (humanoid's stand centroid)222.6 ± 16.75225.71 ± 14.57145.14 ± 12.5892.72 ± 9.2346.42 ± 9.52

We would be grateful if you would consider these additions in your assessment.

Reviewer_QbHdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CLASS learns category-structured latent actions while providing body or scene information separately to an action decoder. After training, category centroids serve as fixed conditioning signals for closed-loop control.

Critical strengths
  • The paper evaluates the functional role of the representation through centroid driving, latent switching, and zero/random-latent controls, rather than relying solely on embedding visualizations.

  • The held-out body–motion results in Table 15 are promising and provide evidence of compositional generalization beyond transferring representations between already learned tasks.

Critical weaknesses
  • The incremental benefit is insufficiently established. Shared skill representations and executable skill centers already exist. For a small set of predefined categories, fixed-centroid conditioning is close to using learned task embeddings. The paper needs stronger evidence that its particular combination improves data efficiency, transfer, or controllability.

  • The generalization evidence is narrow. Only two held-out body–motion combinations are quantitatively evaluated. Cross-benchmark experiments involve jointly trained bodies, while adaptation to a new walker reaches approximately half of full-training performance. These results show partial transfer but do not establish a broadly useful adaptation advantage.

  • Manipulation adds limited support. The experiments vary object positions for one arm, and performance is comparable to task-conditioned BC within seed variability. Removing contrastive learning reduces mean success only from 85.5% to 82.7%, leaving its contribution beyond category conditioning unclear.

Questions for authors

See weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer QbHd (1/3): W1 Advantage over existing methods is not established

Thank you for the careful review. We have added experiments in response to your three points. In short, the advantage of CLASS over a task-embedding BC lies in body–motion combinations absent from training, and in manipulation CLASS matches or exceeds one-hot BC when paired by seed. Here is where the revised PDF (changes in red) addresses your review; the comments below give our point-by-point reply. Table numbers are those of the submission; added tables and figures are R1–R11 in Appendix M.

  • W1 (advantage over existing methods): Sec. 1, Sec. 2, Sec. 4.4 with Tables 15 and R1.
  • W2 (generalization evidence): Sec. 4.4 with Tables R2 and R10, Limitations.
  • W3 (manipulation): Sec. 4.2 with Tables 2 and R3, App. C with Tables 8 and R4.

Reply numbers such as #1-1 or #2-2 refer to our replies; the first digit is the reviewer (1 = QbHd, 2 = yft1, 3 = JZsk).

#1-1 (W1) Advantage over existing methods is not established

We agree that shared skill representations and executable skill centers exist in prior work, and that a fixed centroid and a task embedding give comparable reward on trained body–motion combinations. Beyond this, the difference from the prior work lies in how the driving centroid is obtained. In cross-embodiment skill representations such as XSkill and UniVLA, a per-body decoder holds the mapping from latent action to commands, so only the recognition of a motion is shared across bodies. A task embedding is one vector per category learned from action prediction alone, so no embedding exists for a combination absent from training.

By contrast, a CLASS centroid is the mean of latent actions that the IDM infers from the transitions of several bodies with different state and action dimensions and that contrastive learning gathers into one cluster per motion, and a single decoder executes it on any body by changing only the body condition dd. Because it is computed from the transitions of other bodies, a centroid exists for held-out combinations and persists when the training data are reduced. The advantage of CLASS therefore lies in body–motion combinations absent from training, which it drives with another body's centroid, without additional training and with little training data.

The comparison with task-embedding BC is as follows.

  1. Trained combinations: On walker walk, CLASS reaches 390.2, one-hot BC 367.8, and learned-embedding BC 396.1 (Tables 12 and 16); the three are comparable.
  2. Transfer: Held-out combinations are driven directly with another body's centroid. In Table 15, CLASS reaches about 222, whereas one-hot BC reaches 128–144 and learned-embedding BC 93–110.
  3. Data efficiency: With 10% of the episodes per task, CLASS keeps 84–88% of its full-data reward on held-out combinations, whereas one-hot BC drops to 37–47%, the same range as z=0z=0 (42–67). As Table R1 and Figure R1 show, CLASS with 10% of the data exceeds one-hot BC with 100%.
  4. Controllability: Motions are switched by interpolating between centroids, and the fall rate at switching drops from about 98% to 13–19% (Table 20). One unlabeled demonstration also provides a driving signal, reaching 84% of centroid driving (Table 11).
  5. Adding a new body: Adapting about 32 parameters reaches 51–53% of full training on walker, and 53–59% on humanoid and cheetah (Table 12 and Table R2 in #1-2).

In the revision we add one sentence to the contributions stating the advantages over task-embedding BC in transfer, data efficiency, controllability, and body addition, move Table 15 to the main text, and add Table R1 to the appendix.

Table R1 CLASS and one-hot BC with varying amounts of training data. Rows = fraction of episodes per task.

training dataincluded: walker walkheld out: humanoid walkheld out: walker stand
CLASSone-hot BCCLASSone-hot BCCLASSone-hot BC
10%255.32 ± 25.24252.53 ± 32.54196.41 ± 34.7360.83 ± 24.21187.44 ± 29.7253.15 ± 33.26
25%306.88 ± 35.46279.75 ± 41.46209.38 ± 29.4773.6 ± 20.46206.36 ± 24.1773.47 ± 26.57
50%338.63 ± 32.73306.37 ± 25.82218.64 ± 27.71101.54 ± 28.73214.96 ± 19.63111.24 ± 35.15
100%392.24 ± 24.53375.6 ± 48.3222.2 ± 25.73128.3 ± 24.62222.6 ± 16.75144.3 ± 50.03
AuthorsOpenReview ↗

Response to Reviewer QbHd (2/3): W2 Generalization evidence is narrow

#1-2 (W2) Generalization evidence is narrow

We agree that two held-out combinations are evaluated quantitatively, that the cross-benchmark driving in Table 9 is between jointly trained bodies, and that the new walker reaches about half of full training. What the paper shows is that a motion centroid computed from the transitions of one body drives another body on a combination that was never trained, with no additional training, and that a new body is added by adapting about 32 parameters with the latent space and decoder frozen. Existing methods with a per-body decoder share only the recognition of a motion; CLASS shares the driving signal itself, which is what makes both possible. Adding a new body gives the same result for humanoid and cheetah as for walker, but is harder for a body of very different shape such as ant, so how easily a body can be added depends on its shape similarity to the trained bodies.

Two results show this.

  1. Held-out combinations: When humanoid walk and walker stand are excluded from training and driven with the same-motion centroid of another body, both reach about 222 in Table 15, above one-hot BC (128–144) and z=0z=0 (42–67).
  2. New body: For a model trained without walker, adapting about 32 parameters on 500 transitions, with the latent space and decoder frozen, reaches 51–53% of full training (Table 12). Without adaptation the reward is 37–38%, and with d=0d=0 it is 14–15%, so half of the reward comes from the adapted dd. The adapted dd also works with centroids of motions that were not used for adaptation, reaching 58–78% of direct adaptation (Table 13). Humanoid and cheetah reach 53–59% with dd-only adaptation, on par with walker, whereas ant drops from 49% to 37% (Table R2).

The gap to full training is the cost of describing how the new body moves with only about 32 parameters; the motion representation and the decoder learned on the trained bodies are reused unchanged, for humanoid and cheetah as well as walker.

We state as limitations that two held-out combinations were evaluated and that for ant, whose shape differs greatly, adapting dd lowers the reward, and we add Table R2 to the appendix with this explanation.

Table R2 Adding a new body, repeated for other bodies. Parentheses = ratio to full training.

body / motion held out and addedsame body, full training (denominator)no adaptationd only (500 transitions)d + decoder last layer (500 transitions)
humanoid run334.15 ± 21.17124.35 ± 15.75(37%)180.12 ± 17.34(54%)212.24 ± 14.64(64%)
humanoid stand289.73 ± 17.32102.53 ± 14.25(35%)164.51 ± 15.46(57%)186.46 ± 17.54(64%)
humanoid walk297.54 ± 22.74105.46 ± 13.57(35%)175.75 ± 14.75(59%)184.68 ± 20.61(62%)
cheetah run353.64 ± 18.63131.31 ± 12.75(37%)186.31 ± 13.76(53%)252.15 ± 17.41(71%)
ant run334.24 ± 40.21164.41 ± 18.52(49%)125.23 ± 15.58(37%)112.32 ± 19.43(34%)
walker walk (Table 12)390.2 ± 26.6146.6 ± 6.0(38%)200.8 ± 14.75(51%)216.7 ± 9.43(56%)
walker run (Table 12)395.4 ± 83.7151.4 ± 4.8(38%)211.4 ± 21.54(53%)234.8 ± 16.38(59%)
AuthorsOpenReview ↗

Response to Reviewer QbHd (3/3): W3 Manipulation adds limited support

#1-3 (W3) Manipulation adds limited support

We agree that the contribution of contrastive learning is small in manipulation. At the same time, what the manipulation experiment supports is not a large margin over BC but two points. First, the same construction, a centroid plus a separate condition, carries over to the scene condition; the evidence is success equal to or higher than that of label-conditioned BC. Second, CLASS adds what BC cannot provide, transfer to held-out object positions and label-free specification; the evidence is higher success at the excluded object positions and driving from one unlabeled demonstration.

The evidence and the reasons are as follows.

  1. Driving with the scene as the condition: In Table 2, driving with a fixed centroid and the scene variable cc reaches 85.5% mean success over object positions on the whole table. Success drops to 31.6% with z=0z=0 and to 50.6% with c=0.c=0.
  2. Difference from task-conditioned BC: Paired by training seed, CLASS is higher by 5.8 points in the four-action mean and by 27.2 points in pick, and both 95% confidence intervals exclude zero (Table R3). Only for push is one-hot BC higher, by 6.4 points, and that interval includes zero.
  3. Contribution of contrastive learning: Removing contrastive learning lowers success from 85.5% to 82.7%, whereas training on the raw ztz_t lowers it to 54.6% (Table R4). In locomotion, removing contrastive learning lowers the cross-body driving reward by about 88% (Table 5).
  4. Held-out object positions and label-free specification: When trained on band-excluded data, CLASS reaches 86.3% on the excluded positions and one-hot BC 73.2%, a difference of 13.2 points with a 95% confidence interval of [+1.6, +24.7] (Table R8 in #2-2). Driving with the mean zz of one unlabeled demonstration reaches 85.3% (Table 2).

Success drops both with z=0z=0 and with c=0c=0, so the centroid carries the action and cc the object position. The difference from task-conditioned BC is not hidden by seed variability. Removing contrastive learning barely lowers success because, in manipulation, the decoder receives the category mean as input both in training and at driving time, and this input does not change when the ztz_t space changes. What determines success in manipulation is this substitution by the category mean; the contribution of contrastive learning appears in locomotion, where the motions of several bodies must be aligned into one cluster.

In the revision, Table 2 will gain columns with the paired differences and their 95% confidence intervals, and Tables R3 and R4 will be added to the appendix.

Table R3 Per-seed differences between CLASS and one-hot BC. Difference = CLASS −- one-hot BC.

actionCLASSone-hot BCmean difference95% CI of difference
reach100.0 ± 0.0098.4 ± 3.20+1.6[−2.8, +6.0]
pick65.0 ± 13.8137.8 ± 27.58+27.2[+3.1, +51.3]
place91.6 ± 7.1190.8 ± 6.49+0.8[−13.0, +14.7]
push85.2 ± 3.8791.6 ± 6.12−6.4[−17.5, +4.7]
4-action mean85.5 ± 2.6179.7 ± 5.74+5.8[+1.0, +10.6]

We would be grateful if you would consider these additions in your assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Novelty and significance, and summary of the PDF revision

We thank the three reviewers for their careful reading. We have added experiments and analyses in response to the reviews and updated the PDF (changes in red). This general response summarizes the novelty and significance of the work and the changes in the PDF. Our replies are numbered #r-k, where r is the reviewer (1 = QbHd, 2 = yft1, 3 = JZsk) and k the item in the order of the review.

Novelty and significance

In existing work on latent actions across bodies, a per-body decoder holds the mapping from latent action to commands. What is shared across bodies is therefore the recognition of a motion, and no point in the latent space means "run" for every body. By contrast, CLASS gathers latent actions into one cluster per motion across bodies by supervised contrastive learning, and supplies body and scene information to a single decoder as separate conditions. The center of each cluster then becomes a driving signal executable on any body by changing only the condition. The novelty of this work is that what is shared across bodies changes from the recognition of a motion to the driving signal itself. Both parts are necessary: removing the body condition lowers the cross-body driving reward by about 80%, and removing contrastive learning by about 88%.

The paper shows four points: (1) body–motion combinations absent from training are driven by the centroids of other bodies without additional training, and the margin over a task-embedding BC persists with a tenth of the training data; (2) a new body is added by adapting about 32 parameters, with the latent space and decoder frozen; (3) the same construction holds in object manipulation with the scene as the condition; (4) a centroid can be specified by a category name, by one unlabeled demonstration, or by interpolation between centroids, so the agent can select and switch motions by itself. The evaluation covers four locomotion bodies on two benchmarks and a Franka arm in simulation; within this scope, the paper shows an action representation reusable across bodies and scenes.

Summary of the PDF revision (table numbers as in the submission; added tables and figures are R1–R11 in Appendix M)

Added

  • Sec. 1: the definition of "separated" (body and scene information is supplied to the decoder as a separate condition, not removed from ztz_t; what is shared is the centroid), and one sentence on the advantages over a task-embedding BC
  • Sec. 2: existing methods share the recognition of a motion whereas CLASS makes the centroid itself the driving signal; the relation to XSkill, UniSkill, and UniVLA; the class-prototype baseline
  • Sec. 4.2, Table 2: a row with the paired per-seed differences CLASS −- one-hot BC and their 95% confidence intervals (Table R3)
  • Sec. 4.2, Table 4: body information in ztz_t versus cross-body driving (R5), centroid distances across bodies (R11), and per-action R2R^2 of the object position (R6)
  • Sec. 4.3, Table 9: the explanation of the asymmetry, a ratio column with the per-body models as the reference, and five-seed results (R9)
  • Sec. 4.4: Table 15 moved to the main text, with its five-seed results (R10), the 10% training-data results (R1), one-hot BC on the excluded bands (R8), and body addition for humanoid, cheetah, and ant (R2)
  • App. C, Table 8: three decoder inputs (R4). App. L: per-object success in LIBERO (R7)

Corrected

  • Sec. 4.2: "ztz_t does not absorb the object position" →\to "ztz_t encodes the action and part of the object position, and the fixed centroid used for driving carries no within-action position information"
  • Sec. 4.2: the remaining body information "may be one reason" for the lower cross-body driving, "but removing it lowers that driving"
  • Sec. 4.4, Table 14: the CLASS values on the excluded bands (pick and mean) are updated to the re-evaluation of Table R8 (61 ±\pm 35 →\to 80 ±\pm 9; 81.6 ±\pm 5.8 →\to 86.3 ±\pm 4.8); App. F reports the same values

Reorganized

  • Table 15 moved from App. G to Sec. 4.4 without renumbering; the main text stays within 10 pages

Scope of the claims

  • What is independent of body and scene is the driving centroid; the per-step ztz_t retains body and scene information (Sec. 1, Sec. 4.2, Limitations)
  • In manipulation, CLASS with a fixed centroid matches or exceeds a category-conditioned BC in success rate; the added value is specification from an unlabeled demonstration and higher success at object positions excluded from training (Sec. 4.2, Conclusion)
  • Generalization is shown for two held-out combinations, three bodies added with the latent space and decoder frozen (51–59% of full training), and LIBERO grasping of box-shaped objects; ant and the can, whose shapes differ greatly, are the current limits (Limitations)
  • The cross-benchmark asymmetry comes from the reference used for the ratio, and cross-benchmark driving holds among jointly trained bodies (Sec. 4.3)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.