acceptodds
Withdrawn from ICLR 2027

CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

21 views

Abstract

Robot demonstrations contain reusable manipulation knowledge, but transferring this knowledge across embodiments requires representations that bridge differences in control interfaces. Action reconstruction alone does not ensure that similar behaviors across robots share a representation. To address this gap, we introduce CIALLO, an action codec that aligns actions, visual changes, and observation-grounded language in a shared space of short-horizon behavioral intent. An embodiment-conditioned decoder maps this intent back to robot-specific actions, allowing a shared representation to support different control interfaces. After pretraining on eight embodiments, CIALLO adapts to a new robot from one recording of at most 10 seconds by updating only its embodiment embeddings. The calibrated codec then remains frozen while a policy learns to predict intent from observations and instructions. We instantiate this policy with Qwen3.5-4B to form Qwen-CIALLO, whose backbone requires no large-scale robot-action pretraining before policy finetuning. With 20 demonstrations per task, Qwen-CIALLO achieves 75.0% average success across real SO-101 and single-arm Piper robots, compared with 68.3% for fully finetuned . In simulation, we additionally finetune the action decoder, achieving 97.3% success on LIBERO and 52.7% on RoboTwin 2.0. The LIBERO policy also achieves 80.8% on LIBERO-Plus without further finetuning. Together, these results support a shared action codec as a transferable control interface that separates embodiment calibration from task learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
cross-embodiment learning, vision-language-action models, multimodal alignment, action representation learning, robot manipulation, embodiment adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_REpgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CIALLO, a shared action codec for cross-embodiment manipulation. CIALLO maps heterogeneous robot actions, visual changes, and observation-grounded language into a shared short-horizon intent space, while an embodiment-conditioned decoder maps the intent back to robot-specific actions. The codec is pretrained across eight embodiments and can be calibrated to a new robot using a short recording by updating only embodiment-specific parameters. The calibrated codec is then frozen while a Qwen3.5-4B policy learns to predict intent from observations and instructions.

Critical strengths
  • CIALLO provides a clear approach to handling heterogeneous action spaces by aligning actions with visual changes and grounded language, while retaining embodiment-specific decoding for different control interfaces.

  • The method is evaluated across LIBERO, LIBERO-Plus, RoboTwin 2.0, and two real robot platforms, covering both simulation benchmarks and adaptation to new real-world embodiments.

Critical weaknesses
  • Table 4 is an important real-robot evaluation, but it covers only three relatively simple manipulation tasks with 10 rollouts per task and robot. The overall improvement from 68.3% to 75.0% corresponds to only 4 additional successful rollouts out of 60. Moreover, the improvement is concentrated on the wooden-fish task, while Qwen-CIALLO performs slightly worse than pi_0 on the other two tasks overall. This makes it difficult to assess how consistent the real-world advantage is across tasks.

  • Lack of a more rigorous cross-embodiment transfer setting. Eac robot is trained with 20 demonstrations of every evaluated task, making it difficult to disentangle cross-embodiment transfer from few-shot learning on the target robot, especially given the small performance gap in Table 4. Evaluating tasks unseen on the target robot but available on other robots would more directly demonstrate cross-embodiment task transfer.

Questions for authors
  • Why does CIALLO appear to provide substantially larger improvements in simulation than in the real-robot experiments? What factors might explain this gap between simulation and real-world performance?

I find the topic interesting and would also like to see more qualitative visualizations and failure-case analyses to better understand the learned representation and its limitations. If the concerns raised in the weaknesses can be adequately addressed, I would be willing to raise my score.

Reviewer_9UJJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CIALLO, an action codec for transferring manipulation knowledge across robot embodiments. It aligns actions, visual changes, and grounded language in a shared intent space, independent of each robot's control interface, so that the codec can be transferred to other robots. A new robot is calibrated from one recording of at most 10 seconds by updating only its embodiment embeddings, and the experiments aim to show that embodiment calibration can be separated from task learning. Qwen-CIALLO, which pairs the codec with a Qwen3.5-4B backbone that has no large-scale robot-action pretraining, reaches 97.3% on LIBERO and 75.0% (vs. 68.3% for π0\pi_0) on two real robots, suggesting strong generality and adaptability.

Critical strengths
  • The motivation to separate a shared intent space from robot-specific control interfaces is clear. In particular, adapting to a new robot from a recording of about 10 seconds would be a large practical benefit in real robot deployment.
  • The codec goes beyond action reconstruction. It maps visual change and instructions into the same intent space, so that different controllers can share similar behaviors. Anchoring actions to their visual effects and task context is a well-motivated design.
  • Policy learning builds on an open-source general VLM whose backbone needs no large-scale robot-action pretraining. This makes the approach practical and lowers the cost of building new policies.
Critical weaknesses

W1. Cross-embodiment sharing is neither enforced nor measured.

  • The paper motivates CIALLO by saying that action reconstruction alone does not make similar behaviors on different robots share a representation. The objective does not directly address this.
  • In each InfoNCE term, the only positive pair is two modalities of the same sample, for example one Panda moment and its own visual change. No term explicitly pulls together similar behaviors from different robots.
  • Every other sample in the batch is a negative, including other robots doing the same task. If two robots' similar behaviors map to the same intent, the loss increases. The objective can therefore push against the sharing the paper wants.
  • The evidence does not show cross-robot sharing either. Retrieval (Table 6) matches a sample to its own pair, and in the t-SNE (Fig. 5) Panda, WidowX, and Google Robot form separate clusters. The five RoboTwin robots overlap, but the paper attributes this to their shared tasks and simulated scenes, not to shared behavior.
  • This matters because the title and the problem statement rest on this property. The key question for cross-embodiment learning is whether different robots doing the same task share a representation. The paper neither targets this nor measures it.

W2. The intent is not an information bottleneck.

  • The paper calls the intent a "compact bottleneck". However, the intent (2×512=1,0242 \times 512 = 1{,}024 numbers) is larger than the action chunk it encodes (35 to 240 values).
  • Nothing forces the model to drop robot-specific information. Any sharing must come from the objective, which has the issue in W1.

W3. The evidence for new-robot adaptation is limited.

  • Table 4 is the only experiment that tests 10-second calibration with a frozen codec. LIBERO and RoboTwin use pretraining robots, and the decoder is also finetuned there.
  • The gain comes from one task. π0\pi_0 is equal or better in 4 of the 6 robot-task cells. Excluding the wooden-fish task, π0\pi_0 is ahead (37 vs. 34 successes).
  • Only SO-101 is truly unseen. Piper appears in pretraining as a simulated dual-arm robot. On SO-101, π0\pi_0 is better on 2 of 3 tasks.
  • Three tasks with 10 trials per cell are too few to support claims of robustness.

W4. The comparison with π0\pi_0 does not isolate the source of the gain.

  • Qwen-CIALLO and π0\pi_0 differ in backbone, pretraining, and execution horizon. It is unclear whether the gain comes from the shared intent or from this overall combination.
  • The execution horizon alone could matter. π0\pi_0 executes 50 steps (about 1.7 s) per chunk, while Qwen-CIALLO executes 15 steps (0.5 s) (App. C.2). This can affect a precise grasp such as picking up a thin screwdriver.
  • On the real robots, there is no same-backbone baseline without the codec, and no run without calibration.

W5. The action head is not ablated.

  • The TransConv bin head is independent of the shared intent and could be attached to any policy. Without a standard regression head as a baseline, we cannot tell whether the results in Section 4 come from the shared intent or from the head.
Questions for authors
  1. In RoboTwin, are intents from different robots doing the same task phase (e.g., Aloha AgileX and UR5) closer to each other than to other samples? How does this change with and without the contrastive terms?

  2. Can an action from robot A be encoded and then decoded as robot B, so that B performs the same behavior in simulation? The decoder takes the target action schema as input, so this seems possible.

  3. Table 9 uses 2k steps and reports only reconstruction error. Is this ablation available at the full 16k steps, and in terms of policy or adaptation performance?

  4. What happens on the real robots without calibration, for example with a pretraining robot's EE or a random EE?

  5. How does the same Qwen3.5-4B backbone perform with a standard action head and no codec, trained on the same data? What is π0\pi_0's success rate with a 15-step execution horizon?

  6. How does performance change if the TransConv bin head is replaced by a standard regression head?

Reviewer_ZQADOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CIALLO, an embodiment-conditioned action codec that aligns demonstrated actions, visual changes, and observation-grounded language in a shared intent space. A shared decoder reconstructs actions from each representation. For a new robot, a short recording calibrates embodiment-specific embeddings; a Qwen-based policy then learns to predict intent from observations and instructions. The codec remains frozen during real-robot policy training, while the decoder is additionally finetuned in simulation. The method is evaluated on several simulation benchmarks and two physical robot configurations.

Critical strengths
  • Separating embodiment calibration from task-policy learning is an interesting contribution. The real-robot experiments demonstrate that the calibrated codec can remain frozen while supporting downstream task learning, providing initial evidence for a reusable action interface across robot configurations.
Critical weaknesses
  • The policy comparisons do not isolate the contribution of CIALLO’s shared-intent representation. Sections 4.2–4.3 compare the complete Qwen-CIALLO system against other policies, but do not provide a matched comparison using the same Qwen backbone, demonstrations, and training budget with a simpler action head. Consequently, it is difficult to determine whether the reported improvements come from the proposed representation or other differences between the systems. This limits the evidence supporting the paper’s central methodological contribution.
  • The evidence for reliable new-robot adaptation is quite limited, which should be a core contribution of the paper. The calibration recordings’ motion content and selection are insufficiently described, and the real-robot evaluation covers two configurations with few trials per task. Moreover, Appendix C.2 specifies different execution horizons for Qwen-CIALLO and π₀, fifteen versus fifty steps. This can affect control performance independently of the representation.
Questions for authors
  • Are downstream comparisons available that hold the Qwen backbone, demonstrations, and training budget fixed while replacing CIALLO with a simpler action head or an action-only codec?
  • Action chunk size is usually very important to the real-robot performance. How were the different execution horizons for Qwen-CIALLO and π0.5\pi_{0.5} selected? Is there evidence that the reported advantage persists with matched action chunk sizes?
  • What motions are included in the ten-second calibration recording, and how sensitive is performance to its selection and motion coverage?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.