acceptodds
Withdrawn from ICLR 2027

LinguaMotion: Expressing the Language of Motion for Real-World Robot Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Translating high-level instructions into manipulations is a fundamental capability in robotics, yet achieving this across diverse tasks and embodiments in real world remains challenging. While Vision-Language Models (VLMs) offer rich world knowledge, existing VLM-based policies often couple action prediction to specific robot kinematics and action spaces, failing to fully leverage pretrained knowledge and limiting the generality. We introduce LinguaMotion, a novel framework reinterpreting manipulation as structured generation in the language of motion. Instead of low-level robot actions, our model represents manipulation via object-centric motion directly within the native language space of VLMs. Given an image and instruction, it generates a hierarchical description of task-relevant objects, their spatial states, and how they should evolve to accomplish the task, thus bridging high-level semantic understanding with physical motion planning. To support this formulation, we further introduce a numeric value loss to restore geometric fidelity in tokenized motion representation, and a cycle-consistent learning framework to enforce semantic alignment between motion and language while enabling inference-time self-verification. On challenging real-world manipulation tasks, LinguaMotion consistently outperforms mainstream generalizable baselines across embodiments.

Primary area:
applications to robotics, autonomy, planning
Keywords:
real-world robot manipulation, vision-language models, object-centric motion modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wrpQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper demonstrates that a pretrained VLM can be fine-tuned to directly autoregress a structured, object-centric 6-DoF motion representation, providing an embodiment-independent alternative to dedicated trajectory decoders. It augments token cross-entropy with a typical numerical regression objective and uses bidirectional motion-language training to derive a cycle-reconstruction error that serves as an inference-time confidence measure. None of these contributions is particularly innovative. Probably the largest contribution is described in the appendix: the data generation pipeline that runs a large set of existing methods to get the training labels for the fine-tuning. This substantial, labeled dataset most likely explains the improvement over baselines.

Critical strengths

Using a large zoo of existing methods for segmentation, tracking, depth estimation, etc. a comparatively strong automatic labeling of 150k samples is achieved, which provides a strong basis for fine-tuning the VLM.

Critical weaknesses

The paper is written in very vague terms from the beginning to the end rather than saying precisely what it is doing, namely simply fine-tuning a VLM for object-trajectory prediction without using additional generative components. The argument why the latter is "bad" is weak and without evidence. There is a lot of framing happening with the wording, which leads to overclaiming, for example "hierarchical representation", which is simply a sequence of the defined steps "predict relevant objects + bbox/category/role; predict current and desired object poses; predict intermediate poses". Directly generating spatial plans from a VLM is established in the field. The paper does not explicitly claim any novelty, but the writing frames it as if the paper was full of new ideas. It is not. In fact, I did not find any substantial novelty that is of scientific value in this paper.

Questions for authors

No questions.

Reviewer_UKF1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper fine-tunes a VLM to express manipulation directly in its native language space. The model first outputs task-relevant objects and their roles, then predicts the current and goal 6D poses of the source object and a sequence of pose waypoints; the motion is then executed using AnyGrasp and cuRobo. The authors state that: (i) outputting in the native language space preserves pretrained knowledge; (ii) a numeric value loss and an inverse task that recovers the instruction from the motion improve accuracy; and (iii) the cycle residual can serve as a verification signal at inference time. The authors report zero-shot success rates of 70–90% across two robotic arms and seven real-world tasks.

Critical strengths

S1. The interface is clear and embodiment-agnostic, making it possible to identify the step at which a failure occurs. (Section 3.1; Figure 1) S2. The real-robot evaluation involves substantial effort: two robotic arms, one of which is unseen during training; seven task categories that include tool use; and six baselines that share the same grasp detector and motion planner. (Section 4.1; Figures 5–6)

Critical weaknesses

W1. The paper claims to employ a “cycle-consistent learning framework”, but the actual training appears to involve only bidirectional supervision. The introduction states that the model is “trained to recover the instruction from a predicted motion sequence, which in turn conditions a second motion prediction, forming a closed loop” (Section 1). However, the loss function consists solely of L = L_fwd + L_inv (Section 3.4), and during training, the model switches to the inverse task with a probability of 0.2 (Section 4.1). The round-trip generation seems to be used only during inference (Section 4.3), with no cycle-consistency constraint imposed during training. W2. The explanation that native-language motion representation confers advantages still lacks controlled validation. The abstract and Section 3.2 argue that representing motion within the VLM’s native language space better preserves pretrained knowledge, thereby improving manipulation generalization. However, in the ablation study of Section 4.4 (Table 2), the flat baseline also outputs motion as text tokens (“a flat motion predictor trained with standard cross-entropy”), and the variants only compare designs within the text representation itself, without contrasting them against a dedicated motion decoder under the same backbone and dataset. In the real-world comparisons (Section 4.1; Figures 5–6), the baselines differ from this work in backbone architecture, training data, and output format; moreover, RoboTracer and VLA-R1 are also based on VLMs, yet this paper still demonstrates clear advantages over them, suggesting that the performance gains may also stem from other design choices, such as the object-centric representation, the training data, or the backbone itself. While the current results support the overall effectiveness of the proposed approach, they are insufficient to isolate and validate the mechanism emphasized by the authors.

Questions for authors
  1. During training, does the inverse task take ground-truth motions or model-predicted motions as input? Is a cycle-consistency loss applied to the round-trip outputs, or is the round-trip generation performed only during inference? Please clarify the training pipeline and the inference-time verification mechanism. (W1)
  2. Could you conduct a comparison with a dedicated motion decoder under the same backbone and dataset? If this is not feasible, could you narrow down the claim that “native language representations better preserve pretrained knowledge”? (W2)
  3. The introduction states that VLA methods typically couple action representations with specific robot embodiments; however, the cited Gemini Robotics 1.5 (Team et al., 2025) has already demonstrated cross-embodiment skill transfer via Motion Transfer. What exactly does “coupling” refer to here, and how does this reference support the subsequent assertion that cross-embodiment applications are limited?
  4. Section 4.1 employs AnyGrasp and cuRobo for grasp estimation and motion planning, but is the environment (e.g., obstacles) modeled during planning? How are potential collisions addressed during execution?
  5. For [u, v, z, rx, ry, rz], which reference frame is used to represent the object’s pose? During deployment, how is the predicted motion transformed into the base coordinate frame of different robots, and which module performs this transformation? When switching to a different robot embodiment, what additional calibration or configuration steps are required?
Reviewer_En2xOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LinguaMotion represents manipulation through structured descriptions of task-relevant objects, their current and goal states, and object-centric position and orientation trajectories. A fine-tuned VLM generates this hierarchy as text, and a grasping and planning stack translates the predicted motion into robot execution. The method adds a numeric value loss and bidirectional instruction–motion training, with a motion round-trip residual used for inference-time verification. Evaluation includes seven task categories on Franka Research 3 and Agilex Piper, alongside trajectory ablations and analyses of general vision-language knowledge retention.

Critical strengths
  1. The object-centric representation is well motivated. Describing which objects matter, their intended states, and how they should move provides a useful interface between instruction understanding and execution. Including orientation as well as translation makes the representation relevant to a broader range of manipulation behaviors. The hierarchy is clear and offers a plausible way to reuse vision-language knowledge while separating motion prediction from robot-specific control.
  2. The real-world evaluation gives the approach practical substance. It covers two platforms and tasks involving attribute grounding, spatial relations, implicit goals, and tool use, with comparisons to several relevant motion- and video-based methods under a shared execution stack. The reported success-rate advantages, component ablations, and knowledge-retention analysis together provide useful evidence for the framework. In particular, the ablations show a clear reduction in orientation error.
Critical weaknesses
  1. The motivation and interpretation of cycle consistency are not fully justified. My main concern is the assumption underlying the motion-to-instruction task. Even given the initial image, a motion trajectory does not uniquely determine the instruction that produced it. Different task intentions can lead to the same trajectory, so recovering a plausible instruction does not necessarily demonstrate that the motion preserves the original intent.
  2. The novelty and the source of the improvements need clearer support. The related-work discussion does not adequately position LinguaMotion within research on language-based action representations. RT-H, CLIP-RT, and LaDA are relevant omissions, while the cited Actions as Language shares the motivation of preserving pretrained VLM capabilities through language-compatible action representations. LinguaMotion’s focus on object-centric pose trajectories is a meaningful distinction, but the paper should explain more precisely what its formulation adds to these established ideas.
  3. The experimental evidence does not clearly isolate the benefit of the proposed motion representation. The reported gains could also reflect the VLM backbone, LoRA adaptation, or additional object and goal supervision. A controlled comparison using the same backbone, training data, and adaptation setup with a continuous trajectory predictor would more directly test the claimed advantage of expressing motion in the VLM’s language space. Moreover, the component ablations primarily measure offline prediction accuracy, leaving their individual contributions to real-world task success unclear.
Questions for authors
  1. How does ambiguous instruction reconstruction affect verification? Can the residual detect a self-consistent trajectory that violates the original instruction?
  2. Is training paired forward/inverse supervision, or an explicit round-trip objective using generated motion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.