acceptodds
Under review as a conference paper at ICLR 2027

Motion Chain of Thought: Disentangling Motion Semantics from Visual Appearance for Robotic Manipulation

Abstract

World Action Models (WAMs) aim to construct a unified architecture capable of understanding world state evolution and guiding action generation. However, existing visual branches focus on predicting static visual observation, rather than reflecting potential transition information that captures the evolution of world states under motion interactions. This leads to representational entanglement between high-level physical condition evolution and low-level action trajectory generation within the Action Model, creating a structural bottleneck while weakening the predictive capability of world evolution modeling for action generation. We propose PILOT (Physical Inference for Latent Optimized Trajectories), whose core Representational Deduction (RD) bridges this gap by integrating motion chain-of-thought (CoT) guidance as a native model capability. Specifically, RD aims to encourage the action branch to explicitly model potential state transition tokens, which are retained as CoT in the reasoning space to guide fine-grained motion trajectory. Experiments demonstrate that RD not only improves the success rate and generalization ability of WAMs in complex robotic manipulation tasks but also enhances the model's physical interpretability by decoupling high-level motion semantics from low-level trajectory details. Furthermore, the abundant state transition supervision signals introduced by RD effectively alleviate the sparse supervision in action generation, enabling it to serve as an efficient few-shot real-robot fine-tuning strategy.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action model, latent action model, robot manipulation, joint embedding predictive architecture

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FwbJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PILOT, a world action model that uses learned Motion-CoT tokens to guide action generation. It combines future-representation supervision with asymmetric attention to separate motion semantics from trajectory refinement, while avoiding explicit future-frame generation during inference. Experiments on simulation benchmarks and real-world robotic tasks evaluate its manipulation performance, robustness, and data efficiency.

Critical strengths
  1. The paper presents a clear motivation supported by qualitative analyses. The method is coherent, well justified, and closely aligned with the identified problem.
  2. The evaluation covers multiple simulation benchmarks and real-world tasks, demonstrating competitive performance. The ablation studies further support the effectiveness of the proposed components.
Critical weaknesses

1.The real-world evidence for generalization and data efficiency is limited. Although eight standard tasks are evaluated, both the generalization and few-shot experiments are restricted to the Pen task. Broader evaluations across different manipulation skills and demonstration budgets are needed to establish whether the reported benefits extend beyond this specific setting.

Questions for authors

1.Could the authors evaluate whether Motion-CoT representations remain consistent under appearance changes that preserve the underlying motion, while remaining sensitive to changes in motion under similar visual conditions? 2. Could the authors provide more direct comparisons with motion-based methods such as HiF-VLA, CoWVLA,. etc? These comparisons would help clarify the advantages of Motion-CoT in representation learning and downstream control.

3 comments

Summary of Responses and Revisions: We sincerely thank you for your thoughtful review and suggestions on real-world generalization, motion-versus-appearance sensitivity, and motion-based VLA comparisons. In response, we added Book experiments under visual perturbations and reduced-data training, provided representation-distance and query-replacement analyses, and compared PILOT with HiF-VLA and CoWVLA in performance and motion-guidance design. We also replaced “few-shot” with “reduced-data” terminology and specified the demonstration counts. We have revised the manuscript based on your feedback and re-uploaded the updated version, with revisions highlighted in blue.

Weakness1: Real-world generalization and data efficiency need broader experiments.

Answer: Thank you for your suggestion. We additionally evaluate generalization and reduced-data performance on the "Book upright-placement" task, which requires grasping, orientation adjustment, and precise placement into a storage rack. For reduced-data training, PILOT and pi0.5 use the same fixed subset of 400 Book trajectories. The Standard column retains the previous full-data results from 50 rollouts. The next three columns evaluate lighting and camera changes (F+C), a changed desk surface/background (Desk), and changed object color (Color), with 20 rollouts per condition. The final column evaluates the reduced-data models under standard visual conditions, also with 20 rollouts.

Book success (%)StandardF+CDeskColorReduced-Data (400 demonstrations)
pi0.58425354020
PILOT7235505545

PILOT improves over pi0.5 by 10/15/15 points under F+C/Desk/Color and by 25 points in the reduced-data setting. It retains 62.5% of its full-data success with 400 demonstrations, compared with 23.8% for pi0.5. Despite pi0.5's higher full-data score, PILOT performs better in every generalization and reduced-data condition. These results demonstrate PILOT's stronger robustness to visual changes and better data efficiency on orientation-sensitive placement. Thank you for this helpful suggestion. We agree that "reduced-data training" is a more accurate description of the 400-demonstration setting than "few-shot learning." We have revised this terminology throughout the manuscript and explicitly stated the demonstration counts to clarify the scope of our data-efficiency evidence.

Thank you for your suggestion. We have added the discussion and additional experiments on Book-task generalization and reduced-data performance to Appendix A.4 (p. 18) of the revised manuscript.

Question1: Is Motion-CoT stable to appearance changes and sensitive to motion changes?

Answer: Thank you for proposing this targeted analysis. Using a frozen PILOT model trained on RoboCasa-GR1, we designed two experiments to quantify how stable Motion-CoT is under appearance changes and how sensitive it is to motion changes.

First, we separately vary motion instructions and object appearance to quantify Motion-CoT's sensitivity to motion semantics and visual appearance using cosine distance. Second, we construct two sets of replacement query keys/values: one generated for the same pick-and-place action type but a visually different target receptacle in the same scene, and another generated for a different action under unchanged object appearance. We substitute each set into the Receiver while keeping its original input fixed. Comparing task success under these interventions tests how action generation responds to changes in the instruction-conditioned query content while the Receiver's own inputs remain fixed.

1. Appearance sensitivity versus motion-instruction sensitivity. For each task, we select 40 episodes with different initial scenes. From each episode, we construct two variants through randomly sampled changes. The motion variant changes the action instruction while preserving the original visual observation and robot state. The appearance variant changes the target object's visual appearance while preserving the motion-instruction, object positions, robot configuration, and camera pose. We run frozen PILOT on the three matched initial inputs and extract their Motion-CoT representations.

We compare each variant with the original using cosine distance between their flattened Motion-CoT representations. Smaller distances indicate greater representational consistency. We average the appearance-induced and motion-induced distances separately across the 40 episodes.

TaskScenesAppearance variant distanceMotion variant distanceRatio
CuttingboardToPot400.13380.42093.15
PlacematToBowl400.05190.14372.77

Appearance variations preserve what the robot should do and leave Motion-CoT comparatively close to its original representation. Changing the requested movement produces a larger shift in both tasks, with a pooled distance 3.04 times that induced by appearance changes. This pattern suggests that Motion-CoT emphasizes information about how the robot should act while remaining relatively stable to visual differences that do not change the task.

2. Replacing query content. Receiver and Donor are two inference passes through the same frozen model. The Receiver and Donor receive identical visual inputs and noisy action tokens. The Receiver uses the original task instruction, while the Donor uses a modified instruction. At each Receiver observation, the Donor uses the identical image and robot state but receives a modified instruction. At each layer, we replace the keys/values of the Receiver’s query tokens with the corresponding keys/values from the Donor. The Receiver retains its original visual-language context, proprioception, and action noise.

The identical-input control gives the Donor the Receiver's original instruction and reproduces the original actions. We then compare two content interventions:

  • Same action type, different target receptacle: Both instructions require picking and placing the same manipulated object, but refer to visually different target receptacles in the same scene. For example, the Receiver places the eggplant in the pot, whereas the Donor places it in the bowl. The visual observation is unchanged; the intervention changes which receptacle the instruction designates as the destination.
  • Same object, different action type: Both instructions refer to the same object, but require incompatible operations. For example, the Receiver places the eggplant in the pot, while the Donor cuts the eggplant with a knife.

Each condition uses 12 paired scenes per task. Success is measured against the Receiver's original instruction.

TaskOriginalSame action type, different destinationSame object, different action type
CuttingboardToPot8/126/120/12
TrayToPlate10/1210/121/12

Preserving the action type while changing the destination retains substantial task performance, with success changing from 66.7% to 50.0% on CuttingboardToPot and remaining at 83.3% on TrayToPlate. Changing the action type while retaining the same object reduces success to 0% and 8.3%, respectively. Thus, incompatible action content in the queries disrupts execution much more strongly than a conflicting destination, even though the Receiver retains its original observation and instruction. These results show that Motion-CoT encodes how to act, rather than merely what objects look like, providing motion-specific guidance for action generation.

Question2: How does PILOT compare with HiF-VLA and CoWVLA in downstream control?

Answer: Thank you for recommending these motion-based methods. We add their reported results on standard LIBERO:

Success (%)SpatialObjectGoalLongAverage
HiF-VLA98.899.497.496.498.0
CoWVLA97.297.894.692.895.6
PILOT97.199.298.197.297.9

PILOT achieves comparable overall performance to HiF-VLA, 97.9% versus 98.0%, with higher success on Goal and Long by 0.7 and 0.8 percentage points. Compared with CoWVLA, PILOT improves the average by 2.3 points, with gains of 1.4/3.5/4.4 points on Object/Goal/Long. Its strongest results are therefore on goal-conditioned and long-horizon manipulation, where coordinating motion intent with trajectory execution is especially important.

The methods also differ in how motion guidance is learned. HiF-VLA integrates motion-vector representations with action learning, while CoWVLA learns latent motion from a video representation factorized into structure and motion. PILOT learns its queries inside the flow-based action model and supervises them as conditions for current-to-future representation prediction. This lets the same action model learn predictive motion guidance and fine-grained trajectory generation, with asymmetric attention keeping the guidance independent of action sampling noise. We have included these results and the method comparison in the revised manuscript.

Thank you for your suggestion. We have added the comparative results and discussion of HiF-VLA and CoWVLA to Appendix A.5 (p. 19) of the revised manuscript.

References

  • [HiF-VLA] Lin, Minghui, et al. "HiF-VLA: Hindsight, Insight and Foresight through Motion Representation for Vision-Language-Action Models." arXiv preprint arXiv:2512.09928, 2025. Table 4.
  • [CoWVLA] Yang, Fuxiang, et al. "Chain of World: World Model Thinking in Latent Motion." arXiv preprint arXiv:2603.03195, 2026. Table 1.
Reviewer_n9NtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PILOT, a robotic manipulation framework that combines a Cosmos-Predict2.5 backbone with a flow-matching Action-Perceiver and a Representational Deduction (RD) objective. Sixty-four learned queries extract representations from the current observation, instruction, and proprioceptive state. An asymmetric attention pattern allows action tokens to attend to these queries while preventing action noise and flow-time conditioning from entering the learnable query. RD supervises the final query representations through a trainable Causal Dynamics Engine (CDE) initialized from the V-JEPA 2-AC predictor, using frozen V-JEPA features of future observations as targets (Assran et al., 2025). The training process incorporates action prediction as well as auxiliary prediction of latent representations for future video frames. During deployment, only the encoding of the current context and the generation of actions are required. this work evaluates manipulation performance across LIBERO, LIBERO-Plus, RoboCasa-GR1, and eight real-world tasks, supplemented by component ablation studies, visualizations of representations, and comparisons of inference latency.

Ref

Critical strengths
  1. A well-motivated integration of predictive representation learning with efficient action generation. The specialized 'motion-cot' proposed in this study provides a concrete architectural mechanism that allows predictive representations, which conditioned on the current context and state to remain invariant to action noise. Future observations serve solely to supervise this branch and are not fed as inputs into the policy model used for actual deployment. Since the V-JEPA encoder and the conditional decoder/estimator are not required during inference, it is fully compatible with the requirements for efficient deployment. The design: "prediction only during training" is supported by existing research, which indicates that video co-training can effectively enhance control performance, even when no "imagination" (predictive simulation) is performed during the testing time (Yuan et al., 2026). In a direct comparison with the Cosmos system (Table 8), the proposed approach reduces latency from 1,329.95 ms to 111.60 ms on a single H20 GPU, while increasing the success rate on RoboCasa tasks from 56.5% to 58.3%. By combining explicit constraints on information flow with practical advantages for deployment, this work makes a substantial contribution.
  2. The empirical support extends beyond a single benchmark and includes meaningful real-world gains. PILOT achieves a success rate of 97.9% on LIBERO (Liu et al., 2023), 80.3% on LIBERO-Plus, and 58.3% on RoboCasa-GR1. Moreover, across eight real-world tasks, PILOT attains an average success rate of 82.5%, compared to 73.8% for the previous best-performing baseline model. Table 5 also provides compelling evidence regarding the framework’s internal components: incorporating RD/CDE into a query-based configuration boosts the success rate on LIBERO tasks from 94.5% to 96.1%, while the further addition of a decoupled attention mechanism raises it to 96.9%. These results validate the practical efficacy of the training strategy and architectural design employed in this work. Given these outcomes, combined with the framework's advantages in deployment, even with some remaining limitations in the verification of specific mechanisms, I believe these contributions are sufficient for a "weak acceptance".

Ref

Critical weaknesses
  1. The central interpretation of Motion-CoT is less established. Actions have direct access to visual context; as noted in Appendix B.4, the query path and the action path share attention mechanisms and feed-forward network parameters. Consequently, in the ablation studies, it is hard to spot whether the policy leverages "predictive query" or benefits from "auxiliary supervision via shared representations and parameters." Similarly, t-SNE and PCA visualizations fail to demonstrate that the query content actually governs behavior, nor do they prove that it has been effectively decoupled from visual appearance.
  2. The precision and scope of some empirical conclusions remain limited. Neither the main results tables nor the ablation studies report the variability (such as confidence intervals). Furthermore, as noted in Appendix B.11, the baseline models cited in the paper adhere to their original training and evaluation protocols. Consequently, marginal differences such as the 97.9% versus 97.7% performance observed on LIBERO tasks cannot serve as robust evidence for establishing a reliable ranking of the models. Besides, all quantitative experiments concerning "real-world generalization" and "reduced-data" scenarios are confined exclusively to the "Pen" task. Moreover, given the 3,965 trajectories for the "Pen" task documented in Table 10, the "10% data" setting actually corresponds to approximately 400 demonstrations. While this demonstrates that the model can adapt to a specific task using a limited amount of data, the scope of this evidence is too narrow to support the broader claim that the model possesses "few-shot adaptation capabilities across a diverse range of manipulation tasks."
Questions for authors
  1. Targeted validation of the proposed motion cot mechanism. Could an intervention be designed to verify that the action generation process genuinely utilizes the content within the "motion-queries"? For instance, one could block the relevant "query-to-action" branch or swap intermediate query representations between matched observations (corresponding to different targets). As noted in Appendix B.4, the action generation process reads intermediate query states, while the final Motion-CoT output is generated alongside the action output. Therefore, such an intervention should target these intermediate pathways, which accounting for the indirect "query → state token → action" sequence, while ensuring that proprioceptive information remains intact. This would help clarify whether Motion-CoT truly plays an active role in the control process, rather than merely serving as an auxiliary benefit derived from the training regimen.

  2. Targeted validation of the predictive content. How much does the CDE's future-feature prediction improve when conditioned on Motion-CoT, compared with copying current features or predicting from current features and robot state alone? A quantitative comparison would help establish whether the learned queries provide state-transition information beyond the current scene. This is particularly relevant given related future-state representation learning (Sun et al., 2026) and the effectiveness of training-time video supervision without test-time prediction (Yuan et al., 2026).

  3. Clarification of experimental setting and reduced-data training. How many independent training runs are the results in Table 5 based on? For the reported performance gains attributed to individual components, is there an accompanying assessment of variability (such as variance or confidence intervals)? Regarding the "Pen 10%" experiment, please clarify the exact demonstration count, model initialization, whether the data restriction applies to all Pen-specific training stages, and whether the compared methods use the same training subset. These details are crucial for evaluating the magnitude of the reported performance improvements and for interpreting "data efficiency."

Ref

6 comments

Summary of Responses and Revisions: We sincerely appreciate your rigorous review and targeted questions about Motion-CoT’s contributions, predictive information, and experimental reliability. In response, we added attention measurements, pathway-blocking and query-replacement experiments, frozen-CDE analyses, and predictive-supervision ablations. We also reported evaluation-seed variability, distinguished it from training variability, clarified the reduced-data protocols, and extended real-world evaluation to Book upright placement. These analyses clarify how the trained policy uses Motion-CoT and the scope of our evidence. We have revised the manuscript based on your feedback and re-uploaded the updated version, with revisions highlighted in blue.

Question1 & Weakness1: Quantifying the contributions of Motion-CoT and world-model context to action generation.

Answer: Thank you for proposing this targeted analysis. Using frozen PILOT weights trained on RoboCasa-GR1, we designed three experiments to quantify the contributions of Motion-CoT and world-model context to action generation. First, we measure attention across all action-model layers to quantify how action tokens distribute attention between motion queries and world-model context. Second, we separately block action tokens from reading Motion-CoT or world-model context, while preserving proprioceptive input, to evaluate how well the remaining pathway supports successful execution. Third, we construct two sets of replacement query keys/values: one generated for the same pick-and-place action type but a visually different target receptacle in the same scene, and another generated for a different action under unchanged object appearance. We substitute each set into the Receiver while keeping its original input fixed. Comparing task success under these interventions tests how action generation responds to changes in the instruction-conditioned query content while the Receiver's own inputs remain fixed.

1. Measuring attention. Motion queries (Q), world-model context (C), state (S), and actions (A) contain 64/196/1/16 tokens. For each action row, we apply softmax to the scaled query-key logits over all 277 source tokens, then sum the weights within each group. We average these group totals over 16 action rows, 16 heads, 16 layers, 20 flow steps, and 60 policy calls per trajectory, then average trajectories equally. Thus, a Q mass of 40% means that motion queries receive 40% of action attention. Q/token and C/token divide the respective masses by 64 and 196, and the final column reports their ratio.

TaskTrajectoriesQ mass (%)C mass (%)S mass (%)A mass (%)Q/token (%)C/token (%)Q/C per token
CuttingboardToBasket1039.4453.711.145.710.6160.2742.248
PlacematToPlate1040.3752.781.175.680.6310.2692.343
PlateToBowl1039.9353.131.145.800.6240.2712.302
CuttingboardToPan1039.8753.291.145.710.6230.2722.291
TrayToCardboardbox1039.7753.341.135.760.6210.2722.283

Across five tasks with 10 trajectories per task, the 64 motion-query tokens receive 39.44-40.37% of total action-token attention, compared with 52.78-53.71% for the 196 world-model context tokens. The corresponding per-token weights are 0.616-0.631% and 0.269-0.274%, giving motion queries 2.248-2.343 times the attention per token of context. Action tokens thus allocate substantial attention to both sources, with consistently greater per-token attention to Motion-CoT.

2. Separately block information. The ''w/o query-kv'' variant prevents action tokens from reading motion-query keys/values while retaining their world-model context. The ''w/o direct wm-context'' variant prevents action tokens from directly reading world-model context while retaining their motion-query input. Both variants preserve raw proprioception and action self-attention. We also block the corresponding reads by state tokens in every layer, which closes the indirect query/context-to-state-to-action route.

TaskQuery-batch originalW/o query-kvContext-batch originalW/o direct wm-context
CuttingboardToPot8/121/1211/120/12
TrayToPlate10/120/128/120/12

Removing query access changes success from 66.7% to 8.3% on CuttingboardToPot and from 83.3% to 0% on TrayToPlate. Removing direct context access changes success from 91.7% to 0% and from 66.7% to 0%, respectively. Keeping visual-language context does not recover performance when query access is removed. Conversely, keeping context-conditioned queries does not recover performance when direct context access is removed. Successful execution in the trained policy depends on both reading pathways.

(This response continues in Part 2.)

(Part 2: Continuing our response to Question 1 & Weakness 1.)

3. Replacing query content. Receiver and Donor are two inference passes through the same frozen model. The Receiver and Donor receive identical visual inputs and noisy action tokens. The Receiver uses the original task instruction, while the Donor uses a modified instruction. At each Receiver observation, the Donor uses the identical image and robot state but receives a modified instruction. At each layer, we replace the keys/values of the Receiver’s query tokens with the corresponding keys/values from the Donor. The Receiver retains its original visual-language context, proprioception, and action noise.

The identical-input control gives the Donor the Receiver's original instruction and reproduces the original actions. We then compare two content interventions:

  • Same action type, different target receptacle: Both instructions require picking and placing the same manipulated object, but refer to visually different target receptacles in the same scene. For example, the Receiver places the eggplant in the pot, whereas the Donor places it in the bowl. The visual observation is unchanged; the intervention changes which receptacle the instruction designates as the destination.
  • Same object, different action type: Both instructions refer to the same object, but require incompatible operations. For example, the Receiver places the eggplant in the pot, while the Donor cuts the eggplant with a knife.

Each condition uses 12 paired scenes per task. Success is measured against the Receiver's original instruction.

TaskOriginalSame action type, different destinationSame object, different action type
CuttingboardToPot8/126/120/12
TrayToPlate10/1210/121/12

Preserving the action type while changing the destination retains substantial task performance, with success changing from 66.7% to 50.0% on CuttingboardToPot and remaining at 83.3% on TrayToPlate. Changing the action type while retaining the same object reduces success to 0% and 8.3%, respectively. Thus, incompatible action content in the queries disrupts execution much more strongly than a conflicting destination, even though the Receiver retains its original observation and instruction. These results show that Motion-CoT encodes how to act, rather than merely what objects look like, providing motion-specific guidance for action generation.

Together, these experiments show that PILOT uses both world-model context and Motion-CoT to generate actions. Blocking either source reduces task success. Replacing Motion-CoT with queries for a different action disrupts execution despite identical visual inputs. In contrast, replacing Motion-CoT with queries for the same pick-and-place action type but a different target receptacle causes only a slight drop in task performance. These results suggest that Motion-CoT is more sensitive to motion semantics associated with dynamic changes than to static visual appearance.

Thank you for your suggestion. We have added the discussion and additional experiments on the contributions of Motion-CoT and world-model context to Appendix A.1 (pp. 15–16) of the revised manuscript.

Question2: What predictive information does Motion-CoT add beyond the current scene and robot state?

Answer: Thank you for this question. We evaluate both the dependence of the pretrained CDE on Motion-CoT and the effect of training with different predictive inputs.

1. Representation analysis with the frozen pretrained model on RoboCasa. For each task, we select 5 reference trajectories and 25 evaluation trajectories. From each trajectory, we sample four times t∈{0,120,360,600}t\in\{0,120,360,600\}. Each time defines a current-future observation pair (It,It+16)(I_t,I_{t+16}), giving 20 reference pairs and 100 evaluation pairs per task. The frozen encoder produces zt=E(It)z_t=E(I_t) and zt+16=E(It+16)z_{t+16}=E(I_{t+16}). The frozen policy produces mtm_t from the current observation, instruction, and proprioceptive state sts_t.

We compare four variants:

  • Original: G(zt,st,mt)G(z_t,s_t,m_t) uses the Motion-CoT produced for that specific observation.
  • Zero query: G(zt,st,0)G(z_t,s_t,0) replaces Motion-CoT with zeros while retaining current visual features and robot state.
  • Reference mean: G(zt,st,mˉtask)G(z_t,s_t,\bar m_{\mathrm{task}}) uses one fixed representation for the task, obtained by averaging the 20 reference Motion-CoTs at each corresponding slot and feature dimension.
  • Copy current: z^t+16=zt\hat z_{t+16}=z_t directly uses current features as the future prediction.
Future-feature SmoothL1 errorOriginalZero queryReference meanCopy current
CuttingboardToPot0.0154 +/- 0.00120.1042 +/- 0.00750.0261 +/- 0.00230.2082 +/- 0.0161
TrayToPlate0.0151 +/- 0.00150.0953 +/- 0.00820.0228 +/- 0.00260.2079 +/- 0.0227

Compared with "Zero query", "Reference mean" substantially improves prediction, suggesting that the motion pattern shared across episodes provides a useful task-level prior for future-state prediction. However, "Original" removes a further 33.8–41.0%, showing that episode-specific conditioning is more than a minor refinement. A shared motion pattern must still be adapted to the current scene and robot configuration. Meanwhile, "Zero query" outperforms "Copy current" because the current visual features and robot state already provide cues about future evolution. Together, these results suggest that Motion-CoT captures not only the motion pattern shared across a task, but also how that motion should unfold in the current scene. This episode-specific guidance helps the CDE predict state changes more accurately than a fixed task prior alone.

2. Training with different predictive inputs. All three variants retain query-tokens as context for action generation. They differ in whether the CDE reads those queries and whether CDE-loss trains query-token:

  • Vanilla Query Token: Action generation reads the query-tokens, which are trained only through action loss. A separate CDE predicts future features from current features and robot state as z^t+Δ=Gϕ(zt,st)\hat z_{t+\Delta}=G_\phi(z_t,s_t). It neither reads nor supervises the query-tokens.
  • Stop Gradient: Both action generation and the CDE read query-tokens. The CDE predicts z^t+Δ=Gϕ(zt,st,stopgrad⁡(mt))\hat z_{t+\Delta}=G_\phi(z_t,s_t,\operatorname{stopgrad}(m_t)), where mt=Fθ(current context,st)m_t=F_\theta(\mathrm{current\ context},s_t). Stop-gradient prevents CDE-loss gradients from flowing into the query tokens, which are trained only through action loss.
  • MotionCoT (PILOT): Both action generation and the CDE read the query-tokens, and query-tokens are jointly trained by action loss and CDE loss. The prediction is z^t+Δ=Gϕ(zt,st,mt)\hat z_{t+\Delta}=G_\phi(z_t,s_t,m_t).
MetricVanilla Query TokenStop GradientMotionCoT
CDE reconstruction loss0.0253 +/- 0.0050.0191 +/- 0.0030.0185 +/- 0.002
LIBERO success (%)94.594.597.9

Action-trained query-tokens reduce reconstruction loss of CDE by 24.5% relative to current features and state alone, showing that action learning organizes information useful for predicting scene evolution. With RD supervision, reconstruction loss falls a further 3.1%, while control success increases by 3.4 percentage points. The key result is that Motion-CoT's value is not captured by prediction error alone: joint predictive and action supervision makes the representation useful for both anticipating state changes and generating the trajectories that realize them. This is the motivation for learning high-level representations and low-level actions inside the same action model.

Thank you for your suggestion. We have added the discussion and additional experiments on Motion-CoT's predictive information and the effects of predictive supervision to Appendix A.2 (pp. 16–17) of the revised manuscript.

Reviewer_Si55OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PILOT (Physical Inference for Latent Optimized Trajectories) enhances World Action Models (WAMs) by integrating motion chain-of-thought guidance to improve state transition modeling and action generation. Its Representational Deduction decouples high-level motion semantics from low-level trajectory details, boosting success rates, generalization in robotic tasks, physical interpretability, and enabling efficient few-shot fine-tuning with abundant supervision signals.

Critical strengths
  • The proposed method is relatively effective, with validation results from several datasets and real device experiments.
  • The demo video in the supplementary materials demonstrates strong model generalization ability.
Critical weaknesses
  • It is unclear whether the motionCoT proposed in the article is a core factor driving improvement. Although Table 5 provides experimental results from an ablation study, I am filled with doubts about these results. First, what is the baseline for this work? The method on Libero with a success rate of 91.3 has too low performance; most VLA methods have already exceeded 95%. Conducting ablation studies on such a low baseline seems unreasonable. Second, Libero's ablation study can only demonstrate fitting ability, not generalization ability. Third, there is a lack of comparison between motionCoT and other CoT methods in VLA. Prediction and CoT are very common in VLA; to prove the effectiveness of this method, it must be thoroughly compared with visual CoT, language CoT, and other world model approaches.

  • The experimental performance in this paper is not particularly outstanding; many recent works are not included for comparison—for example, pi0.5, Gr00t N1.7, being H0.7—and there is also a lack of experimental results on mainstream benchmarks like RobotWin2.0.

  • Recently, many papers have claimed to combine world models, vision-language models, and action models to improve task success rates. This article is one such work. A technical discussion is needed to clarify the fundamental differences between the proposed method in this paper and those approaches.

Questions for authors

The proposed module motionCoT is more like a visual feature alignment rather than directly representing in the action space. Therefore, why perform such alignment at the action head?

5 comments

Summary of Responses and Revisions: We sincerely thank you for your careful review and constructive suggestions regarding baseline strength, generalization, broader comparisons, and PILOT’s distinction from JEPA-based methods. In response, we clarified the internal baseline, extended component ablations to LIBERO-Plus perturbations, and added comparisons with ECoT, CoT-VLA, StarVLA-alpha, TwinBrainVLA, and ABot-M0. We also clarified how PILOT jointly learns transition-conditioning queries and trajectories inside the action expert. We have revised the manuscript based on your feedback and re-uploaded the updated version, with revisions highlighted in blue.

Weakness1: The 91.3% internal baseline is substantially weaker than existing VLAs/WAMs.

Answer: Thank you for raising this question. The 91.3% baseline is obtained by removing PILOT's proposed mechanisms from our architecture, rather than adopting an existing method. Crucially, this variant also removes future-prediction supervision, a central objective of WAMs (which explains the low starting performance noted in the review). Restoring this objective raises success to 93.6%, slightly below the cited 95% level. Without the specialized designs of existing VLA/WAM systems, this configuration provides a vanilla-WAM reference for evaluating PILOT's components.

The ablations compare backbone future prediction with state-transition learning inside the action model. The former reaches 93.6%, whereas complete Motion-CoT without backbone future prediction reaches 96.9%. Combining both reaches 97.9%, a 4.3-point gain over the future-supervised baseline.

Weakness2: Generalization of PILOT's components under distribution shifts of LIBERO-Plus.

Answer: Thank you for this suggestion. We extended the five ablated configurations and full PILOT to the Camera, Robot, and Language perturbations of LIBERO-Plus, as shown below. Following the benchmark protocols, these policies are trained on standard LIBERO and evaluated on LIBERO-Plus without fine-tuning on the perturbed conditions. The "MotionCoT ✓, RD/CDE ×" configuration indicates that the query tokens receive supervision solely through gradients from the action loss, whereas the "MotionCoT ✓, RD/CDE ✓" configuration indicates that they receive supervision through gradients from both the action loss and the CDE reconstruction loss.

Fut.Pred.MotionCoTRDCDEDecoupleLIBEROCameraRobotLanguage
××---91.344.345.660.1
✓×---93.660.749.871.7
×✓×××94.549.147.562.4
×✓✓✓×96.165.954.377.9
×✓✓✓✓96.969.362.680.5
✓✓✓✓✓97.976.070.284.1

Future prediction alone improves Camera/Robot/Language by 16.4/4.2/11.6 percentage points, averaging 10.7 points. Adding queries without RD/CDE yields only 4.8/1.9/2.3 points, averaging 3.0 points. In contrast, the complete Motion-CoT mechanism without future-frame supervision improves these categories by 25.0/17.0/20.4 points, averaging 20.8 points. Thus, the generalization benefit comes primarily from how the queries are supervised, rather than simply adding query tokens. Specifically, adding RD/CDE supervision to the query-only configuration improves Camera/Robot/Language performance by 16.8/6.8/15.5 percentage points, averaging 13.0 points. Decoupled attention adds another 3.4/8.3/2.6 points, with the largest gain under Robot perturbations. Finally, adding future-frame supervision to the complete Motion-CoT mechanism yields a further 6.7/7.6/3.6-point improvement, supporting their complementary benefits under the tested distribution shifts. We have added these expanded results to the Table 5 in the revised manuscript.

Weakness3: Comparisons with textual and visual CoT methods.

Answer: Thank you for suggesting these comparisons. We add ECoT and CoT-VLA to compare explicit textual reasoning, visual subgoal prediction, and PILOT's latent motion guidance. ECoT generates textual reasoning about plans, subtasks, motions, and visually grounded quantities before predicting actions. CoT-VLA autoregressively predicts future images as visual subgoals, then generates actions to reach them. ECoT's LIBERO-Plus results below are taken from ERVLA, whereas CoT-VLA's standard-LIBERO results are taken from its original paper.

Standard LIBERO success (%)SpatialObjectGoalLong
CoT-VLA (Visual CoT)87.591.687.669.0
PILOT (Motion-CoT)97.199.298.197.2

PILOT improves over CoT-VLA by 9.6/7.6/10.5/28.2 percentage points on Spatial/Object/Goal/Long. The largest gain is on Long, where successful execution requires maintaining task progress across multiple manipulation steps. PILOT learns motion guidance within the action model and retains it throughout trajectory refinement, directly targeting this requirement.

LIBERO-Plus success (%)CameraRobotLanguageLightBackgroundNoiseLayoutAverage
ECoT (Textual CoT, reported by ERVLA)0.326.840.242.616.410.236.924.3
PILOT (Motion-CoT)76.070.284.192.885.178.382.080.3

PILOT improves over ECoT in every category. The gains are 75.7 points for Camera, 68.7 for Background, and 68.1 for Noise, showing strong robustness to changes in the observed scene. Gains of 43.4/43.9/45.1 points on Robot/Language/Layout extend this advantage to changed robot configurations, instructions, and spatial arrangements. PILOT also improves Light by 50.2 points. Its updated overall success is 80.3%, using the manuscript's evaluation-condition weights, compared with ECoT's reported 24.3%. Together, these comparisons demonstrate PILOT's strong downstream performance on standard LIBERO and under LIBERO-Plus perturbations.

We have added the results of ECoT and CoT-VLA to Tables 1 and 2 in the revised manuscript.

References

  • [ECoT] Zawalski, Michal, et al. "Robotic Control via Embodied Chain-of-Thought Reasoning." arXiv preprint arXiv:2407.08693, 2024.
  • [ERVLA] Sun, Nan, et al. "Revisiting Embodied Chain-of-Thought for Generalizable Robot Manipulation." arXiv preprint arXiv:2606.03784, 2026.
  • [CoT-VLA] Zhao, Qingqing, et al. "CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.

Weakness4: Comparisons with recent strong baselines.

Answer: Thank you for recommending broader comparisons. We add recent StarVLA-alpha, TwinBrainVLA, and ABot-M0 on the RoboCasa-GR1. As shown below, we group all 24 tasks into five categories and average the published task-level results within each category. PILOT's updated evaluation reaches 59.75%, improving on the manuscript's 58.3%. The corresponding checkpoint and training/evaluation code are available through our anonymous project website. All Baseline results are from their respective manuscripts.

MethodPnP (6)Cuttingboard (5)Placemat (4)Tray (5)Plate (4)Average
StarVLA-alpha Specialist49.0055.4046.7553.8066.0053.79
StarVLA-alpha Generalist54.0057.2059.0053.8065.0057.29
TwinBrainVLA Qwen2.5-VL56.6754.0051.2546.4059.5053.54
TwinBrainVLA Qwen3-VL56.6759.2046.5049.6060.0054.58
ABot-M061.6761.6051.0056.8058.0058.25
PILOT47.6770.0054.5073.2053.5059.75

PILOT has the highest overall success among these models and is particularly strong on Cuttingboard and Tray. Its 70.00% and 73.20% exceed the strongest added baseline in each category by 8.40 and 16.40 points, respectively. These categories require grasping novel objects from varied source locations and transporting them to different receptacles, combining target-conditioned transfer with precise grasping and placement. Being-H0.7 reports 49.2% overall on RoboCasa-GR1, compared with PILOT's 59.75%. For the other specifically requested baselines, the manuscript already reports that PI0.5 underperforms PILOT on LIBERO (96.9% vs. 97.9%), and pi0.5/GR00T-N1.7 at 73.8%/55.5% across eight real-world tasks versus PILOT's 82.5%.

RoboTwin 2.0 is a valuable additional benchmark. However, training and evaluating another WAM at this scale exceeds our available GPU budget. We selected RoboCasa-GR1 because its 24-task tabletop suite combines a dual-arm humanoid embodiment, dexterous hands, novel-object transfers, and diverse source-target relationships, complementing the Franka-based LIBERO evaluation. We will extend PILOT's evaluation to RoboTwin 2.0 in future work.

We have added the Robocasa results of StarVLA-alpha, TwinBrainVLA and ABot-M0 to Table 3 in the revised manuscript.

References

  • [StarVLA-alpha] Ye, Jinhui, et al. "StarVLA-alpha: Reducing Complexity in Vision-Language-Action Systems." arXiv preprint arXiv:2604.11757, 2026.
  • [TwinBrainVLA] Yu, Bin, et al. "TwinBrainVLA: Unleashing the Potential of Generalist VLMs for Embodied Tasks via Asymmetric Mixture-of-Transformers." arXiv preprint arXiv:2601.14133, 2026.
  • [ABot-M0] AMAP CV Lab. "ABot-M0: VLA Foundation Model for Robotic Manipulation with Action Manifold Learning." arXiv preprint arXiv:2602.11236, 2026.
  • [Being-H0.7] BeingBeyond Team. "Being-H0.7: A Latent World-Action Model from Egocentric Videos." arXiv preprint arXiv:2605.00078, 2026.

Comments on the paper

1 comment

Dear Reviewers,

Please take time to review the responses from the authors and engage with them for further questions and comments if any.

Time left is not much and please do shortly.

Thanks

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.