acceptodds
Under review as a conference paper at ICLR 2027

Adapting Video Models For Robot Control via Motion-guided Asymmetric Prediction

Abstract

World Action Models (WAMs) leverage pretrained video generative models for robotic control by jointly predicting future visual observations and actions. The visual prediction objective plays a key role in adapting pretrained video representations to the target robot domain. Yet standard visual prediction inherited from video pretraining treats all visual regions uniformly, despite their differing relevance to robotic control. We therefore investigate the spatial allocation of visual supervision in WAMs and show that focusing supervision on dynamic regions can substantially improve policy performance. Building on this finding, we introduce Motion-Guided Asymmetric Prediction (MoGAP), which augments WAM training with motion-focused visual prediction. MoGAP allocates noise asymmetrically across regions, training the model to reconstruct more heavily corrupted dynamic regions using less corrupted static regions as context. MoGAP is trained alongside standard video–action prediction through a training-only auxiliary prediction path within the shared video diffusion backbone. It uses motion cues derived directly from training videos, requiring neither an external teacher nor additional modalities. Across robot manipulation benchmarks and WAM architectures, MoGAP consistently improves policy performance, increasing success rates on the GR-1 tabletop benchmark from 38.8% to 47.5% for Fast-WAM (Wan2.2-5B) and from 52.3% to 62.3% for Cosmos3-Nano. We further demonstrate MoGAP’s effectiveness in physical deployment through evaluation on a real humanoid robot.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World action models, Robotic foundation models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_48hCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Motion-Guided Asymmetric Prediction (MoGAP) to improve visual prediction supervision in World Action Models (WAMs). The method identifies dynamic regions using temporal differences between video latents, applies stronger noise to dynamic regions and weaker noise to static regions, and trains the model to predict dynamic regions using relatively clean static context. MoGAP is incorporated into existing WAMs through a training-only auxiliary prediction path without introducing additional inference overhead. Experiments on RoboCasa GR-1 Tabletop, RoboCasa-Kitchen, and real-world robotic manipulation tasks demonstrate improved task success rates and better adaptation to target object relocation during execution.

Critical strengths

(1) Clear research motivation. Rather than simply introducing additional visual supervision, the paper investigates how the spatial allocation of supervision influences robotic control. The motivating study provides empirical support for this research direction.

(2) Meaningful performance improvements. MoGAP improves the average success rate of Cosmos3 from 52.3% to 62.3% on GR-1 Tabletop and also improves performance when applied to Fast-WAM. Comparisons with Geometry Forcing, AGRA, TSA, and VideoREPA further strengthen the empirical evaluation.

(3) Well-targeted ablation studies. The authors examine the auxiliary prediction path, noise asymmetry, and the direction of asymmetric noise allocation. A compute-matched comparison is also provided. These experiments directly support several key components of the proposed method.

Critical weaknesses

(1) Limited methodological novelty. The main technical contribution is concentrated in Eqs. (8)–(11), which adjust noise levels, distinguish dynamic and static regions through binary thresholding, apply asymmetric noise, and introduce a region-selective flow-matching objective. While this constitutes a meaningful training modification, region-wise noise allocation and flow matching have already been investigated in prior work. Therefore, the methodological novelty appears moderate rather than substantial.

(2) Simplistic dynamic-region selection with unclear reliability. Equation (9) uses the spatial mean motion score within each temporal slice as a threshold to distinguish dynamic and static regions. However, motion does not necessarily indicate task relevance, while subtle task-relevant movements may not be reliably identified. Although Table 4a provides evidence that the motion scores distinguish moving regions, the sensitivity of this thresholding strategy to local visual noise, camera perturbations, and threshold selection has not been sufficiently investigated.

(3) Insufficient disentanglement of asymmetric noise allocation and other training components. The dynamic-region loss reweighting approach in Section 2.2 already improves the Cosmos3 success rate from 52.3% to 57.7%, while MoGAP further improves it to 62.3%. Although Table 3 includes relevant ablations, the two approaches also differ in their use of an auxiliary prediction path. Consequently, the additional 4.6 percentage-point improvement cannot be directly attributed to asymmetric noise allocation alone.

(4) Incomplete explanation of the underlying mechanism. Figure 8 and Table 4 show that MoGAP encourages action tokens to attend more strongly to the robot arm and manipulated objects. However, these observations mainly provide correlational evidence. They do not fully explain why this change leads to improved control performance or why asymmetric noise allocation provides greater benefits than dynamic-region loss reweighting alone.

(5) Unclear generality under complex observation conditions. In RoboCasa-Kitchen, the authors introduce a special treatment for the wrist-camera view because camera motion can dominate the latent motion scores. This raises concerns about the generality of the proposed motion-guided region selection strategy under moving-camera conditions and more complex visual environments.

(6) Limited analysis of experimental variability. Although the paper includes extensive simulated and real-world evaluations, it does not sufficiently report performance variability across independent training seeds. Additional repeated training runs would help establish the consistency of the reported improvements, particularly in real-world generalization settings.

Questions for authors

(1) Methodological novelty. Could the authors clarify the technical differences between MoGAP and existing token-wise noise assignment methods? A direct comparison with alternative noise allocation strategies under the same WAM architecture and training conditions would help establish the independent contribution of motion-guided asymmetric prediction.

(2) Reliability of dynamic-region selection. Could the authors investigate the sensitivity of the thresholding strategy in Eq. (9)? Comparisons among mean thresholding, Top-K selection, and continuous soft weighting, together with evaluations under local visual noise and camera perturbations, would help assess the reliability of dynamic-region identification and its effect on downstream control performance.

(3) Sources of performance improvement. Could the authors further disentangle the contributions of dynamic-region loss reweighting, asymmetric noise allocation, and the auxiliary prediction path? For example, applying dynamic-region loss reweighting and asymmetric noise prediction within the same auxiliary architecture and under matched computational budgets would help clarify the source of the additional improvement over the motivating study in Section 2.2.

(4) Underlying mechanism. Beyond attention visualizations, could the authors provide more direct evidence explaining how improved dynamic-region prediction affects action generation? For example, analyzing the relationship between dynamic-region prediction errors and task success rates, or evaluating action sensitivity under controlled perturbations of task-relevant regions, could strengthen the mechanistic interpretation.

(5) Generalization and experimental reliability. What motivates assigning all wrist-camera tokens to the high-noise group? Could the authors provide an ablation of this design choice and investigate whether motion-guided region selection remains effective under substantial camera motion? Additionally, reporting results across multiple independent training seeds would help establish the consistency of the performance gains.

4 comments

Dear reviewer 48hC,

We sincerely appreciate your efforts and insightful comments in reviewing our manuscript. Below, we address each comment in detail. In the revised manuscript, we mark our major revisions as “blue”.


[W1, Q1] Comparison to alternative noise allocation strategies.

MoGAP introduces token-wise noise assignment for the spatial asymmetry of dynamic and static regions, motivated by our finding that emphasizing dynamic regions in visual supervision improves WAM manipulation performance. Existing approaches such as Diffusion Forcing [1] and Self-Flow [2] employ frame-wise sequential noise allocation and randomized noise assignment, respectively. We conducted additional comparisons using these noise allocation strategies in the same training framework proposed at MoGAP. While alternative noise allocation strategies also improve performance over the baseline, motion-guided regional allocation achieves the highest success rate, suggesting its effectiveness for WAM manipulation. We have also incorporated these results into the revised manuscript (Table 9 in Appendix E.3). Thank you again for the meaningful suggestion.

\begin{array}{lc} \hline \text{Method} & \text{GR-1 Tabletop} \newline \hline \text{Baseline} & 52.3\% \newline \text{Self-Flow (random allocation)} & 53.8\% \newline \text{Diffusion Forcing (framewise allocation)} & 56.1\% \newline \textbf{MoGAP (regional allocation)} & \mathbf{62.3\%} \newline \hline \end{array}

[1] Chen et al., Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion, NeurIPS 2024

[2] Chefer et al., Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis, ICML 2026


[W2, Q2] Regarding the dynamic-region selection.

We thank the reviewer for raising this concern. As the reviewer noted, we provide quantitative and qualitative analysis about the motion scores at Table 4a and Figure 8, effectively distinguishing moving regions. We used a simple spatial mean threshold consistently across our experiments, which gave a reliable downstream control performance on diverse simulation and real robot benchmarks. To further investigate this sensitivity, we conducted additional comparisons with alternative threshold strategies under the same training setup.

\begin{array}{lc} \hline \text{Method} & \text{GR-1 Tabletop} \newline \hline \text{Baseline} & 52.3\% \newline \text{Random mask} & 53.8\% \newline \text{Continuous soft weighting} & 54.8\% \newline \text{Threshold = 0} & 55.3\% \newline \text{Threshold = mean - }\sigma & 55.9\% \newline \textbf{Mean thresholding} & \mathbf{62.3\%} \newline \hline \end{array}

We further investigate the reliability to local visual noise, camera perturbations and movements at our response to [W4, Q4] and [W5, Q5].


[W3, Q3] Disentanglement of asymmetric noise allocation and dynamic-region loss reweighting.

To disentangle the contributions of asymmetric noise allocation and dynamic-region loss reweighting, we conducted an additional controlled experiment using the same auxiliary prediction architecture under matched computational budgets. Adding the auxiliary prediction path to loss reweighting alone does not improve performance, whereas MoGAP's asymmetric prediction objective achieves a substantially higher success rate.

\begin{array}{lc} \hline \text{Method} & \text{GR-1 Tabletop} \newline \hline \text{Loss reweighting} & 57.7\% \newline \text{Loss reweighting + auxiliary prediction path} & 57.1\% \newline \textbf{MoGAP} & \mathbf{62.3\%} \newline \hline \end{array}

[W4, Q4] Further explanation and analysis of the mechanism.

MoGAP introduces a more challenging prediction task for dynamic regions while preserving static regions as context. This encourages the video backbone to better model manipulation dynamics, which in turn benefits action prediction through WAM’s joint video-action learning. To further investigate this mechanism, we measure region-wise visual prediction errors over four episodes for each of the 24 GR-1 Tabletop tasks.

We observe that dynamic regions exhibit higher prediction errors than static regions, with particularly large errors in regions containing the robot. MoGAP reduces prediction errors in dynamic and robot regions while leaving static-region errors nearly unchanged. This observation is consistent with our hypothesis that asymmetric prediction encourages the video backbone to better model manipulation dynamics, which may contribute to improved action prediction. This suggests that MoGAP selectively improves the video modeling of challenging regions, which is consistent with improved task success through joint video-action learning.

\begin{array}{lcccc} \hline \text{Method} & \text{Static region error} & \text{Dynamic region error} & \text{Robot-region error} & \text{Success Rate} \newline \hline \text{Baseline} & 0.031 & 0.170 & 0.241 & 52.3\% \newline \textbf{MoGAP} & 0.031 & \mathbf{0.165} & \mathbf{0.233} & \mathbf{62.3\%} \newline \hline \end{array}

We additionally conducted perturbation experiments at task-relevant regions, especially at the robot’s arm, as the reviewer suggested to better understand the scope and reliability of MoGAP. For noise and lighting perturbation, we add a localized gaussian noise in the region around the robot arm, and darken the same region, respectively in GR-1 Tabletop. For camera perturbation, we include moving camera experiments in our response to [W5, Q5]. We find that MoGAP consistently outperforms the baseline model under such perturbations.

\begin{array}{lccc} \hline \text{Method} & \text{Noise perturbation} & \text{Lighting perturbation} & \text{No perturbation} \newline \hline \text{Baseline} & 43.7\% & 45.5\% & 52.3\% \newline \textbf{MoGAP} & \mathbf{58.2\%} & \mathbf{54.0\%} & \mathbf{62.3\%} \newline \hline \end{array}


[W5, Q5] Generality under moving camera conditions.

We thank the reviewer for raising this concern. In RoboCasa-Kitchen, strong wrist-camera motion can dominate the latent motion scores, making it difficult to identify robot movements in the other exterior camera views. We therefore assign all wrist-camera tokens to the high-noise group, preserving motion-guided region selection in the external views while retaining visual prediction supervision for the wrist view. This provides a practical solution for particularly strong camera motion, but does not imply that MoGAP requires special treatment under all moving-camera conditions.

To further investigate its applicability during training, we separately trained the baseline and MoGAP in two additional settings. (1) egocentric camera movement on our OpenArm humanoid, where the robot continuously rotates its neck through a total range of 30° during manipulation, and (2) a mobile manipulation scenario on the SteamInMicrowave task from RoboCasa365 [3], where a mobile manipulator must place an object into a bowl, pick up the bowl, navigate to a microwave, place the bowl inside, and close its door.

\begin{array}{lcc} \hline \text{Method} & \text{OpenArm moving camera SR} & \text{Mobile manipulation SR} \newline \hline \text{Baseline} & 79.2\% & 58.0\% \newline \textbf{MoGAP} & \mathbf{87.5\%} & \mathbf{66.0\%} \newline \hline \end{array}

The results demonstrate that MoGAP remains effective under moving-camera and mobile manipulation settings. By deriving motion scores from consecutive latent slices in the spatially compressed VAE space, manipulation-related dynamics can remain distinguishable under camera movement, as also illustrated in Figure 8. We have also incorporated these results into the revised manuscript (Table 7 in Appendix E.1). Thank you again for the meaningful suggestion.

[3] Nasiriany et al., RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots, ICLR 2026

[W6] Repeated training runs on real-world generalization settings.

We conducted additional training runs with independent training seeds for our real-world experiments, and evaluated on the Object Relocation II task illustrated in Figure 7. We evaluated the two additional training seeds (0 and 7) over 12 trials per task, while retaining the original results for seed 42. We find that MoGAP consistently achieves meaningful performance gains across all three training seeds.

\begin{array}{lc} \hline \text{Method (Training seeds)} & \text{Object Relocation II Task success} \newline \hline \text{Baseline (0 / 7 / 42)} & 33.3\% / 41.7\% / 37.5\% \newline \textbf{MoGAP (0 / 7 / 42)} & \mathbf{75.0\% / 66.7\% / 79.2\%} \newline \hline \end{array}


We sincerely appreciate the time and effort you took to engage with our work. If there are any remaining concerns, we would be grateful if you could share them.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer 48hC,

Thank you again for your time and thoughtful efforts in reviewing our paper.

As the discussion period is coming to a close, we would like to kindly check whether you have any remaining questions or concerns. If so, please feel free to let us know.

Many thanks,
Authors

Reviewer_oUiaOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Motion-Guided Asymmetric Prediction (MoGAP) with an auxiliary visual prediction objective for World Action Models (WAMs). MoGAP uses temporal differences in video latents to identify dynamic regions and applies asymmetric noise allocation, encouraging the model to reconstruct heavily corrupted dynamic regions from less corrupted static context. The method improves manipulation success rates across two WAM architectures, from 38.8% to 47.5% for Fast-WAM and from 52.3% to 62.3% for Cosmos3-Nano on GR-1 Tabletop, with additional validation on a real humanoid robot.

Critical strengths

A well-motivated research question with practical significance.

The paper investigates an important problem in video-based robot learning: how to adapt the rich visual representations of pretrained generative models to the specific requirements of robotic control. Rather than introducing additional modalities or external supervision, the authors focus on improving the supervision already available within the video prediction objective. This is a meaningful and practically relevant direction.

Critical weaknesses

The motion-based definition of task relevance is restrictive.

The method implicitly assumes that regions exhibiting large temporal changes are more relevant to robotic control. While this assumption is reasonable for some manipulation tasks, motion magnitude and task relevance are not necessarily equivalent. For instance, stationary target objects can be critical for grasp planning, while robot self-motion, camera movement, and moving distractors may produce substantial visual changes without corresponding task relevance. The authors partially address camera motion in the multi-view experiments by assigning wrist-camera tokens to the high-noise group. However, this practically means for head cameras with movements the proposed method does not apply anymore.

Questions for authors

Generalization to different observation configurations.

The multi-view experiments require special treatment of wrist-camera tokens. How would the method behave in more strongly egocentric settings, where camera movement dominates frame differences, or in mobile manipulation scenarios?

Sensitivity to motion-mask quality.

How sensitive is MoGAP to the quality of the latent motion masks? In particular, how does performance change when masks contain false positives from camera movement or dynamic distractors? A comparison against random masks, optical-flow-based masks, or alternative motion thresholds would be informative.

3 comments

Dear reviewer oUia,

We sincerely appreciate your efforts and insightful comments in reviewing our manuscript. Below, we address each comment in detail. In the revised manuscript, we mark our major revisions as “blue”.


[W1, Q1] Applicability of the motion scores

We appreciate the reviewer’s concern regarding the applicability of the motion scores. In many manipulation scenarios, camera observations maintain a stable view of the manipulation scene to support precise and reliable control. Nevertheless, we acknowledge that under strong camera motion, such as in wrist-camera views or at very low frame rates, the spatial discrimination of latent motion scores can become less reliable.

To further investigate the applicability of MoGAP under stronger camera movements, we separately trained both the baseline and MoGAP on two additional settings.(1) egocentric camera movement on our OpenArm humanoid, where the robot continuously rotates its neck by 30° during manipulation, and (2) a mobile manipulation scenario on the SteamInMicrowave task from RoboCasa365 [1], where a mobile manipulator must place an object into a bowl, pick up the bowl, navigate to a microwave, place the bowl inside, and close its door.

\begin{array}{lcc} \hline \text{Method} & \text{OpenArm moving camera SR} & \text{Mobile manipulation SR} \newline \hline \text{Baseline} & 79.2\% & 58.0\% \newline \textbf{MoGAP} & \mathbf{87.5\%} & \mathbf{66.0\%} \newline \hline \end{array}

Moving camera demonstration video: https://anonymous.4open.science/r/iclr2027-mogap/OpenArm_moving_camera_example.mp4

Mobile manipulation demonstration video: https://anonymous.4open.science/r/iclr2027-mogap/RoboCasa365_mobile_manipulation_example.mp4

We find that MoGAP remains effective under the evaluated egocentric camera movement and mobile manipulation settings. We attribute this in part to computing motion scores from consecutive latent slices in the compressed VAE space, where manipulation-related dynamics can remain distinguishable under moderate camera movement, as illustrated at the waist-motion example in Figure 8 (row 3). Importantly, MoGAP does not use motion scores as explicit prediction targets or conditioning signals, but rather as coarse indicators to guide regional noise allocation during training. Therefore, MoGAP can remain effective as long as the motion scores preserve sufficient discrimination between moving and static regions in the latent visual tokens, even under camera motion. We have also incorporated these results with experimental details into the revised manuscript (Table 7 in Appendix E.1).

[1] Nasiriany et al., RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots, ICLR 2026


[Q2] Sensitivity to motion-mask quality.

We thank the reviewer for the suggestion. We conducted additional experiments to investigate the sensitivity of MoGAP to motion-mask quality. As shown in Table 3(a) of the original manuscript, removing motion-guided asymmetry (no mask) substantially reduces the gains of MoGAP, while using random masks can further degrade the performance. In contrast, tracking-based motion masks constructed using CoTracker3 can achieve comparable performance to latent motion scores on GR-1 Tabletop, where the latent scores already provide reliable signals for manipulation dynamics.

\begin{array}{lc} \hline \text{Method} & \text{GR-1 Tabletop} \newline \hline \text{No mask} & 55.3\% \newline \text{Random mask} & 53.8\% \newline \text{Tracking-based mask} & 61.7\% \newline \textbf{Latent motion scores} & \mathbf{62.3\%} \newline \hline \end{array}

On RoboCasa-Kitchen, strong wrist-camera motion makes latent motion scores less discriminative. As a practical solution, our original method assigns all wrist-camera tokens to the high-noise group while preserving motion-guided region selection in the exterior views. We further find that improved motion masks from optical flow and tracking can increase performance without requiring this special treatment. These results further support our central finding that spatially allocating visual supervision toward manipulation dynamics benefits WAMs, while demonstrating that more accurate motion estimation can extend MoGAP to challenging camera-motion settings.

\begin{array}{lc} \hline \text{Method} & \text{RoboCasa-Kitchen} \newline \hline \text{Baseline} & 62.7\% \newline \text{Latent motion scores} & 65.8\% \newline \quad \text{w/ wrist cam high-noise} & \newline \text{Optical-flow-based mask} & 66.3\% \newline \textbf{Tracking-based mask} & \mathbf{67.3\%} \newline \hline \end{array}

We have also incorporated these results into the revised manuscript (Table 8 in Appendix E.2). Thank you again for the meaningful suggestion.


We sincerely appreciate the time and effort you took to engage with our work. If there are any remaining concerns, we would be grateful if you could share them.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer oUia,

Thank you again for your time and thoughtful efforts in reviewing our paper.

As the discussion period is coming to a close, we would like to kindly check whether you have any remaining questions or concerns. If so, please feel free to let us know.

Many thanks,
Authors

I thank the authors for their responses and additional experiments, which have addressed my concerns regarding sensitivity to motion-mask quality.

However, I remain concerned that the motion-based definition of task relevance is overly restrictive, particularly in egocentric settings with wrist-mounted cameras or head-mounted cameras that move continuously to track targets. Under the proposed framework, these settings would appear to fall into the high-noise category, limiting the method’s applicability to common robotic configurations.

I appreciate the authors’ efforts, but I maintain my original score.

Reviewer_kPSDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoGAP, a motion-guided asymmetric prediction method for adapting pretrained video models to robot control. It applies stronger noise to dynamic regions and uses static regions as context for visual prediction. The method adds an auxiliary training objective without changing inference. Experiments on RoboCasa and real humanoid robots show substantial improvements in manipulation success rates.

Critical strengths
  1. Clear motivation and intuitive design.

The paper presents an interesting idea of focusing visual supervision on motion-related regions rather than treating all regions equally. The method is simple and does not require additional teacher models or inference overhead.

  1. Strong experimental results.

The paper demonstrates consistent improvements across different WAM architectures, simulation benchmarks, and real-robot tasks. The ablation studies also provide convincing evidence for the effectiveness of asymmetric noise allocation.

Critical weaknesses
  1. Motion saliency does not necessarily reflect task relevance.

MoGAP uses motion scores from future video latents without explicitly considering language-conditioned task relevance. In pick-and-place tasks, the target object is often static before grasping. The paper does not sufficiently explain how this supervision helps the model select the correct target among multiple static objects. There may also be a risk of learning motion biases from training demonstrations rather than generalizable task-relevant representations.

  1. Limited methodological novelty.

Motion-based supervision, regional noise allocation, and auxiliary denoising are closely related to existing techniques. Although their combination is effective, the methodological novelty appears relatively limited. A more direct comparison with attention-based or language-guided supervision would help clarify the contribution of MoGAP.

Questions for authors

Q1. Motion-Guided Supervision vs. Task Relevance

MoGAP emphasizes motion saliency rather than explicitly modeling language-conditioned task relevance. In pick-and-place tasks such as RoboCasa, the target object is usually static before grasping. I believe accurate localization and selection of the correct object among multiple static candidates are critical for success.

During training, the motion scores are computed from ground-truth future videos, which may highlight the target object after it has been manipulated. However, at inference time, this future information is unavailable, and the motion-guided supervision itself is not conditioned on the task instruction.

How does this supervision help the policy identify the correct static target before grasping? Could the model simply learn motion patterns or object-selection biases from the training demonstrations, leading to overfitting when the target object or scene layout changes?

Q2. Comparison with Attention-Based Guidance

MoGAP seems related to implicit attention guidance. Have the authors considered using action-to-vision attention or language-conditioned attention to select important regions? A comparison under similar training budgets would help clarify whether motion-based supervision is more effective than task-aware region selection.

Q3. Additional Experiments

I would like to see two controlled experiments:

  • Multiple static targets: Place several static objects in the same scene and vary the language instruction to select different targets. Does MoGAP improve correct-target selection and pre-grasp success?
  • Irrelevant motion distractors: Introduce moving background objects while keeping the target stationary. Does the motion-guided supervision remain effective, or does irrelevant motion reduce performance?

If the authors can convincingly address these concerns, I would consider raising my rating.

5 comments

Dear reviewer kPSD,

We sincerely appreciate your efforts and insightful comments in reviewing our manuscript. Below, we address each comment in detail. In the revised manuscript, we mark our major revisions as “blue”.


[W1-a, Q1] Further explanation of the motion-guided supervision and task relevance

We thank the reviewer for raising this question. We understand the central concern to be how motion-guided supervision helps the policy manipulate initially stationary target objects.

For successful task execution, WAMs must both generate video-action predictions consistent with the language instruction (instruction following) and accurately generate the corresponding future video and actions (manipulation accuracy). While instruction following is an important challenge that many works seek to address, our primary focus is on improving manipulation accuracy. By using motion scores from future training videos to guide regional noise allocation and auxiliary supervision, MoGAP encourages the model to learn to better predict how the robot arm approaches and grasps static target objects at future frames, without requiring the future information at inference.

We would also like to clarify that our notion of task-relevance primarily refers to physical changes associated with robot motion and robot-object interactions, rather than semantic relevance to the language instruction. We apologize for this ambiguity and we have modified the terminology in our revised manuscript.


[W1-b, Q2] Language-conditioned task relevance and attention-based guidance

We conducted additional experiments comparing language-conditioned and action-to-vision attention with MoGAP under the same auxiliary prediction framework we’ve proposed, changing only the region selection strategy.

\begin{array}{lc} \hline \text{Method} & \text{GR-1 Tabletop (\%)} \newline \hline \text{Baseline} & 52.3 \newline \text{Language-conditioned attention} & 56.7 \newline \text{Action-to-vision attention} & 59.3 \newline \textbf{MoGAP} & \mathbf{62.3} \newline \hline \end{array}

GR-1 Tabletop includes tasks with distractor objects, which language-conditioned attention can provide useful guidance. However, we observe many failures regarding manipulation accuracy (e.g., incorrect grasping, dropping objects, and imprecise placing), and MoGAP achieves the highest success rate, suggesting that emphasizing manipulation-related dynamics provides an effective supervision. We have also incorporated these results with further details about the experiment in the revised manuscript (Appendix E.3).


[Q3] Additional Experiments

We thank the reviewer for the suggestion. We conducted both requested experiments on our real-robot platform used in Sec 3.3. For multiple static targets, we place two to three objects in the same scene and measure instruction following and full task success separately. For irrelevant motion distractors, we introduce irrelevant motion by having a person wave their hand, or wave a held object throughout task execution while keeping the target object stationary.

(1) Multiple static targets \begin{array}{lcc} \hline \text{Method} & \text{Instruction following} & \text{Full task success} \newline \hline \text{Baseline} & 75.0\% \ (36/48) & 39.6\% \ (19/48) \newline \textbf{MoGAP} & \mathbf{79.2\%} \ (38/48) & \mathbf{60.4\%} \ (29/48) \newline \hline \end{array} Demonstration video: https://anonymous.4open.science/r/iclr2027-mogap/OpenArm_instruction_following_example.mp4

(2) Irrelevant motion distractors \begin{array}{lc} \hline \text{Method} & \text{Motion distraction task success} \newline \hline \text{Baseline} & 43.8\% \ (21/48) \newline \textbf{MoGAP} & \mathbf{64.6\%} \ (31/48) \newline \hline \end{array}

Demonstration video: https://anonymous.4open.science/r/iclr2027-mogap/OpenArm_motion_distractor_example.mp4

MoGAP maintains the instruction following accuracy of the baseline, while largely improving full success, suggesting that the main benefits come from more precise and reliable manipulation. For motion distractors, we find MoGAP remains effective under the perturbations, also consistently outperforming the baseline. We have included the results with further details and visualizations of the tasks, at Appendix D (Table 6, Figures 15–16).

We also appreciate the concern regarding overfitting when the target object or scene layout changes at [Q1]. We believe the improvements on unseen objects and environments (Figure 6), together with these additional experiments, provide further evidence that MoGAP generalizes beyond the object configurations and motion patterns observed during training.


We sincerely appreciate the time and effort you took to engage with our work. If there are any remaining concerns, we would be grateful if you could share them.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer kPSD,

Thank you again for your time and thoughtful efforts in reviewing our paper.

As the discussion period is coming to a close, we would like to kindly check whether you have any remaining questions or concerns. If so, please feel free to let us know.

Many thanks,
Authors

Thank you to the authors for the detailed responses and additional experiments, which have addressed most of my concerns.

However, I still have some reservations about the methodological novelty and motivation. In pick-and-place tasks, target objects are typically stationary before grasping. The authors explain that MoGAP mainly improves manipulation accuracy by emphasizing robot motion and interactions. However, the attention visualizations in Figure 8 appear to show increased attention to the manipulated objects as well.

This raises a remaining question: does motion-guided supervision actually improve the representation of static targets before grasping, or does it primarily improve the prediction of object motion after interaction?

I appreciate the authors' efforts, but I believe this distinction remains insufficiently explained and somewhat limits the claimed methodological contribution.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and Area Chair,

We deeply appreciate your valuable time and effort spent reviewing our manuscript.

As the reviewers highlighted, our work investigates an important research question in adapting pretrained video models to robot control (Reviewers kPSD, oUia, 48hC): how to adapt their rich visual representations to better support downstream policy learning. Our proposed method MoGAP, provides a intuitive and practical approach (Reviewers kPSD, oUia) to emphasizing manipulation-related dynamics through spatially asymmetric visual prediction. We are encouraged that the reviewers recognized the clear and meaningful research motivation (All Reviewers) and strong experimental results (Reviewers kPSD, 48hC) across WAM architectures and diverse manipulation settings, together with well-targeted ablation studies for convincing evidence (Reviewers kPSD 48hC).

In the rebuttal, we have carefully considered the reviewers' suggestions and addressed them in our responses. As a result, we have revised and enhanced the manuscript with the following additional discussions and experiments:

  • Comparisons with attention-based guidance (Appendix E.3)
  • Instruction following and motion distractor experiments (Appendix D)
  • Moving-camera and mobile manipulation experiments (Appendix E.1)
  • Motion-mask quality analysis (Appendix E.2)
  • Updated terminology: replacing “task-relevant” to “manipulation-relevant”

In the revised manuscript, these updates are temporarily highlighted in blue\color{blue}{\textbf{blue}} for your convenience. We hope our responses and revisions sincerely address all the reviewers' concerns and further clarify the contribution of our work.

Thank you once again for your valuable contributions.

Warm regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.