acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Manipulation with World-Action Models via Counterfactual Planning

33 views

Abstract

World–Action models (WAMs) trained on static demonstrations often fail to manipulate moving targets even when they possess the required manipulation skills. We attribute this failure to target-response collapse: as execution advances, the policy becomes increasingly biased toward the learned continuation of its ongoing behavior and less responsive to target relocation. To bridge the gap between what the model has learned and what it can generate from the current context, we formulate dynamic manipulation as counterfactual planning by decoupling the context used for plan generation from the physical state used for execution. Our framework, Dynamic Predictive Planning (DPP), first uses the WAM's predictive rollout to estimate when an interaction is expected to occur, and combines this timing estimate with observed target motion to predict the target's future interaction position. DPP then constructs a counterfactual observation that places this predicted target position in a familiar robot context, allowing the model to invoke an existing manipulation skill rather than generate a recovery behavior from an unfamiliar robot–target configuration. The resulting plan is connected to the robot's actual state during execution. DPP enables real-time dynamic manipulation on a single consumer GPU without additional training on dynamic data. Experiments in simulation and on a real robot demonstrate consistent improvements across diverse target motions, with simulation performance surpassing all evaluated baselines, including methods additionally trained on dynamic data.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dynamic Robot Manipulation, Vision-Language-Action Model, World Action Models, Counterfactual Observation, Motion-Aware Planning, Test-Time Adaptation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_Tqy4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why world–action models (WAMs) trained on static demonstrations struggle to manipulate moving targets. The authors identify a failure mode called target-response collapse, where the policy gradually loses its ability to respond to target relocation as execution proceeds. To address this, they propose Dynamic Predictive Planning (DPP), which estimates the future interaction time and target position, constructs counterfactual observations using canonical robot configurations, and aligns the resulting plans with ongoing execution. The framework aims to reuse pretrained manipulation skills without additional training on dynamic tasks. Experiments on RoboTwin2.0-Dyn, more complex motion scenarios, and real robots show substantial improvements over several reactive, future-conditioned, and dynamically trained baselines.

Critical strengths
  1. The paper identifies and carefully analyzes an interesting failure mode of existing WAMs. I found the matched-prefix experiments in Section 3 and Appendix B particularly informative. For FastWAM, the mean directional response gain drops from 0.897 to 0.078 after 48 executed actions, while the response-hull area decreases to only 1.47% of its initial value (Table 6). A similar trend is observed for ImageWAM in Table 7. These controlled experiments provide fairly convincing evidence of target-response collapse. More importantly, they offer a useful perspective on why policies trained on static demonstrations may fail in dynamic environments, rather than simply reporting low success rates on moving-target tasks.

  2. The proposed framework provides an interesting way to reuse pretrained manipulation skills without additional training. DPP predicts where the target will be at the expected interaction time and combines this prediction with a familiar robot configuration to generate a new plan. The plan is then aligned with the robot's current execution state. Compared with SIDO (Shin et al., 2026), which relies on counterfactual action augmentation during training, and ReOI (Chen et al., 2026b), which modifies visual distractors at inference time, DPP uses canonical observation conditioning to reuse the existing WAM policy. The ablation in Table 2 also supports this design choice: removing canonical-state conditioning reduces Task 1 success from 83.33% to 51.67%. I think this training-free formulation is one of the more appealing aspects of the work.

  3. The experimental results are strong across different tasks and model backbones. In Table 1, FastWAM-DPP and ImageWAM-DPP achieve 83.33% and 78.67% success on bimanual grasping, respectively, compared with 52.00% for the dynamically trained DynamicWAM baseline. Improvements are also reported for pick-and-place and additional task types in Appendix A. The real-robot results are encouraging as well, with success rates improving from 0% to 76.7% on Task A and from 40.0% to 86.7% on Task B (Figure 4). The relatively consistent gains across these settings suggest that the proposed approach has practical value beyond a single task or backbone.

  4. The paper pays attention to practical execution issues, which is important for this type of planning framework. In particular, the authors address the mismatch between a trajectory planned from a canonical state and the robot's actual state through trajectory bridging and asynchronous plan activation. Table 11 reports that generating a 32-action chunk takes approximately 0.495 seconds, while executing it takes 3.2 seconds at 10 Hz. This suggests that planning can be performed asynchronously without necessarily interrupting execution. The real-robot deployment using an RTX 4090, together with the controller details in Appendix C, also makes the implementation more convincing and easier to reproduce.

Critical weaknesses
  1. The benchmark setup raises some concerns about the fairness of comparisons with dynamically trained baselines. According to Appendix E.2, the evaluation regions are selected based on FastWAM's performance in static validation. This may favor configurations that are already well supported by FastWAM, while other independently trained policies may have different regions of competence. Another concern is the change in the motion-to-physics handoff protocol between RoboTwin2.0-Dyn and DOMINO. The difference appears to matter: PUMA achieves 86.67% success with DOMINO-style contact stopping in Table 10, compared with 50.33% in Table 1. DPP also uses a timing adjustment specific to this protocol. Although DPP still performs better in the reported experiments, I think the comparison would be more convincing with static-support controls for each baseline and evaluations under a more consistent handoff protocol.

  2. It is unclear how much of the improvement comes specifically from WAM-derived interaction-time prediction. A key part of DPP is using the interaction event predicted by the initial WAM rollout to estimate the future target position (Eq. 8). However, this component is not directly isolated in the experiments. The fixed-horizon Future-Conditioned baselines use different controllers, and the fixed-time VLA-style alternative in Appendix F.7 also changes the trajectory generation and execution procedure. As a result, these comparisons do not really show whether the WAM-predicted interaction time is better than a fixed or calibrated time estimate when the rest of DPP remains unchanged. Since interaction timing is central to the method, I would like to see a more controlled ablation of this component.

  3. The explanation of target-response collapse is plausible, but the underlying cause is not fully established. Section 3 suggests that the reduced responsiveness comes from robot–target configurations that are poorly covered by static training demonstrations. The matched-prefix experiments clearly demonstrate the behavioral effect, but they do not directly measure whether these configurations are outside the training distribution. There may also be other explanations. For example, the robot could become less able to recover from target changes due to kinematic constraints or execution progress, or the model's predictions may become less reliable at later states. Appendix H.7 acknowledges this limitation, which is good. Still, I think the paper should distinguish more carefully between establishing target-response collapse as an observed phenomenon and explaining its primary cause.

  4. The generalization results for more complex target motion rely on assumptions that may not hold in less structured environments. The base DPP controller uses a constant-velocity estimate that is frozen after bootstrap (Appendix F.3). For the complex-motion experiments in Figure 5, the method uses predefined motion-family information, late canonical planning, and conditional spatial warping. Several motion checks in the base controller are also relaxed for this evaluation. These adaptations are reasonable, and the results are useful, but they make it difficult to assess how robust DPP would be when the target motion is not known in advance. The real-robot experiments also use controlled motor-driven object trajectories. I would therefore be cautious about generalizing the conclusions to unpredictable motion, noisy trajectory changes, or more unconstrained interactions.

Questions for authors
  1. Could the authors report the static-task success rates of each backbone and dynamically trained baseline in the selected evaluation regions? I would also be interested in whether the relative rankings change when using regions not selected based on FastWAM, or when applying the same motion-to-physics handoff protocol to all methods.

  2. How does WAM-derived interaction timing compare with fixed, empirically calibrated, or oracle interaction times when all other DPP components are kept unchanged? It would also be helpful to report the timing prediction error and how sensitive task success is to errors in the estimated interaction time.

  3. Could the authors provide further evidence that target-response collapse is mainly caused by insufficient coverage of joint robot–target states in the training data? How can this be distinguished from reduced recoverability or increasing prediction errors at later execution stages?

  4. For the complex-motion experiments in Figure 5, how much do the improvements depend on knowing the motion family in advance and relaxing the controller's motion checks? Would similar results be expected with automatic motion-model selection and the same controller constraints as the base setting?

2 comments
Reviewer_Tqy4OpenReview ↗

Official Comment to Authors

Thanks for the responses, which fully address my concerns. Thus, I retain my positive rating of 3.

We sincerely thank the reviewer for their thoughtful feedback and for taking the time to carefully consider our responses. We are pleased that our clarifications have fully addressed your concerns, and we greatly appreciate your continued positive assessment of our work.

Best regards,
The Authors

Reviewer_fB8mOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates how world action models trained on static demonstrations can manipulate moving targets without additional dynamic training. It identifies "target response collapse", where models become less responsive to target relocation as execution progresses and increasingly follow trajectories associated with their current robot configurations. The proposed Dynamic Predictive Planning (DPP) predicts the interaction time and future target position, then constructs a counterfactual observation pairing that position with a familiar robot configuration to generate a plan. DPP connects this plan to ongoing execution and refines it using fresh observations, with ablations supporting the importance of the familiar planning context and concurrent execution. Across two model backbones, DPP outperforms all selected simulation baselines, and improves success on real robot tasks involving moving targets.

Critical strengths

The analysis in Section 3 provides interesting insights into why WAMs trained on static demonstrations struggle to respond to moving targets.

Critical weaknesses

The method makes several strong assumptions, but the paper does not adequately explain why they are reasonable or when they hold in practice.

  • DPP assumes that a "canonical robot state" provides a familiar context in which the frozen WAM can reliably generate a skill for the repositioned target. It remains unclear how to select and validate such a reference for pretrained WAMs with unknown training coverage.
  • The base DPP method estimates target motion from a short initial observation window and keeps this estimate fixed throughout execution, assuming that it remains accurate until interaction. The nonlinear extensions also rely on predefined motion families, but the paper does not explain how the appropriate family would be known in advance for a new task.
  • DPP uses the interaction time predicted from a static-scene WAM rollout as the deadline for the dynamic task, even though reaching the target's future position may take longer. Although DPP checks whether a candidate plan can be connected to ongoing execution, it remains unclear how often the chosen deadline makes this connection infeasible and how much this affects performance.

Together, these assumptions raise a key question: whether this narrowly defined setting warrants such a complex system. I would like to see a clearer account of which components or insights transfer to other manipulation problems.

Questions for authors

How many static demonstrations were used to train the WAM? How would a standard WAM trained on the same number of dynamic demonstrations perform without DPP?

Reviewer_5S4fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Dynamic Predictive Planning (DPP) to reuse static manipulation skills for moving targets without additional training. It identifies target-response collapse, where a policy becomes less responsive to target relocation as execution progresses. DPP addresses this by predicting the interaction time and future target position, generating a plan from a familiar robot context, and bridging that plan to the actual execution state. Its main contributions are the empirical diagnosis of this failure mode and the decoupling of planning context from execution state. Experiments across two WAM backbones, simulation benchmarks, and two real-robot tasks show substantial improvements over evaluated baselines. Ablations support the importance of canonical-context planning and asynchronous execution. The results depend on predictable target motion and existing static manipulation skills.

Critical strengths

A well-motivated approach to skill reuse. The paper characterizes target-response collapse through controlled target-relocation experiments and proposes generating plans by combining a familiar robot context with the predicted future target position, then connecting them to actual execution. This provides a concrete way to adapt frozen models to dynamic environments without additional dynamic-data training, constituting the paper’s strongest conceptual contribution.

Critical weaknesses

The complex-motion evaluation confounds motion adaptation with relaxed execution constraints. Appendix F.3 states that the nonlinear extension disables Cartesian-limit rejection for prepayment and canonical bridges and does not apply the base controller’s numerical joint-speed and acceleration rejection limits. Figure 5 therefore evaluates several changes together, leaving the contribution of motion adaptation insufficiently isolated. A comparison under matched execution constraints is needed to establish whether the reported gains persist under the same feasibility requirements.

Questions for authors
  1. Clarification of the evaluation protocol: What effective velocity and acceleration limits were enforced for DPP and each baseline in Figure 5? Appendix F.3 states that some DPP rejection checks were disabled while MPLib/TOPP limits remained active. Were the resulting execution constraints comparable across methods? This would determine whether the comparison isolates planning performance.

  2. Validation of the current robustness claim: Do the complex-motion gains persist when the base controller’s feasibility checks are restored, with the other nonlinear extensions unchanged? Please provide success and plan-rejection rates if available, or clarify whether this comparison was performed. This evidence would materially affect my confidence in the robustness claims.

Reviewer_F1knOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies target‑response collapse, a key failure: pretrained world‑action models trained on static demos lose responsiveness to moving targets as robot execution proceeds. The authors propose DPP, an inference‑time framework without retraining. DPP decouples plan‑generation context from physical execution state: it infers interaction timing via WAM rollout, estimates future target position, builds counterfactual canonical observations to invoke learned static skills, and bridges the canonical trajectory to real‑time robot states.

Critical strengths
  1. This work clearly characterizes target‑response collapse with quantitative relocation probes. It provides a clean behavioral explanation for why static‑trained WAMs fail dynamic manipulation, offering useful insights for embodied model deployment.
  2. DPP is an inference‑only pipeline requiring no additional policy training. It reuses existing static manipulation skills, which lowers deployment cost compared to retraining on dynamic datasets.
  3. Comprehensive evaluations include simulation benchmarks, real‑robot hardware tests, extensive ablations and oracle diagnostic studies. Multiple backbones validate the generalizability of the proposed planning logic.
Critical weaknesses
  1. DPP assumes predictable target motion. Performance will degrade heavily under abrupt, unmodeled target motion, and the nonlinear extension only supports predefined motion families.
  2. The whole pipeline relies on multi‑component perception stack. Perception noise, tracking drift or severe occlusions will corrupt motion estimation and counterfactual observation synthesis. Real‑world robustness under heavy occlusion lacks sufficient testing.
  3. Comparisons against closely related inference‑time methods are mostly mechanistic in appendix, lacking head‑to‑head numerical benchmark results in main text.
Questions for authors
  1. How does DPP behave under sudden, unforeseen target acceleration or direction changes outside your predefined motion families? Could you add stress‑test results for highly‑irregular object motion in rebuttal?
  2. What is the measured failure rate of your perception pipeline under heavy object occlusion? How robust is DPP when tracking outputs contain large pose noise?
  3. Can you quantify how visual artifacts from counterfactual image editing affect planning success rate?

Comments on the paper

1 comment

Dear Reviewers,

Please take a moment to review the authors' rebuttal and assess whether your questions and concerns have been adequately addressed. If necessary, please engage in further discussion with the authors.

Kindly provide your final recommendation as soon as possible.

Thank you for your time and efforts!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.