acceptodds
Under review as a conference paper at ICLR 2027

DynamicManip: Enabling Dynamic Manipulation from a Single Static Demonstration

Abstract

Dynamic manipulation is a critical capability for robots operating in complex and dynamic environments, where robots must interact with objects that are moving or require rapid adjustments. However, learning models for dynamic manipulation tasks face two major challenges: (1) the combinatorial complexity of dynamic scenarios leads to substantial data requirements, and (2) rapid variations in dynamics require real-time and accurate policy execution. In this paper, we propose DynamicManip to address these challenges through an efficient data augmentation pipeline and a low-latency imitation policy. We first propose a static-to-dynamic augmentation pipeline that synthesizes diverse dynamic manipulation demonstrations from a single static demonstration. Second, we introduce a dynamic-aware adaptive policy that adjusts its inference frequency according to task dynamics, enabling responsive and effective dynamic manipulation. Third, we build a dynamic manipulation benchmark, which includes diverse dynamic tasks with an automatic evaluation system for scalable and consistent assessment. Extensive experiments in both simulation and the real world demonstrate that DynamicManip not only provides significant improvements in data efficiency but also achieves better performance in dynamic manipulation tasks, with a mean success rate 18.4 percentage points higher and policy-query latency 32.9% lower.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotics, Manipulation, Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bgQDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

DynamicManip proposes and implements a new system for dynamic manipulation. The key contributions are twofold:

  1. It introduces a data augmentation pipeline that synthesizes trajectories for dynamic manipulation tasks from a single static demonstration, using a combination of heuristically defined rules and motion-planning methods.
  2. It introduces an adaptive policy that dynamically adjusts the feedback interval and computational allocation during inference.

The authors demonstrate clear improvements in inference latency and task performance on their proposed benchmark.

Critical strengths

The proposed methods are technically sound and could be valuable to the robotics community, particularly if the codebase and benchmark suite are made publicly available.

Critical weaknesses
  • Limited methodological novelty. The proposed methods are relatively intuitive and offer limited originality. While applying this data augmentation pipeline to dynamic manipulation tasks may be novel, the underlying approach of manual keyframe selection and trajectory planning between keyframes is well established in the literature. Similarly, adaptive inference for reducing latency and improving computational efficiency has been widely explored in prior work.
  • Limited scope of experimental evaluation. Although the authors demonstrate significant improvements on their proposed benchmark, they do not evaluate their method on existing dynamic manipulation benchmarks, such as DOMINO. Furthermore, the main policy comparisons primarily focus on DP3, with limited comparisons against alternative approaches to dynamic manipulation.
Questions for authors
  1. Could the authors evaluate their data augmentation method on dynamic manipulation tasks from existing benchmarks, such as DOMINO, to further demonstrate its generalizability?
  2. Could the authors provide additional experiments or discuss the following comparison between two approaches?
  • Trajectory synthesis + IL: Generate 100 trajectories using the proposed data augmentation pipeline, followed by imitation learning.
  • Demonstration-guided RL: Start with 5 simulation demonstrations and use demonstration-guided RL (e.g., RLPD) to learn a policy through exploration and environmental feedback.

The central question is whether the exploration required for dynamic manipulation is better guided by manually designed rules and motion planning, as in the proposed method, or learned autonomously through RL. How do these approaches compare in terms of policy performance, human engineering effort, and computational cost?

  1. In Table 1, when controlling for dataset size (50 trajectories), the proposed augmentation method achieves substantially higher success rates than conventional simulated demonstrations on several tasks, particularly Dynamic Tapping (45% vs. 19%) and Object Catching (70% vs. 37%). Could the authors explain why the generated trajectories lead to substantially better downstream policy performance despite having the same number of training examples? Additionally, this performance gap seems to become much smaller when the dataset size increases to 200 trajectories. Could the authors explain this scaling behavior?
8 comments

We thank the reviewer for recognizing the technical soundness and potential value of DynamicManip. Below, we clarify the methodological contribution, present the additional DOMINO and demonstration-guided RL experiments, and explain the data-scaling results.

Weakness 1: Methodological novelty

Our contribution is a unified interaction-stage formulation that connects static-to-dynamic trajectory synthesis, stage-supervised policy learning, and adaptive execution. We agree that keyframe-based trajectory editing, motion planning, and inference acceleration are established techniques. The technical distinction lies in how these components address the temporal requirements of dynamic manipulation together (Sections 3.2–3.3).

1. Dynamic synthesis requires temporal coordination. A spatially valid motion can fail when the target moves before contact. Our pipeline samples object motions and constructs time-varying interaction targets, so the robot's alignment motion is coordinated with the moving object. The four stage operators assign different roles to planning and demonstration reuse: planning generates feasible alignment and transition motions, while anchor-based replay preserves short, contact-rich interaction patterns that are difficult to specify through motion planning alone. This decomposition supports new dynamic episodes from one static demonstration while retaining its task-relevant interaction structure.

2. The synthesis process also supplies supervision for adaptive execution. Each generated timestep inherits the identity of its generating stage operator. These labels supervise a lightweight stage-prediction head alongside action learning, without requiring additional per-timestep annotation. At deployment, the policy predicts the stage from observations; it does not require ground-truth stage labels. Thus, the same interaction structure used to generate demonstrations also teaches the policy when different execution behaviors are needed.

3. Stage-conditioned inference addresses different control requirements. The predicted stage selects the Taylor recomputation interval, approximation order, and action-chunk execution length through stage-conditioned rules. Dynamic alignment uses short action chunks and accelerated inference for frequent feedback; contact execution uses more frequent full computation and longer action prefixes to preserve local motion consistency. These choices follow the distinct requirements of the stages used during synthesis. Table 4 provides supporting evidence: with the same checkpoint and evaluation episodes, stage-aware scheduling achieves 64% versus 34% for fixed Taylor acceleration on Object Catching, and 88% versus 67% on Goal Blocking.

We will sharpen this distinction in the revision: the contribution is the shared stage structure and its concrete use across data generation, supervision, and execution. We will also make clear which underlying techniques are inherited from prior work and that the scheduling rules are heuristic.

Weakness 2: Evaluation beyond our benchmark and DP3

Thank you for highlighting the need to make this evidence more prominent.Our evaluation extends beyond DP3: we add DOMINO experiments and already report π₀ transfer in the submitted appendix.

External benchmark: DOMINO. We evaluate two tasks over 100 test episodes each:

MethodHandover Mic (%)Move Can Pot (%)Average (%)
DP3: 50 conventional expert demonstrations321624.0
DynamicManip: augmented training + stage-aware scheduling515151.0

Question 1 provides the intermediate configurations. These results extend the evaluation to two tasks from an independently developed benchmark.

Transfer to a VLA policy. Appendix F.3 and Table 17 evaluate our stage-prediction and stage-conditioned replanning strategy with π₀:

PolicyGoal Blocking (%)Object Catching (%)
π₀2228
π₀ + our stage-aware learning and replanning6240

This experiment supports transfer of the stage-aware learning and replanning component to another policy family, complementing the point-cloud data-augmentation evaluation.

DP3 is the main controlled baseline because it directly consumes the generated point clouds. We will highlight the existing comparisons in the main text and add the DOMINO results in the revision.

Question 1: Generalization to DOMINO

On two DOMINO tasks, mean success increases from 24.0% for conventional DP3 to 44.5% with augmentation-based training and 51.0% with stage-aware scheduling. We evaluate Handover Mic and Move Can Pot over 100 test episodes per task.

Experimental settingSpecification
Conventional baselineDP3 trained on 50 dynamic expert demonstrations per task.
Augmentation-based training50 generated trajectories from one source demonstration per task, with automatic stage labels for auxiliary supervision.
Inference variantsStandard inference; fixed Taylor acceleration and action-chunk scheduling; predicted-stage scheduling.
Evaluation100 test episodes per task and configuration.
ConfigurationHandover Mic (%)Move Can Pot (%)Average (%)
DP3: 50 conventional expert demonstrations321624.0
Augmented training + standard inference434644.5
Augmented training + fixed scheduling444745.5
Augmented training + stage-aware scheduling515151.0

The comparisons distinguish the training pipeline from the additional benefit of inference scheduling:

ComparisonHandover Mic gain (pp)Move Can Pot gain (pp)Mean gain (pp)
Augmented training + standard inference vs. conventional DP3+11+30+20.5
Stage-aware vs. validation-tuned fixed scheduling+7+4+5.5
Stage-aware vs. standard inference on augmented training+8+5+6.5

Here, pp denotes percentage points. The first comparison reflects the augmentation-based training pipeline, including auxiliary stage supervision. The latter comparisons assess the additional gains from stage-aware inference.

We will include these settings and results in the revision, identifying their scope as two external tasks rather than the full DOMINO suite.

Reviewer_c3P8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses two core bottlenecks in learning dynamic robot manipulation skills: the high cost of collecting diverse dynamic demonstrations and the high inference latency of diffusion-based visuomotor policies. The authors propose DynamicManip, which consists of three main components:

  1. Static-to-Dynamic Data Augmentation Pipeline: Starting from a single static human teleoperated demonstration, the pipeline reconstructs 3D object meshes from RGB images (via Hyper3D and OBB+ICP registration), extracts sparse phase keyframes, and composes four phase operators (Φstatic,Φdyn,Φinter,Φtrans\Phi_{\text{static}}, \Phi_{\text{dyn}}, \Phi_{\text{inter}}, \Phi_{\text{trans}}) using task-specific target-generation rules and motion planning (CuRobo) to synthesize diverse dynamic trajectories with automatic timestep-level stage labels.
  2. Dynamic-Aware Imitation Learning Policy: Built upon 3D Diffusion Policy (DP3), the model trains an auxiliary lightweight MLP head to predict the current interaction stage online and uses a heuristic stage-conditioned scheduler to adaptively adjust TaylorSeer feature extrapolation intervals (τ\tau), polynomial order (KK), and executed action-chunk lengths (nexecn_{\text{exec}}).
  3. DynamicManip Benchmark: Built on RoboTwin 2.0 (SAPIEN), the benchmark introduces five simulated dynamic manipulation tasks with a latency-aware automatic evaluation protocol (Ndelay=round(tinfer/Δtsim)N_{\text{delay}} = \text{round}(t_{\text{infer}}/\Delta t_{\text{sim}})) alongside four real-world tasks on AgileX Piper arms. Empirical results show that training on 200 augmented episodes from a single static demo matches or outperforms training on 200 simulated demos or 50 real-world dynamic teleoperated demos, while reducing mean inference latency by 32.9% and improving simulation task success rate by 18.4 percentage points over vanilla DP3.
Critical strengths
  • Well-Motivated Coupling of Data Generation and Inference Scheduling.
    Connecting the phase structure of trajectory augmentation (Φstatic,Φdyn,Φinter,Φtrans\Phi_{\text{static}}, \Phi_{\text{dyn}}, \Phi_{\text{inter}}, \Phi_{\text{trans}}) directly to downstream inference scheduling is a clean and practical system design. Using the synthetic phase identities as free supervision (Lstage\mathcal{L}_{\text{stage}}) avoids extra manual labeling while enabling phase-adaptive compute allocation at test time.
  • Substantial Reduction in Human Teleoperation Effort.
    Collecting dynamic teleoperation demonstrations is notoriously difficult because human operators struggle with teleoperation lag during fast motion. As documented in Tables 2, 6, and 7, converting 1 static demonstration (~11 minutes of total human setup/annotation effort + 31 minutes of GPU compute) into 200 dynamic episodes yields dramatic improvements over 50 manual dynamic demos (especially on difficult tasks like Boat Loading, improving from 3.33% to 66.7%).
Critical weaknesses
  • Inadequate Ablation Studies of the Adaptive Inference Policy.
    The central claim is that stage-conditioned allocation of denoising computation and action execution improves the latency–success trade-off. Table 3 compares the complete adaptive policy against standard DP3 inference, but this comparison alone does not establish the benefit of stage conditioning over simpler acceleration and replanning choices. Tables 4 and 11 provide useful same-checkpoint scheduling ablations; however, the reduced-DDIM, fixed-Taylor, and fixed-short-chunk baselines are not specified with complete sampling/execution configurations, tuning procedures, or measured latencies. A reasonably tuned joint baseline combining fixed Taylor acceleration with a fixed execution horizon is also not clearly documented. Consequently, the reported advantage may depend on the selected fixed configurations, and the independent value of stage conditioning remains insufficiently established. A controlled comparison against such a fixed joint baseline, reporting both success and measured latency on the existing ablation tasks, is important for substantiating this central contribution.

  • Missing Real-World Ablation for the Dynamic-Aware Adaptive Policy.
    In Section 5.1 (line 376) and Section 5.2, the data-efficiency experiments (Tables 1 and 2) fix the baseline DP3 architecture and vary only the training data. Meanwhile, Section 5.3 (Table 3) evaluates the dynamic-aware adaptive policy against DP3 only in simulation under the latency-aware evaluator. There is no real-world experiment comparing the dynamic-aware adaptive policy against standard DP3 inference on physical hardware, where camera frame rates (20 Hz), point-cloud preprocessing latency, and physical actuator delays interact.

Questions for authors
  • Can you provide real-world latency and success rate comparisons between standard DP3 and your dynamic-aware adaptive policy on at least 1–2 real-world tasks?
  • Why do 50 manual dynamic demonstrations fail on real-world Boat Loading (3.33%)? In Table 2, 50 manual dynamic demos achieve ~70% on Dynamic Tapping, Belt Picking, and Bottle Catching, yet only 3.33% (1/30) on Boat Loading, whereas 1 static demo augmented to 50 episodes reaches 23.3% (and 66.7% at 200 episodes). What causes human dynamic teleoperation to fail so drastically on Boat Loading?
  • What does each adaptive component contribute beyond a fixed schedule? On a four-stage task such as Mole Whacking, could you use the same checkpoint and evaluation episodes to compare:
    1. Fixed TaylorSeer + fixed nexecn_{\mathrm{exec}};
    2. Adaptive TaylorSeer + fixed nexecn_{\mathrm{exec}};
    3. Fixed TaylorSeer + adaptive nexecn_{\mathrm{exec}};
    4. Both adaptive? Please specify the fixed settings and report success rate and policy-query latency (ms) for each configuration, so that the contribution of each adaptive component can be assessed separately.
  • How do the ablation columns in Table 4 / Table 10 map to Φstatic,Φdyn,Φinter,Φtrans\Phi_{\text{static}}, \Phi_{\text{dyn}}, \Phi_{\text{inter}}, \Phi_{\text{trans}}, and what happened on Object Catching? Could you explicitly write out how Spatial Only, Quasi-static, and Planned Motion are defined in terms of the four Φ\Phi operators from Section 3.2? Specifically, since Appendix B.5.1 (line 1088) defines Object Catching as Ξcatch=Φstatic→Φdyn\Xi_{\text{catch}} = \Phi_{\text{static}} \rightarrow \Phi_{\text{dyn}} with no Φinter\Phi_{\text{inter}} phase, why does adding local interaction replay (Ours vs. Planned Motion) boost Object Catching from 53% to 64% (or 68% in Table 10)? Also, why do the final numbers for Ours differ between Table 4 (64% / 88%) and Tables 10–11 (68% / 87%)?
9 comments

Weakness

Weakness 1

We thank the reviewer for identifying this gap and apologize that our original ablations did not sufficiently isolate the contribution of stage-conditioned scheduling. Our additional four-way ablation on Mole Whacking shows that adaptive Taylor scheduling and adaptive chunk execution each improve success over the jointly fixed baseline. Combining both raises success from 29% to 44%, with a 0.58 ms increase in policy-query latency (38.46 to 39.04 ms).

All four scheduling configurations use 10 DDIM denoising steps and a maximum action horizon of 8. Fixed parameters were selected through validation-set tuning, while adaptive schedules follow the stage-specific rules in Appendix Table 5. The four entries in each schedule correspond to the static, dynamic, interaction, and transition phases, respectively.

ConfigurationTaylor interval τ\tauTaylor order KKExecution steps
Fixed Taylor + Fixed Chunk5 / 5 / 5 / 52 / 2 / 2 / 26 / 6 / 6 / 6
Adaptive Taylor + Fixed Chunk3 / 7 / 3 / 81 / 1 / 1 / 26 / 6 / 6 / 6
Fixed Taylor + Adaptive Chunk5 / 5 / 5 / 52 / 2 / 2 / 22 / 2 / 8 / 8
Both Adaptive (Ours)3 / 7 / 3 / 81 / 1 / 1 / 22 / 2 / 8 / 8

The four-way Mole Whacking results are:

ConfigurationSuccess rate (%)Latency (ms)
Fixed Taylor + Fixed Chunk2938.46
Adaptive Taylor + Fixed Chunk3637.78
Fixed Taylor + Adaptive Chunk3437.76
Both Adaptive (Ours)4439.04

Adaptive Taylor acceleration and adaptive chunk execution improve success by 7 and 5 percentage points, respectively. Combining both improves success by 15 percentage points over the fixed joint baseline, with a 0.58 ms increase in latency.

Due to hardware availability constraints, latency measurements for this additional ablation were conducted on a different hardware configuration from those reported in Table 3. The absolute latency values may therefore differ.

Weakness 2

We thank the reviewer for emphasizing the importance of real-world evaluation. We evaluated the dynamic-aware adaptive policy against standard DP3 inference on the real-world Belt Picking task. The comparison shows a 13.3-percentage-point increase in success rate and a 1.89× speedup in mean policy-query latency. The success-rate, policy-query latency, and shared system-latency results are detailed in our response to Question 1.

Question 1

We thank the reviewer for emphasizing the importance of evaluating both inference latency and task success on physical hardware. We conducted an additional real-world comparison between standard DP3 and our dynamic-aware adaptive policy on Belt Picking. Both policies were trained for 300 epochs using the same 200 augmented demonstrations and evaluated over 30 real-world trials under the same protocol.

The experimental setup differed from that used for Table 2 because of an unavoidable change in the physical setup. The success rates below should therefore be compared between the two policies in this experiment, rather than directly against the Table 2 results.

MethodTrialsSuccessful trialsSuccess rate
Standard DP3302170.0%
Dynamic-aware DP3302583.3%
MethodMean policy-query latency (ms)P95 policy-query latency (ms)
Standard DP376.8778.14
Dynamic-aware DP340.7851.75

The dynamic-aware policy improves success by 13.3 percentage points. It reduces mean policy-query latency by 46.9%, corresponding to a 1.89× speedup, and reduces P95 latency by 33.8%. Latency is measured from prepared observation inputs to returned action predictions over 30 recorded deployments per method.

We also profiled shared sensing and control overheads associated with real-world execution:

Shared system stepReference latency (ms)
Camera read and depth-map alignment5.59
Point-cloud preprocessing (fully warmed)5.90
Joint-feedback cache read0.14
Control computation and robot-command send/return0.84
Total shared overhead12.47

These shared-path measurements are reported separately from policy-query latency.

Question 2

We thank the reviewer for drawing attention to the low success rate of manual demonstrations on Boat Loading. In Table 2, the policy trained on 50 manual demonstrations succeeded in 3.33% of evaluation trials (1/30). Policies trained on episodes generated from one static demonstration achieved 23.3% success with 50 episodes and 66.7% with 200 episodes.

During policy evaluation, the gripper repeatedly released the block before reaching the boat, causing it to fall into the water. To investigate this failure, we reviewed the 50 training demonstrations and compared them with the augmented trajectories. We believe the low success rate reflects both limited spatial coverage and the difficulty of collecting successful demonstrations for this fast-moving target.

First, the release positions in human demonstrations can be concentrated around an operator's habitual choices. A policy trained on this narrow distribution may fail when the boat follows an unfamiliar trajectory. Our augmentation procedure samples a broader range of object motions and release configurations.

Second, the fast boat requires the operator to anticipate the release time. The reviewed demonstrations often lack a distinct follow behavior in which the gripper tracks the boat before release. Our augmented trajectories explicitly model this tracking relationship, providing temporal cues that can help the policy decide when to release the block.


Question 3

We thank the reviewer for this helpful suggestion and apologize that our original presentation did not clearly isolate the contributions of the adaptive components. We evaluated all four configurations using the same policy checkpoint and the same evaluation episodes. Each adaptive component improved success over the jointly fixed schedule. Combining both yields the highest success on Mole Whacking.

All four configurations use 10 DDIM denoising steps and a maximum action horizon of 8. Fixed parameters were selected through validation-set tuning, while adaptive schedules follow the stage-specific rules in Appendix Table 5. The four entries in each schedule below correspond to the static, dynamic, interaction, and transition phases, respectively.

ConfigurationTaylor interval τ\tauTaylor order KKExecution steps
Fixed Taylor + Fixed Chunk5 / 5 / 5 / 52 / 2 / 2 / 26 / 6 / 6 / 6
Adaptive Taylor + Fixed Chunk3 / 7 / 3 / 81 / 1 / 1 / 26 / 6 / 6 / 6
Fixed Taylor + Adaptive Chunk5 / 5 / 5 / 52 / 2 / 2 / 22 / 2 / 8 / 8
Both Adaptive (Ours)3 / 7 / 3 / 81 / 1 / 1 / 22 / 2 / 8 / 8
ConfigurationMole Whacking success rate (%)Latency (ms)
Fixed Taylor + Fixed Chunk2938.46
Adaptive Taylor + Fixed Chunk3637.78
Fixed Taylor + Adaptive Chunk3437.76
Both Adaptive (Ours)4439.04

Adaptive Taylor scheduling and adaptive chunk execution individually improve success by 7 and 5 percentage points, respectively. Combining both improves success by 15 percentage points over the jointly fixed baseline (44% vs. 29%), with a 0.58 ms increase in latency (39.04 vs. 38.46 ms).

Due to hardware availability constraints during the rebuttal, latency measurements for this additional ablation were conducted on a different hardware configuration from those reported in Table 3. The absolute latency values may therefore differ.

Reviewer_oSqdOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work tackles two central bottlenecks in learning-based dynamic robot manipulation: the prohibitive cost of collecting diverse dynamic demonstration data, and the high inference latency of existing visuomotor policies that impairs real-time reactivity. Experiments demonstrate that the method substantially improves data efficiency, reduces average policy query latency by 32.9%, and lifts mean task success rate by 18.4 percentage points over a 3D diffusion policy baseline.

Critical strengths
  • The core premise—deriving rich dynamic training data from a single static demonstration—addresses a major pain point in robotic learning. The cost comparison (≈11 minutes of human effort for 200 augmented episodes vs. ≈400 minutes for manual dynamic teleoperation) clearly quantifies the practical savings, making the approach highly relevant for real-world deployment.
  • The stage-aware scheduling strategy is well-motivated: different manipulation phases have fundamentally different requirements for reactivity vs. motion smoothness. By combining a lightweight stage classification head with Taylor-accelerated DDIM and adaptive action chunking, the method achieves simultaneous improvements in both latency and success rate, which is an uncommon and valuable result.
  • The paper provides detailed network architectures, training hyperparameters, inference procedures, and hardware specifications, and commits to releasing code upon acceptance. This meets high reproducibility standards for robotics learning research.
Critical weaknesses
  • The stage-conditioned inference parameters (Taylor interval, expansion order, chunk length) are set via manually defined heuristic rules, tuned for the benchmark tasks. There is no evidence that these handcrafted schedules transfer to unseen task categories (e.g., dynamic assembly, dexterous in-hand manipulation). Although the stage classifier is learned, the scheduling policy itself is not data-driven, which caps the framework’s scalability to more complex and diverse dynamic tasks.
  • More papers about Dynamic Manipulation should be cited and discussed to highlight the innovativeness of this paper. [1] Iterative Residual Policy for Goal-Conditioned Dynamic Manipulation of Deformable Objects. Robotics: Science and Systems 2022 [2] F2F-AP: Flow-to-Future Asynchronous Policy for Real-time Dynamic Manipulation. CoRL 2026 [3] Action-aware Dynamic Pruning for Efficient Vision-Language-Action Manipulation. ICLR 2026
  • All benchmark tasks use parallel-jaw grippers and revolve around a small set of simple interaction primitives (striking, catching, picking, blocking). There are no contact-rich dynamic assembly tasks, deformable objects, multi-object interactions, or highly unpredictable motion patterns. While the limitations section acknowledges this, the narrow scope weakens the claim of enabling "generalizable dynamic manipulation". The dynamic variations explored are mostly parameterizable velocity and position changes, which may not reflect unstructured real-world dynamics.
  • The inference scheduling ablation compares only three simplified variants (uniform DDIM, fixed Taylor, fixed short chunks) against the full method. It does not include an oracle-stage baseline (using ground-truth stage labels instead of predicted ones) to isolate the performance cost of stage prediction errors.
Questions for authors
  • Could you elaborate on the key algorithmic differences between your trajectory editing pipeline and prior work such as MimicGen, R2RGen and DemoGen, specifically in terms of modeling temporal dynamic coordination rather than quasi-static spatial variation?
  • Why was a heuristic scheduling strategy chosen over a learned scheduling policy? Do you expect that learning the scheduling parameters end-to-end would further improve performance and generalization?
  • What mechanisms ensure the physical plausibility of augmented dynamic interactions beyond geometric feasibility? Have you observed failure modes where physically unrealistic augmented trajectories degrade policy performance?
  • How would the framework handle dynamic tasks with highly unpredictable, non-parameterizable object motion (e.g., bouncing objects, external disturbances) that cannot be easily sampled in the augmentation pipeline?
  • If the authors resolve all my concerns, I will consider raising the final score.
6 comments
AuthorsOpenReview ↗

Reply to Weakness Mentioned by Reviewer oSqd

We sincerely thank Reviewer oSqd for the insightful and constructive feedback, which has significantly strengthened the quality and rigor of our work. We hope that our responses and additional experiments have thoroughly addressed your concerns and warrant a more positive evaluation of our paper.

Weakness 1

We thank the reviewer for noting the heuristic components of our method. We tested the generalizability of our method on a new real-world task using a dexterous hand, beyond our original benchmark. We conducted an additional pull-back-car experiment using the BrainCo Revo2 hand and PiPER across three surface conditions with the same protocol.

Surface ConditionSuccess Trials
Dry9/10
Wet8/10
Towel-coverd9/10

We adopt heuristic scheduling as a deliberate trade-off between pipeline simplicity, interpretability, computational cost and performance. Our ablation results also demonstrate that the proposed stage-aware scheduling is effective without requiring additional scheduler training, which is shown in Table 4.

We've also considered learning scheduling parameters through RL. However, this would introduce substantial additional optimization costs, particularly for real-world robotic tasks, without guaranteeing improvements in cross-task generalization.

The corresponding changes will be incorporated into the revised manuscript, which will be uploaded after we have addressed all reviewers' comments.


Weakness 2

We thank the reviewer for pointing out these relevant works. We will incorporate citations to all three suggested papers and include a brief discussion of them in the Related Work section of the revised manuscript.

Overall, IRP addresses deformable-object manipulation through iterative action refinement. F2F-AP compensates for asynchronous latency using flow-predicted future observations, while ADP accelerates VLAs through action-aware visual token pruning. DynamicManip complements these efforts through temporally coordinated static-to-dynamic demonstration synthesis and stage-conditioned scheduling of diffusion computation and action-chunk execution length.

The corresponding changes will be incorporated into the revised manuscript, which will be uploaded after we have addressed all reviewers' comments.


Weakness 3

Regarding unpredictable and non-parameterizable object motion, our framework focuses on modeling the relative temporal coupling between the end-effector and the moving object, rather than precisely reproducing object trajectories.

This is illustrated by the real-world Boat Loading task, in which the target moves irregularly on water. We explicitly modeled stage follow in our augmentation pipeline, and that made our augmented data well-distributed enough to enable the policy to react to irregular object motion through closed-loop feedback, as demonstrated by the real-world Boat Loading experiment.

Our additional dexterous-hand experiment further broadens the evaluation to a new task and embodiment, though we acknowledge that some dynamic tasks remain beyond the scope of our current evaluation.


Weakness 4

We appreciate the reviewer's constructive suggestion. Overall, our additional ablations indicate that our learned stage predictor achieves performance close to oracle-stage scheduling, while more accurate stage prediction could provide further improvements.

We have added an oracle-stage baseline to isolate the impact of stage prediction errors. Specifically, we replace the predicted stage labels with ground-truth labels during inference, while keeping the policy and stage-conditioned scheduling rules unchanged. The results are shown below:

TaskPredicted Stage (Ours)Oracle StageImprovement
Mole Whacking44%49%+5 pp
Object Catching64%66%+2 pp
Goal Blocking88%91%+3 pp

Oracle-stage scheduling does improve success rates across all three tasks, with gains of 2–5 percentage points. The largest improvement occurs in Mole Whacking, which involves frequent transitions between target tracking, interaction, and recovery stages. In contrast, Object Catching involves fewer stage transitions, while Goal Blocking already achieves a high success rate with predicted stages, leaving relatively limited space for further improvement.

AuthorsOpenReview ↗

Reply to Questions Raised by Reviewer oSqd (Part 1)

Question 1

Our key distinction from MimicGen, DemoGen, and R2RGen lies in explicitly modeling temporal coordination between robot actions and moving objects, rather than primarily augmenting spatial configurations.

In terms of modeling temporal dynamic coordination rather than quasi-static spatial variation, we synthesize time-varying object trajectories and generate corresponding robot motions through tracking and replanning, with contact-aligned interaction replay in tasks that include an interaction phase. Our method couples robot motions with changing object states instead of simply adapting demonstrations to new spatial configurations.

During revision, we found that some historical configuration records were missing. To maintain rigor, we have updated the following comparison by removing the Planned Dynamic Motion entry.

MethodObject CatchingGoal BlockingMole Whacking
Spatial Transforms2%4%15%
Quasi-static Editing26%34%29%
Ours68%87%80%

Additional design choices in our pipeline include:

  1. Offline augmentation: Generating dynamic demonstrations without additional on-robot rollouts.
  2. Geometric reconstruction: Maintaining geometrically consistent observations under edited object configurations.
  3. Kinematically constrained motion planning: Generating feasible robot motions instead of rigidly transforming entire trajectories.

Question 2

We completely agree that end-to-end scheduling optimization holds strong potential for further boosting performance and generalization. At the same time, minimizing sample complexity and ensuring predictable, interpretable behavior are equally critical. Our current heuristic design strikes a practical balance—delivering strong empirical gains (Table 4).

To further validate our heuristic scheduling strategy, we conducted a controlled ablation on Mole Whacking, which encompasses all four manipulation stages, using the same checkpoint and evaluation episodes.

The results show that adaptive TaylorSeer and action chunking independently improve success rates by 7 and 5 percentage points, respectively. Combining both achieves a 15-point improvement over the fixed configuration (44% vs. 29%), with comparable policy-query latency (39.04 vs. 38.46 ms). This demonstrates the effectiveness of stage-aware scheduling without requiring a separately trained scheduler.

TaylorSeer SchedulingAction Chunk SchedulingSuccess Rate (%)Policy-Query Latency (ms)
FixedFixed2938.46
AdaptiveFixed3637.78
FixedAdaptive3437.76
AdaptiveAdaptive4439.04

We have added a discussion on this trade-off in the Limitation/Future Work section.

Due to hardware availability constraints during rebuttal, the latency measurements in this ablation were conducted on a different hardware configuration from Table 3, resulting in differences in the reported absolute latency values.

The corresponding changes will be incorporated into the revised manuscript, which will be uploaded after we have addressed all reviewers' comments.


AuthorsOpenReview ↗

Reply to Questions Raised by Reviewer oSqd (part 2)

Question 3

Our pipeline mitigates physically implausible interactions through following mechanisms:

  1. In Stage Interact, we reuse local contact motions and interaction timing from the original demonstration, leveraging the valuable priors that human teleoperators hold about the physical world.
  2. In Stage Follow, we apply velocity-constrained motion planning to generate smooth and kinematically feasible trajectories
  3. In Stage Interact, we align robot and object velocity during contact to reduce excessive relative contact velocity.

Within our evaluated tasks, we do not identify severe performance degradation directly attributable to physically unrealistic augmented trajectories. However, we acknowledged that our method does not explicitly model full and complex contact dynamics such as impacts and unexpected object slippage. As discussed in appendix H, the rigid-grasp assumption limits its applicability to sliding or non-rigid contacts. Overall, physically unrealistic augmentation is, indeed, expected to degrade model's performance, so plenty of our pipeline design focuses on eliminating these cases, though in some cases (which is mentioned in Limitation) our pipeline is not applicable and may be expected to be degradation.


Question 4

Rather than precisely modeling the irregular motion, we instead focus on the relative temporal coupling between the end-effector and the object. As discovered in our researching process, more efforts should instead be made to model the relative temporal correlation between the gripper's end-effector and the object, namely the modeling of the following stage. As shown in our real-world settings, the Boat Loading task serves as strong evidence that our methods are capable of handling irregular and difficult-to-predict object motion in real-world settings.

For the Boat Loading task, the setting is a simple aerodynamic boat that is free from human intervention and cannot be remotely controlled; Driven by a tilted aerodynamic propeller, it moves freely around the water through irregular circular motion. We explicitly modeled stage follow in our augmentation pipeline, and that made our augmented data well-distributed enough that the model developed the ability to react based on the distance or the relative temporal association between the end-effector and the boat. This enables the policy to react to irregular object motion through closed-loop feedback, as demonstrated by the real-world Boat Loading experiment.

Comments on the paper

2 comments

Dear reviewers, the discussion period is ending. Can you please engage with the rebuttal submitted by the authors? Thanks, your AC

Dear Area Chair,

Thank you for helping facilitate the discussion. We have uploaded the revised manuscript, with changes highlighted in blue, incorporating the additional experiments and clarifications from our rebuttal.

We have posted a revision summary in each reviewer’s discussion thread, with section and table references to help locate the changes. We appreciate your time and coordination and remain available to clarify any outstanding questions from you or the reviewers.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.