acceptodds
Under review as a conference paper at ICLR 2027

DSDyn-VLA: A Dual-Stream Dynamic Manipulation Framework with Motion Perception, Future Awareness, and Realtime Correction

14 views

Abstract

While Vision-Language-Action (VLA) models excel in static tasks, they struggle in dynamic environments where objects are in motion (e.g., conveyor belt manipulation). We identify three fundamental limitations hindering current VLAs in these scenarios: the perception gap, where static visual inputs lack temporal motion cues; the latency gap, where inference delays render actions obsolete; and the control gap, caused by the open-loop action chunk execution without real-time adjustment. In this work, we propose DSDyn-VLA, a Slow-Fast Dual-Stream Dynamic manipulation framework that integrates motion-aware foresighted planning with real-time residual correction. The slow Flow-Planner serves as a macro-planner. By enhancing the VLA with optical flow for temporal perception and a future state awareness mechanism to preemptively offset inference latency, it produces globally consistent, motion-aware action chunks. Complementing this, the fast Res-Refiner employs a lightweight RL policy to inject high-frequency, closed-loop corrections into the planned action chunks based on real-time observations. In addition, we introduce DynBench, a MuJoCo-based benchmark for dynamic object manipulation that comprises nine tasks. Extensive experiments demonstrate that DSDyn-VLA reduces the failure rate by over 76% compared to current SOTA method in high-latency setting on the Kinetix dynamic benchmark, while achieving about 6 the success rate of PI0.5 in real-world dynamic settings and about 5 on DynBench. We will open-source all the code and weights.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Embodied AI, Vision-Language-Action Model, Dynamic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VKj6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper combines a VLA planner that uses optical flow, a prediction of the robot state when actions will run, and a fast RL controller that corrects those actions. It is tested on Kinetix, nine DynBench tasks and four real-world dynamic manipulation tasks.

Critical strengths
  1. The method addresses delays when controlling robots around moving objects. Average real-world success rises from 38.1% with DynamicVLA to 49.0% (Table 1). Tests that remove individual modules also help explain the gain.
Critical weaknesses
  1. Equation 5 predicts the future robot state using pending planner actions, but the robot executes actions with residual corrections. The paper does not measure the prediction error caused by this difference or explain how it is handled. This matters because the predicted state is a key input to the planner.
  2. The polar-mapping test also changes the encoder. Raw flow uses a ResNet-34 trained from scratch, while mapped flow uses CLIP (Section 4.3). The gain could therefore come from the encoder or its pretraining. A comparison with matched encoders would better test polar mapping itself.
  3. The delay tests use state-based Kinetix with an MLP planner and no optical flow (Section 4.1). They support the control design, but do not test how the full visual VLA system handles delays. Similar tests with delayed or irregular inputs on visual manipulation tasks would make that claim stronger.
Questions for authors
  1. How is Eq. 5 implemented for the robot's action representation? How accurate is the predicted state with the residual controller enabled, and are its pending corrections included?
  2. Please report demonstration counts, online environment steps and reward-labeling effort for each method, including the stronger Table 11 baselines. Are these budgets matched?
3 comments

We sincerely thank the reviewer for the constructive feedback and for recognizing our clear presentation, effective delay compensation in dynamic manipulation, substantial real-world performance gains, and informative module ablations. Below, we address each concern in detail and summarize our additional evidence.

W1 & Q1 Implementation details of future proprioceptive state prediction in Eq. (5) and impact of residual corrections

Thank you for these insightful questions on Eq. (5) and state prediction accuracy. Below, we clarify the implementation of fkinf _ {\text{kin}}, how residual corrections are handled, and the actual prediction accuracy:

1. Implementation of fkinf _ {\text{kin}} in Eq. (5): Equation (5) is written as a general kinematics rollout formulation from a relative-action perspective. In our actual implementation, however, because both the proprioceptive state q\mathbf{q} and Flow-Planner's predicted action space are absolute EEF poses in the robot base frame (3D position, orientation, and gripper width), Abuffer[Tlat]\mathbf{A} _ {\text{buffer}}[T _ {\text{lat}}] naturally represents the target proprioceptive state at t+Tlatt + T _ {\text{lat}}. Therefore, predicting the future state q^future\hat{\mathbf{q}} _ {\text{future}} (at t+Tlatt + T _ {\text{lat}}) simplifies directly to taking q^t+Tlat=Abuffer[Tlat]\hat{\mathbf{q}} _ {t + T _ {\text{lat}}} = \mathbf{A} _ {\text{buffer}}[T _ {\text{lat}}], avoiding multi-step rollout errors with zero compute overhead.

2. Are pending residual corrections included? No—residual corrections are not included in Eq. (5), and including them is unnecessary:

  • Causality: Res-Refiner computes each residual online from real-time camera images at execution. When Flow-Planner starts inference at time tt, future visual observations over [t,t+Tlat][t, t + T _ {\text{lat}}] are not yet available, so future residuals cannot be obtained in advance.
  • Minimal difference caused by residuals: Because Abuffer\mathbf{A} _ {\text{buffer}} specifies absolute target poses, residual corrections never accumulate across steps—each step's residual only locally adjusts that step's target pose and does not carry over to subsequent steps. Moreover, each single-step residual is strictly bounded within 10% of the action range, so omitting them results in only a minimal difference from the actual state at t+Tlatt + T _ {\text{lat}}.

3. State prediction accuracy with Res-Refiner enabled: Across 50 complete real-world trajectories with Res-Refiner enabled, we measured the mean error between the predicted state q^t+Tlat\hat{\mathbf{q}} _ {t + T _ {\text{lat}}} and the robot's true physical EEF pose qt+Tlat\mathbf{q} _ {t + T _ {\text{lat}}} at execution time:

State Comparison (Real-World, w/ Res-Refiner Enabled)Position Error (mm)Orientation Error (deg)
Unaligned inference-time state qt\mathbf{q} _ t vs. true execution-time state qt+Tlat\mathbf{q} _ {t + T _ {\text{lat}}}42.66.8
Predicted future state q^t+Tlat\hat{\mathbf{q}} _ {t + T _ {\text{lat}}} vs. true execution-time state qt+Tlat\mathbf{q} _ {t + T _ {\text{lat}}}3.10.6

Conditioning on the unaligned state qt\mathbf{q} _ t causes a 42.6 mm42.6\text{ mm} (>4 cm> 4\text{ cm}) mismatch during inference latency, whereas our prediction—even with Res-Refiner enabled and without modeling future residuals—reduces the error to just 3.1 mm and 0.6 deg. This millimeter-level offset has negligible impact on Flow-Planner's macro-planning, and any remaining local error at execution is corrected online by our 30 Hz closed-loop Res-Refiner.

W2 Controlled ablation study on optical flow polar mapping

We appreciate the suggestion to test polar mapping with matched encoders. To evaluate the effect of polar mapping alone while keeping the encoder architecture and pretraining identical, we added a controlled variant: Raw Flow (Zero-Padded) + Pretrained CLIP ViT. We linearly normalize 2-channel raw flow (u,v)(u, v) to [0,1][0, 1], append a zero channel to form (u,v,0)(u, v, 0), and feed it into the exact same pretrained CLIP ViT as our full model—making polar mapping the sole difference:

Method VariantFlow RepresentationEncoder & PretrainingDynBench (SR %)Real-World (SR %)
w/o Polar Map (Original ablation, ResNet-34)2-channel raw flow (u,v)(u, v)ResNet-34 (from scratch)37.233.0
w/o Polar Map (Zero-padded raw flow, controlled)3-channel padded raw flow (u,v,0)(u, v, 0)CLIP ViT (pretrained)39.136.0
DSDyn-VLA (Full)3-channel polar color map (Hue & Sat.)CLIP ViT (pretrained)48.448.5

Even with the identical pretrained CLIP ViT, zero-padded raw flow (u,v,0)(u, v, 0) still performs significantly worse than polar mapping (39.1% vs. 48.4% on DynBench; 36.0% vs. 48.5% real-world), as raw flow deviates sharply from natural RGB distributions and cannot be effectively processed by the pretrained CLIP encoder. Polar color mapping yields +9.3 and +12.5 percentage-point gains under the same encoder, confirming that the gain stems from Polar Mapping's feature-distribution alignment rather than encoder or pretraining differences (added to Section 4.3, Table 4).

W3 Delay robustness evaluation of the complete visual system on visual manipulation tasks

Thank you for the valuable suggestion to evaluate delay robustness on visual tasks. First, our real-world experiments already run under actual system latency, and our visual simulation benchmark DynBench simulates an inference delay of δ=3\delta = 3 steps by default (clarified in Section 4.2). Together, these two sets of experiments verify the delay robustness of our complete visual system.

To further test DSDyn-VLA's delay robustness on a vision-based benchmark, we additionally evaluated varying delay steps δ∈{2,3,4,5}\delta \in \{2, 3, 4, 5\} on DynBench:

Methodδ=2\delta=2 (SR %)δ=3\delta=3 (Default, SR %)δ=4\delta=4 (SR %)δ=5\delta=5 (SR %)
π0.5\pi _ {0.5} (Baseline)13.09.77.43.6
RTC22.919.416.710.3
VLASH29.727.124.819.0
DynamicVLA38.134.330.925.4
DSDyn-VLA (Ours)49.348.448.047.6

The results show that as visual delay steps increase, all baselines degrade noticeably (e.g., DynamicVLA drops from 38.1% to 25.4%), whereas DSDyn-VLA consistently maintains stable, leading success rates (49.3% →\to 47.6%).

Q2 Clarification on training budget comparability and reward-labeling effort

We appreciate the request to report exact budgets and reward-labeling effort. In our real-world comparisons with the strengthened baselines in Appendix D Table 11, all methods use the exact same 50 offline demos/task. Because PPO is less sample-efficient, we gave the PPO baselines 100 online interaction episodes/task, whereas our RLPD-trained Res-Refiner uses only 70 online interaction episodes/task:

Setting (Real-World, Table 11)Offline Demos / TaskOnline Interaction Episodes / TaskManual Reward Labels / TaskAvg. Success (%)
RTC + history frames + PPO50100 (<40,000< 40{,}000 steps)100 (1 button press / ep.)24.0
VLASH + history frames + PPO50100 (<40,000< 40{,}000 steps)100 (1 button press / ep.)32.5
DynamicVLA + history frames + PPO50100 (<40,000< 40{,}000 steps)100 (1 button press / ep.)39.0
DSDyn-VLA (Ours)5070 (<28,000< 28{,}000 steps)70 (1 button press / ep.)48.5

Simulation rewards are assigned automatically (0 manual labels); real-world RL uses sparse episode-end binary rewards (r∈{0,1}r \in \{0, 1\}) requiring 1 key press/episode, totaling 70 button presses/task vs. strengthened PPO's 100. Thus, DSDyn-VLA achieves higher success (48.5% vs. 39.0%) with identical offline data and strictly fewer online interaction episodes and manual labels. We have added these details to the revised manuscript.

Dear Reviewer VKj6,

As the author–reviewer discussion period is coming to a close (Nov 18 AoE), we wanted to briefly follow up to see if our responses and newly added experiments have addressed your questions. In particular, we have included:

  1. Implementation details of Eq. (5) and real-world prediction accuracy with Res-Refiner active, showing that in absolute EEF space residuals do not accumulate and latency-induced mismatch drops from 42.6 mm / 6.8° to 3.1 mm / 0.6° across 50 real-world rollouts (Sec. 3.3.2, Appendix G.5 Table 17).
  2. Matched-encoder Polar Mapping ablation feeding normalized zero-padded raw flow into the exact same pretrained CLIP ViT (39.1% vs. 48.4% on DynBench; 36.0% vs. 48.5% real-world, Sec. 4.3 Table 4, Appendix G.3 Table 15).
  3. Visual inference delay evaluations on DynBench confirming stable performance across increasing visual delays (49.3% → 47.6%, Appendix G.1 Table 13), along with exact online step counts and episode-end button-press labeling budgets (70 vs. 100 presses/task for PPO baselines, Appendix D Table 11).

We would be glad to answer any final questions you might have before the discussion window closes. Thank you very much for your constructive suggestions!

Best regards, Authors of Submission #16046

Reviewer_igsBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DSDyn-VLA addresses dynamic manipulation through a slow planner and a fast residual controller. The planner augments a VLA with optical-flow inputs and future proprioceptive-state conditioning to handle motion and inference delay. An RL-trained residual policy corrects actions using current observations during chunk execution. The paper introduces DynBench and reports improvements across simulation benchmarks and four real-world tasks.

Critical strengths
  1. The empirical evaluation provides meaningful support for the system. Average real-world success increases from 38.1% with DynamicVLA to 49.0%, with repeated trials across four tasks. Results on DynBench and external benchmarks provide additional evidence beyond a single setting.
  2. The ablations support contributions from motion information, future-state conditioning, and residual correction. Appendix D strengthens the evaluation by including history-frame and additional-RL baselines, as well as a frame-stacking alternative.
Critical weaknesses

The distinction from the closest prior methods needs stronger treatment. A2C2 already performs observation-conditioned, per-step residual correction of VLA action chunks and evaluates on the same Kinetix suite. VLASH already uses future-state-aware asynchronous inference. Although these works are cited, A2C2 is absent from the direct comparisons. This limits how much novelty can be attributed to the slow–fast structure itself; the contribution is better supported as the integration of motion cues, latency compensation, and RL-based correction.

Training-budget comparability and the scope of generalization remain unclear. The additional PPO baselines in Appendix D are useful, but equal training time does not establish equal environment interactions or data access. The paper should also clarify whether residual policies are trained separately for each task and whether they are retrained on LIBERO and DOMINO. These details affect whether the results demonstrate a broadly reusable controller or effective task-specific adaptation.

Questions for authors

How does the Res-Refiner with SFT differ from A2C2 in inputs, architecture, and training targets? Does this ablation test the same correction mechanism? Is one residual policy shared across tasks, or is a separate policy trained for each task? Were the residual policies retrained for LIBERO and DOMINO? What are the online interaction counts and offline-data budgets for DSDyn-VLA and the strengthened PPO baselines in Appendix D?

4 comments

We sincerely thank the reviewer for the constructive feedback and for recognizing our solid empirical results across real-robot and diverse simulation benchmarks, as well as our thorough module ablations and strengthened baseline comparisons. Below, we address each concern in detail and summarize our additional evidence.

W1 & Q1 Relationship to prior work, detailed comparison with A2C2, and core contributions

We appreciate the request for a clearer comparison with VLASH and A2C2. We have expanded the Related Work section in the main text to thoroughly discuss both methods. Below, we clarify the technical differences between our DSDyn-VLA and these closely related methods—including how Res-Refiner (w/ SFT) differs from A2C2—and highlight our core contributions.

Comparison with VLASH: Both VLASH and DSDyn-VLA aim to align the policy to the state at the future execution moment to mitigate action staleness. This future state consists of two parts: the robot's own proprioceptive state and the external environment state. However, VLASH aligns only the former while overlooking the latter. In dynamic manipulation, the future external environment state (especially moving targets) at execution time is far more critical than future proprioception, as it directly dictates the actual manipulation target. DSDyn-VLA aligns both: beyond future proprioception, optical-flow velocity cues let the policy implicitly model the positions moving objects will reach at execution time, achieving future compensation for external dynamics (implicitly aligning the future external environment state).

Comparison with A2C2: Our Res-Refiner differs fundamentally from A2C2 in three aspects:

  1. Training targets & state distribution: A2C2 uses SFT on the offline demonstration distribution to regress (expert−base)(\text{expert} - \text{base}) actions, seeing only demonstration-trajectory states. At deployment, every residual correction compounded with latency alters subsequent states, driving the robot into out-of-distribution (OOD) states never encountered in demonstrations. Conversely, Res-Refiner learns via RL directly inside the (base policy+residual+actual latency)(\text{base policy} + \text{residual} + \text{actual latency}) closed-loop system, keeping learning and deployment state distributions completely consistent.
  2. Supervision source: A2C2 is supervised by expert trajectories, where inevitable human teleoperation reaction delays impose a ceiling on A2C2's performance; Res-Refiner uses task-success-rewarded RL and is not limited by demonstration quality.
  3. Inputs, architecture & base-policy coupling: A2C2 relies on stale latent features zz from the base VLA's previous inference pass (which encode outdated visual observations) and uses a larger, slower Transformer head. Conversely, Res-Refiner treats the VLA strictly as a black box, taking only real-time images, proprioception, and base actions to eliminate the impact of outdated visual information. In addition, it uses a lightweight ResNet-10 + Spatial Embedding + MLP architecture, enabling fast inference in just 3 ms per step.

Distinction between Res-Refiner (w/ SFT) and A2C2: Res-Refiner (w/ SFT) in Table 4 is not equivalent to A2C2. Although both share the same training target, Res-Refiner (w/ SFT) keeps the same inputs and architecture as Res-Refiner (taking no stale base-model latent features and using the lightweight ResNet-10 + Spatial Embedding + MLP architecture). To provide the most direct comparison, in the table below we jointly report comparative results for Res-Refiner (w/ SFT), the original A2C2 applied directly to π0.5\pi_{0.5}, and the variant replacing Res-Refiner with the original A2C2:

Method / VariantDynBench (SR %)DOMINO (SR %)DOMINO (MS)
π0.5\pi_{0.5} (Baseline)9.79.6326.19
π0.5\pi_{0.5} + VLASH27.114.0132.29
π0.5\pi_{0.5} + A2C213.010.6828.05
DSDyn-VLA (replace Res-Refiner w/ A2C2)44.618.2735.90
DSDyn-VLA (Res-Refiner w/ SFT)45.318.9237.45
Our DSDyn-VLA48.420.7441.64

This isolates both sources of gain: Res-Refiner's real-time black-box architecture outperforms A2C2 under identical SFT targets (44.6% →\to 45.3% SR on DynBench; 18.27% →\to 18.92% SR and 35.90 →\to 37.45 MS on DOMINO), while closed-loop RL further raises performance to 48.4% SR on DynBench and 20.74% SR / 41.64 MS on DOMINO.

To address the missing Kinetix comparison you noted, we evaluated A2C2 on Kinetix under the exact same latency (δ\delta) and execution horizon (KK) settings:

Kinetix (12 Tasks Avg.)δ=0,K=1\delta=0, K=1δ=0,K=8\delta=0, K=8δ=1,K=1\delta=1, K=1δ=1,K=7\delta=1, K=7δ=2,K=2\delta=2, K=2δ=2,K=6\delta=2, K=6δ=3,K=3\delta=3, K=3δ=3,K=5\delta=3, K=5δ=4,K=4\delta=4, K=4
A2C290.3%87.8%89.5%87.8%87.4%86.6%87.4%86.0%86.5%
Our DSDyn-VLA97.2%97.2%97.3%97.1%96.9%97.3%97.0%97.3%97.1%

DSDyn-VLA outperforms A2C2 by +6.9 to +11.3 percentage points across all settings with virtually zero degradation as δ\delta grows.

Core Contributions: As you noted, our contribution is not confined to an isolated module. The results above confirm that dynamic manipulation cannot be solved by any standalone component. Rather than the design of any single module, we believe our main contribution to the field lies in clearly answering "what core problems must be solved to achieve dynamic manipulation." Through a thorough, first-principles analysis, we identify the three indispensable and tightly coupled bottlenecks of dynamic manipulation—the Perception Gap, Latency Gap, and Control Gap—and build a complete, tightly coordinated system around them. Showing the community what truly bottlenecks dynamic manipulation, establishing a clear analytical framework, and demonstrating that solving all three together brings a major leap in performance represent the core value of our work beyond individual component designs.

W2.1 & Q3 Clarification on training budget comparability

Thank you for asking for a clearer breakdown of the training and interaction budgets. In our real-world comparisons with the strengthened baselines in Appendix D Table 11, all methods use the exact same 50 offline demos/task. Because PPO is less sample-efficient, we gave the PPO baselines 100 online interaction episodes/task, whereas our RLPD-trained Res-Refiner uses only 70 online interaction episodes/task:

Setting (Real-World, Table 11)Offline Demos / TaskOnline Interaction Episodes / TaskAvg. Success (%)
RTC + history frames + PPO50100 (<40,000< 40{,}000 steps)24.0
VLASH + history frames + PPO50100 (<40,000< 40{,}000 steps)32.5
DynamicVLA + history frames + PPO50100 (<40,000< 40{,}000 steps)39.0
DSDyn-VLA (Ours)5070 (<28,000< 28{,}000 steps)48.5

DSDyn-VLA achieves higher success (48.5% vs. 39.0%) with identical offline data and strictly fewer online interaction episodes (70 vs. 100), confirming that the gain stems from our system architecture rather than extra interaction data. We have added these budget details to the revised manuscript.

W2.2 & Q2 Generalization scope and retraining setup of Res-Refiner

Thank you for this valuable question on the task-sharing scope of Res-Refiner. First, Res-Refiner is multi-task shared within each benchmark (e.g., one Res-Refiner covers all 40 LIBERO tasks, and one covers all 35 DOMINO tasks), confirming multi-task generalization. Second, yes, it was retrained on LIBERO and DOMINO: since each benchmark uses a distinct post-trained Flow-Planner, our default setup trains one Res-Refiner per benchmark via RL on top of its corresponding Flow-Planner. We also tested training a single unified Res-Refiner across all three simulation benchmarks simultaneously (with one Flow-Planner per benchmark):

Method / VariantDynBench (SR %)LIBERO-10 (SR %)DOMINO (SR %)
Flow-Planner only (w/o Refiner)42.791.7314.77
Full DSDyn-VLA (w/ Unified Res-Refiner)46.193.0719.22
Full DSDyn-VLA (Default, per-benchmark Res-Refiner)48.493.3020.74

Although slightly below dedicated per-benchmark Res-Refiners, a single unified Res-Refiner still transfers effectively across all three benchmarks, achieving substantial gains over Flow-Planner alone (e.g., 42.7% →\to 46.1% on DynBench and 14.77% →\to 19.22% on DOMINO). This generality comes mainly from formulating both proprioceptive states and residual actions in absolute EEF space, which naturally compensates for embodiment differences.

Dear Reviewer igsB,

With the discussion period approaching its deadline (Nov 18 AoE), we wanted to gently follow up to check whether our rebuttal and new experiments have resolved your questions regarding soundness and experimental scope. In summary, we have provided:

  1. Direct A2C2 evaluations and exact comparisons with Res-Refiner (w/ SFT) across Kinetix (+6.9 to +11.3 pp across all latency/horizon settings), DynBench, and DOMINO, isolating the gains from both our real-time black-box architecture and closed-loop RL (Sec. 2, Table 2, Appendix F Table 12).
  2. Exact offline and online interaction budgets showing DSDyn-VLA achieves higher real-world success (48.5% vs. 39.0%) with identical offline demos (50/task) and strictly fewer online episodes than the strengthened PPO baselines (70 vs. 100 episodes/task, Appendix D Table 11).
  3. Clarification on multi-task sharing and cross-benchmark generality, including a new unified Res-Refiner trained jointly across DynBench, LIBERO, and DOMINO (42.7% → 46.1% on DynBench; 14.77% → 19.22% on DOMINO, Appendix G.4 Table 16).

If anything remains unclear before the discussion window closes, we are standing by and happy to discuss further. Thank you again for your thoughtful review and support!

Best regards, Authors of Submission #16046

Reviewer_RkufOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DSDyn-VLA, which uses a low-frequency Flow-Planner to generate action sequences and a high-frequency Res-Refiner to correct actions based on real-time observations. The Flow-Planner incorporates optical flow encoding and future robot proprioceptive state conditions into π0.5\pi_{0.5}, while the Res-Refiner is trained via reinforcement learning. The authors report results on Kinetix, the self-built DynBench, four real-robot tasks, as well as LIBERO and DOMINO in the appendix, and provide module ablations and partial reinforcement baseline comparisons.

Critical strengths

The paper selects a dynamic manipulation problem with application value. The system structure is clear. And the three modules, incuding the perception, latency compensation, and real-time feedback module are related to distinct execution needs. Experiments include both simulation and real robot tasks, demonstrating performance improvements of the combined system over the selected baselines. The supplementary external dynamic benchmarks, historical frame inputs, and additional reinforcement learning comparisons in the appendix also provide some support for the system's effectiveness.

Critical weaknesses

First, the demonstration of novelty is insufficient. Future state compensation is directly related to VLASH, real-time correction of VLA action chunks is highly relevant to A2C2, and stacking learned residuals on a base policy already has mature research. The related work in the main text is only one paragraph; although the appendix has an extension, it still does not fully discuss the technical differences between these direct neighbors and this paper.

Second, the ablations are insufficient to support mechanistic interpretation. The optical flow encoding comparison simultaneously changes the representation method, encoder structure, and pre-training conditions, making it impossible to clearly attribute the gains to polar mapping.

Finally, the correspondence between experiments and core claims is limited. Kinetix uses state input and removes optical flow, unable to verify the complete vision architecture; DynBench lacks sufficient description of motion distribution, train-test splits, and success criteria, and real-machine experiments also do not systematically verify adaptation capabilities under unseen motion conditions. Improvements on external benchmarks are worth reporting, but cannot replace direct neighbor comparisons and rigorous contribution attribution.

Questions for authors

See above.

6 comments

We sincerely thank the reviewer for the constructive feedback and for recognizing our well-motivated system design and empirical gains across real-robot and simulation benchmarks. Below, we address each concern in detail and summarize our additional evidence.

W1 Relationship to prior work and clarification of our core contributions

Thank you for encouraging us to further clarify our connections to VLASH, A2C2, and prior residual policies. We have expanded the Related Work section to thoroughly discuss them. Below, we detail the technical differences between our DSDyn-VLA and these closely related methods and highlight our core contributions.

Comparison with VLASH: Both VLASH and DSDyn-VLA aim to align the policy to the state at the future execution moment to mitigate action staleness. This future state consists of two parts: the robot's own proprioceptive state and the external environment state. However, VLASH aligns only the former while overlooking the latter. In dynamic manipulation, the future external environment state (especially moving targets) at execution time is far more critical than future proprioception, as it directly dictates the actual manipulation target. DSDyn-VLA aligns both: beyond future proprioception, optical-flow velocity cues let the policy implicitly model the positions moving objects will reach at execution time, achieving future compensation for external dynamics (implicitly aligning the future external environment state).

Comparison with A2C2: Our Res-Refiner differs fundamentally from A2C2 in three aspects:

  1. Training state distribution: A2C2 uses SFT on the offline demonstration distribution to regress (expert−base)(\text{expert} - \text{base}) actions, seeing only demonstration-trajectory states. At deployment, every residual correction alters subsequent states, driving the robot into out-of-distribution (OOD) states never encountered in demonstrations. Conversely, Res-Refiner learns via RL directly inside the (base policy+residual+actual latency)(\text{base policy} + \text{residual} + \text{actual latency}) closed-loop system, keeping learning and deployment state distributions completely consistent.
  2. Supervision source: A2C2 is supervised by expert trajectories, where inevitable human teleoperation reaction delays impose a ceiling on A2C2's performance; Res-Refiner uses task-success-rewarded RL and is not limited by demonstration quality.
  3. Coupling with the base policy: A2C2 relies on stale latent features zz from the base VLA's previous inference pass (which encode outdated visual observations), whereas Res-Refiner treats the VLA as a black box, taking only real-time images, proprioception, and base actions to eliminate the impact of outdated visual information.

We also provide direct experimental comparisons with A2C2 in the table below.

Relation to prior residual policy learning: While prior residual policy learning mainly targets high-precision manipulation in static tasks, DSDyn-VLA is, to our knowledge, the first to successfully introduce residual RL into VLA for dynamic manipulation. Directly applying prior residual methods cannot solve dynamic VLA manipulation due to a scale–stability dilemma: enlarging the residual bound causes jerky motion and unstable RL exploration, whereas keeping a tight bound (e.g., 10% action range) for smooth control leaves local residuals powerless when unaligned base actions fail to even enter the moving target's 10% neighborhood. By decoupling macro motion/latency alignment from local intra-chunk refinement, our tightly coordinated system is the prerequisite for residual learning to succeed in dynamic manipulation (verified below), laying a foundation and offering insights for future work.

Method / VariantDynBench (SR %)DOMINO (SR %)DOMINO (MS)
π0.5\pi_{0.5} (Baseline)9.79.6326.19
π0.5\pi_{0.5}+VLASH27.114.0132.29
π0.5\pi_{0.5}+A2C213.010.6828.05
π0.5\pi_{0.5}+Residual RL (Res-Refiner on unaligned base)17.312.3829.43
Our DSDyn-VLA (Flow-Planner + Res-Refiner)48.420.7441.64
DSDyn-VLA (replace Res-Refiner w/ A2C2)44.618.2735.90

Core Contributions: As shown above, dynamic manipulation cannot be solved by any isolated module. Rather than the design of any single module, we believe our main contribution to the field lies in clearly answering "what core problems must be solved to achieve dynamic manipulation." Through a thorough, first-principles analysis, we identify the three indispensable and tightly coupled bottlenecks of dynamic manipulation—the Perception Gap, Latency Gap, and Control Gap—and build a complete, tightly coordinated system around them. Showing the community what truly bottlenecks dynamic manipulation, establishing a clear analytical framework, and demonstrating that solving all three together brings a major leap in performance represent the core value of our work beyond individual component designs.

W2 Controlled ablation study on optical flow polar mapping

We appreciate the suggestion of a cleaner controlled comparison for polar mapping. To evaluate the effect of polar mapping alone while keeping the encoder architecture and pretraining identical, we added a controlled variant: Raw Flow (Zero-Padded) + Pretrained CLIP ViT. We linearly normalize 2-channel raw flow (u,v)(u, v) to [0,1][0, 1], append a zero channel to form (u,v,0)(u, v, 0), and feed it into the exact same pretrained CLIP ViT as our full model—making polar mapping the sole difference:

Method VariantFlow RepresentationEncoder & PretrainingDynBench (SR %)Real-World (SR %)
w/o Polar Map (Original ablation, ResNet-34)2-channel raw flow (u,v)(u, v)ResNet-34 (from scratch)37.233.0
w/o Polar Map (Zero-padded raw flow, controlled)3-channel padded raw flow (u,v,0)(u, v, 0)CLIP ViT (pretrained)39.136.0
DSDyn-VLA (Full)3-channel polar color map (Hue & Sat.)CLIP ViT (pretrained)48.448.5

Even with the identical pretrained CLIP ViT, zero-padded raw flow (u,v,0)(u, v, 0) still performs significantly worse than polar mapping (39.1% vs. 48.4% on DynBench; 36.0% vs. 48.5% real-world), because raw flow deviates sharply from natural RGB distributions and cannot be effectively processed by the pretrained CLIP encoder. Polar color mapping yields +9.3 and +12.5 percentage-point gains under the same encoder, confirming that the gain stems from Polar Mapping's feature-distribution alignment rather than encoder or pretraining differences (added to Section 4.3, Table 4).

W3.1 Role of Kinetix and validation of the complete visual architecture

Thank you for raising this important point on the role of Kinetix. We fully agree that as a purely state-based benchmark, Kinetix does not include visual inputs. However, since it is the recognized standard benchmark for prior real-time manipulation policies (e.g., RTC, VLASH, and A2C2), we evaluate on Kinetix primarily to ensure a fair comparison with existing methods under the exact same evaluation protocol. Precisely to address Kinetix's lack of vision, we built our vision-based DynBench and further evaluated on the external visual dynamic benchmark DOMINO and real-world robot tasks—together providing rigorous validation of our complete visual architecture.

W3.2 Motion distributions, train-test splits, and success criteria of DynBench

Thank you for pointing out that the DynBench protocol should be made more explicit. Full DynBench evaluation protocols have been added to the Appendix:

  • Motion Distribution: (i) Conveyor / linear-traction (Pick, Drop, Stack, Pour, Sort, Insert): per-episode target speed sampled uniformly in [0.10,0.25] m/s[0.10, 0.25]\text{ m/s}; (ii) Free-rolling (Ball, Can, GreenBall): initial speed sampled uniformly in [0.15,0.30] m/s[0.15, 0.30]\text{ m/s}. Across all tasks, conveyor placement or motion angle ∈[−15∘,+15∘]\in [-15^\circ, +15^\circ] and perpendicular initial lateral offset ∈[−10,+10] cm\in [-10, +10]\text{ cm} are sampled uniformly.
  • Train-Test Splits: 100 training demos and 100 evaluation trials per task are sampled independently from these continuous distributions using strictly disjoint random seeds, so every test-time object trajectory is unseen during training.
  • Success Criteria (auto-evaluated in MuJoCo): (i) Grasping (Pick, Ball, Can, GreenBall): grasp the correct target, lift >10 cm> 10\text{ cm} above the tabletop, and hold stably for >2 s> 2\text{ s}; (ii) Sort / Drop: land inside the target bin or moving container and remain stably inside for >2 s> 2\text{ s} without bouncing out; (iii) Stack: place upper cube onto moving lower cube with horizontal center offset <2 cm< 2\text{ cm}, moving stably together on the conveyor without falling for >2 s> 2\text{ s}; (iv) Insert: insert cylinder into the moving base's hole to depth >2 cm> 2\text{ cm} for >2 s> 2\text{ s}; (v) Pour: retain ≥80%\ge 80\% of simulated liquid particles inside the moving cup at episode end.

W3.3 Real-world adaptation evaluation under unseen motion conditions

Thank you for the valuable suggestion to evaluate unseen motion conditions in the real world. Since our physical circular conveyor runs at a fixed speed, we evaluate unseen motion adaptation by altering the conveyor's overall position and orientation—applying random rotations in [−15∘,+15∘][-15^\circ, +15^\circ] and planar shifts in [−10,+10] cm[-10, +10]\text{ cm} along both horizontal axes relative to training placement. This substantially shifts object velocity direction vectors in both camera and robot base frames. Deploying our trained models zero-shot without retraining across all 4 real-world tasks (50 trials/task) yields:

MethodStandard Setting (Avg. SR %)Unseen Motion Conditions (±15∘\pm 15^\circ rot. & ±10 cm\pm 10\text{ cm} shift, Avg. SR %)
π0.5\pi_{0.5} (Baseline)8.58.5
VLASH26.522.0
DynamicVLA37.530.5
DSDyn-VLA (Ours)48.547.5

While baselines degrade noticeably under unseen motion shifts (e.g., DynamicVLA: 37.5% →\to 30.5%), DSDyn-VLA remains nearly unchanged (47.5% unseen vs. 48.5% standard), demonstrating strong adaptability under unseen motion conditions.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We sincerely thank Reviewers Rkuf, igsB, and VKj6 for their constructive feedback. We have uploaded a revised manuscript (changes highlighted in blue) and posted point-by-point responses addressing every question with new experiments and analyses. Below is a brief summary of our key updates:

  • Clarified Core Contributions & Direct Comparisons with VLASH / A2C2 (Rkuf, igsB — Sec. 2, Table 2, Appendix F Table 12): We expanded Related Work to clarify how DSDyn-VLA goes beyond VLASH (aligning both proprioception and external object dynamics via optical flow) and A2C2 (closed-loop RL distribution matching, reward-driven supervision without teleoperation lag, and a 3 ms real-time black-box design without stale VLA latents). Direct evaluations show DSDyn-VLA outperforms A2C2 by +6.9 to +11.3 pp across all nine (δ,K)(\delta, K) settings on Kinetix (97.1% vs. 86.5% at δ=4\delta=4), surpasses both A2C2 (44.6%) and SFT-trained Res-Refiner (45.3%) on DynBench (48.4%) and DOMINO (20.74% SR / 41.64 MS), and confirms that no isolated module solves dynamic manipulation (PI0.5 + A2C2: 13.0%; unaligned PI0.5 + Residual RL: 17.3%).
  • Encoder-Matched Optical-Flow Polar Mapping Ablation (Rkuf, VKj6 — Sec. 4.3 Table 4, Appendix G.3 Table 15): Feeding 3-channel zero-padded normalized raw flow (u,v,0)(u, v, 0) into the identical pretrained CLIP ViT yields 39.1% on DynBench and 36.0% on real hardware, versus 48.4% (+9.3 pp) and 48.5% (+12.5 pp) with Polar Mapping—confirming the gain comes from feature-distribution alignment rather than encoder or pretraining differences.
  • Visual Delay Robustness, Unseen Real-World Motion & DynBench Protocol (Rkuf, VKj6 — Appendices E, G.1 Table 13, G.2 Table 14): We detailed DynBench's motion distributions, disjoint-seed unseen test trajectories, and automated MuJoCo thresholds (Appendix E). Across visual inference delays δ∈{2,3,4,5}\delta \in \{2, 3, 4, 5\} on DynBench, DSDyn-VLA stays stable (49.3% → 47.6%) while DynamicVLA drops sharply (38.1% → 25.4%). Under zero-shot unseen real-world conveyor poses (±15∘\pm 15^\circ rotation and ±10 cm\pm 10\text{ cm} shifts), DSDyn-VLA maintains 47.5% (vs. 48.5% standard), whereas DynamicVLA drops from 37.5% to 30.5%.
  • Future Proprioceptive Prediction Accuracy with Res-Refiner Active (VKj6 — Sec. 3.3.2, Appendix G.5 Table 17): We clarified Eq. (5) in absolute EEF space (q^t+Tlat=Abuffer[Tlat]\hat{\mathbf{q}}_{t+T_{\text{lat}}} = \mathbf{A}_{\text{buffer}}[T_{\text{lat}}], where bounded 10% local residuals do not accumulate across steps). Across 50 real-world rollouts with Res-Refiner enabled, our alignment reduces latency-induced EEF mismatch from 42.6 mm / 6.8° to 3.1 mm / 0.6°.
  • Transparent Training/Labeling Budgets & Multi-Task / Cross-Benchmark Generality (igsB, VKj6 — Sec. 4.3, Appendix D Table 11, Appendix G.4 Table 16): With the exact same 50 offline demos/task, DSDyn-VLA achieves higher real-world success (48.5% vs. 39.0%) using strictly fewer online episodes (70 episodes / <28,000 steps and 70 episode-end key presses per task vs. 100 episodes / <40,000 steps and 100 presses for strengthened PPO baselines). We also clarified multi-task Res-Refiner sharing per benchmark and verified that a single unified Res-Refiner trained jointly across DynBench, LIBERO, and DOMINO transfers effectively (42.7% → 46.1% on DynBench; 14.77% → 19.22% on DOMINO).

We warmly welcome any further feedback or questions during the discussion period.

Best regards, Authors of Submission #16046

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.