- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper combines a VLA planner that uses optical flow, a prediction of the robot state when actions will run, and a fast RL controller that corrects those actions. It is tested on Kinetix, nine DynBench tasks and four real-world dynamic manipulation tasks.
Critical strengths
- The method addresses delays when controlling robots around moving objects. Average real-world success rises from 38.1% with DynamicVLA to 49.0% (Table 1). Tests that remove individual modules also help explain the gain.
Critical weaknesses
- Equation 5 predicts the future robot state using pending planner actions, but the robot executes actions with residual corrections. The paper does not measure the prediction error caused by this difference or explain how it is handled. This matters because the predicted state is a key input to the planner.
- The polar-mapping test also changes the encoder. Raw flow uses a ResNet-34 trained from scratch, while mapped flow uses CLIP (Section 4.3). The gain could therefore come from the encoder or its pretraining. A comparison with matched encoders would better test polar mapping itself.
- The delay tests use state-based Kinetix with an MLP planner and no optical flow (Section 4.1). They support the control design, but do not test how the full visual VLA system handles delays. Similar tests with delayed or irregular inputs on visual manipulation tasks would make that claim stronger.
Questions for authors
- How is Eq. 5 implemented for the robot's action representation? How accurate is the predicted state with the residual controller enabled, and are its pending corrections included?
- Please report demonstration counts, online environment steps and reward-labeling effort for each method, including the stronger Table 11 baselines. Are these budgets matched?
3 comments
W2 Controlled ablation study on optical flow polar mapping
We appreciate the suggestion to test polar mapping with matched encoders. To evaluate the effect of polar mapping alone while keeping the encoder architecture and pretraining identical, we added a controlled variant: Raw Flow (Zero-Padded) + Pretrained CLIP ViT. We linearly normalize 2-channel raw flow to , append a zero channel to form , and feed it into the exact same pretrained CLIP ViT as our full model—making polar mapping the sole difference:
| Method Variant | Flow Representation | Encoder & Pretraining | DynBench (SR %) | Real-World (SR %) |
|---|---|---|---|---|
| w/o Polar Map (Original ablation, ResNet-34) | 2-channel raw flow | ResNet-34 (from scratch) | 37.2 | 33.0 |
| w/o Polar Map (Zero-padded raw flow, controlled) | 3-channel padded raw flow | CLIP ViT (pretrained) | 39.1 | 36.0 |
| DSDyn-VLA (Full) | 3-channel polar color map (Hue & Sat.) | CLIP ViT (pretrained) | 48.4 | 48.5 |
Even with the identical pretrained CLIP ViT, zero-padded raw flow still performs significantly worse than polar mapping (39.1% vs. 48.4% on DynBench; 36.0% vs. 48.5% real-world), as raw flow deviates sharply from natural RGB distributions and cannot be effectively processed by the pretrained CLIP encoder. Polar color mapping yields +9.3 and +12.5 percentage-point gains under the same encoder, confirming that the gain stems from Polar Mapping's feature-distribution alignment rather than encoder or pretraining differences (added to Section 4.3, Table 4).
W3 Delay robustness evaluation of the complete visual system on visual manipulation tasks
Thank you for the valuable suggestion to evaluate delay robustness on visual tasks. First, our real-world experiments already run under actual system latency, and our visual simulation benchmark DynBench simulates an inference delay of steps by default (clarified in Section 4.2). Together, these two sets of experiments verify the delay robustness of our complete visual system.
To further test DSDyn-VLA's delay robustness on a vision-based benchmark, we additionally evaluated varying delay steps on DynBench:
| Method | (SR %) | (Default, SR %) | (SR %) | (SR %) |
|---|---|---|---|---|
| (Baseline) | 13.0 | 9.7 | 7.4 | 3.6 |
| RTC | 22.9 | 19.4 | 16.7 | 10.3 |
| VLASH | 29.7 | 27.1 | 24.8 | 19.0 |
| DynamicVLA | 38.1 | 34.3 | 30.9 | 25.4 |
| DSDyn-VLA (Ours) | 49.3 | 48.4 | 48.0 | 47.6 |
The results show that as visual delay steps increase, all baselines degrade noticeably (e.g., DynamicVLA drops from 38.1% to 25.4%), whereas DSDyn-VLA consistently maintains stable, leading success rates (49.3% 47.6%).
Q2 Clarification on training budget comparability and reward-labeling effort
We appreciate the request to report exact budgets and reward-labeling effort. In our real-world comparisons with the strengthened baselines in Appendix D Table 11, all methods use the exact same 50 offline demos/task. Because PPO is less sample-efficient, we gave the PPO baselines 100 online interaction episodes/task, whereas our RLPD-trained Res-Refiner uses only 70 online interaction episodes/task:
| Setting (Real-World, Table 11) | Offline Demos / Task | Online Interaction Episodes / Task | Manual Reward Labels / Task | Avg. Success (%) |
|---|---|---|---|---|
| RTC + history frames + PPO | 50 | 100 ( steps) | 100 (1 button press / ep.) | 24.0 |
| VLASH + history frames + PPO | 50 | 100 ( steps) | 100 (1 button press / ep.) | 32.5 |
| DynamicVLA + history frames + PPO | 50 | 100 ( steps) | 100 (1 button press / ep.) | 39.0 |
| DSDyn-VLA (Ours) | 50 | 70 ( steps) | 70 (1 button press / ep.) | 48.5 |
Simulation rewards are assigned automatically (0 manual labels); real-world RL uses sparse episode-end binary rewards () requiring 1 key press/episode, totaling 70 button presses/task vs. strengthened PPO's 100. Thus, DSDyn-VLA achieves higher success (48.5% vs. 39.0%) with identical offline data and strictly fewer online interaction episodes and manual labels. We have added these details to the revised manuscript.
Dear Reviewer VKj6,
As the author–reviewer discussion period is coming to a close (Nov 18 AoE), we wanted to briefly follow up to see if our responses and newly added experiments have addressed your questions. In particular, we have included:
- Implementation details of Eq. (5) and real-world prediction accuracy with Res-Refiner active, showing that in absolute EEF space residuals do not accumulate and latency-induced mismatch drops from 42.6 mm / 6.8° to 3.1 mm / 0.6° across 50 real-world rollouts (Sec. 3.3.2, Appendix G.5 Table 17).
- Matched-encoder Polar Mapping ablation feeding normalized zero-padded raw flow into the exact same pretrained CLIP ViT (39.1% vs. 48.4% on DynBench; 36.0% vs. 48.5% real-world, Sec. 4.3 Table 4, Appendix G.3 Table 15).
- Visual inference delay evaluations on DynBench confirming stable performance across increasing visual delays (49.3% → 47.6%, Appendix G.1 Table 13), along with exact online step counts and episode-end button-press labeling budgets (70 vs. 100 presses/task for PPO baselines, Appendix D Table 11).
We would be glad to answer any final questions you might have before the discussion window closes. Thank you very much for your constructive suggestions!
Best regards, Authors of Submission #16046
We sincerely thank the reviewer for the constructive feedback and for recognizing our clear presentation, effective delay compensation in dynamic manipulation, substantial real-world performance gains, and informative module ablations. Below, we address each concern in detail and summarize our additional evidence.
Thank you for these insightful questions on Eq. (5) and state prediction accuracy. Below, we clarify the implementation of fkin, how residual corrections are handled, and the actual prediction accuracy:
1. Implementation of fkin in Eq. (5): Equation (5) is written as a general kinematics rollout formulation from a relative-action perspective. In our actual implementation, however, because both the proprioceptive state q and Flow-Planner's predicted action space are absolute EEF poses in the robot base frame (3D position, orientation, and gripper width), Abuffer[Tlat] naturally represents the target proprioceptive state at t+Tlat. Therefore, predicting the future state q^future (at t+Tlat) simplifies directly to taking q^t+Tlat=Abuffer[Tlat], avoiding multi-step rollout errors with zero compute overhead.
2. Are pending residual corrections included? No—residual corrections are not included in Eq. (5), and including them is unnecessary:
3. State prediction accuracy with Res-Refiner enabled: Across 50 complete real-world trajectories with Res-Refiner enabled, we measured the mean error between the predicted state q^t+Tlat and the robot's true physical EEF pose qt+Tlat at execution time:
Conditioning on the unaligned state qt causes a 42.6 mm (>4 cm) mismatch during inference latency, whereas our prediction—even with Res-Refiner enabled and without modeling future residuals—reduces the error to just 3.1 mm and 0.6 deg. This millimeter-level offset has negligible impact on Flow-Planner's macro-planning, and any remaining local error at execution is corrected online by our 30 Hz closed-loop Res-Refiner.