- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores an important and practical challenge in the deployment of VLAs, where the deviation between commanded actions and the actual execution outcomes of the robot. The idea of compensating for such execution errors is both attractive and well justified, as it directly addresses a real-world problem that has not yet received sufficient attention. In addition, it introduce RoboStress, a simulation benchmark that models four joint-level sources of execution error, thereby systematically evaluating the robustness of VLAs under different deployment conditions. Extensive experiments conducted in simulation and on real robots validate the effectiveness of the proposed compensation method. Although its technical approach is relatively simple and intuitive, this paper is clearly motivated and thoroughly evaluated. And most importantly, it highly focuses on practical deployment applications in robotics.
Critical strengths
- Practical and well-motivated problem. The paper addresses an important real-world challenge in VLA deployment. The proposed self-compensation method is simple yet appealing, requiring neither task rewards nor modifications to low-level controllers.
- Solid benchmark and extensive experiments. RoboStress provides a physically motivated benchmark with four joint-level error models. Extensive simulation and real-world experiments demonstrate the effectiveness of the method, making the work a solid and practically relevant contribution.
Critical weaknesses
The analysis of online adaptation efficiency is limited. Although the proposed method demonstrates promising improvements through online adaptation, it remains unclear how many episodes are needed to achieve meaningful performance gains. The current evaluation uses 20 episodes per task in simulation and 25 episodes per task on the real robot, and averages the success rates over these episodes. However, it does not show how performance evolves during this adaptation process. Providing success rate curves as a function of the number of adaptation iterations would help clarify the adaptation speed and better demonstrate the practical effectiveness of the method.
Questions for authors
- Limited real-world evaluation under varying execution conditions. The real-world generalization experiments focus on appearance variations of bowls and plates. However, since the paper primarily addresses execution errors caused by physical factors, evaluating objects with different weights instead would be more relevant and provide stronger evidence of the method's effectiveness under varying execution conditions.
- The proposed pseudo-targets assume that compensating for execution errors improves task performance, which may not always hold when the original actions are suboptimal or during contact-rich interactions. Although the ablations demonstrate empirical effectiveness, further analysis of the relationship between execution error reduction and task success would help clarify the reliability and limitations of this approach.
3 comments
Question 2-1. Relationship between execution error reduction and task success
We agree that compensating for execution errors may not improve task performance when the action intended by the policy is itself suboptimal, since the task can fail even if the robot executes it accurately. We would like to note that our method aims to make the robot execute the action intended by the policy, not to correct the intended action itself. It is thus complementary to methods that improve action selection.
As suggested, we examined the relationship between execution error reduction and task success. Specifically, on RoboStress, we computed the reduction in execution error and the gain in success rate of our method over the base policy on each of the ten tasks, averaged over the suites. The table below reports the correlation between the two.
| Pearson | Spearman | |
|---|---|---|
| Correlation | 0.83 | 0.68 |
The two are positively correlated, which indicates that the success rate improves more on the tasks where the execution error is reduced more. We have included this analysis and discussion in Appendix D.8 of the revised paper.
Question 2-2. Reliability under contact-rich interactions
Regarding contact-rich interactions, we evaluated our method under constrained contact by adding a pressing phase to four pick-and-place tasks. In the pressing phase, which lasts for 60 steps after grasping, we keep the policy active and add a downward offset to its commands so that the gripper is pushed against the table and its downward motion is blocked. Video R1 shows this setting, with the object interaction and pressing phases marked. To examine whether the residuals measured during contact mislead adaptation, we also evaluated two variants of our method that ignore them when updating the policy. The first ignores the residuals measured while the gripper holds the object, during both object interaction and pressing. The second ignores only those measured during pressing. Both use contact labels from the simulator to identify these residuals. Without pressing, the second variant is identical to our method. The table below reports the success rate (%).
| Method | Without pressing | With pressing |
|---|---|---|
| Base | 54.6 | 33.6 |
| Ours | 55.7 | 39.8 |
| Ours ignoring all contact residuals | 53.4 | 35.0 |
| Ours ignoring pressing residuals | – | 34.5 |
As shown in the table, our method outperforms the base policy under pressing, and ignoring the contact residuals does not lead to better results. These results suggest that our method remains effective under contact, and that the residuals during contact do not mislead adaptation in this setting. Our method also improves the success rate of the drawer-opening task on both arms in Table 6, where the gripper motion is constrained by the drawer. We have included this analysis in Appendix D.5 and Table D11 of the revised paper, and will add this contact setting to RoboStress.
A gentle reminder
Dear Reviewer zJx3,
We sincerely appreciate your effort and the time you have dedicated to reviewing our submission. We have addressed your concerns in our rebuttal, and it would be helpful if you could give us any further feedback before the author-reviewer discussion phase ends. Your opinion would be invaluable to us in improving our work, and we are more than willing to provide further clarifications or respond to any additional questions you might have. Thank you for your consideration.
Best regards,
Authors
We appreciate your insightful feedback and constructive suggestions, which help improve our paper substantially. We have addressed all the comments and included the additional experiments in the revised paper, where the changes are marked in blue. Please find our responses to the comments below. Unless otherwise stated, all additional experiments were conducted using π0.5. Simulation results are averaged over the seven deployment scenarios of RoboStress, and real-robot results are obtained on Task 1 with Robot A.
Weakness 1. Analysis of online adaptation efficiency
Thank you for the constructive suggestion. We report how the success rate changes as the numbers of episodes and updates increase, both on the real robot and in simulation. The results are also shown in Figure R4.
On the real robot, the table below and Figure R4(a) report the success rate (%) over 25 consecutive episodes in five-episode windows. Our method achieves the same success rate as the base policy in the first five episodes and outperforms it in every window after the first.
In simulation, the table below and Figure R4(b) report the success rate (%) on the first task of LIBERO-Spatial in four-episode windows, averaged over the seven scenarios. Our method is comparable to the base policy in the first 12 episodes and outperforms it by 7.1 and 10.7 points in the last two windows.
Figure R4(c) shows how the gain changes when the adaptation continues beyond the first task. It covers the ten tasks of each suite, across which the LoRA parameters are carried. The curve is the success rate of our method minus that of the base policy over all episodes so far. The gain increases overall as the number of updates increases.
In short, the gain of our method appears within a single task, after about 5 episodes on the real robot (about 60 updates) and after about 12 episodes in simulation (about 40 updates). It then continues to increase as the updates accumulate over the ten tasks. We have included these analyses in Section 5.3 of the revised paper, with Table 7 and Figure 6, and the details in Appendix D.7.
Question 1. Real-world evaluation with different object weights
Thank you for the valuable comment. As suggested, we conducted a real-robot experiment where weights of 500 g and 1,000 g are placed in the bowl, so that the load changes while the bowl and the grasp remain the same. The setup is shown in Figure R5, and the table below reports the success rate (%) over 25 episodes for each added weight.
As shown in the table, our method outperforms the base policy by 36, 40, and 40 points with added weights of 0 g, 500 g, and 1,000 g, respectively. As the weight increases, the success rate of the base policy decreases from 48% to 32%, whereas our method achieves at least 72%. These results suggest that our method is effective under varying execution conditions. We have included this experiment in Section 5.3 and Figure 9 of the revised paper.