- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper bootstraps a residual action correction solely from the rollouts of a frozen VLA policy. An advantage regularizer is designed to ground a critic on expected returns from the base policy before finding local corrective actions that would improve upon executing the VLA policy directly.
Critical strengths
The paper proposes an advantage-based regularization mechanism that improves critic calibration and enables selective correction of VLA actions without finetuning the base policy. The method consistently outperforms standard residual RL across five unseen LIBERO-90 tasks, demonstrating substantial improvements in task success rates and the effectiveness of repairing critical decision errors.
Critical weaknesses
ABR² is most effective when the base policy is partially competent and makes isolated critical errors, but provides limited benefits for tasks requiring continuous corrections and can significantly underperform standard residual RL when the base policy is weak.
Questions for authors
- How sensitive is ABR² to the warm-up duration and regularization weights? Can these hyper-parameters be selected automatically without relying on task-specific evaluation or extensive tuning?
- The paper evaluates ABR² exclusively in simulation. Could the authors conduct real-world robotic reinforcement learning experiments to validate the effectiveness and practicality of the proposed method, thereby strengthening its contribution to real-world robotic applications?
2 comments
The authors have addressed most of my concerns, although I would have preferred to see results from real-robot experiments. I am willing to raise my score to weak accept.
Thank you for these questions. We clarify the intended scope and address hyperparameter sensitivity and physical deployment below.
1. Limited benefit from continuous corrections:
We agree with the reviewer that ABR2 is most effective when the base policy is partially competent and its failures can be addressed through targeted corrections. This is an intentional aspect of our problem formulation rather than a claim that ABR2 universally outperforms standard residual RL. Our objective is to investigate how an agent can leverage existing, but imperfect, policy competence to efficiently learn a novel task online, without requiring expert demonstrations.
ABR2 is designed to preserve useful behavior from the base policy while selectively learning deviations that improve task performance. This introduces an inductive bias toward relying on the base policy, which is beneficial when the base policy already provides a reasonable behavioral prior. However, when the base policy is weak or requires continuous correction, this same bias can become restrictive, as successful behavior may require learning a substantially different policy rather than repairing individual decisions. In these settings, other methods may be better suited to learning broader behavioral changes.
2. Hyperparameters:
There are two hyperparameters:
The value of λδ, which penalizes large corrections to the base policy action.
The percentage of the budget dedicated to the warm up period.
We will add additional details on how the hyperparameters were selected to the appendix.
We did not implement automatic warm up stopping in this paper, but it is something we plan to explore in future work. One possible idea would be to stop the warm up when the loss of the value function plateaus to automatically end the warm up phase.
3. Real-world experiments:
We present a controlled, interpretable domain where the base policy is a goal conditioned reinforcement learning policy and the agent is tasked with learning to achieve an unseen goal online. We then show that this method scales up to prompt conditioned robot manipulation tasks, where the base policy is instead an embodiment finetuned VLA and the agent must learn to solve tasks with unseen prompts and environments. The controlled domain environment was used to thoroughly investigate the properties of ABR2, including the hypothesis that corrections will primarily be low magnitude and the base policy will act as the primary driver of executed actions and critical errors are acutely corrected by the residual. The robot manipulation domain is used to show viability of this method in realistic systems where agents may encounter tasks not seen in training. We agree that physical experiments would strengthen the evidence for practical deployment. We plan to explore this in future work, but cannot complete a rigorous real-world evaluation within the discussion period.