acceptodds
Under review as a conference paper at ICLR 2027

: Advantage based Regularization for Online Policy Repair

Abstract

Vision-Language-Action (VLA) models are popular for zero-shot manipulation across diverse tasks, but their effectiveness is conditioned on exposure during training to **both** the deployment embodiment **and** the specific task domain. We observe that embodiment-grounded VLA policies frequently exhibit _partial competence_ in unseen tasks, even in those closely related to tasks seen in training. These policies complete a task in some non-zero proportion of attempts but frequently make non-recoverable errors, preventing zero-shot deployment of the policy. To address this we propose Advantage-based Regularization for Online Policy Repair (), a method that learns to correct the critical errors that lead to task failure when deploying VLA policies in unseen tasks. Our method requires neither expert demonstrations, reward shaping, nor VLA finetuning. bootstraps a residual action correction solely from the rollouts of a frozen base policy. An advantage regularizer grounds a critic on expected returns from the base policy before finding local corrective actions that would improve upon executing the VLA policy directly. We evaluate the ability of to repair policies on both small scale goal conditioned control environments and VLA policies in unseen manipulation and decision making tasks. Across five held-out LIBERO-90 tasks with 21.8–54.7% base success, improves final success to 55.6–96.3%, outperforming residual RL without advantage regularization on every task. excels at correcting acute critical errors that lead to policy failure, increases the sample efficiency of policy repair, and accommodates multi-step corrections for perturbed configurations of tasks seen during training.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Policy Repair, Online Learning, Residual Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9py5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper bootstraps a residual action correction solely from the rollouts of a frozen VLA policy. An advantage regularizer is designed to ground a critic on expected returns from the base policy before finding local corrective actions that would improve upon executing the VLA policy directly.

Critical strengths

The paper proposes an advantage-based regularization mechanism that improves critic calibration and enables selective correction of VLA actions without finetuning the base policy. The method consistently outperforms standard residual RL across five unseen LIBERO-90 tasks, demonstrating substantial improvements in task success rates and the effectiveness of repairing critical decision errors.

Critical weaknesses

ABR² is most effective when the base policy is partially competent and makes isolated critical errors, but provides limited benefits for tasks requiring continuous corrections and can significantly underperform standard residual RL when the base policy is weak.

Questions for authors
  1. How sensitive is ABR² to the warm-up duration and regularization weights? Can these hyper-parameters be selected automatically without relying on task-specific evaluation or extensive tuning?
  2. The paper evaluates ABR² exclusively in simulation. Could the authors conduct real-world robotic reinforcement learning experiments to validate the effectiveness and practicality of the proposed method, thereby strengthening its contribution to real-world robotic applications?
2 comments

Thank you for these questions. We clarify the intended scope and address hyperparameter sensitivity and physical deployment below.

1. Limited benefit from continuous corrections:

We agree with the reviewer that ABR2\text{ABR}^2 is most effective when the base policy is partially competent and its failures can be addressed through targeted corrections. This is an intentional aspect of our problem formulation rather than a claim that ABR2\text{ABR}^2 universally outperforms standard residual RL. Our objective is to investigate how an agent can leverage existing, but imperfect, policy competence to efficiently learn a novel task online, without requiring expert demonstrations.

ABR2\text{ABR}^2 is designed to preserve useful behavior from the base policy while selectively learning deviations that improve task performance. This introduces an inductive bias toward relying on the base policy, which is beneficial when the base policy already provides a reasonable behavioral prior. However, when the base policy is weak or requires continuous correction, this same bias can become restrictive, as successful behavior may require learning a substantially different policy rather than repairing individual decisions. In these settings, other methods may be better suited to learning broader behavioral changes.

2. Hyperparameters:

There are two hyperparameters:

  • The value of λδ\lambda_\delta, which penalizes large corrections to the base policy action.

    • For the selection of λδ\lambda_\delta, we found this was method agnostic and primarily correlated with the task and the action space bounds. λδ\lambda_\delta primarily serves to keep corrections reasonable and effectively controls how large a correction can be made by the residual policy.
    • Attempts at automatically setting this value resulted in fixed recommendations for a particular task settings, rather than general prescriptive function.
    • In a short hyperparameter sweep we found a strong value for each task and kept it frozen and uniform between methods. We observed no benefit for either method in making this parameter method specific.
  • The percentage of the budget dedicated to the warm up period.

    • We found that the choice of the warm up period for ABR2\text{ABR}^2 primarily serves to manage the learning of the base policy value function. This is seen most clearly in the LIBERO-PRO experiments where warm up periods shorter than 500 episodes have noticeably worse performance relative to the baseline.
    • We found that the choice of the warm up period on LIBERO tasks can be pushed as low as 50 episodes with ~2% regression in performance.
    • We see that the baseline is agnostic to choice of warm up period in both LIBERO and LIBERO-PRO as the warm up period only serves to pre-populate the replay buffer.

We will add additional details on how the hyperparameters were selected to the appendix.

We did not implement automatic warm up stopping in this paper, but it is something we plan to explore in future work. One possible idea would be to stop the warm up when the loss of the value function plateaus to automatically end the warm up phase.

3. Real-world experiments:

We present a controlled, interpretable domain where the base policy is a goal conditioned reinforcement learning policy and the agent is tasked with learning to achieve an unseen goal online. We then show that this method scales up to prompt conditioned robot manipulation tasks, where the base policy is instead an embodiment finetuned VLA and the agent must learn to solve tasks with unseen prompts and environments. The controlled domain environment was used to thoroughly investigate the properties of ABR2\text{ABR}^2, including the hypothesis that corrections will primarily be low magnitude and the base policy will act as the primary driver of executed actions and critical errors are acutely corrected by the residual. The robot manipulation domain is used to show viability of this method in realistic systems where agents may encounter tasks not seen in training. We agree that physical experiments would strengthen the evidence for practical deployment. We plan to explore this in future work, but cannot complete a rigorous real-world evaluation within the discussion period.

The authors have addressed most of my concerns, although I would have preferred to see results from real-robot experiments. I am willing to raise my score to weak accept.

Reviewer_wWCsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ABR², a residual RL method that repairs a frozen, partially competent VLA policy without target-task demonstrations or VLA finetuning. The main idea is to regularize the critic using a learned base-policy value function and use the critic to select between the original and corrected actions. Experiments on simple control environments and LIBERO show improvements on unseen tasks

Critical strengths

The method addresses a useful setting where a pretrained VLA sometimes succeeds but is not reliable enough, without requiring additional demonstrations or updating the large model. The regularizer improves over the residual baseline on unseen LIBERO tasks.

Critical weaknesses
  • The VLA evaluation uses one base model and five selected unseen tasks, and Table 3 shows no consistent benefit under position perturbations, including a substantial drop in the hardest setting.
  • Selecting the best warm-up length and regularization weight separately for each task makes the practical tuning requirements and fairness of the comparison unclear
Questions for authors

How were the per-task hyperparameters selected, was evaluation kept separate from this selection, and did the residual baseline receive the same tuning budget?

3 comments

Thank you for the feedback. We address the evaluation scope, perturbations, and your questions below.

1. Evaluation scope

In this work we start with a controlled domain (4.1) to focus more clearly on the components of ABR², where the base policy is a goal-conditioned reinforcement learning policy. We use this domain to study the properties of online task learning when starting with a base policy that is broadly capable, but not reliable in accomplishing an unseen goal. In this experiment, we show that ABR² enables successful generalization to new goals by acting according to the base policy while occasionally relying on a residual to steer the base policy with short, large-magnitude corrections. Analysis of the characteristics of the corrections and executed actions when using ABR² supporting this conclusion will be added to 4.1.

We then show that this method scales up to prompt-conditioned robot manipulation tasks (4.2), where the base policy is instead an embodiment-finetuned VLA. We evaluate ABR² on five unseen tasks. These tasks were selected because they represent tasks in which the base policy is partially competent, meaning that the base policy contains the skills necessary to solve the task, but requires active guidance to deploy those skills in a task-relevant manner. ABR² is able to repair the base policy with regard to these novel prompts and scenes.

2. Perturbations

We distinguish two classes of domain shift that a policy may see at deployment time. The first class is domain shift due to encountering tasks that have not yet been learned, requiring online adaptation or learning of new skills. The second class is a change in the distribution of states for already learned tasks, requiring a policy that is robust to covariate shift. We assess ABR²’s applicability to both classes of domain shift and find it aids in online task learning; however, it does not address robustness to test-time perturbations of the same task. This reflects the intended goal of ABR²: leveraging useful but imperfect base-policy behavior through targeted corrections, without expert demonstrations. When the base policy is weak or requires sustained correction, grounding can become restrictive and alternative methods may be more appropriate. We will make the distinction between the experiments in 4.2.1 and 4.2.2 more explicit in the paper.

3. Hyperparameter selection and comparison fairness

There are two hyperparameters in ABR²:

1) The value of λδλ_δ, which penalizes large corrections to the base-policy action.

  • For the selection of λδλ_δ, we found this was method-agnostic and primarily correlated with the task and the action-space bounds. λδλ_δ primarily serves to keep corrections reasonable and effectively controls how large a correction can be made by the residual policy.
  • In a hyperparameter sweep we found a strong value for each task and kept it frozen and uniform between methods. We observed no benefit either way in making this parameter method-specific.

2) The percentage of the budget dedicated to the warmup period.

  • We found that the choice of the warmup period for ABR² primarily serves to manage the learning of the base-policy value function. This is seen most clearly in the LIBERO-PRO experiments, where warmup periods shorter than 500 episodes have noticeably worse performance relative to the baseline.
  • We found that the choice of the warmup period on LIBERO tasks can be pushed as low as 50 episodes with ~2% regression.
  • The baseline is agnostic to the choice of warmup period in both LIBERO and LIBERO-PRO, as the warmup period only serves to pre-populate the replay buffer.

We will add additional details on how the hyperparameters were selected to the appendix.

Thanks for the update. The rebuttal clarifies the intended scope and the shared action-penalty setting, but it remains unclear whether per-task hyperparameter selection was separated from final evaluation and whether the baseline received a comparable tuning budget, so I’m keeping my rating at 2.

Thanks for responding. If we did not adequately answer your questions with the previous response, would it be possible for you to clarify what you mean by "per-task hyperparameter selection was separated from final evaluation" and "whether the baseline received a comparable tuning budget"?

On selecting λδ\lambda_\delta, we did a sweep of values from [0.1, 0.3, 1, 3, 10] on each task for both methods. We found that on the tasks we evaluated, the same choice of λδ\lambda_\delta resulted in the best steady state performance for each method. We attribute this to the fact that λδ\lambda_\delta primarily serves to keep corrections reasonable relative to the action-space bounds and task constraints. We also note that this hyperparameter is not unique to our method and prior work like RLToken use a similar term to constraint the magnitude of corrections.

For the choice of the length of the warm up period, this parameter is only relevant to ABR2\text{ABR}^2 as it manages the amount of samples the base policy value function is trained on before being frozen. For the baseline, this period is used to populate the replay buffer. We tried values from [50, 250, 500] for each task again for both methods. For the baseline, there was no change in steady state performance from this choice. However, it does matter for ABR2\text{ABR}^2. As stated in the prior comment, if the warm up period is too short, ABR2\text{ABR}^2 can under perform, and there can be a marginal gain to going longer in some tasks.

Results reported in the paper were collected after these hyperparameters were frozen from the per task hyperparameter sweeps described above.

Reviewer_Z15ROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to learn residual corrections to a pre-trained frozen VLA policy. The setting is that this VLA policy has somewhat correct required behavior in a new task but still cannot fully complete the task. The new task doesn't come with demo data but it can rollout policies with reward feedback. The main idea is a critic regularizer that is anchored to estimated base-policy returns. Experiments in simulation show improved repair on five selected unseen manipulation tasks.

Critical strengths
  1. Timely topic (how to continually improve VLAs' capabilities).

  2. The deployment setting is useful. A pretrained VLA already has somewhat of required behavior but is still not perfect on a new task where collecting new demos is undesirable.

  3. The proposed change is relatively simple.

Critical weaknesses
  1. The biggest issue of this paper is that the experiment setting is too limited. It only tries one VLA backbone in already saturated LIBERO simulation. Only 5 tasks and no real-world experiments. Moreover, the proposed regularizer has no consistent benefit under scene perturbations. Considering how crowded the VLA fine-tuning community is now, this is the biggest issue of this paper.

  2. There is no comparison with many existing RL residual learning methods in the VLA settings. For example, RL token and PLD.

  3. The proposed explanation "improved advantage estimation" is not well justified. The frozen value estimation could also become inaccurate on states reached after residual corrections.

Questions for authors

Please see the weakness part.

1 comment

Thank you for the feedback. We agree that evaluation breadth and the reliability of the frozen value estimator are important considerations. We clarify our setting, baseline, and claims below.

1. Evaluation scope

In this work, we start with a controlled, interpretable domain (4.1), where the base policy is a goal-conditioned reinforcement learning policy. We study online learning from a base policy that is capable but unreliable on an unseen goal. This includes confirming the hypothesis that acting primarily according to the base policy, combined with a few short corrections, is sufficient to repair the policy for a novel goal. We then scale up to prompt-conditioned robot manipulation with an embodiment-finetuned VLA (4.2). We chose LIBERO because finetuning a generalist pretrained policy like π0.5\pi_{0.5} on a subset of LIBERO data does not saturate the whole benchmark. Instead, it leaves the policy in a state of partial competence on a number of unseen tasks that we show can be repaired without human intervention using ABR2\mathrm{ABR}^2. Real-world deployment is outside our scope. We demonstrate ABR2\mathrm{ABR}^2's benefit on two classes of base policy, while recognizing that more evaluations would be ideal. We believe the experiments support the utility of advantage regularization for self-guided policy repair without human intervention.

2. Perturbations

We distinguish two classes of deployment-time domain shift. The first is encountering tasks that have not yet been learned, requiring online adaptation or learning new skills (4.2.1). The second is a change in the distribution of states for already learned tasks, requiring a policy robust to covariate shift (4.2.2). We assess ABR2\mathrm{ABR}^2 on both and find it aids online task learning, but does not address robustness to test-time perturbations of the same task. This reflects the intended goal of ABR2\mathrm{ABR}^2: leveraging useful but imperfect base-policy behavior through targeted corrections, without expert demonstrations. When the base policy is weak or requires sustained correction, grounding can become restrictive and alternative methods may be more appropriate. We will make the distinction between Sections 4.2.1 and 4.2.2 more explicit in the paper.

3. Comparison to other recent methods

Our primary VLA baseline derives from RL Token, and ABR2\text{ABR}^2 also leverages the same RL-token encoder for context compression. However, the full RL Token method requires human intervention in a number of ways which we don't believe is fair to directly compare to in our setting of online task learning without expert guidance. We adapt the original protocol from RL Token to our setting by removing the following:

- Human expert demonstrations on the target-task
- Fine tuning of the base policy to the target task using expert data
- Training the RL Token specifically with regard to the target task using expert data
- Human annotated handoffs points between the residual and base policy

In our experiments, both ABR2\text{ABR}^2 and the baseline use the same frozen RL Token encoder trained on the same data used to fine tune π0.5\pi_{0.5} to LIBERO rather than data collected from the unseen task. RL Token encoder training details can be found in Appendix C. We will make the baseline’s relationship to RL Token and these protocol differences more explicit in the paper.

4. Advantage estimation and Frozen value estimation after corrections

ABR2\text{ABR}^2 is designed to preserve useful behavior from the base policy while selectively learning deviations that improve performance on unseen tasks. This introduces an inductive bias toward relying on the base policy, which is beneficial when the base policy already provides a reasonable behavioral prior. The base-policy value function estimates the utility of acting according to the base policy so that we can find corrections that exceed that expectation. Therefore, it should remain frozen when residual actions may be executed to preserve this estimate. We agree that residual corrections may lead to states where the frozen value function may be inaccurate, but our formulation is designed to lean on the base policy for the vast majority of executed actions with short residual corrections to address critical errors. We believe that this limits impact of inaccurate value estimation on off-base-policy states. As part of the interpretable-domain experiments (4.1), we performed an analysis on the magnitude and duration of corrections made using ABR2\text{ABR}^2 and the baseline. We found that corrections produced by the residual in ABR2\text{ABR}^2 were either very small magnitude or large and short in duration as opposed to the baseline where the magnitude and length of correction was more normally distributed. We will add these supporting results to section 4.1. We believe that more effective ways to learn these value functions would be useful follow on work that builds upon ABR2\text{ABR}^2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.