acceptodds
Under review as a conference paper at ICLR 2027

Relative Bellman Constraints for Value Learning in Residual Reinforcement Learning

Abstract

Residual reinforcement learning (RL) adapts pretrained robot policies through bounded action corrections, making it appealing for real-robot learning where exploration and intervention are costly. However, value learning in residual RL typically focuses on absolute returns, whereas adaptation with a frozen base policy depends on resolving the value variations induced by these corrections. This motivates making value differences an explicit focus of supervision over the replay distribution, rather than learning them only indirectly from pointwise value targets. In this paper, we propose **ReVRA** (Relative Value Learning for Residual Adaptation), a plug-and-play value-learning module that explicitly supervises critic-induced value differences. ReVRA incorporates relative Bellman constraints alongside standard TD learning, using paired replay transitions to supervise value differences within the existing critic. Across simulated locomotion, manipulation, and VLA-based adaptation, ReVRA consistently improves residual RL performance. On real-robot manipulation tasks, it achieves higher task success with fewer interventions under the same limited interaction budget. The project code is available at https://anonymous.4open.science/r/ReVRA-DF70.

Primary area:
reinforcement learning
Keywords:
Residual Reinforcement Learning, Relative Value Learning, Robot Policy Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gNohOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ReVRA (Relative Value Learning for Residual Adaptation), a value-learning module for residual reinforcement learning for post-training existing policies.The main contribution is an additional term in the critic loss (called a relative Bellman loss), which fits the difference between critic predictions on pairs of replay transitions to a relative Bellman target. The paper claims that this helps the critic rank the small residual corrections around a frozen base policy, which leads to better residual policy improvement. The method is evaluated on Gym, Robomimic, ManiSkill2, LIBERO (with SmolVLA), and three real-robot tasks.

Critical strengths
  • The method is a simple additional regularized that can be added to most existing pipelines. It does not need separate networks or any other changes to the actor.
  • The evaluation is broad, and covers several configurations of base policies, two critic backbones, and includes real-robot experiments. Though, I have some questions on the rigorousness of experiments and the reported results.
  • The paper includes ablations on the loss components, the relative-loss weight β, and LayerNorm.
Critical weaknesses

While the paper is promising, I have several concerns regarding theoretical results, presentation, novelty, and experiments.

  • The value-dependent component of the actor gradient (Eq. 4) depends only on how QQ varies over actions at a fixed state. However, the relative loss pairs inputs from different states. The authors argue that relative Bellman consistency “is defined over arbitrary critic-input pairs and therefore does not require such counterfactual samples”. The constraint being well-defined for arbitrary pairs does not imply that it helps compare actions at the same state. Does it make sense theoretically to supervise value differences across different state-action pairs, when policy improvement only requires comparisons between actions at the same state?

  • The theory sections (Sec. 3 and 4) are highly unclear and generally poorly written. Notation is used before it is introduced, and it is often confusing or inconsistent. I highly encourage the authors to add a dedicated notation section, and rewrite both sections with a clear and logical flow of arguments. I elaborate below.

    • Several symbols are not introduced. ϕ\phi first appears in Eq. 3 and θ\theta at the end of Sec. 3, without being defined as the residual actor and critic parameters. xx is used in Eq. 5 but only defined as (s,a)(s, a) in Sec. 4.2. The policy π\pi in Eq. 5–6 is only specified in Sec. 4.2 as the executed-action policy.
    • The phrase “critic-induced value differences” is used throughout. Since the critic is the value function, this phrase is redundant. If the intended meaning is the action gap (Bellemare et al., 2016), please use the standard term. Note, however, that the action gap is defined at a single state, while the relative loss is computed across different states.
    • Eqs. 2–4 write aϕ(s)a_\phi(s) and ares,ϕ(s)a_{\text{res},\phi}(s) as deterministic functions of the state, while the residual policy is stochastic (Eq. 13, Eq. 16, Alg. 1). Please use standard policy notation and rewrite for stochastic policies.
    • Eq. 1 and the definition of QπQ^\pi use the standard RL objective without an entropy term. The method is built on SAC, and Eqs. 14 and 16 include αlog⁡π\alpha \log \pi. As a result, Def. 4.1, Prop. 4.2 and Thm. 4.3 are stated for the hard Bellman operator, while the implementation uses soft backups. Can you please state the maximum entropy objective and give the theoretical results for the soft Bellman operator?
    • Eqs. 13, 14 and 16 use only the residual log-probability log⁡πres,ϕ(ares∣s)\log \pi_{\text{res},\phi}(a_{\text{res}} \mid s). Is the base policy deterministic during RL training?
    • The theoretical contribution is limited. Several core theorems follow those of Höftmann et al. (2026), moved from V to Q. Please cite these results and clarify whats new.
  • The empirical evaluation is underspecified and potentially misleading.
    • The real-robot results seem to come from a single run with few trials, and the gaps to the strongest baseline are small. How many runs and trials were used? Are the improvements statistically significant?
    • Table 2 (LIBERO) reports no seeds, variance, or number of evaluation episodes. How many seeds were used? Is the residual policy trained per task, per suite, or across all suites?
    • Policy Decorator performs far below the frozen base policy on all Robomimic tasks, from the start of training. A residual method should start close to its base policy. Was this baseline tuned for this codebase?
Questions for authors

Please see weaknesses.

10 comments
AuthorsOpenReview ↗

Response to Reviewer gNoh (1/4): W1

We thank the reviewer for raising this point. We agree that being well-defined on arbitrary pairs does not by itself justify cross-state supervision, and the revised manuscript no longer uses this property as a justification (L314–320). Our answer is that policy improvement is local in the state, but policy evaluation, which is what the critic loss performs, is not: the same-state comparisons used by the actor are determined, through the Bellman equation, by value differences between different successor states. We make this precise in (1)–(3) and summarize the empirical evidence in (4).

(1) Same-state action comparisons are cross-state value comparisons one step later. Setting si=sj=ss_i=s_j=s in the fixed-point equation of Eq. 10 gives

Qπ(s,a)−Qπ(s,a~)=r(s,a)−r(s,a~)+γE[Qπ(s′,a′)−Qπ(s~′,a~′)],Q^\pi(s,a)-Q^\pi(s,\tilde a)=r(s,a)-r(s,\tilde a)+\gamma\mathbb{E}\big[Q^\pi(s',a')-Q^\pi(\tilde s',\tilde a')\big],

where s′∼P(⋅∣s,a)s'\sim\mathcal P(\cdot\mid s,a) and s~′∼P(⋅∣s,a~)\tilde s'\sim\mathcal P(\cdot\mid s,\tilde a). Two actions at the same state lead to different successor states, so the same-state comparison needed for policy improvement is determined by a cross-state difference at the next step. Same-state pairs are therefore not closed under the relative Bellman operator: a Bellman relation restricted to them is not self-contained, and the contraction and unique fixed point of Theorem 4.3 require the full pairwise space. Under sparse rewards, as in our real-robot tasks, the reward difference vanishes almost everywhere, and the advantage of one correction over another is entirely a cross-state difference between successors. For the same reason, state-dependent errors at successor inputs, which do not enter the actor gradient directly, become action-dependent errors at their predecessors through bootstrapping.

(2) Consistency on all pairs implies correctness at every state. Remark 4.4 shows that if ΔQ=Δπ\Delta_Q=\Delta^\pi on all pairs, then Q=Qπ+cQ=Q^\pi+c, hence ∇aQ(s,a)=∇aQπ(s,a)\nabla_a Q(s,a)=\nabla_a Q^\pi(s,a) at every state. Moreover, Δθ\Delta_\theta is induced by the single scalar critic (Eq. 9) rather than a separate pairwise network, so cross-state constraints act on the same function that the actor differentiates in Eq. 4.

(3) TD learning has no same-state supervision either. In continuous control with off-policy replay, a state is observed with essentially one action, so ∇aQθ\nabla_a Q_\theta in SAC is always obtained through generalization of a critic fitted across different states. ReVRA keeps this mechanism and the same data, and changes only which error is penalized: the relative term penalizes the error difference directly and is unaffected by errors shared by both predictions (L213–215). We do not claim that cross-state pairs replace counterfactual same-state samples, and we state the distinction from the single-state action gap explicitly (L237–238).

(4) Empirical support. Adding the relative term to the same critic, data, and actor update improves learning over absolute-only TD (Figure 4, and Figure 7 in Appendix D.1). The gain is also not reproduced by controls that only reweight TD residuals: on PegInsertionSide, ReVRA approaches 100% success, whereas Scaled TD reaches about 25% and Centered TD stays below 10% (Figure 10 in Appendix D.4).

AuthorsOpenReview ↗

Response to Reviewer gNoh (2/4): W2

We thank the reviewer for the detailed comments. We have revised both sections accordingly and added a dedicated notation section (Appendix A). Every symbol is now defined at its first use, the stochastic policy and the maximum-entropy objective are stated explicitly, the theory is extended to the soft operators used by SAC, and its relation to Höftmann et al. (2026) is made explicit. Line numbers refer to the revised manuscript.

(1). Symbols used before introduction. ϕ\phi is now defined as the residual-policy parameters (L148). θ1,θ2\theta_1,\theta_2 are defined as the critic parameters, with Qˉ\bar Q denoting the averaged target critic and QθQ_\theta either critic (L158–164). The critic input x=(s,a)x=(s,a) and the executed-action policy π\pi are defined at their first use in Eq. 5 (L191–196).

(2). "Critic-induced value differences" and the action gap. We agree and replace the phrase by "pairwise value differences" throughout (L19, L169–170, L225). We do not refer to the action gap, which compares actions at a single state, whereas our pairs are drawn from the replay buffer and generally come from different states. We state this distinction after Eq. 9 (L237–238).

(3). Stochastic policy notation. We now write the base and residual actions as samples, abase∼πbase(⋅∣s)a_{\mathrm{base}}\sim\pi_{\mathrm{base}}(\cdot\mid s) and ares∼πres,ϕ(⋅∣s)a_{\mathrm{res}}\sim\pi_{\mathrm{res},\phi}(\cdot\mid s), combined as a=abase+λaresa=a_{\mathrm{base}}+\lambda a_{\mathrm{res}} (Eq. 2, L149). Successor actions (L162, L288, L321–323) and Alg. 1 follow the same notation, and the actor objective takes the expectation over both sampled actions (Eq. 18, L350). We remove the deterministic notation aϕ(s)a_\phi(s) and ares,ϕ(s)a_{\mathrm{res},\phi}(s). Sec. 4.1 now states that aresa_{\mathrm{res}} is sampled through reparameterization, so Eq. 4 is the per-sample actor gradient (L174–179).

(4). Maximum-entropy objective and soft Bellman operator. Sec. 3 now describes the maximum-entropy objective of SAC and states its soft Bellman target (Eq. 3, L157–164). The new Corollary 4.5 defines the soft Bellman operator and the soft action-value function QsoftπQ^\pi_{\mathrm{soft}} (Eq. 13), and extends Proposition 4.2, Theorem 4.3, and Remark 4.4 to the soft operators by absorbing the entropy term into a bounded reward, as in the soft policy evaluation of SAC (L280–290, proof in Appendix B.3). The analysis in Sec. 4.1 thus applies with QπQ^\pi replaced by QsoftπQ^\pi_{\mathrm{soft}} (L302–303), and the first case of Eq. 16 applies the soft relative operator with sampled entropies, as in the SAC target (L331–333).

(5). Is the base policy deterministic? Not in all settings. It is deterministic for Behavior Transformer and stochastic for Diffusion Policy and VLA (L151–152), but always frozen (L147). From the residual policy's perspective, the frozen base policy is thus a fixed part of the environment, and our method is standard SAC over residual actions. Accordingly, only the residual policy is entropy-regularized (L163–164), and Eqs. 3, 15, 16, and 18 evaluate the critic at the combined action a=abase+λaresa=a_{\mathrm{base}}+\lambda a_{\mathrm{res}} while using the log-probability of the same residual sample aresa_{\mathrm{res}}.

(6). Relation to Höftmann et al. (2026). We now credit this work explicitly. Sec. 4.2 states that Definition 4.1 and Theorem 4.3 follow RVL (L225–229), Definition 4.1 is marked as adapted from RVL (L243), and the terminal cases of Eq. 16 are attributed to RVL (L335–337). Our new elements are threefold.

  • (i) We extend relative consistency to soft action values in off-policy SAC with continuous actions (Corollary 4.5), whereas RVL treats state values in on-policy PPO on discrete-action Atari.
  • (ii) We induce value differences from the existing scalar critic and train it with both absolute and relative supervision, whereas RVL learns a separate antisymmetric difference network without absolute targets. Proposition 4.2 shows that relative targets from a scalar critic are consistent with its standard Bellman updates, and Remark 4.4 shows that the absolute term fixes the global offset that relative consistency leaves unspecified.
  • (iii) We motivate explicit supervision of value differences in residual RL (L183–221).

The theory thus shows that, under exact evaluation, the coupled objective has a unique solution consistent with standard Bellman evaluation, while our main contribution is the method and its evaluation for residual RL.

AuthorsOpenReview ↗

Response to Reviewer gNoh (3/4): W3 (Part 1)

We thank the reviewer for the insightful comments. We address the three points in turn.

(1) Real-robot results.

Runs and trials. Following common real-robot RL practice (e.g., HIL-SERL [1], ConRFT [2]), we train one policy per method from the same Diffusion Policy checkpoint under the same 5k-step budget. To quantify evaluation variability, we evaluate each trained policy in 5 rounds of 30 fully autonomous trials (150 trials per method and task). The original Table 3 reported a single round. For this rebuttal, we evaluated the same trained policies for four additional rounds.

Statistical significance. The revised table (Table 2 of the revised manuscript, formerly Table 3) is shown below.

MethodPush Cube Success ↑Push Cube Intervention ↓Grasp Cube Success ↑Grasp Cube Intervention ↓Stack Cube Success ↑Stack Cube Intervention ↓
Diffusion Policy92.0 ± 5.1--52.0 ± 5.6--28.7 ± 7.7--
+ Policy Decorator99.3 ± 1.57.967.3 ± 10.115.245.3 ± 6.921.9
+ Vanilla DAWN100.0 ± 0.06.468.7 ± 11.211.352.0 ± 5.620.3
+ ReVRA (Ours)100.0 ± 0.05.383.3 ± 5.3*8.568.7 ± 5.1*17.2
pp-value----0.040*--0.001*--

Success rates are fully autonomous: mean ± std over 5 evaluation rounds of 30 trials each. Intervention rates are computed over the 5k-step online training budget as Nhuman/5000×100N_{\mathrm{human}}/5000 \times 100%. pp-values are from two-sided Welch's t-tests between ReVRA and Vanilla DAWN, and * denotes p<0.05p<0.05.

On the two harder tasks, ReVRA improves over the strongest baseline (Vanilla DAWN) by 14.6 percentage points on Grasp Cube and 16.7 on Stack Cube, and both improvements are statistically significant (p<0.05p<0.05). On Push Cube, all residual methods saturate near 100% success, and ReVRA's advantage lies in requiring the fewest interventions. ReVRA also achieves the lowest intervention rate on all three tasks.

(2) LIBERO results.

Seeds and evaluation episodes. Each LIBERO suite (Spatial, Object, Goal, Long) contains 10 tasks. The original Table 2 was obtained from a single training seed and evaluated with 10 episodes per task (100 episodes per suite) from LIBERO's fixed initial states. For this rebuttal, we trained two additional seeds (three in total) and increased the evaluation to 20 episodes per task (200 episodes per suite and seed). The revised table (Table 1 of the revised manuscript, formerly Table 2) is shown below.

MethodSpatialObjectGoalLongAvg.
SmolVLA (0.45B)84.8 ± 3.594.0 ± 0.083.7 ± 1.451.0 ± 5.378.4 ± 1.0
+ Policy Decorator85.0 ± 3.394.8 ± 0.383.3 ± 2.351.8 ± 2.078.8 ± 1.9
+ Vanilla DAWN84.7 ± 2.893.0 ± 2.283.8 ± 4.352.8 ± 2.878.6 ± 0.6
+ DAWN85.5 ± 2.694.5 ± 1.085.5 ± 0.954.8 ± 2.080.1 ± 1.6
+ DAWN + ReVRA (Ours)88.5 ± 0.996.0 ± 1.086.7 ± 0.657.8 ± 2.582.3 ± 0.5

Success rates (%): mean ± std over three training seeds, each evaluated on 200 episodes per suite. For the frozen SmolVLA base policy, the std is over three evaluation seeds.

Training details. The residual policy is trained per suite: for each suite, a single residual actor and twin critic are trained jointly on all 10 tasks on top of the same frozen SmolVLA checkpoint, and the critic additionally takes a one-hot encoding of the task ID as input to accommodate multi-task training. We have added this description to Section 5.3.

[1] J. Luo et al. Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. Science Robotics, 2025.

[2] Y. Chen et al. ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy. RSS, 2025.

Reviewer_rvKWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReVRA augments a residual-SAC critic with a relative Bellman loss on pairs of replay transitions. The base policy remains frozen, value differences are induced by the existing scalar critic, and the actor objective is unchanged. The paper derives a fixed-policy relative Bellman operator and its contraction/fixed-point properties, then implements soft targets with explicit mixed-terminal cases. Evaluation covers locomotion, manipulation with diffusion and transformer base policies, SmolVLA adaptation, and three real-robot tasks. Ablations compare absolute-only, relative-only, and coupled losses, critic normalization, and integration with full DAWN.

Critical strengths
  1. The modification is small and the evaluation is broad. The backbone-matched comparisons, the experiments without LayerNorm, and the full-DAWN comparison in Appendix C.3 help establish that the gains are not simply due to a different critic architecture. This is a practical advantage for a method intended as a plug-in.

  2. The results extend to VLA and real-robot adaptation. LIBERO average success is 81.75%, compared with 77.50% for Vanilla DAWN (Table 2), and the robot experiments report improved Grasp/Stack success under the same stated interaction budget. The absolute-only and relative-only ablations also show why retaining both losses can be useful.

Critical weaknesses
  1. The local-action motivation is not yet directly supported by the random-pair experiments. For nonterminal pairs formed with the same individual targets, the relative residual is e_i−e_j; for independent identically distributed pairs its squared expectation is 2 Var(e). The loss can therefore emphasize differences in TD error without supplying information about alternative actions at the same state. That may still be effective, but a matched loss-scale or centered-residual control would help distinguish the explanation. The theoretical contribution also needs precise positioning relative to the cited Relative Value Learning paper, which already introduces a pairwise Bellman operator and contraction result.

  2. The real-robot evaluation needs a clearer account of human assistance. Appendix B.3 permits intervention during training and evaluation, but Table 3 does not say whether its successes are autonomous, what the intervention-rate denominator is, or how intervention decisions are standardized. Tables 2–3 also lack the evaluation counts and repeated-run uncertainty needed to interpret these gains, unlike Figure 3. This is important because a success obtained after an evaluation takeover is a different outcome from autonomous policy success.

Questions for authors
  1. How does the relative loss compare with a matched-scale TD loss or a centered-TD-residual penalty? A direct local-action ranking diagnostic would also help, but I would view it as supporting evidence rather than a requirement for the practical contribution.

  2. Which theoretical claims and target constructions are new relative to Höftmann et al.? Please distinguish the fixed-policy operator analysis from any implication about nonlinear off-policy SAC convergence.

  3. Were Table 3 successes counted after human takeovers? Please give autonomous success separately if assistance was allowed, define the intervention rate, and report evaluation counts and independent training runs for Tables 2–3.

3 comments
AuthorsOpenReview ↗

Response to Reviewer rvKW (1/2): W1, Q1 & Q2

Response to W1 and Q1.

Motivation. We agree that ReVRA supplies no additional actions at the same state. It changes how the critic is supervised on the same replay data, and we revised Sec. 4.1 so that the motivation no longer relies on comparing alternative actions at the same state. Following RVL, policy improvement is unaffected by a constant offset of the critic and thus depends on value differences (L180–182). For any two critic inputs, the predicted difference deviates from the true one by the error difference, which pointwise regression constrains only indirectly and relative supervision penalizes directly (L191–203, L213–215). This explicit signal is particularly valuable in residual RL, where the critic must represent variations induced by the scaled residual (L215–218). Since the argument applies to arbitrary pairs, random replay pairs supervise the critic's difference function as a whole.

Matched-scale and centered controls. Following the reviewer, let ek=Qθ(xk)−ykQe_k=Q_\theta(x_k)-y^Q_k denote the TD residual, which differs from the estimation error ϵk\epsilon_k against the true value in Sec. 4.1. The reduction to 2Var(e)2\mathrm{Var}(e) assumes that the relative target is formed from the same individual TD targets. With exact targets, this agrees with our idealized analysis in Eq. 8. In practice, however, ReVRA bootstraps its relative targets through relative Bellman consistency, and they differ from differences of TD targets even for nonterminal pairs. The SAC target bootstraps from the minimum of the two target critics, whereas the relative target bootstraps from their average (Eqs. 3 and 16). In addition, pairs in which exactly one transition terminates use the terminal identity instead of the two TD targets. The relative residual is thus not ei−eje_i-e_j, and the objective does not reduce to 2Var(e)2\mathrm{Var}(e).

To test this directly, we add both suggested controls to DAWN with the same β=1\beta=1 as ReVRA. Scaled TD minimizes (1+2β)E[e2](1+2\beta)\mathbb{E}[e^2], which matches the loss scale, and Centered TD minimizes E[e2]+2βVar(e)\mathbb{E}[e^2]+2\beta\mathrm{Var}(e) with the variance taken over the minibatch, which is exactly the reduced objective. Implementation details and learning curves are given in Appendix D.4. On PushChair, Centered TD performs on par with ReVRA and Scaled TD learns more slowly, consistent with the reviewer's view that emphasizing TD-error differences can be effective. On PegInsertionSide, however, ReVRA approaches 100% success, whereas Scaled TD reaches only about 25% and Centered TD stays below 10%. The reduced objective thus does not reproduce the gain of ReVRA on this task. Since both controls share all other components with ReVRA and are run with the same β\beta over 5 seeds, the gain is not explained by reweighting TD residuals alone and depends on how the relative targets are constructed.

Response to Q2 (and the positioning in W1).

What follows RVL. The pairwise Bellman operator, its contraction with a unique fixed point (Definition 4.1, Theorem 4.3), and the treatment of mixed terminal pairs in Eq. 16 follow Höftmann et al. (2026). The revised manuscript now states this explicitly (L225–229, L243, L335–337).

What is new.

  • (i) Corollary 4.5 extends the operator from state values to soft action values, as required by off-policy SAC with continuous actions.
  • (ii) Value differences are induced by the existing scalar critic and trained jointly with absolute TD targets (Eqs. 9 and 17), whereas RVL learns a separate antisymmetric network without absolute targets. Proposition 4.2 and Remark 4.4 characterize this coupling.
  • (iii) The relative target in Eq. 16 bootstraps from the difference of the averaged target critics with sampled entropy terms (L324–333).

Scope. All results concern exact evaluation of a fixed policy. We do not establish convergence of off-policy SAC with function approximation, sampled targets, and a changing policy, which we now state explicitly (L303–305).

AuthorsOpenReview ↗

Response to Reviewer rvKW (2/2): W2 & Q3

We thank the reviewer for these questions and for pointing out the ambiguity in Appendix B.3.

1. All success rates in the original Table 3 (now Table 2) were obtained under fully autonomous evaluation. Human intervention via the SpaceMouse was used only during online training, and no human takeover occurred in any evaluation trial, so every reported success was achieved by the policy on its own. We have clarified this distinction in Appendix C.3 of the revised manuscript (formerly Appendix B.3).

2. Intervention rate and intervention protocol. The intervention rate is measured during training only, as Nhuman/5000×100N_{\mathrm{human}}/5000 \times 100%, where NhumanN_{\mathrm{human}} is the number of training steps in which the operator overrides the policy's action. To keep interventions comparable across methods, the same operator trained all methods under a fixed rule set: the operator intervenes only when (i) a collision or unsafe motion is imminent, (ii) the end-effector leaves the workspace, or (iii) the policy makes no progress for more than three seconds. The operator then provides a brief corrective motion and immediately returns control to the policy. We have added this protocol to Appendix C.3 of the revised manuscript (formerly Appendix B.3).

3. Real-robot results (original Table 3, now Table 2): trial counts and uncertainty. Following common real-robot RL practice [1,2], we train one policy per method from the same Diffusion Policy checkpoint under the same budget. We evaluated each trained policy in 5 evaluation rounds of 30 autonomous trials (150 trials per method and task). The original Table 3 reported a single round of 30 trials, and for this rebuttal we evaluated the same trained policies for four additional rounds. Table 2 of the revised manuscript (formerly Table 3) reports the mean ± std over rounds, together with pp-values that assess whether ReVRA's improvement over the strongest baseline is statistically significant.

MethodPush Cube Success ↑Push Cube Intervention ↓Grasp Cube Success ↑Grasp Cube Intervention ↓Stack Cube Success ↑Stack Cube Intervention ↓
Diffusion Policy92.0 ± 5.1--52.0 ± 5.6--28.7 ± 7.7--
+ Policy Decorator99.3 ± 1.57.967.3 ± 10.115.245.3 ± 6.921.9
+ Vanilla DAWN100.0 ± 0.06.468.7 ± 11.211.352.0 ± 5.620.3
+ ReVRA (Ours)100.0 ± 0.05.383.3 ± 5.3*8.568.7 ± 5.1*17.2
pp-value----0.040*--0.001*--

pp-values are from two-sided Welch's t-tests between ReVRA and Vanilla DAWN, and * denotes p<0.05p<0.05.

In summary, ReVRA improves autonomous success over the strongest baseline (Vanilla DAWN) by 14.6 percentage points on Grasp Cube and 16.7 on Stack Cube, and requires the fewest interventions on all three tasks.

4. LIBERO results (original Table 2, now Table 1): evaluation counts and independent training runs. One residual policy is trained per suite and shared across its 10 tasks. The original Table 2 used a single training run evaluated with 10 trials per task (100 episodes per suite). For this rebuttal, we trained two additional runs per method (3 independent runs in total) and increased evaluation to 20 trials per task (200 episodes per suite and run). Table 1 of the revised manuscript (formerly Table 2) reports the mean ± std over the 3 runs.

MethodSpatialObjectGoalLongAvg.
SmolVLA (0.45B)84.8 ± 3.594.0 ± 0.083.7 ± 1.451.0 ± 5.378.4 ± 1.0
+ Policy Decorator85.0 ± 3.394.8 ± 0.383.3 ± 2.351.8 ± 2.078.8 ± 1.9
+ Vanilla DAWN84.7 ± 2.893.0 ± 2.283.8 ± 4.352.8 ± 2.878.6 ± 0.6
+ DAWN85.5 ± 2.694.5 ± 1.085.5 ± 0.954.8 ± 2.080.1 ± 1.6
+ DAWN + ReVRA (Ours)88.5 ± 0.996.0 ± 1.086.7 ± 0.657.8 ± 2.582.3 ± 0.5

In summary, ReVRA achieves the highest success rate on all four suites and improves its matched baseline, full DAWN, by 2.2 percentage points on average.

[1] J. Luo et al. Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. Science Robotics, 2025.

[2] Y. Chen et al. ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy. RSS, 2025.


We thank the reviewer again for the constructive feedback. We hope that our responses and the revised manuscript address your concerns. If anything remains unclear, we would be glad to provide further clarification during the discussion period.

Thanks for the detailed and thoughtful rebuttal, which effectively addresses most of my concerns and provides valuable clarifications. I will maintain my score.

Reviewer_Pd2BOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReVRA, a critic-learning module for residual reinforcement learning with a frozen base policy. It augments standard TD regression with a loss on value differences between replay transitions paired through a cyclic minibatch shift, without adding a pairwise network or changing the actor objective. The paper establishes consistency and contraction properties for fixed-policy relative Bellman evaluation and implements entropy-adjusted, terminal-aware targets in residual SAC.

Critical strengths
  1. ReVRA adds a value-difference loss to the existing critic without changing the actor or collecting extra environment transitions. This makes it a practical way to improve a frozen robot policy without fine-tuning the base model.

  2. The clearest benefit is faster learning on difficult control tasks, including PegInsertionSide, alongside gains on HalfCheetah, Walker2d, and Square-Visual. Results with Diffusion Policy, Behavior Transformer, and SmolVLA suggest that the modification is not tied to one policy architecture (Figure 3; Table 2).

Critical weaknesses
  1. The gains over strong residual methods are not equally clear across tasks. Can and Hopper show little difference between the critic objectives (Appendix C.1). On LIBERO-Long, ReVRA improves Vanilla DAWN from 53% to 55%, while the four-suite average improves by 4.25 percentage points (Table 2). These are useful results, but Table 2 gives no evaluation counts or variation across training runs. The full-DAWN comparison also covers only two locomotion tasks. More evidence is needed to know how reliably ReVRA improves a strong residual learner.

  2. It is unclear whether the real-robot success rates include human help. Appendix B.3 allows intervention during evaluation, but Table 3 does not separate assisted and autonomous success. It also omits trial counts and variation across training runs. The lower intervention rates are encouraging, but the table does not yet show how much better the robot performs on its own.

Questions for authors
  1. How many training runs and evaluation episodes were used for Table 2? Are the gains over Vanilla DAWN consistent across runs? Does ReVRA also improve full DAWN on manipulation or VLA tasks?

  2. Does Table 3 count successes achieved with human help? Please report autonomous success, trial counts, and how intervention rates are calculated.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Pd2B (1/2): W1 & Q1

We thank the reviewer for raising this point. To address how reliably ReVRA improves strong residual learners, we have substantially extended our evaluation: LIBERO is now evaluated over three training seeds with 20 evaluation episodes per task (200 per suite and seed, up from 100), and the full-DAWN comparison now covers locomotion, manipulation, and VLA tasks. Across all these settings, ReVRA consistently improves strong residual learners.

(1) LIBERO: more seeds, more evaluation episodes, consistent gains. The original Table 2 used a single training seed and 10 evaluation episodes for each of the 10 tasks per suite (100 episodes per suite). We now train three seeds and evaluate 20 episodes per task (200 episodes per suite and seed). Due to time and computational constraints, we currently report three seeds and will extend them to five seeds in the final version.

MethodSpatialObjectGoalLongAvg.
SmolVLA (0.45B)84.8 ± 3.594.0 ± 0.083.7 ± 1.451.0 ± 5.378.4 ± 1.0
+ Policy Decorator85.0 ± 3.394.8 ± 0.383.3 ± 2.351.8 ± 2.078.8 ± 1.9
+ Vanilla DAWN84.7 ± 2.893.0 ± 2.283.8 ± 4.352.8 ± 2.878.6 ± 0.6
+ DAWN85.5 ± 2.694.5 ± 1.085.5 ± 0.954.8 ± 2.080.1 ± 1.6
+ DAWN + ReVRA (Ours)88.5 ± 0.996.0 ± 1.086.7 ± 0.657.8 ± 2.582.3 ± 0.5

Success rates (%): mean ± std over three training seeds, each evaluated on 200 episodes per suite.

Under this stricter protocol, ReVRA achieves the highest success rate on all four suites with low variance across seeds. It improves its matched baseline, full DAWN, by 2.2 percentage points on average and 3.0 on LIBERO-Long, and Vanilla DAWN (without data warmup) by 3.7 and 5.0 percentage points.

(2) ReVRA improves full DAWN on locomotion, manipulation, and VLA tasks. We extended the full-DAWN comparison (Appendix D.3, Figure 9) to additionally cover two ManiSkill2 manipulation tasks, and added full DAWN as a baseline on LIBERO. ReVRA improves full DAWN in every setting:

  • Locomotion. DAWN + ReVRA achieves higher returns on HalfCheetah and Walker2d.
  • Manipulation. On PegInsertionSide, DAWN + ReVRA reaches near-perfect success, whereas full DAWN remains below 50% after 10M steps with large variance across seeds. On PushChair, DAWN + ReVRA converges to a higher success rate with lower variance.
  • VLA-based Adaptation. DAWN + ReVRA outperforms full DAWN on all four LIBERO suites, raising the average from 80.1% to 82.3% and LIBERO-Long from 54.8% to 57.8%.

(3) Gains concentrate on harder tasks. The smaller differences on Can and Hopper arise because these tasks are near saturation: all critic objectives quickly converge to the same level, and ReVRA matches the strong baselines without any degradation. Where strong baselines leave headroom, ReVRA's gains are substantial, both in simulation (HalfCheetah, Walker2d, Square-Visual, PegInsertionSide, and StackCube in Figure 3) and on the real robot, where ReVRA improves over Vanilla DAWN by 14.6 and 16.7 percentage points on Grasp Cube and Stack Cube with statistical significance (p<0.05p<0.05, see our response to W2). We also note that Appendix D.1 (formerly C.1) ablates the two components of the ReVRA objective. On Can and Hopper, even the relative loss alone matches the full objective.

(4) Consistency across architectures. Strong baselines are setting-dependent: Policy Decorator is strong on ManiSkill2 with Behavior Transformer but unstable on Robomimic with Diffusion Policy, and Vanilla DAWN converges notably more slowly on several ManiSkill2 tasks once its normalization is transferred to the MLP critic of Policy Decorator (Figure 3). In contrast, ReVRA achieves the best or near-best performance across three base policies, two critic architectures, and both simulated and real-world tasks, and remains effective without LayerNorm (Appendix D.2, formerly C.2). This makes ReVRA a reliable plug-and-play improvement for residual RL.

AuthorsOpenReview ↗

Response to Reviewer Pd2B (2/2): W2 & Q2

We thank the reviewer for pointing out this ambiguity.

1. All success rates in Table 2 of the revised manuscript (formerly Table 3) were obtained under fully autonomous evaluation. We apologize for the ambiguity in our original description. Human intervention via the SpaceMouse was used only during online training, and no human intervention occurred in any evaluation trial. We have clarified this distinction in Appendix C.3 of the revised manuscript (formerly Appendix B.3).

2. Trial counts and evaluation protocol. Following most real-robot RL practices (e.g., HIL-SERL [1], ConRFT [2]), we train one policy per method from the same Diffusion Policy checkpoint and under the same budget. To report trial counts and evaluation variability, we evaluated each trained policy in 5 evaluation rounds of 30 autonomous trials each (150 trials per method and task). The original Table 3 reported a single round of 30 trials. For this rebuttal, we evaluated the same trained policies for four additional rounds. Table 2 of the revised manuscript (formerly Table 3) reports the mean ± std over rounds, together with pp-values that assess whether ReVRA's improvement over the strongest baseline is statistically significant.

MethodPush Cube Success ↑Push Cube Intervention ↓Grasp Cube Success ↑Grasp Cube Intervention ↓Stack Cube Success ↑Stack Cube Intervention ↓
Diffusion Policy92.0 ± 5.1--52.0 ± 5.6--28.7 ± 7.7--
+ Policy Decorator99.3 ± 1.57.967.3 ± 10.115.245.3 ± 6.921.9
+ Vanilla DAWN100.0 ± 0.06.468.7 ± 11.211.352.0 ± 5.620.3
+ ReVRA (Ours)100.0 ± 0.05.383.3 ± 5.3*8.568.7 ± 5.1*17.2
pp-value----0.040*--0.001*--

Success rates are fully autonomous: mean ± std over 5 evaluation rounds of 30 trials each, with no human intervention during evaluation. Intervention rates are computed over the full 5k-step online training budget as Nhuman/5000×100N_{\mathrm{human}}/5000 \times 100%, where NhumanN_{\mathrm{human}} is the number of training steps during which the operator overrides the policy's action. pp-values are from two-sided Welch's t-tests between ReVRA and Vanilla DAWN, and * denotes p<0.05p<0.05.

3. Autonomous performance gains. Compared with the strongest baseline (Vanilla DAWN), ReVRA improves autonomous success by 14.6 percentage points on Grasp Cube and 16.7 on Stack Cube, with fewer training interventions. Both gains are statistically significant (p<0.05p<0.05), i.e., unlikely to arise from evaluation randomness alone.

[1] J. Luo et al. Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. Science Robotics, 2025.

[2] Y. Chen et al. ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy. RSS, 2025.


We thank the reviewer again for the constructive feedback. We hope that our responses and the revised manuscript address your concerns. If anything remains unclear, we would be glad to provide further clarification during the discussion period.

AuthorsOpenReview ↗

Follow-up: Updated Five-Seed Results for LIBERO and Real-Robot Experiments

We thank the reviewer again for the constructive feedback. Following the questions on variability across training runs (Q1 and Q2), we have completed additional experiments and now report five independent seeds for both LIBERO and the real-robot tasks, with 95% confidence intervals and pp-values. These results supersede the tables in our earlier responses and are included in the revised manuscript.

1. LIBERO (Table 1 of the revised manuscript). All methods now use 5 seeds, each evaluated on 200 episodes per suite. For each seed, SmolVLA is fine-tuned as the base policy and all residual methods are trained on top of it, so all rows share the same seeds.

MethodSpatialObjectGoalLongAvg.
SmolVLA (0.45B)84.2 ± 3.493.5 ± 0.983.4 ± 1.350.2 ± 4.977.8 ± 1.3
+ Policy Decorator85.0 ± 3.494.0 ± 1.683.5 ± 2.052.6 ± 2.278.8 ± 1.7
+ Vanilla DAWN84.2 ± 2.792.9 ± 1.983.7 ± 3.854.0 ± 3.178.7 ± 0.9
+ DAWN85.3 ± 2.394.2 ± 1.085.5 ± 0.955.5 ± 2.280.1 ± 1.4
+ DAWN + ReVRA (Ours)88.5 ± 0.8*96.1 ± 0.9*86.3 ± 1.057.8 ± 2.382.2 ± 0.5*
pp-value (vs. DAWN)0.017*0.005*0.1420.0750.012*

Mean success rates (%) over 5 seeds with 95% confidence intervals. pp-values are from two-sided Welch's t-tests between DAWN + ReVRA and DAWN, and * denotes p<0.05p<0.05.

Since ReVRA is applied on top of full DAWN in the revised Table 1, we assess its gains against this matched baseline. ReVRA improves the average success over DAWN by 2.1 points (p=0.012p=0.012). ReVRA improves DAWN on all four suites, by 3.2, 1.9, 0.8, and 2.3 points on Spatial, Object, Goal, and Long, respectively. On Goal and Long, these gains are small relative to seed variability. Beyond LIBERO, the full-DAWN comparison now also covers manipulation (Appendix D.3, Figure 9), where DAWN + ReVRA reaches near-perfect success on PegInsertionSide while full DAWN remains below 50% after 10M steps.

2. Real robot (Table 2 of the revised manuscript). Our earlier response reported 5 evaluation rounds of a single trained policy per method. We now train 5 independent seeds per method. Each seed trains its own Diffusion Policy, and all residual methods are trained on top of it under the same 5k-step budget. Each trained policy is evaluated with 30 fully autonomous trials, with no human intervention during evaluation.

MethodPush Cube Success ↑Push Cube Intervention ↓Grasp Cube Success ↑Grasp Cube Intervention ↓Stack Cube Success ↑Stack Cube Intervention ↓
Diffusion Policy95.3 ± 6.9--49.3 ± 8.0--25.3 ± 10.0--
+ Policy Decorator97.3 ± 3.58.2 ± 1.066.0 ± 14.519.2 ± 4.844.7 ± 11.219.3 ± 2.8
+ Vanilla DAWN98.7 ± 2.35.9 ± 0.569.3 ± 13.512.5 ± 4.754.0 ± 9.022.1 ± 4.5
+ ReVRA (Ours)100.0 ± 0.05.0 ± 0.384.0 ± 7.4*7.7 ± 1.966.7 ± 6.5*13.9 ± 3.2
pp-value0.178--0.037*--0.015*--

Mean success and intervention rates (%) over 5 seeds with 95% confidence intervals. pp-values are from two-sided Welch's t-tests on success rates between ReVRA and Vanilla DAWN, and * denotes p<0.05p<0.05.

Across training runs, ReVRA achieves the highest mean autonomous success on all three tasks. Its improvements over Vanilla DAWN are statistically significant on Grasp Cube (+14.7 points, p=0.037p=0.037) and Stack Cube (+12.7 points, p=0.015p=0.015). These values replace the single-run estimates in our earlier response. Intervention rates are measured per training run and averaged over the five seeds. Since the operator was not blinded to the method being trained, we treat them as supporting evidence and state this caveat in Appendix C.3.


As the discussion period is approaching its end, we would greatly appreciate it if the reviewer could let us know whether these updates address the concerns. We would be glad to provide any further clarification in the remaining time. If the concerns have been resolved, we would be grateful if the reviewer could take these results into account in the final assessment.

Thank you for the substantial additional experiments. The five-seed results address my concerns about training variability, and the expanded full-DAWN comparisons strengthen the empirical evidence. I also appreciate the independent real-robot training runs and confirmation that evaluation was fully autonomous.

The average LIBERO gain remains modest, and improvements on Goal and Long are not statistically significant. Please retain these qualifications and treat intervention rates as supporting evidence given the unblinded operator.

Overall, my main concerns have been addressed, and I will raise my rating from 2 to 3. Please ensure that the revised manuscript consistently reflects the updated results, protocols, and limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.