- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes ReVRA (Relative Value Learning for Residual Adaptation), a value-learning module for residual reinforcement learning for post-training existing policies.The main contribution is an additional term in the critic loss (called a relative Bellman loss), which fits the difference between critic predictions on pairs of replay transitions to a relative Bellman target. The paper claims that this helps the critic rank the small residual corrections around a frozen base policy, which leads to better residual policy improvement. The method is evaluated on Gym, Robomimic, ManiSkill2, LIBERO (with SmolVLA), and three real-robot tasks.
Critical strengths
- The method is a simple additional regularized that can be added to most existing pipelines. It does not need separate networks or any other changes to the actor.
- The evaluation is broad, and covers several configurations of base policies, two critic backbones, and includes real-robot experiments. Though, I have some questions on the rigorousness of experiments and the reported results.
- The paper includes ablations on the loss components, the relative-loss weight β, and LayerNorm.
Critical weaknesses
While the paper is promising, I have several concerns regarding theoretical results, presentation, novelty, and experiments.
-
The value-dependent component of the actor gradient (Eq. 4) depends only on how varies over actions at a fixed state. However, the relative loss pairs inputs from different states. The authors argue that relative Bellman consistency “is defined over arbitrary critic-input pairs and therefore does not require such counterfactual samples”. The constraint being well-defined for arbitrary pairs does not imply that it helps compare actions at the same state. Does it make sense theoretically to supervise value differences across different state-action pairs, when policy improvement only requires comparisons between actions at the same state?
-
The theory sections (Sec. 3 and 4) are highly unclear and generally poorly written. Notation is used before it is introduced, and it is often confusing or inconsistent. I highly encourage the authors to add a dedicated notation section, and rewrite both sections with a clear and logical flow of arguments. I elaborate below.
- Several symbols are not introduced. first appears in Eq. 3 and at the end of Sec. 3, without being defined as the residual actor and critic parameters. is used in Eq. 5 but only defined as in Sec. 4.2. The policy in Eq. 5–6 is only specified in Sec. 4.2 as the executed-action policy.
- The phrase “critic-induced value differences” is used throughout. Since the critic is the value function, this phrase is redundant. If the intended meaning is the action gap (Bellemare et al., 2016), please use the standard term. Note, however, that the action gap is defined at a single state, while the relative loss is computed across different states.
- Eqs. 2–4 write and as deterministic functions of the state, while the residual policy is stochastic (Eq. 13, Eq. 16, Alg. 1). Please use standard policy notation and rewrite for stochastic policies.
- Eq. 1 and the definition of use the standard RL objective without an entropy term. The method is built on SAC, and Eqs. 14 and 16 include . As a result, Def. 4.1, Prop. 4.2 and Thm. 4.3 are stated for the hard Bellman operator, while the implementation uses soft backups. Can you please state the maximum entropy objective and give the theoretical results for the soft Bellman operator?
- Eqs. 13, 14 and 16 use only the residual log-probability . Is the base policy deterministic during RL training?
- The theoretical contribution is limited. Several core theorems follow those of Höftmann et al. (2026), moved from V to Q. Please cite these results and clarify whats new.
- The empirical evaluation is underspecified and potentially misleading.
- The real-robot results seem to come from a single run with few trials, and the gaps to the strongest baseline are small. How many runs and trials were used? Are the improvements statistically significant?
- Table 2 (LIBERO) reports no seeds, variance, or number of evaluation episodes. How many seeds were used? Is the residual policy trained per task, per suite, or across all suites?
- Policy Decorator performs far below the frozen base policy on all Robomimic tasks, from the start of training. A residual method should start close to its base policy. Was this baseline tuned for this codebase?
Questions for authors
Please see weaknesses.
10 comments
Response to Reviewer gNoh (2/4): W2
We thank the reviewer for the detailed comments. We have revised both sections accordingly and added a dedicated notation section (Appendix A). Every symbol is now defined at its first use, the stochastic policy and the maximum-entropy objective are stated explicitly, the theory is extended to the soft operators used by SAC, and its relation to Höftmann et al. (2026) is made explicit. Line numbers refer to the revised manuscript.
(1). Symbols used before introduction. is now defined as the residual-policy parameters (L148). are defined as the critic parameters, with denoting the averaged target critic and either critic (L158–164). The critic input and the executed-action policy are defined at their first use in Eq. 5 (L191–196).
(2). "Critic-induced value differences" and the action gap. We agree and replace the phrase by "pairwise value differences" throughout (L19, L169–170, L225). We do not refer to the action gap, which compares actions at a single state, whereas our pairs are drawn from the replay buffer and generally come from different states. We state this distinction after Eq. 9 (L237–238).
(3). Stochastic policy notation. We now write the base and residual actions as samples, and , combined as (Eq. 2, L149). Successor actions (L162, L288, L321–323) and Alg. 1 follow the same notation, and the actor objective takes the expectation over both sampled actions (Eq. 18, L350). We remove the deterministic notation and . Sec. 4.1 now states that is sampled through reparameterization, so Eq. 4 is the per-sample actor gradient (L174–179).
(4). Maximum-entropy objective and soft Bellman operator. Sec. 3 now describes the maximum-entropy objective of SAC and states its soft Bellman target (Eq. 3, L157–164). The new Corollary 4.5 defines the soft Bellman operator and the soft action-value function (Eq. 13), and extends Proposition 4.2, Theorem 4.3, and Remark 4.4 to the soft operators by absorbing the entropy term into a bounded reward, as in the soft policy evaluation of SAC (L280–290, proof in Appendix B.3). The analysis in Sec. 4.1 thus applies with replaced by (L302–303), and the first case of Eq. 16 applies the soft relative operator with sampled entropies, as in the SAC target (L331–333).
(5). Is the base policy deterministic? Not in all settings. It is deterministic for Behavior Transformer and stochastic for Diffusion Policy and VLA (L151–152), but always frozen (L147). From the residual policy's perspective, the frozen base policy is thus a fixed part of the environment, and our method is standard SAC over residual actions. Accordingly, only the residual policy is entropy-regularized (L163–164), and Eqs. 3, 15, 16, and 18 evaluate the critic at the combined action while using the log-probability of the same residual sample .
(6). Relation to Höftmann et al. (2026). We now credit this work explicitly. Sec. 4.2 states that Definition 4.1 and Theorem 4.3 follow RVL (L225–229), Definition 4.1 is marked as adapted from RVL (L243), and the terminal cases of Eq. 16 are attributed to RVL (L335–337). Our new elements are threefold.
- (i) We extend relative consistency to soft action values in off-policy SAC with continuous actions (Corollary 4.5), whereas RVL treats state values in on-policy PPO on discrete-action Atari.
- (ii) We induce value differences from the existing scalar critic and train it with both absolute and relative supervision, whereas RVL learns a separate antisymmetric difference network without absolute targets. Proposition 4.2 shows that relative targets from a scalar critic are consistent with its standard Bellman updates, and Remark 4.4 shows that the absolute term fixes the global offset that relative consistency leaves unspecified.
- (iii) We motivate explicit supervision of value differences in residual RL (L183–221).
The theory thus shows that, under exact evaluation, the coupled objective has a unique solution consistent with standard Bellman evaluation, while our main contribution is the method and its evaluation for residual RL.
Response to Reviewer gNoh (3/4): W3 (Part 1)
We thank the reviewer for the insightful comments. We address the three points in turn.
(1) Real-robot results.
Runs and trials. Following common real-robot RL practice (e.g., HIL-SERL [1], ConRFT [2]), we train one policy per method from the same Diffusion Policy checkpoint under the same 5k-step budget. To quantify evaluation variability, we evaluate each trained policy in 5 rounds of 30 fully autonomous trials (150 trials per method and task). The original Table 3 reported a single round. For this rebuttal, we evaluated the same trained policies for four additional rounds.
Statistical significance. The revised table (Table 2 of the revised manuscript, formerly Table 3) is shown below.
| Method | Push Cube Success ↑ | Push Cube Intervention ↓ | Grasp Cube Success ↑ | Grasp Cube Intervention ↓ | Stack Cube Success ↑ | Stack Cube Intervention ↓ |
|---|---|---|---|---|---|---|
| Diffusion Policy | 92.0 ± 5.1 | -- | 52.0 ± 5.6 | -- | 28.7 ± 7.7 | -- |
| + Policy Decorator | 99.3 ± 1.5 | 7.9 | 67.3 ± 10.1 | 15.2 | 45.3 ± 6.9 | 21.9 |
| + Vanilla DAWN | 100.0 ± 0.0 | 6.4 | 68.7 ± 11.2 | 11.3 | 52.0 ± 5.6 | 20.3 |
| + ReVRA (Ours) | 100.0 ± 0.0 | 5.3 | 83.3 ± 5.3* | 8.5 | 68.7 ± 5.1* | 17.2 |
| -value | -- | -- | 0.040* | -- | 0.001* | -- |
Success rates are fully autonomous: mean ± std over 5 evaluation rounds of 30 trials each. Intervention rates are computed over the 5k-step online training budget as %. -values are from two-sided Welch's t-tests between ReVRA and Vanilla DAWN, and * denotes .
On the two harder tasks, ReVRA improves over the strongest baseline (Vanilla DAWN) by 14.6 percentage points on Grasp Cube and 16.7 on Stack Cube, and both improvements are statistically significant (). On Push Cube, all residual methods saturate near 100% success, and ReVRA's advantage lies in requiring the fewest interventions. ReVRA also achieves the lowest intervention rate on all three tasks.
(2) LIBERO results.
Seeds and evaluation episodes. Each LIBERO suite (Spatial, Object, Goal, Long) contains 10 tasks. The original Table 2 was obtained from a single training seed and evaluated with 10 episodes per task (100 episodes per suite) from LIBERO's fixed initial states. For this rebuttal, we trained two additional seeds (three in total) and increased the evaluation to 20 episodes per task (200 episodes per suite and seed). The revised table (Table 1 of the revised manuscript, formerly Table 2) is shown below.
| Method | Spatial | Object | Goal | Long | Avg. |
|---|---|---|---|---|---|
| SmolVLA (0.45B) | 84.8 ± 3.5 | 94.0 ± 0.0 | 83.7 ± 1.4 | 51.0 ± 5.3 | 78.4 ± 1.0 |
| + Policy Decorator | 85.0 ± 3.3 | 94.8 ± 0.3 | 83.3 ± 2.3 | 51.8 ± 2.0 | 78.8 ± 1.9 |
| + Vanilla DAWN | 84.7 ± 2.8 | 93.0 ± 2.2 | 83.8 ± 4.3 | 52.8 ± 2.8 | 78.6 ± 0.6 |
| + DAWN | 85.5 ± 2.6 | 94.5 ± 1.0 | 85.5 ± 0.9 | 54.8 ± 2.0 | 80.1 ± 1.6 |
| + DAWN + ReVRA (Ours) | 88.5 ± 0.9 | 96.0 ± 1.0 | 86.7 ± 0.6 | 57.8 ± 2.5 | 82.3 ± 0.5 |
Success rates (%): mean ± std over three training seeds, each evaluated on 200 episodes per suite. For the frozen SmolVLA base policy, the std is over three evaluation seeds.
Training details. The residual policy is trained per suite: for each suite, a single residual actor and twin critic are trained jointly on all 10 tasks on top of the same frozen SmolVLA checkpoint, and the critic additionally takes a one-hot encoding of the task ID as input to accommodate multi-task training. We have added this description to Section 5.3.
[1] J. Luo et al. Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. Science Robotics, 2025.
[2] Y. Chen et al. ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy. RSS, 2025.
Response to Reviewer gNoh (1/4): W1
We thank the reviewer for raising this point. We agree that being well-defined on arbitrary pairs does not by itself justify cross-state supervision, and the revised manuscript no longer uses this property as a justification (L314–320). Our answer is that policy improvement is local in the state, but policy evaluation, which is what the critic loss performs, is not: the same-state comparisons used by the actor are determined, through the Bellman equation, by value differences between different successor states. We make this precise in (1)–(3) and summarize the empirical evidence in (4).
(1) Same-state action comparisons are cross-state value comparisons one step later. Setting si=sj=s in the fixed-point equation of Eq. 10 gives
Qπ(s,a)−Qπ(s,a~)=r(s,a)−r(s,a~)+γE[Qπ(s′,a′)−Qπ(s~′,a~′)],
where s′∼P(⋅∣s,a) and s~′∼P(⋅∣s,a~). Two actions at the same state lead to different successor states, so the same-state comparison needed for policy improvement is determined by a cross-state difference at the next step. Same-state pairs are therefore not closed under the relative Bellman operator: a Bellman relation restricted to them is not self-contained, and the contraction and unique fixed point of Theorem 4.3 require the full pairwise space. Under sparse rewards, as in our real-robot tasks, the reward difference vanishes almost everywhere, and the advantage of one correction over another is entirely a cross-state difference between successors. For the same reason, state-dependent errors at successor inputs, which do not enter the actor gradient directly, become action-dependent errors at their predecessors through bootstrapping.
(2) Consistency on all pairs implies correctness at every state. Remark 4.4 shows that if ΔQ=Δπ on all pairs, then Q=Qπ+c, hence ∇aQ(s,a)=∇aQπ(s,a) at every state. Moreover, Δθ is induced by the single scalar critic (Eq. 9) rather than a separate pairwise network, so cross-state constraints act on the same function that the actor differentiates in Eq. 4.
(3) TD learning has no same-state supervision either. In continuous control with off-policy replay, a state is observed with essentially one action, so ∇aQθ in SAC is always obtained through generalization of a critic fitted across different states. ReVRA keeps this mechanism and the same data, and changes only which error is penalized: the relative term penalizes the error difference directly and is unaffected by errors shared by both predictions (L213–215). We do not claim that cross-state pairs replace counterfactual same-state samples, and we state the distinction from the single-state action gap explicitly (L237–238).
(4) Empirical support. Adding the relative term to the same critic, data, and actor update improves learning over absolute-only TD (Figure 4, and Figure 7 in Appendix D.1). The gain is also not reproduced by controls that only reweight TD residuals: on PegInsertionSide, ReVRA approaches 100% success, whereas Scaled TD reaches about 25% and Centered TD stays below 10% (Figure 10 in Appendix D.4).