acceptodds
Withdrawn from ICLR 2027

Bellman residual minimization for sample-efficient deep reinforcement learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent work on improving the sample efficiency of deep reinforcement learning has largely focused on network architecture design, normalization schemes, and optimizer choices. The loss function itself, however, has received comparatively little attention, even though it plays a central role in shaping training dynamics. We show that enforcing an additional Bellman consistency objective on top of the semi-gradient loss stabilizes training at high replay ratios. Our method demonstrates competitive performance with concurrent methods on the hardest DeepMind Control Suite tasks like humanoid and dog, without relying on network resets or additional normalization techniques.

Primary area:
reinforcement learning
Keywords:
sample-efficient reinforcement learning, residual algorithms

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Xxa3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes BRM-SAC, which augments the standard SAC critic loss with a Bellman residual objective. The method differentiates through both sides of the Bellman error and adaptively adjusts the residual loss coefficient based on critic Q-values. Experiments on DeepMind Control Suite show that BRM-SAC stabilizes training at a replay ratio of 32 and achieves competitive performance on several challenging humanoid and dog tasks.

Critical strengths
  • Interesting direction for improving critic optimization: Revisiting Bellman residual minimization in modern deep RL is worthwhile. Modifying the critic objective, rather than relying on architectural changes, could be a useful contribution.
  • Promising results at high replay ratios: The experiments show that BRM-SAC suppresses Q-value and gradient norm divergence, and the ablations support the importance of both the residual objective and its adaptive coefficient.
Critical weaknesses
  • Missing comparisons with modern architectures: I disagree with the authors' justification for excluding SimBa and its variants in Section 5.1. While parameter scaling is indeed a major motivation behind SimBaV2, these methods also explicitly address optimization stability and high replay ratios. In particular, SimBaV2, XQC, and FlashSAC all emphasize controlling gradient or feature norms as an important factor in stable training. Since BRM-SAC also aims to stabilize critic gradients at high replay ratios, these are highly relevant comparisons, regardless of whether their primary contributions involve architectural scaling.
  • Unclear benefit beyond existing stabilization techniques: The current experiments mainly establish that BRM improves standard SAC. It remains unclear whether the residual objective adds value when combined with modern architectures that already stabilize critic optimization. This is particularly important because the paper presents BRM as complementary to these techniques, but does not experimentally validate this claim.
Questions for authors

Can the authors evaluate BRM with SimBaV2, XQC, or FlashSAC, particularly at high replay ratios? I would be interested in whether the residual objective still improves performance when gradient norms are already stabilized through architectural or normalization techniques.

I find the idea of revisiting the Bellman residual objective interesting, and the current results are promising. However, I do not find the argument for excluding modern architectures convincing. The relevant question is not simply whether BRM differs from parameter scaling, but whether it provides additional benefits over existing approaches to stabilizing high-replay-ratio training.

I currently lean toward weak rejection. I would be willing to raise my score to acceptance if the authors demonstrate that BRM remains effective when combined with modern architectures such as SimBaV2, XQC, or FlashSAC.

Reviewer_1oqtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The main drawback of the current version of this work is the lack of focus. The authors argue that "we should revisit the critic update", but the question that should be answered in the introduction is "why do we need that?" and how the proposed method addresses it. The paragraph that follows this question establishes loose connections to the hypothesis raised. Instead of listing examples, the Related Work starts again by loosely connecting works on sample efficiency to the current work. Not even the baselines chosen in the experiment setup seem to address the same sort of problem.

Critical strengths

None

Critical weaknesses

The paper needs a major revision.

Questions for authors

Why does a high replay rate matter to the setting you are addressing?

Reviewer_JXayOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper revisits the true Bellman equation gradient in deep reinforcement learning and proposes optimizing it as a form of auxillary loss to improve critic 'consistency'. They do this by assuming deterministic dynamics, which only requires one transition 'sample' to be recorded. They consider the high UTD setting and look at DMCS tasks, building on SAC and comparing the improved SAC baselines as well as model-base methods.

Critical strengths

S1: Sound idea that is straightforward to implement. Target networks slow down deep RL and so revisiting the critic loss gradient is sensible. The auxiliary loss approach keeps the standard algorithm intact, so it's easy to add to any DDPG-based method.

Critical weaknesses

W1: Cherry-picked results. In Appendix A, BRM-SAC performs worst on acrobot, dog-run, fish-swin, quadruped-run, quadruped-walk and walker-run. This would be picked up in an aggregate plot across all DMCS tasks (like other papers do) but the cherry-picked environments in the main paper don't reflect this. These results are not mentioned in the concluding text.

W2: Empirical performance not statistically significant. Even after cherry-picking, the performance in Figure 1 is not compelling in terms of statistical significance, can the authors argue other benefits like wall-clock time, FLOPs or memory requirements from the baseline?

W3: Cited baselines not in empirical results. The paper cites more recent SOTA methods like SimBa-v2 and XQC but does not compare to them empirically.

W4. Scholarship on off-policy deep RL. The BRM idea applies to any DDPG-based method, such as TD3, SAC and so on, so focusing so much on BRM-SAC which is really just the practical implementation for the experiments, limits the scope of the paper. However, DDPG and TD3 aren't even discussed or cited.


Minor comments.

  • Figure 1 caption needs more details, i.e. number of seeds and definition of confidence interval
  • The paper wastes a lot of space explaining the Bellman residual idea and repeating similar equations in Sections 2 and 4. I think these sections could be rewritten to reduce repetition and generate a lot more space.
  • Re: the section 4 title, it is not good practice to use acronyms in subtitles, as it is not immediately clear to the reader what they mean unless they've read the paper
Questions for authors

See W2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.