- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes BRM-SAC, which augments the standard SAC critic loss with a Bellman residual objective. The method differentiates through both sides of the Bellman error and adaptively adjusts the residual loss coefficient based on critic Q-values. Experiments on DeepMind Control Suite show that BRM-SAC stabilizes training at a replay ratio of 32 and achieves competitive performance on several challenging humanoid and dog tasks.
Critical strengths
- Interesting direction for improving critic optimization: Revisiting Bellman residual minimization in modern deep RL is worthwhile. Modifying the critic objective, rather than relying on architectural changes, could be a useful contribution.
- Promising results at high replay ratios: The experiments show that BRM-SAC suppresses Q-value and gradient norm divergence, and the ablations support the importance of both the residual objective and its adaptive coefficient.
Critical weaknesses
- Missing comparisons with modern architectures: I disagree with the authors' justification for excluding SimBa and its variants in Section 5.1. While parameter scaling is indeed a major motivation behind SimBaV2, these methods also explicitly address optimization stability and high replay ratios. In particular, SimBaV2, XQC, and FlashSAC all emphasize controlling gradient or feature norms as an important factor in stable training. Since BRM-SAC also aims to stabilize critic gradients at high replay ratios, these are highly relevant comparisons, regardless of whether their primary contributions involve architectural scaling.
- Unclear benefit beyond existing stabilization techniques: The current experiments mainly establish that BRM improves standard SAC. It remains unclear whether the residual objective adds value when combined with modern architectures that already stabilize critic optimization. This is particularly important because the paper presents BRM as complementary to these techniques, but does not experimentally validate this claim.
Questions for authors
Can the authors evaluate BRM with SimBaV2, XQC, or FlashSAC, particularly at high replay ratios? I would be interested in whether the residual objective still improves performance when gradient norms are already stabilized through architectural or normalization techniques.
I find the idea of revisiting the Bellman residual objective interesting, and the current results are promising. However, I do not find the argument for excluding modern architectures convincing. The relevant question is not simply whether BRM differs from parameter scaling, but whether it provides additional benefits over existing approaches to stabilizing high-replay-ratio training.
I currently lean toward weak rejection. I would be willing to raise my score to acceptance if the authors demonstrate that BRM remains effective when combined with modern architectures such as SimBaV2, XQC, or FlashSAC.