acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Spectral Bulk: Spike Acceleration for Critic Learning in Deep Reinforcement Learning

Abstract

Value-function approximation can destabilize deep reinforcement learning because critic errors affect both bootstrapped targets and subsequent policy updates.Existing methods mainly stabilize learning targets, reduce estimation error, or constrain critic parameters, but do not directly promote weak task-related directionsthat remain unresolved within the spectral bulk. We propose Spike Spectral Acceleration Regularization (SSAR), which selectively amplifies critic singular modesnear the estimated upper bulk edge and removes the additional pressure after sufficient spectral separation.SSAR is motivated by the Baik–Ben Arous–P´ ech ´e (BBP) transition, under which an emerging low-rank signal becomes increasingly aligned with an empirical singular mode as it approaches and crosses the separation threshold. For a fixedtarget critic update, we decompose the SSAR-induced Bellman-loss change intoa task-aligned descent term and a remainder, and characterize their scaling acrossthe BBP transition. Under a rank-one spiked model, the two terms have the sameasymptotic order in the fixed-subcritical regime, providing no sign guarantee. Atnon-degenerate criticality and in the fixed-supercritical regime, the aligned termdominates, making the additional SSAR step loss-decreasing with probabilitytending to one.This analysis shows that spectral proximity determines where SSAR acts, whilethe emergence of task alignment determines when the additional update becomesbeneficial. Controlled experiments verify the predicted finite-width scaling andloss-sign transition. We further evaluate SSAR with Soft Actor-Critic on sixcontinuous-control tasks from the DeepMind Control Suite against vanilla SACand matched L2 critic regularization.

Primary area:
reinforcement learning
Keywords:
Deep Reinforcement Learning, Spectral Regularization, Random Matrix Theory, BBP Transition

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_epRqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Spike Spectral Acceleration Regularization (SSAR), a regularizer for deep RL critics based on random matrix theory. The core idea is to boost singular values of the critic’s penultimate layer that sit just above the edge of the random spectral bulk to help task-relevant directions break out sooner. It switches off once enough modes have separated. Using the BBP phase transition, the authors prove that this extra update has no guaranteed effect on the loss below the threshold, but reliably lowers the Bellman loss at and above it. Synthetic experiments match the predicted scaling, with 98% of updates lowering the loss in those regimes. With SAC on six DeepMind Control tasks, SSAR beats vanilla SAC and L2 regularization in sample efficiency.

Critical strengths

The idea of using Random Matrix theory as a framework to analyze and regularize the critic might be potentially useful but the results do not support the motivation and conclusions.

Critical weaknesses
  1. The paper is difficult to read. Details are presented before the big picture which makes it difficult to understand the methodology.
  2. Results are reported on only 6 simple control tasks with no clear benefit, even over limited baselines. See for example: https://arxiv.org/abs/2602.19373
  3. Comparisons should be provided to many other methods addressing regularization in DRL, and highlighting the core advantages.
  4. Experiments only used 3 seeds resulting in very large standard deviations and no clear statistical significance.
Questions for authors

I don't have questions to the authors. I think the paper needs to be rewritten with less LLM use, and more human language and style. Results should be significant and demonstrate the benefits of the approach. At the moment, there is no clear advantage in the method presented.

Reviewer_pTuUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces Spike Spectral Acceleration Regularization (SSAR), which is essentially a loss regularization term for the second last layer of the critic in actor-critic RL. The idea of the method comes from random matrix theory. As neural networks start training, the singular values of the weight matrices seem to be like random noise and stay below some known limit. As the critic starts learning, a few of the singular values grow larger than this limit. SSAR tries to speed this up by computing every 20 critic updates the singular values of the critic’s second last layer and then pushing up, with an additional loss term, those that are close to the limit. This limit is based on the layer’s initialization variance. Once a set number of the singular values are large enough, the loss is turned off. The authors apply this to toy examples and 6 DMC tasks where they show they learn slightly better in 4 of them. Disclaimer: I was not able to follow most of the theory that was introduced and therefore can not assess this part of the paper.

Critical strengths

The main strength from my point of view is that method is cheap and easy to add to existing algorithms. It only touches one layer of the critic and needs a singular value decomposition only every set number of updates. On Cheetah-run, Hopper-hop and Quadruped-run it gets noticeably higher final returns than both baselines.

Critical weaknesses

The spectrum plot (Fig. 3) doesn’t really fit well the story that the authors are telling. At the end of training, all methods have about 300–460 of 1024 singular values above the limit, and the whole spectrum seems like to keep on growing and does not like it converged even yet. SSAR also seems to switch off after only about 15–20% of training. On Hopper-hop, that is before the agent even gets any return. So its hard for me to understand how this minor change at the start of the learning phase has such a drastic impact, especially while some environments are not learning by then at all.

The experiments are too narrow in my opinion. There are many other methods for normalization or regularization cited but the paper does not compare to any of these. Especially simple ones that everyone in the field is using like LayerNorm. Also some hyperparameters like the window width or the L2 strength of the baseline are never mentioned. Also the paper lacks all kinds of ablations for the methods and its hyperparameters. I think this would be a lot stronger if these things would be added, and also if the author could show that this also works well for other base methods like TD3, or even newer methods like Simba, XQC or FastSAC.

My absolute main weakness is that the writing of the theory is extremely hard to follow and hard to understand. Many terms are not properly introduced. I think taking a slower pace on the introduction of the theory could help a lot to better understand the motivations of this work and how and why thinks work.

Questions for authors

What is the window width and how was it chosen? How was N_c chosen and how should it be chosen for different layer size? How does it compare to adding LayerNorm to the critic? Are the results orthogonal? What was the L2 strength of the baseline? Do the improvement also hold for other base RL algorithms?

Reviewer_km41OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Spike Spectral Acceleration Regularization (SSAR), a method to improve critic learning in deep reinforcement learning, by selectively amplifying singular values that sit near the upper edge of the spectral bulk. The idea draws on random matrix theory, in particular the BBP phase transition, which describes when a low-rank signal becomes distinguishable from random spectral noise. The authors propose a regularizer that encourages near-edge singular modes to grow and disables the additional regularization once enough modes have separated from the bulk. The paper provides a theoretical analysis of the extra update's effect on the Bellman loss under a spiked random-matrix model. The results suggest that the update becomes more and more likely to decrease the loss as the underlying task signal aligns with the selected singular direction. The authors validate these predictions in controlled synthetic experiments and evaluate SSAR with SAC on six continuous-control tasks from DeepMind control suite, reporting improvements over vanilla SAC and L2 regularization.

Critical strengths
  • The idea is interesting and well-motivated. The paper proposes an interesting connection between random matrix theory and critic optimization in deep RL. The core idea of selectively speeding up emerging spectral modes is simple and potentially useful.
  • Promising results. SSAR improves sample efficiency on several continuous environments, with especially strong gains on Hopper-hop and Quadruped-run. The experiments uses five seeds and compare against both vanilla SAC and a matched L2 baseline.
  • Solid theoretical analysis within the modeling assumptions. The authors provide a theoretical characterization of the additional spectral update across different BBP regimes. The controlled synthetic experiments also support the predicted scaling behavior and the transition toward loss decreasing updates. Also the theoretical analysis is particularly interesting because it distinguishes regimes where spectral amplification may or may not be beneficial, rather than assuming a universally positive effect. The controlled experiments provide useful evidence for these predictions.
Critical weaknesses
  • The connexion between the theoretical analysis and actual critic learning is not fully established. The analysis relies on a spiked random-matrix model and on assumptions about how task-relevant the underlying signal is. Assumption 4 in particular requires the population signal direction to benefit the Bellman objective. That’s reasonable in the synthetic setting, but it’s unclear whether it holds for trained neural-network critics. The method picks singular directions based on how close they are to the estimated spectral edge, without checking whether those directions actually matter for the task. The spectral edge is also computed from the initialization variance, even though the critic’s weight distribution changes over training, so it’s not clear that this fixed reference still reflects the bulk later on. Overall, the synthetic experiments confirm the theory under controlled conditions, but the same mechanism isn’t directly shown to be at work in the full RL experiments.
  • The experiments don’t fully isolate what the spectral selection contributes. The results are encouraging, but the comparisons are limited to vanilla SAC and L2 regularization. It would help to include simpler spectral baselines, such as amplifying the top singular values without the BBP-inspired window. Likewise, the stopping rule that turns off the regularizer once enough modes have separated isn’t ablated, so its importance is hard to judge. The authors do show that SSAR speeds up spectral separation, but Walker-run illustrates that this doesn’t always lead to better performance. As a result, it’s still unclear how much of the improvement comes from the proposed selection mechanism specifically, as opposed to spectral amplification more generally.
  • Another concern is that the spectral criterion depends on the network parameterization. Since the critic uses ReLU activations, the selected weight matrix can be rescaled while compensating in the following layer, leaving the represented Q-function unchanged but modifying its singular values. Thus, the spectral window and stopping criterion are not invariant to functionally equivalent parameterizations. It would be interesting to understand how this affects the interpretation of the selected modes as emerging task-relevant structure.
  • I think overall the writing is pretty dense, hard to read and would benefit from quite some rewriting.
Questions for authors

1 - The theory relies on a spiked random-matrix model and assumes the underlying task direction helps the Bellman objective. How well do these assumptions hold for trained critics? Can you show that the modes SSAR selects are actually task-relevant?

2 - How much of the improvement comes from the BBP-inspired window itself? It would help to see comparisons with simpler variants, such as amplifying the top singular values without an edge window, or dropping the stopping rule.

3 - The spectral edge is computed from the initialization variance. How well does this fixed reference track the bulk as the critic changes during training, and would estimating the edge from the current weights change the results?

4 - The regularizer uses a hard spectral window, which creates a discontinuity at its lower boundary. Does this affect optimization stability, and would a smoother window behave similarly?

5 - SSAR is motivated as a way to speed up weak task-relevant directions hidden inside the bulk, yet the regularizer only acts on singular values at or above the upper edge. Why not extend the window slightly below the edge? Could you explain this choice, and whether such a change would help emerging directions separate earlier?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.