acceptodds
Withdrawn from ICLR 2027

Bellman Demand and Critic Response: A Spectral View of Continual RL

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual reinforcement learning requires agents to retain the ability to learn as tasks change over time, yet plasticity loss in neural networks can impede learning on subsequent tasks. In SAC, policy improvement relies on value estimates provided by the critic; therefore, whether the critic can learn the value function of a new task within a finite update budget is critical to continual adaptation. We study this problem through the discrepancy between the Bellman target and the current prediction, examining how the value corrections required by a new task align with the critic's learning responses across different directions. When the dominant correction demand of a new task is concentrated in directions to which the critic responds slowly, this mismatch can hinder value correction. To quantify this difficulty, we derive a demand-weighted spectral burden that combines the fraction of demand carried by each direction with its relative response speed, characterizing the relative directional difficulty of correcting the current residual. Motivated by this analysis, we adopt Periodic Spectral Projection (PSP) in continual SAC, constraining the singular values of each linear layer in both critics to a prescribed interval to modulate their learning responses at the weight level. Experiments on Meta-World and CompoSuite show improved subsequent-task acquisition with PSP. The spectral analyses characterize how the current Bellman correction demand is distributed across the critic's response directions during adaptation.

Primary area:
reinforcement learning
Keywords:
Continual Reinforcement Learning, continual learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_okpeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual adaptation in SAC from the alignment between Bellman correction demand and the critic's local learning spectrum. It defines a demand-weighted spectral burden based on the eigenspectrum of an empirical Jacobian kernel, derives its closed-form expression under a fixed-target affine model, and proposes Periodic Spectral Projection (PSP), which clips critic weight singular values to a fixed interval at task boundaries and periodically during subsequent-task training. Experiments on Meta-World and CompoSuite show improved subsequent-task acquisition and reduced spectral burden.

Critical strengths
  1. The demand-response formulation provides an intuitive and potentially useful perspective on critic plasticity.
  2. The paper includes several useful ablations on clipping direction, projection scope, spectral interval, and projection frequency.
Critical weaknesses
  1. Insufficient methodological novelty. The proposed PSP is highly similar to the recently introduced SingularClip, which periodically clips singular values of weight matrices to preserve plasticity in continual and reinforcement learning. The paper cites this work but does not perform a direct comparison. The authors need to clearly distinguish PSP from SingularClip and establish whether the proposed method provides additional benefit beyond applying SingularClip to SAC critics.

  2. Theory does not model the actual SAC optimization. Proposition 4.1 assumes a fixed Jacobian, fixed Bellman target, and full-batch gradient descent, whereas the actual algorithm uses Adam, minibatches, changing target critics, a changing actor, and a changing replay distribution. Thus, the proposed learning kernel dynamics are not the dynamics followed by the actual training algorithm. A quantitative test comparing the theoretically predicted residual decay with the empirical SAC dynamics would substantially strengthen the paper.

  3. Limited continual-RL setting. The replay buffer is cleared at task transitions and Meta-World uses task-specific policy heads with known task boundaries. Furthermore, evaluation focuses primarily on subsequent-task acquisition rather than retention of previously learned tasks. This makes the setting closer to continual adaptation/forward plasticity than the broader continual RL problem.

  4. Missing related work. The paper lacks literature on spectral analysis of some recent research on continual reinforcement learning. For example, the paper should also discuss [1], which studies spectral plasticity in continual reinforcement learning using NTK-based analysis and spectral regularization on Meta-World.

[1] Luo, Lirui, et al. "SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning." ICML (2026).

Questions for authors

Can the authors provide a detailed distinction between the method used in this paper and in the references?

Reviewer_jRnpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The manuscript discusses why reinforcement learning agents struggle to learn new tasks after training on previous ones. It argues that this can happens because the critic cannot correct its value predictions in proper learning directions. Also it introduces a spectral measure to analyse this problem and apply a projection method to restricts the singular values of the critic's weight matrices.

Critical strengths

The paper provides a mathematical analysis of critic adaptation. The experiments provide comparisons with current plasticity methods.

Critical weaknesses

I have some concers about this work:

1- I think the main issue of this work is the gap between the analysis and the proposed method. In the paper, Eq.11, identifies learning difficulty through the eigenvalues and residual directions of the critic learning filter. However, the model in Eq. 12 clips the singular values of each weight matrices, without considering the Bellman correction demand or the kernel directions identified earlier. As I understand, two critics can have similar weight singular values but very different learning kernels and correction demands. So, why should the same spectral constraint improve the specific weak response directions which identified by some analysis? Moreover, I am not convinced that the proposed spectral burden or Eq. 11 is a reliable measure of critic plasticity, because, If a model can learn successfully despite having a high spectral burden, how this distinguish between critics that have lost plasticity and those that can still adapt?

2- As I understand, Eq. 8 describe the convergence of GD under fixed Bellman targets, fixed inputs, and a fixed Jacobian. However, the model is applied during training, where these quantities change continuously. Moreover, the proposition does not analyse the effect of spectral projection or reduces the residual error compared with general training.

Questions for authors

Please see the weaknesses.

Reviewer_reunOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper explores the role of the NTK and its eigenspectrum in continual reinforcement learning. Directions of small eigenvalues are hypothesized to impair learning if the loss gradient is aligned with them. Some theory is developed to demonstrate slow convergence rates in these directions. A practical algorithm is proposed, which bounds the singular values of parameter matrices, and is shown to help in continual RL tasks.

Critical strengths

The paper is clear and the contributions are easy to follow, with a good blend of theory and experiments.

Critical weaknesses
  • Novelty is the main weakness in my view. The learning dynamics with a fixed kernel matrix are well understood from classic machine learning theory, including the rates of convergence for each eigendirection. From that perspective, those results are standard and I did not see additional theory built on it. Moreover, the proposed algorithm involving singular value clipping has already been explored in RL in recent times [1,2] and the experiments did not seem to produce novel findings.

  • The main assumption for the theoretical analysis is to consider the NTK at a fixed time, but it is known that this NTK evolves as features are learned (e.g. see [1]). So it is unclear how applicable this assumption is in practice with neural networks.

[1] "SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learning" Kastner et al. (2026) [2] "FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoff" Han et al. (2026) [3] "Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel", Fort et al. (2020)

Questions for authors

I believe the paper would need some additional, interesting insights to be published. I would consider expanding the analysis on how the NTK's spectrum may be linked to the critic's learning process. Some possible directions could include investigating the change in the NTK over time (e.g. [1] investigate the alignment of features and the targets) or how the learning process may be influenced by TD learning (e.g. [2] investigates some of the learning biases of TD). Currently, much of the theory in the paper looks to be not specific to RL. These changes may require substantial revisions to the paper though.

[1] "Representation Alignment in Neural Networks", Imani et al. (2021) [2] "Learning Dynamics and Generalization in Reinforcement Learning", Lyle et al. (2022)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.