- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies continual adaptation in SAC from the alignment between Bellman correction demand and the critic's local learning spectrum. It defines a demand-weighted spectral burden based on the eigenspectrum of an empirical Jacobian kernel, derives its closed-form expression under a fixed-target affine model, and proposes Periodic Spectral Projection (PSP), which clips critic weight singular values to a fixed interval at task boundaries and periodically during subsequent-task training. Experiments on Meta-World and CompoSuite show improved subsequent-task acquisition and reduced spectral burden.
Critical strengths
- The demand-response formulation provides an intuitive and potentially useful perspective on critic plasticity.
- The paper includes several useful ablations on clipping direction, projection scope, spectral interval, and projection frequency.
Critical weaknesses
-
Insufficient methodological novelty. The proposed PSP is highly similar to the recently introduced SingularClip, which periodically clips singular values of weight matrices to preserve plasticity in continual and reinforcement learning. The paper cites this work but does not perform a direct comparison. The authors need to clearly distinguish PSP from SingularClip and establish whether the proposed method provides additional benefit beyond applying SingularClip to SAC critics.
-
Theory does not model the actual SAC optimization. Proposition 4.1 assumes a fixed Jacobian, fixed Bellman target, and full-batch gradient descent, whereas the actual algorithm uses Adam, minibatches, changing target critics, a changing actor, and a changing replay distribution. Thus, the proposed learning kernel dynamics are not the dynamics followed by the actual training algorithm. A quantitative test comparing the theoretically predicted residual decay with the empirical SAC dynamics would substantially strengthen the paper.
-
Limited continual-RL setting. The replay buffer is cleared at task transitions and Meta-World uses task-specific policy heads with known task boundaries. Furthermore, evaluation focuses primarily on subsequent-task acquisition rather than retention of previously learned tasks. This makes the setting closer to continual adaptation/forward plasticity than the broader continual RL problem.
-
Missing related work. The paper lacks literature on spectral analysis of some recent research on continual reinforcement learning. For example, the paper should also discuss [1], which studies spectral plasticity in continual reinforcement learning using NTK-based analysis and spectral regularization on Meta-World.
[1] Luo, Lirui, et al. "SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learning." ICML (2026).
Questions for authors
Can the authors provide a detailed distinction between the method used in this paper and in the references?