acceptodds
Under review as a conference paper at ICLR 2027

The Spectral Lazy Regime: Spectral Norms and Generalization in Continual Learning

Abstract

Loss of plasticity, the progressive degradation of a neural network's ability to learn, has been observed in several continual learning settings. This phenomenon can be split into two related ideas: loss of trainability, when a network loses its ability to reduce the training loss, and loss of generalization, when a network can still be trained but its generalization abilities are impaired. We focus on generalization and propose a mechanism for when this can arise: entering the spectral lazy regime. This learning regime is signaled by large increases in the spectral norm of the parameter matrices. When this occurs, the learning dynamics more closely resemble training in the lazy (kernel) regime—representation learning is inhibited, leading to poor generalization. We support this finding by establishing a novel theoretical connection between large spectral norms and the lazy regime. We empirically validate that training a network with constraints on the spectral norm of its weight matrices avoids loss of generalization. Through careful investigations into other norm constraints, we conclude that other choices can be useful but only the spectral norm leads to the radius sensitivity curve being invariant to the width of the matrix. Furthermore, to enable finer control over the spectral norm, we propose two new parameterization of weight matrices, the singular value parameterization and the spectral norm parameterization. We show that these approaches naturally reduce the spectral norm growth compared to standard parameter matrices and can mitigate loss of generalization.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, plasticity, lazy regime, representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_whmdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies declining test performance during continual learning despite preserved training fit. It proposes that growing spectral norms reduce useful representation learning. Under specified scaling and gradient-flow assumptions, Theorem 1 establishes a lazy regime driven by dominant singular values; Theorem 2 bounds spectral norm after an update. The interventions constrain spectral norms or explicitly parameterize norms and singular values through SNP and SVP. Experiments on sequential random regression and SplitTinyImageNet show improved generalization stability, with comparisons across widths, norm constraints, optimizers, and learning rates. The results support spectral control as a practical intervention; its proposed causal mechanism remains conditional.

Critical strengths

The paper clearly distinguishes declining generalization from failure to fit new training data, giving the study a useful and coherent focus. Its theoretical analysis connects this problem to spectral geometry: under stated conditions, a dominant singular scale can induce lazy behavior without uniformly scaling the spectrum. SNP and SVP then translate spectral quantities into parameters that can be directly controlled.

The empirical study is broad, spanning sequential regression and SplitTinyImageNet classification, with comparisons across norm constraints, widths, and optimization settings. The learning-rate and weight-decay analyses help assess whether the observed benefits persist beyond one configuration. This combination of theory, practical parameterizations, and comparative experiments makes spectral control worth considering even while the proposed explanation of natural training trajectories needs further evidence.

Critical weaknesses
  1. The intervention results do not yet identify the proposed mechanism along natural training trajectories. Theorem 1 establishes a conditional lazy regime, and Appendix B.4 explains deliberately induced early laziness; neither directly shows that late checkpoints lose a useful nonlinear-adaptation advantage. The existing learning-rate and optimizer controls support the practical intervention, but do not measure this missing link.

  2. Several interpretations exceed conditions already stated in the paper. Section 2.3 acknowledges the negative aligned term in Theorem 2, so its lower bound does not establish inevitable net growth. Section 3.2 defines SNP on a unit ball, making the scalar's interpretation as the realized norm conditional on saturation. Figure 3 and Section 4.3 also show width alignment for Frobenius constraints, despite the abstract's uniqueness claim. These concerns call for precise interpretation or supporting measurements; they do not imply that the conditional theorem or reported performance gains are incorrect.

Questions for authors
  1. At a few early and late checkpoints from one existing regression setting, compare full-network adaptation with adaptation of its local linearization, starting from identical predictions. Report training and test errors under comparable adaptation budgets for both unconstrained and spectrally controlled runs. Does the nonlinear test-performance advantage diminish in the late baseline and persist under spectral control? Equivalent existing evidence is welcome; otherwise, qualify direct causation as a proposed explanation.

  2. Please reconcile the statement that networks “inevitably fall into this trap” at the beginning of Section 2 (p. 3, line 130) with the negative aligned term acknowledged in Section 2.3. If no additional condition establishes net spectral-norm growth, distinguish upward pressure and empirically observed growth from a general guarantee of entering the lazy regime.

  3. For a representative SNP run, report whether ∥W~∥2\lVert\widetilde W\rVert_2 stays near one during the period used to interpret norm growth. Section 3.2 already specifies the unit-ball constraint. If it is not saturated, distinguish changes in the bound ∣s∣|s| from changes in the realized norm ∣s∣∥W~∥2|s|\lVert\widetilde W\rVert_2 and qualify the statement that growth can occur only through the scalar.

  4. Please reconcile the abstract's statement that only spectral-norm constraints give width-invariant radius sensitivity with the Frobenius alignment reported in Figure 3 and Section 4.3. Appendix B already discusses geometric advantages of spectral control; revise the abstract to state the advantage supported by those comparisons without implying that width alignment is exclusive to spectral constraints.

Reviewer_9XYvOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies loss of plasticity in continual learning, with a specific focus on generalization performance. The authors attribute the loss of generalization to growing spectral norms of the weight matrices, and argue this pushes the network into the lazy (kernel) regime. They provide a lazy training result for MLPs with one (large) singular value per layer, and provide a bound that suggests that gradient updates increase the spectral norm. The authors then support this experimentally by comparing norm-ball constraints (spectral, frobenius, oblique, maxnorm) in a synthetic regression task and SplitTinyImageNet. They propose two ways of reparameterizing the weights: SNP (spectral norm parameterization) writes weight matrices a scalar times a matrix which is constrained to have spectral norm at most 1, and SVP (singular value parameterization) writes weight matrices as A diag(s) B, where A and B are basically orthogonal matrices (Stiefel manifold) and s is a learnable vector representing the singular values. The paper shows that both learn to have lower spectral normals than regular weight matrices and are less sensitive to choice of constraint radius (and constraining the SVs in SVP to have mean 1 and variance 1 further improves results). Experiments mainly use the Muon optimizer, with Adam results in the appendix.

Critical strengths
  • Authors provide a controlled demonstration of the phenomenon (loss of generalization). On both their tasks, the network keeps reaching low training error on the task while test performance starts dropping, and the spectral norm grows with the progression. The baseline that resets ever task shows the performance of what a fresh network would achieve. The observation is useful and well presented.

  • An empirical study of which norm to constrain: the width-scaling test is a simple and cheap way to compare constraints. The appendix also covers a lot: Muon vs Adam, weight decay and cautious weight decay, learning-rate transfer across widths, and four ways of bounding the spectral norm. The proof of Theorem 1 is careful and, as far as I checked, correct under its assumptions.

  • Finally, though I believe most of the arguments have already been made in some form in previous work (see weaknesses), bringing this together in a more up to date, complete version with a clear link from norm growth (or other metrics) to loss of generalization can be useful for the field.

Critical weaknesses
  1. The novelty seems overstated and crucial prior work is missing: most of what the paper presents as new seems already established.
  • Bounding the spectral norm preserves plasticity and generalization: Lewandowski et al. (ICLR 2025) already keep σ_max near 1 to preserve plasticity, and report better generalization in continual learning. The paper cites it (as the arXiv version, please correct to the ICLR version) but never compares against it. The empirical message of Section 4.2 is therefore largely known.

  • Norm growth as the cause: Lyle et al. (NeurIPS 2024) show that parameter norm growth reduces the effective learning rate and hurts plasticity, and fix it with weight projection (NaP). In normalized networks (such as the LayerNorm ResNet used here), this is a simpler explanation of the same observations. The paper neither cites this crucial work, nor rules it out for those architectures. Related to this, Lyle et al. (CoLLAs 2024) already list norm growth as a cause of plasticity loss and use the empirical NTK as a diagnostic.

  • Splitting plasticity loss into trainability and generalization: this split has been studied before, e.g. by Lee et al. (NeurIPS 2023) in RL, and by Xiao, Pennington & Schoenholz (ICML 2020), who use the NTK spectrum to separate trainability from generalization and show regimes with large weights where networks memorize but don't generalize. This should be discussed.

  • Large weights --> lazy training --> poor generalization: Yang, Simon & Bernstein (2023): feature learning requires the spectral norms of weights and updates to scale together. If weights grow while updates stay fixed, learning becomes lazy. This is the paper’s core idea in general form, and it already predicts the width-invariance used in Section 4.3. Chizat et al. (NeurIPS 2019) (the authors cite this): their general laziness criterion largely predicts Theorem 1 once the derivative scalings are computed. Kumar et al. (ICLR 2024) and Liu et al. (ICLR 2023) link large weight norm to lazy training and delayed generalization, and Liu et al. show that constraining the norm mostly removes the delay. None of these papers states “spectral norm growth --> lazy regime --> loss of plasticity” in one line. However, I believe every step already exists, and this is not properly addressed in the paper.

  • “No mechanism has been identified” (Appendix A.1): warm-starting (Ash & Adams, NeurIPS 2020, cited) is the standard case of same training error and worse test error. It is not tested.

  • Based on previous works, I believe it is possible that the whole spectrum matters, not just σ_max: Jia et al. (CVPR 2017; "Orthogonal Deep Neural Networks", arXiv:1905.05929) prove a generalization bound that depends on the whole spectrum and is tightest when singular values are equal (i.e., a scaled isometry or Parseval frame). Pennington et al. (NeurIPS 2017) show the full distribution of Jacobian singular values matters for learning. The spectral-norm generalization bounds of Bartlett et al. (NeurIPS 2017) and Neyshabur et al. (ICLR 2018) are not cited. Rosseau et al. (ICML 2026) link plasticity to the NTK and dynamical isometry, and argue spectral-norm control is only a partial measure. Although this line of work is mostly about trainability, it suggests the whole spectrum, not only σ_max, matters, which challenges the paper's focus on σ_max. The paper’s own Table 2 actually seems to point in the same direction, so I believe this is not properly addressed.

  • The parameterizations are not new: SNP (learned scalar times a spectrally normalized matrix) is σReparam (Zhai et al., ICML 2023). SVP is the SVD parameterization with Stiefel factors of Vorontsov et al. (ICML 2017) and Zhang et al. (ICML 2018). Clipping singular values is Singular Value Bounding (Jia et al., CVPR 2017). None of these are cited, and the techiques are presented as novel.

What seems clearly novel to me is the rank-k version of Chizat’s lazy-training argument and the width-scaling test. However, I believe (at this point) these to be modest contributions for ICLR.

  1. The main claim, that the spectral lazy regime causes the loss of generalization, does not seem well-supported:
  • Laziness seems to be never measured (no NTK change across tasks, e.g. the kernel distance of Fort et al., NeurIPS 2020, and no change in features or relative weights)? The evidence is a correlation between growing spectral norms and dropping test performance. For the LayerNorm ResNet, a decaying effective learning rate (Lyle et al., NeurIPS 2024) is a competing explanation that is not ruled out, e.g. by scaling the learning rate with ||W||_2 or using NaP.

  • The theory does not always seem to match the experiments. Theorem 1 is for gradient flow slowed by eta^{2(L-1)}, but the experiments use Muon, Lion and Adam at fixed learning rates. The centering hides outputs of order eta^L, and spectral norms only grow about 3x (Figure 2).

  • With one large SV per layer, the leading part of the kernel seems to have very low rank and cannot fit the training data. This contradicts "trainability is preserved", and Figures 10-11 indeed show large spikes losing trainability.

  • I am not entirely certain about this, but I believe nothing in the argument actually needs continual learning? Appendix B.4 shows inflating singular values in a fresh network already hurts, so this looks like the known large weights --> lazy --> poor generalization effect? But I could be missing something, and I do believe the continual learning setting is a natural one where these pathologies can easily occur, so this is a minor point.

  1. The paper presents SNP and SVP as successful ways to mitigate loss of generalization, but apart from generic norm-ball versions such as max norm (which equals weight clipping) and weight decay, never compares them with existing plasticity methods, scuch as spectral regularization, Parseval regularization, L2-init, shrink-and-perturb, ReDo, NaP and others (I do not believe an exhaustive comparison of all methods is required, but a few main ones would significantly strengthen the empirical claims). It also evaluates on only two benchmarks, one of them synthetic, with small models. So it is unclear whether SNP and SVP improve on existing methods. It is also unclear whether methods that don’t target the spectral norm work just as well, which would weaken the proposed mechanism.

Minor remarks:

  • The abstract claims only the spectral norm gives width-invariant sensitivity curves, but in Figure 3 the Frobenius-ball curves are also width-invariant?

  • Theorem 2 is a one-line bound and not specific to the spectral norm. Under zero-mean noise, any norm grows in expectation (Jensen’s inequality), and the Frobenius norm grows by exactly E‖G‖²_F. The bound also does not show a tendency, since β₁ can be negative. It does not justify SNP over other parameterizations.

  • The O(c^{−L}) rate in the introduction requires Assumption 3. Without it, the rate is O(η^{−1}).

  • Assumption 3 uses z̃, which the main text never defines. It is very strong (every pre-activation for every example bounded away from zero) and cannot be checked on trained networks.

  • Assumption 1 is ill-posed at 0. σ − z₊ is not twice differentiable there, and boundedness of ξ is used but never stated.

  • Lemma 3(ii) mixes operator and Frobenius norms. This is harmless up to constants.

  • Spectral cap is called “the only computationally efficient” way to project onto the spectral-norm ball, without support.


Overall, the overlap with prior work, the lack of direct evidence for the proposed mechanism, and the missing comparisons with existing plasticity methods are, in my view, too fundamental for the paper in its current form, which is why at this point I recommend rejection.

If the authors can show that laziness actually occurs (e.g. NTK change across tasks), rule out effective learning rate decay for the normalized architecture, compare against (a subset of) existing plasticity methods in a wider range of benchmarks (continual pixel permutation in MNIST or CIFAR-10, for example), which then either support the claim that the spectral norm specifically is the right quantity (e.g. with a non-isometric teacher, and explaining Figure 3, and comparing to other hypotheses like a full uniform spectrum) or tone the claims down according to the new results if they don't support the original claims, I am open to updating my score. (Adding citations alone would not be sufficient: the novelty claims, in particular for SNP and SVP, would need to be repositioned relative to prior work.)

Questions for authors

(Most questions restate comments from the weaknesses section.)

  • How does the NTK distance from initialization evolve across tasks, compared with resets?

  • For the LayerNorm ResNet, does the loss of generalization disappear if the learning rate scales with ‖W‖₂, or with NaP?

  • In Table 2, normalizing the SVs to mean 1 and variance 1 works best, but only this one setting is tested. What happens as the variance goes to 0 (i.e. an orthogonal matrix, full isometry), which would show whether moving toward isometry is what helps?

  • Does a fresh network rescaled to the spectrum of the final task generalize as badly as the trained one?

  • What are the effective rank and the distance from isometry of the weights, across widths and constraints? Effective rank is never mentioned as a metric, I believe, though it is crucial in most analysis of related work.

  • Do the conclusions hold with a teacher network that is not isometric? Otherwise this benchmark might be too narrow to trust the results. (The synthetic teacher is built from orthogonal layers and MaxMin activations, i.e. it is exactly an isometric network with spectral norm 1 per layer, so a spectral constraint near 1 is expected to work best, right?)

  • How do the methods compare with spectral regularization, Parseval regularization, L2-init, shrink-and-perturb and NaP, each tuned, on a wider range of standard plasticity benchmarks?

  • Table 2 has no seeds, error bars or bound values, and only one variance setting is tested for the normalization.

  • How do you reconcile "trainability is preserved" with the training error at large spike scales in Figure 10?

Writing:

In the introduction: "Although the network successfully minimizes test error within any given task, its overall generalization drastically deteriorates across the sequence of tasks." This is confusing. I believe you mean test error decreases within each task but the level reached gets worse across tasks. Please rephrase.

Dohare et al. is cited as the 2021 arXiv, not Nature 2024.

Lewandowski et al. is cited as arXiv, not ICLR 2025.

1 comment

I appreciate the substantive response from the authors, and feel that many of my concerns have been addressed.

While I believe the paper is in much better shape now, I would still have liked to see a more thorough comparison between constraining the full singular value spectrum to be close to 1 and constraining only the spectral norm, particularly in terms of generalization and continual learning. I believe there is an increasing body of work suggesting that the former may provide a more complete picture of generalization than spectral norm constraints alone (see, for example, Rosseau et al., 2026; Jia et al., 2019).

That being said, I have updated my score to 3 (weak accept), as I believe many of the analyses and results presented in the paper are useful.

Reviewer_Ktw2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the loss of trainability and loss of generalization in continual learning of neural network models. It aims to figure out when the generalization becomes worse but the training loss is still controlled. The paper's answer is the spectral lazy regime where the training loss is decaying but the neural network remains in lazy/NTK regime without feature learning. This analysis is done in MLP setting and based on the control of the norm of the change of the weight matrices in a finite time horizon. The authors then empirically show that we should bound spectral norms to prevent loss of generalization. Based on this, the paper introduces spectral norm parameterization (SNP) and singular value parameterization (SVP) to reparameterize the weights to get better generalization.

Critical strengths

The experiment results are interesting and inspiring. Especially the radius sensitivity and rescaling experiments.

Critical weaknesses

The novelty of the theoretical results in the paper is weak. Theorem 1 only control the change of the weight in a finite time horizon but never proves the lazy regime training leads to bad generalization performances. Theorem 2 only shows a lower bound of the singular values but does not establish spectral growth during training. The claim that the spectral norm is growing during lazy training and the generalization is bad, is not fully proved in the MLP model.

Questions for authors
  1. β\beta should be βi\beta_i in Theorem 2.

  2. What is z~\tilde{z} in assumption 3?

  3. Theorem 3 only considers finite time horizon. How about the case when TT is also increasing and scaled with the η\eta?

  4. In Figures 3 and 5, you showed the spectral norm sensitivity but in general how do we find the optimal spectral norm for the best generalization.

  5. In line 1128, add cite?

  6. There are several related works maybe should be discussed in the paper. [1] also used parameterization of weight matrices via their SVD which is similar to SVP in your setting. [2] also the norms of the weight matrices and their updates in a lazy training setting. And [3] also controlled the scaling of the norms of the weight matrices and their updates in a feature learning setting.


[1] Zhang et al. 2018. Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization

[2] Wang et al. 2022. Spectral Evolution and Invariance in Linear-width Neural Networks

[3] Yang et al. 2024. A Spectral Condition for Feature Learning

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.