- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies declining test performance during continual learning despite preserved training fit. It proposes that growing spectral norms reduce useful representation learning. Under specified scaling and gradient-flow assumptions, Theorem 1 establishes a lazy regime driven by dominant singular values; Theorem 2 bounds spectral norm after an update. The interventions constrain spectral norms or explicitly parameterize norms and singular values through SNP and SVP. Experiments on sequential random regression and SplitTinyImageNet show improved generalization stability, with comparisons across widths, norm constraints, optimizers, and learning rates. The results support spectral control as a practical intervention; its proposed causal mechanism remains conditional.
Critical strengths
The paper clearly distinguishes declining generalization from failure to fit new training data, giving the study a useful and coherent focus. Its theoretical analysis connects this problem to spectral geometry: under stated conditions, a dominant singular scale can induce lazy behavior without uniformly scaling the spectrum. SNP and SVP then translate spectral quantities into parameters that can be directly controlled.
The empirical study is broad, spanning sequential regression and SplitTinyImageNet classification, with comparisons across norm constraints, widths, and optimization settings. The learning-rate and weight-decay analyses help assess whether the observed benefits persist beyond one configuration. This combination of theory, practical parameterizations, and comparative experiments makes spectral control worth considering even while the proposed explanation of natural training trajectories needs further evidence.
Critical weaknesses
-
The intervention results do not yet identify the proposed mechanism along natural training trajectories. Theorem 1 establishes a conditional lazy regime, and Appendix B.4 explains deliberately induced early laziness; neither directly shows that late checkpoints lose a useful nonlinear-adaptation advantage. The existing learning-rate and optimizer controls support the practical intervention, but do not measure this missing link.
-
Several interpretations exceed conditions already stated in the paper. Section 2.3 acknowledges the negative aligned term in Theorem 2, so its lower bound does not establish inevitable net growth. Section 3.2 defines SNP on a unit ball, making the scalar's interpretation as the realized norm conditional on saturation. Figure 3 and Section 4.3 also show width alignment for Frobenius constraints, despite the abstract's uniqueness claim. These concerns call for precise interpretation or supporting measurements; they do not imply that the conditional theorem or reported performance gains are incorrect.
Questions for authors
-
At a few early and late checkpoints from one existing regression setting, compare full-network adaptation with adaptation of its local linearization, starting from identical predictions. Report training and test errors under comparable adaptation budgets for both unconstrained and spectrally controlled runs. Does the nonlinear test-performance advantage diminish in the late baseline and persist under spectral control? Equivalent existing evidence is welcome; otherwise, qualify direct causation as a proposed explanation.
-
Please reconcile the statement that networks “inevitably fall into this trap” at the beginning of Section 2 (p. 3, line 130) with the negative aligned term acknowledged in Section 2.3. If no additional condition establishes net spectral-norm growth, distinguish upward pressure and empirically observed growth from a general guarantee of entering the lazy regime.
-
For a representative SNP run, report whether stays near one during the period used to interpret norm growth. Section 3.2 already specifies the unit-ball constraint. If it is not saturated, distinguish changes in the bound from changes in the realized norm and qualify the statement that growth can occur only through the scalar.
-
Please reconcile the abstract's statement that only spectral-norm constraints give width-invariant radius sensitivity with the Frobenius alignment reported in Figure 3 and Section 4.3. Appendix B already discusses geometric advantages of spectral control; revise the abstract to state the advantage supported by those comparisons without implying that width alignment is exclusive to spectral constraints.
I appreciate the substantive response from the authors, and feel that many of my concerns have been addressed.
While I believe the paper is in much better shape now, I would still have liked to see a more thorough comparison between constraining the full singular value spectrum to be close to 1 and constraining only the spectral norm, particularly in terms of generalization and continual learning. I believe there is an increasing body of work suggesting that the former may provide a more complete picture of generalization than spectral norm constraints alone (see, for example, Rosseau et al., 2026; Jia et al., 2019).
That being said, I have updated my score to 3 (weak accept), as I believe many of the analyses and results presented in the paper are useful.