acceptodds
Under review as a conference paper at ICLR 2027

Optimal Explicit and Implicit Regularization Strengths in High-Dimensional Continual Linear Regression

Abstract

We study continual linear regression in a high-dimensional regime, where a learner observes a sequence of tasks and updates its parameters sequentially. In this setting, we derive an explicit closed-form expression for the generalization loss of a model trained with gradient descent for a fixed number of steps per task, which holds for arbitrary teachers. We prove that the optimal fixed step size decays nearly inversely with the number of tasks, specifically as , for a single teacher and, more generally, i.i.d. teachers. Under this scaling, we show that the resulting generalization loss is asymptotically equivalent to that achieved by optimally tuned L2 (isotropic) regularization. Furthermore, we extend our analysis to generalized ridge regression with diagonal regularization matrices, obtaining a closed-form characterization of the generalization loss and identifying necessary and sufficient conditions under which the optimal regularizer is isotropic. Finally, we identify a regime where the optimal step size remains finite even as the number of tasks tends to infinity, thus revealing a non-trivial tradeoff between task variability and implicit regularization, in contrast to the i.i.d. teacher scaling. Our theoretical predictions are corroborated by experiments on both linear models and neural networks, providing practical guidance for the design of continual learning systems.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, multi-task learning, optimization, random matrix theory, marchenko-pastur

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5SEQOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The authors study the properties of various algorithms of continul learning under high-dimensional regression model. The main results can be summarized as follows:

  1. For early-stopping GD, with i.i.d groundtruth weight for each task, the authors derive the determinisitc equivalent of the generalization error after T tasks, and characterize the optimal learning rate for SGD, which is vanishing w.r.t task number T.T.
  2. For early-stopping GD, with sinusoidal groundtruth weight (non i.i.d teacher), the show that the optimal learning rate for SGD is vanishing w.r.t T.T.
  3. For explicit regularization with ridge, the authors characterize the generalization error, and show the benefit of generalized ridge regularization compare to the isotropic one.
Critical strengths
  1. The paper is pretty well-written and easy to follow.
  2. The paper studies a few properties of continual learning under linear regression setup, I think those are interesting questions to study from a theoretical point of view.
Critical weaknesses
  1. The paper focus on linear model, whose results might not be easily transferred to more practical setup. While this could be a common comment for all theoretical works on linear model, I still wonder to what extend the theoretical intuitions obtain in this paper can be generalized to more practical setups, e.g. continual learning with modern neural networks. In particular, the paper characterized a few interesting results, such as the optimal learning rate and the benefit of generalized ridge regularization, I wonde would those results generalized to more practical setup qualitatively?

  2. The technical novelty needs to be elaborated more. The paper characterized a few determinsitic equivalent of the test error, but these seems to be easily follows from standard determinisitic equivalent results for random matrix in proportional regime, see e.g. [1].

  3. Section 6 show that under sinusoidal model the optimal learning rate is not vanishing w.r.t TT, compare to the i.i.d tacher in section 4. But first the characterization of the optimal learning rate is only empirical. Moreover, I feel this is not a fair comparison, in the sense that the sinusoidal model only have at most 1/f1/f different tasks for any TT, and the i.i.d model w.h.p have TT different tasks. Could the authos elaborate more on this point?

[1] Romain Couillet and Zhenyu Liao. "Random matrix methods for machine learning."

Questions for authors
  1. For the early stopping GD with i.i.d teacher, the optimal length of training (Nγ∗N \gamma_*) is ln⁡T/T,\ln T/ T, could the author explain more on the intuition why the scaling is of this vanishing order?

  2. Line 309-310 says "So far, our analysis characterizes how to optimally tune the parameters of early-stopped GD under the i.i.d. teacher setting, showing that the resulting generalization loss is asymptotically equivalent to that of optimally tuned isotropic regularization." However, I can not directly see the equivalence from the charactization of generalization error in Theorem 1 and Theorem 3, could the authors elaborate more on this point?

  3. Theorem 4 characterize the benefit of generalized ridge. Would it be possible to characterize the optimal generalized ridge matrix M,M, and if so, does the optimal structure of MM give any insights? In particular, is it possible to construct such MM without knowing any prior information about the tasks?

3 comments

We thank the reviewer for their effort and for providing a constructive review to improve our paper.

Weakness 1 - The paper focus on linear model, whose results might not be easily transferred to more practical setup. While this could be a common comment for all theoretical works on linear model, I still wonder to what extend the theoretical intuitions obtain in this paper can be generalized to more practical setups, e.g. continual learning with modern neural networks. In particular, the paper characterized a few interesting results, such as the optimal learning rate and the benefit of generalized ridge regularization, I wonder would those results generalized to more practical setup qualitatively?

We believe that the main insights extend qualitatively to more practical continual learning settings. Importantly, we provide empirical evidence supporting this in Section 7. In particular, our theory predicts that the optimal learning rate decreases approximately as γ⋆≍ln⁡T/T\gamma^\star\asymp \ln T/T under a single teacher or i.i.d. teachers. To investigate whether this prediction extends beyond linear models, we conduct experiments on MNIST using a two-layer ReLU neural network, under both single-teacher and i.i.d. teacher settings. As shown in Figure 3, we observe that the empirically optimal learning rate follows the same predicted scaling, despite the nonlinear architecture and the fact that the MNIST features do not satisfy our theoretical assumptions. This provides empirical evidence that the identified scaling is not restricted to linear regression. Additionally, following the reviewer’s question, we have extended our empirical evaluation to include more challenging experiments with CNN on CIFAR-10 based dataset. These results are now presented in Appendix B.2.

Weakness 2 - The technical novelty needs to be elaborated more. The paper characterized a few determinsitic equivalent of the test error, but these seems to be easily follows from standard determinisitic equivalent results for random matrix in proportional regime, see e.g. [1].

We disagree that our deterministic equivalent results for the test error easily follow from [1].

The results in [1] are established in a single-task setting, and extending them to multiple tasks is highly nontrivial. In continual learning, the predictor at each task depends on all previous tasks, leading to products of random matrices and resolvents across tasks, rather than the individual random matrices typically analyzed in single-task settings. Characterizing these products requires additional theoretical developments beyond the existing results in [1]. In particular, our analysis establishes novel results in the random matrix theory framework as detailed in Lemma 4 and Theorem 7.

Weakness 3a - Section 6 show that under sinusoidal model the optimal learning rate is not vanishing w.r.t TT, compared to the i.i.d teacher in section 4. But first the characterization of the optimal learning rate is only empirical.

The characterization of the optimal learning rate under the sinusoidal model uses Theorem 5, which provides an explicit expression for the asymptotic generalization loss under the sinusoidal model. We then numerically optimize this expression to obtain the optimal step size shown in Figure 2. Thus, the numerical results are based on our theoretical characterization rather than empirical simulations. Note that it suffices to verify whether there exists γ\gamma such that G∞(γ)<G∞(0)G_\infty(\gamma)<G_\infty(0). Indeed, we numerically verified that this condition holds for grid points whose optimal value is not attained at the smallest value of γ\gamma in the grid.

Weakness 3b - Moreover, I feel this is not a fair comparison, in the sense that the sinusoidal model only have at most 1/f1/f different tasks for any TT, and the i.i.d model w.h.p have TT different tasks. Could the authors elaborate more on this point?

Note that the sinusoidal model does not necessarily contain only 1/f1/f distinct teachers, as for irrational ff, the sequence does not repeat exactly. Importantly, our purpose in considering the sinusoidal model is to provide an example where the optimal step size remains finite as TT increases, demonstrating that the vanishing scaling established for i.i.d. teachers is not universal and can depend on the underlying task structure.

Questions

Question 1 - For the early stopping GD with i.i.d teacher, the optimal length of training N is T/T could the author explain more on the intuition why the scaling is of this vanishing order?

Intuitively, as the number of tasks increases, we need to use a smaller step size to avoid interference between tasks and reduce the accumulation of errors. This allows the model to better preserve what it has learned from previous tasks while continuing to learn new ones. However, the exact rate at which the step size should decrease is not clear a priori, which motivates our theoretical analysis. Informally, our analysis reveals that the contribution of the initialization error, arising from the difference between the initial parameters and the teacher, decreases exponentially with the cumulative training length TNγTN\gamma, whereas the accumulated effect of task-specific fluctuations is approximately linear in NN in the relevant asymptotic regime. Balancing these two effects yields the optimal scaling Nγ⋆≍ln⁡T/TN\gamma^\star\asymp \ln T/T.

Question 2 - Line 309-310 says "So far, our analysis characterizes how to optimally tune the parameters of early-stopped GD under the i.i.d. teacher setting, showing that the resulting generalization loss is asymptotically equivalent to that of optimally tuned isotropic regularization." However, I can not directly see the equivalence from the characterization of generalization error in Theorem 1 and Theorem 3, could the authors elaborate more on this point?

In Theorem 3, lines 216-217 we write that if γ≍ln⁡TNT\gamma\asymp\frac{\ln T}{NT} then, the resulting generalization loss of the early-stopped GD matches that of optimally tuned L2 regularization up to O(1/T)O(1/T) factor. In particular, it establishes that as T→∞T\to\infty then GTridge→GTesgdG_T^{\mathrm{ridge}}\to G_T^{\mathrm{esgd}} if γ≍ln⁡TNT\gamma\asymp \frac{\ln T}{NT}.

Question 3a - Theorem 4 characterizes the benefit of generalized ridge. Would it be possible to characterize the optimal generalized ridge matrix and if so, does the optimal structure give any insights?

We thank the reviewer for this interesting question. In general, obtaining a closed-form expression for the optimal generalized ridge matrix is not possible. Indeed, minimizing the generalization loss characterized in Theorem 3 generally requires solving a polynomial equation of degree greater than four (see the equation defining m~p(−1)\tilde m_p(-1)), which is provably impossible in the general case. However, our Theorem 3 can provide insights into the optimal structure of generalized ridge regularization. For example, when the squared distances between the teacher and initialization are equal across all coordinates, the optimal diagonal regularizer is isotropic, as established in Theorem 9.

Question 3b - In particular, is it possible to construct such M\mathbf M without knowing any prior information about the tasks?

It is impossible to construct an optimal M\mathbf M without knowing any prior information about the tasks. For example, consider two settings in which the teacher is respectively w⋆=(1,0)\mathbf w^\star=(1,0) and w⋆=(0,1)\mathbf w^\star=(0,1) with initialization w0=0\mathbf w_0=\mathbf 0. In the presence of noise, the first setting favors weaker regularization of the first coordinate and stronger regularization of the second, whereas the opposite holds in the second setting. Thus, the optimal regularization matrix necessarily depends on the underlying task distribution.

Thank you for the detailed clarification and explanation.

I think the rebuttal has resolved/clarified the technical issues I proposed in the review. Thus I would like to increase the score to 3.

Reviewer_PjxbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the lack of a unified theoretical characterization of how regularization should be optimally designed as the number and structure of tasks vary in continual learning, this paper develops a statistical framework for high-dimensional continual linear regression and derives closed-form generalization losses for early-stopped gradient descent and generalized ridge regression. Using these expressions, it characterizes the optimal implicit regularization strength, the conditions under which diagonal or isotropic regularization is optimal, and the behavior of optimal regularization under different teacher sequences. Theoretical findings are further validated through experiments on both linear models and neural networks using synthetic and MNIST-based datasets.

Critical strengths
  1. This paper derives the optimal scaling of the step size for finite-step gradient descent, providing a theoretical correspondence between implicit regularization induced by early stopping and optimally tuned explicit L_2 regularization in high-dimensional continual regression.
  2. This paper theoretically characterizes when direction-dependent diagonal regularization can outperform isotropic regularization, showing that isotropic regularization is not universally optimal and identifying conditions under which it remains optimal.
Critical weaknesses
  1. Eq. (52) appears to be inconsistent with Theorem 3, as the two expressions yield different values under a simple single-task, noiseless specialization. This raises concerns about the mathematical correctness of the diagonal regularization analysis.
  2. The necessity part of Theorem 4 is not fully justified, as Eq. (60) appears to omit the effects of the other diagonal regularization parameters on each coordinate-specific term, which are not shown to vanish by permutation symmetry.
  3. The claimed convergence in probability of the generalization loss in Theorem 1 appears to require additional concentration assumptions on the label noise, as the stated zero-mean, isotropic covariance, and independence assumptions alone do not ensure concentration of the noise-dependent quadratic forms used in the proof.
  4. The empirical experiments do not fully validate the practical implications of the proposed step-size scaling law, as they either change the evaluation objective or re-optimize the step size for each task horizon rather than demonstrating that the theory enables reliable tuning transfer across different horizons.
Questions for authors
  1. Can the consistency between Eq. (52) and Theorem 3 be rigorously established?
  2. Can the necessity claim in Theorem 9 be justified while accounting for the dependence of the coordinate-specific terms on all diagonal regularization parameters?
  3. Does the claimed convergence in probability of the generalization loss in Theorem 1 require additional concentration assumptions on the label noise beyond those stated in Section 3?
  4. Does the characterization of Theorem 2 of the optimal fixed step size for early-stopped gradient descent require the exact optimizer, or does the same result hold for step sizes with the same asymptotic scaling?
  5. Is the result for isotropic regularization in Figure 4a primarily a consequence of the symmetry of the artificial teacher and the theoretical objective, and does the diagonal regularizer still outperform isotropic regularization when the scalar regularizer is also optimally tuned using the same information?
3 comments

We thank the reviewer for their effort and for providing a constructive review.

Below we address the reviewers' concerns and questions raised by the reviewer.

Question 1 + Weakness 1 - Eq. (52) appears to be inconsistent with Theorem 3, as the two expressions yield different values under a simple single-task, noiseless specialization. This raises concerns about the mathematical correctness of the diagonal regularization analysis.

We thank the reviewer for identifying the (minor) typo in Eq. (52) of Appendix D.3, which we have corrected in the revised paper. We emphasize that our derivation is sound. Indeed, combining Eq. (55) with the equivalence established in lines 2409–2416 (or lines 2462-2471, in the revised version) yields the equivalent form now appearing in the revised version of Eq. (52). Setting in this equation Mt=M\mathbf M_t=\mathbf M and Σt=vxI\mathbf\Sigma_t=v_x \mathbf I, recovers exactly Theorem 3.

Question 2 + Weakness 2 - The necessity part of Theorem 4 is not fully justified, as Eq. (60) appears to omit the effects of the other diagonal regularization parameters on each coordinate-specific term, which are not shown to vanish by permutation symmetry.

We assume that the reviewer refers to Theorem 9 rather than Theorem 4, as Eq. (60) appears in Theorem 9. Importantly, the original arguments remain valid. By permutation symmetry, the cross-coordinate derivatives contribute an additional term CT(λ)(∥w⋆−w0∥2−(wi⋆−w0,i)2)C_T(\lambda)(\Vert\mathbf w^\star-\mathbf w_0\Vert^2-(w_i^\star-w_{0,i})^2), where CT(λ)=∂bT,j(m)∂mi∣m=λ1C_T(\lambda)=\left.\frac{\partial b_{T,j}(\mathbf m)}{\partial m_i}\right|_{\mathbf m=\lambda\mathbf 1} for any j≠ij\neq i. Since CT(λ)∥w⋆−w0∥2C_T(\lambda)\Vert\mathbf w^\star-\mathbf w_0\Vert^2 is identical for all coordinates, it can be absorbed into the coordinate-independent term BT(λ)B_T(\lambda). More precisely, defining A~T(λ)=AT(λ)−CT(λ)\widetilde A_T(\lambda)=A_T(\lambda)-C_T(\lambda) and redefining BT(λ)B_T(\lambda) to include this common contribution, we recover the same form of the gradient expression in Eq. (60), which does not affect the conclusion, since the proof relies on differences between coordinate-wise derivatives, for which the common term cancels. In the revision, we included the cross-coordinate derivative term in Eq. (60) and explained the reason it does not affect the conclusion. Moreover, in lines 2670-2676, we simplified the proof by directly establishing the strict positivity of A~T(λ)\widetilde A_T(\lambda) without relying on prior work.

Question 3 + Weakness 3 - The claimed convergence in probability of the generalization loss in Theorem 1 appears to require additional concentration assumptions on the label noise.

We thank the reviewer for pointing this out. As reflected in our data model, we impose standard regularity conditions ensuring concentration in the high-dimensional limit. In particular, we assume finite higher-order moments for the features and uniform integrability of the teacher correlations. We intended the same uniform integrability condition to apply to the label noise as well, but the reviewer’s comment made us realize that this might not have been sufficiently explicit in the data model. We have therefore revised the data model to make the uniform integrability requirement more explicit.

Weakness 4 - The empirical experiments do not fully validate the practical implications of the proposed step-size scaling law, as they either change the evaluation objective or re-optimize the step size for each task horizon rather than demonstrating that the theory enables reliable tuning transfer across different horizons.

In our empirical MNIST experiments, we preserve the original evaluation objective by measuring the mean squared error on a held-out test set, which directly assesses the generalization loss. Moreover, we re-optimize the step size for each task horizon to obtain a reliable estimate of the optimal step size at that horizon. These estimates are then used to empirically validate our theoretically predicted scaling law, γ⋆≍ln⁡T/T.\gamma^\star\asymp \ln T/T.

Question 5a - Is the result for isotropic regularization in Figure 4a primarily a consequence of the symmetry of the artificial teacher and the theoretical objective?

Yes. In this case, the optimal solution is isotropic.

Question 5b - Does the diagonal regularizer still outperform isotropic regularization when the scalar regularizer is also optimally tuned using the same information?

It depends. If the distance of each teacher’s coordinate to initialization is fixed, then we would expect the diagonal regularizer to perform on par with scalar regularization if both are optimally tuned; this is implied by Theorem 9. Conversely, if this condition does not hold, we would expect the diagonal regularizer to outperform the isotropic regularizer. For example, see Figure 4b, where for all lambda, we get the diagonal is better than the scalar.

Thank you to the authors for their efforts in addressing my concerns. However, I believe two issues remain insufficiently resolved. The added uniform integrability condition does not ensure the concentration required for Theorem 1, and the response does not clarify whether Theorem 2's guarantee requires the exact optimal step size or also holds for step sizes with the same asymptotic scaling. Therefore, I maintain my original rating of 2.

We thank the reviewer for the follow up.

We are not sure why the added uniform integrability condition does not ensure the concentration required for Theorem 1. We would like to clarify that the assumptions now on the label noise zt\mathbf z_{t} are E[zt]=0,\mathbb E[\mathbf z_{t}]=0, Cov(zt)=vzI\mathrm{Cov(\mathbf z_{t})}=v_z\mathbf I with entries sampled independently of all tasks (this already includes the task tt itself with its other label noise entries) and the squared label noise entries are uniformly integrable.

Indeed, this is a standard result. We provide here the proof for completeness. Define Xd,i=zk,izj,i,X_{d,i}=z_{k,i}z_{j,i}, by Young's inequality, ∣Xd,i∣≤12(zk,i2+zj,i2).|X_{d,i}|\leq \frac{1}{2}(z_{k,i}^2+z_{j,i}^2). Hence, uniform integrability of the squared coordinates implies uniform integrability of Xd,iX_{d,i}. For M>0M>0, define the truncation Xd,i(M)=Xd,i1∣Xd,i∣≤M,X_{d,i}^{(M)}=X_{d,i}\mathbf{1}\\{|X_{d,i}|\leq M\\}, and split 1d∑i(Xd,i−EXd,i)=Ad,M+Rd,M,\frac{1}{d}\sum_{i}(X_{d,i}-\mathbb{E}X_{d,i})=A_{d,M}+R_{d,M}, where Ad,M=1d∑i(Xd,i(M)−E[Xd,i(M)])A_{d,M}=\frac{1}{d}\sum_{i}(X_{d,i}^{(M)}-\mathbb{E}[X_{d,i}^{(M)}]) and Rd,M=1d∑i[(Xd,i−Xd,i(M))−E(Xd,i−Xd,i(M))].R_{d,M}=\frac{1}{d}\sum_{i}\bigl[(X_{d,i}-X_{d,i}^{(M)})-\mathbb{E}(X_{d,i}-X_{d,i}^{(M)})\bigr].

By independence and Chebyshev's inequality, P(∣Ad,M∣>ϵ)=P(∣1d∑i=1d(Xd,i(M)−E[Xd,i(M)])∣>ϵ)≤M2dϵ2⟶0\mathbb{P}\left(\left|A_{d,M}\right|>\epsilon\right)=\mathbb{P}\left(\left|\frac{1}{d}\sum_{i=1}^{d}\left(X_{d,i}^{(M)}-\mathbb{E}[X_{d,i}^{(M)}]\right)\right|>\epsilon\right)\leq\frac{M^{2}}{d\epsilon^{2}}\longrightarrow0 as d→∞d\to\infty for fixed MM.

Furthermore, by the triangle inequality and Markov’s inequality, sup⁡dP(∣Rd,M∣>ϵ)≤sup⁡dE∣Rd,M∣ϵ→0\sup_d\mathbb{P}(|R_{d,M}|>\epsilon)\le\frac{\sup_d\mathbb{E}|R_{d,M}|}{\epsilon}\to 0 as M→∞M\to\infty this holds because of the uniform integrability, sup⁡dE∣Rd,M∣≤sup⁡d1d∑i=1d(E∣XM,d,i∣+∣EXM,d,i∣)≤sup⁡d2d∑i=1dE[∣XM,d,i∣]→0\sup_d\mathbb{E}|R_{d,M}|\le\sup_d\frac{1}{d}\sum_{i=1}^{d}\left(\mathbb{E}|X_{M,d,i}|+|\mathbb{E}X_{M,d,i}|\right)\le\sup_d\frac{2}{d}\sum_{i=1}^{d}\mathbb{E}\left[|X_{M,d,i}|\right]\to0 where XM,d,i=Xd,i1∣Xd,i∣>MX_{M,d,i}=X_{d,i}\mathbf{1}\\{|X_{d,i}|>M\\}. Taking d→∞d\to\infty followed by M→∞M\to\infty gives 1d∑i=1d(zk,izj,i−E[zk,izj,i])→P0.\frac1d\sum_{i=1}^d\left(z_{k,i}z_{j,i}-\mathbb E[z_{k,i}z_{j,i}]\right)\xrightarrow{P}0. Therefore for k≠jk\neq j, 1dzk⊤zj→P0,\frac1d\mathbf z_k^\top\mathbf z_j\xrightarrow{P}0, and for k=jk=j, 1d∥zk∥2→Pvz.\frac1d\Vert\mathbf z_k\Vert^2\xrightarrow{P}v_z.

Furthermore, the asymptotic equivalence of Theorem 2 holds for all step sizes that scale as ln⁡T/(NT)\ln T/(NT) and does not require the exact optimal step.

We thank the reviewer for his effort in providing a constructive review.

Reviewer_dCktOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies high-dimensional continual linear regression across TT sequential tasks in the proportional asymptotic limit (n/d→α∈(0,∞)n/d \to \alpha \in (0, \infty)) with isotropic i.i.d. features Xt∈Rn×dX_t \in \mathbb{R}^{n \times d} and additive label noise.

Its main contributions are:

  • Generalization & Optimal Scaling of Early-Stopped GD (Theorems 1 & 2). In this isotropic, iid optimal parameter setting, the paper gives the optimal step size.
  • Analysis of Diagonal Regularization & Optimality Conditions (Theorems 3, 4), which complements the isotropic, underparameterized analysis with an overparameterized, diagonal feature-covariance one.
  • Non-Stationary Sinusoidal Teachers (Theorem 5): In a non-iid optimal parameter setting, the papar give a bad case for the predicted optimal step size, showing that the result is (understandably) not yet universal.

It also gives numerical experiments that support its claim.

Critical strengths

I believe the messages from the three contributions are insightful for the CL theory field. The paper is sound and technically interesting to me. The work follows a solid technical line of CL theory work from a prominent lab in the CL theory field.

Critical weaknesses

I would like to slightly complain about this line of CL theory work, including this one. This does not affect my judgment of its quality.

Works like this one tend to consider an isotropic distribution of XX. This always corresponds to the underparameterized regime. In this regime, I doubt it corresponds to the real neural network scenario. This work partially compensates for it with Thm 3, where a diagonal feature covariance Σt\Sigma_t is considered.

Questions for authors

I will vote to accept this paper. Meanwhile, I do have some questions about the technical contribution and the authors' thoughts on the implications of the messages from the isotropic regime for the big picture of CL theory. I am asking these questions mainly because I believe your answers might be beneficial to the CL theory field.

  • Is the introduction of tools including Marchenko-Pastur distribution, the Stieltjes transform and the isotropic resolvent law in Lemma 4 to the CL theory field the main technical contribution of your paper? What technical problem do you solve with these three tools? If they are not the main ones, which one is, and how does it clear the main technical hurdle?
  • How do you think the messages from the isotropic, underparameterized regime help people understand the big picture of CL theory? I understand that a simplified regime eases the proof and makes the result more readable, but it is widely known that neural networks are highly anisotropic and overparameterized. How can your theoretical messages shed light on some parts of the big picture of the learning dynamics of CL?

Also there might be a gap that I do not yet fully understand: In Corollary 1 (Appendix C.3, line 1300), the left-hand side is written as GTridge→P…G_T^{\text{ridge}} \xrightarrow{P} \dots, whereas the corollary and proof actually compute the limit of GTesgdG_T^{\text{esgd}} for Algorithm 2 (Early-Stopped GD).
Is there a typo or am I missing something here?

2 comments

We thank the reviewer for his positive feedback. Below we address the reviewer’s main concerns and questions.

Question 1 - Is the introduction of tools including Marchenko-Pastur distribution, the Stieltjes transform and the isotropic resolvent law in Lemma 4 to the CL theory field the main technical contribution of your paper? What technical problem do you solve with these three tools? If they are not the main ones, which one is, and how does it clear the main technical hurdle?

The Marchenko–Pastur distribution, Stieltjes transform, and isotropic resolvent law are established tools in random matrix theory, and applying them to continual learning is an important ingredient of our analysis. However, our main technical contribution goes beyond introducing these tools to the continual learning setting. Rather, it lies in deriving precise generalization characterizations for continual learning algorithms, which present additional challenges compared with the single-task settings commonly analyzed in the literature [1,2,3].

In particular, the main difficulty is that the predictor at each task depends on all previous tasks, leading to products of random matrices across tasks and correlations between the resulting iterates and the teachers. For early-stopped GD, we use Lemma 4 to characterize these products and obtain an exact generalization expression for arbitrary teacher sequences and label noise (Theorem 1). For generalized ridge regularization, the analysis also requires deterministic equivalents for products of resolvents, not just individual resolvents. We address this through the analysis in Appendix D, which enables the precise generalization characterization in Theorem 3.

Question 2 + Weakness 1 - Works like this one tend to consider an isotropic distribution of XX. This always corresponds to the underparameterized regime. In this regime, I doubt it corresponds to the real neural network scenario. This work partially compensates for it with Thm 3, where a diagonal feature covariance is considered. How do you think the messages from the isotropic, underparameterized regime help people understand the big picture of CL theory? I understand that a simplified regime eases the proof and makes the result more readable, but it is widely known that neural networks are highly anisotropic and overparameterized. How can your theoretical messages shed light on some parts of the big picture of the learning dynamics of CL?

In our work, we assume the isotropy of the population feature covariance matrix, which does not restrict the model to the underparameterized regime. In other words, our analysis applies to both underparameterized and overparameterized settings. This is in contrast to assuming an isotropic empirical feature covariance matrix of X\mathbf X, which always corresponds to the (less realistic) underparameterized regime. Moreover, both Theorems 3 and 8 in Appendix D.3 extend our analysis to anisotropic population feature covariance matrices.

Question 3 - there might be a gap that I do not yet fully understand: In Corollary 1 (Appendix C.3, line 1300), the left-hand side is written as GTridgeG_T^{\mathrm{ridge}}, whereas the corollary and proof actually compute the limit of GTesgdG_T^{\mathrm{esgd}} for Algorithm 2 (Early-Stopped GD).

Indeed, in Corollary 1 (Appendix C.3, line 1300) GTridgeG_T^{\mathrm{ridge}} appears to be a typo; we meant to write GTesgdG_T^{\mathrm{esgd}}. We have fixed this typo in the revision.

[1] Salehi, Fariborz, Ehsan Abbasi, and Babak Hassibi. "The impact of regularization on high-dimensional logistic regression." advances in neural information processing systems 32 (2019).

[2] Moniri, Behrad, and Hamed Hassani. "On the mechanisms of weak-to-strong generalization: A theoretical perspective." Advances in Neural Information Processing Systems 38 (2026): 121409-121444.

[3] Thrampoulidis, Christos, Ehsan Abbasi, and Babak Hassibi. "Precise error analysis of regularized $ M $-estimators in high dimensions." IEEE Transactions on Information Theory 64.8 (2018): 5592-5628.

Thanks for your response. I will keep the score. The population feature covariance matrix explanation is intriguing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.