- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The authors study the properties of various algorithms of continul learning under high-dimensional regression model. The main results can be summarized as follows:
- For early-stopping GD, with i.i.d groundtruth weight for each task, the authors derive the determinisitc equivalent of the generalization error after T tasks, and characterize the optimal learning rate for SGD, which is vanishing w.r.t task number
- For early-stopping GD, with sinusoidal groundtruth weight (non i.i.d teacher), the show that the optimal learning rate for SGD is vanishing w.r.t
- For explicit regularization with ridge, the authors characterize the generalization error, and show the benefit of generalized ridge regularization compare to the isotropic one.
Critical strengths
- The paper is pretty well-written and easy to follow.
- The paper studies a few properties of continual learning under linear regression setup, I think those are interesting questions to study from a theoretical point of view.
Critical weaknesses
-
The paper focus on linear model, whose results might not be easily transferred to more practical setup. While this could be a common comment for all theoretical works on linear model, I still wonder to what extend the theoretical intuitions obtain in this paper can be generalized to more practical setups, e.g. continual learning with modern neural networks. In particular, the paper characterized a few interesting results, such as the optimal learning rate and the benefit of generalized ridge regularization, I wonde would those results generalized to more practical setup qualitatively?
-
The technical novelty needs to be elaborated more. The paper characterized a few determinsitic equivalent of the test error, but these seems to be easily follows from standard determinisitic equivalent results for random matrix in proportional regime, see e.g. [1].
-
Section 6 show that under sinusoidal model the optimal learning rate is not vanishing w.r.t , compare to the i.i.d tacher in section 4. But first the characterization of the optimal learning rate is only empirical. Moreover, I feel this is not a fair comparison, in the sense that the sinusoidal model only have at most different tasks for any , and the i.i.d model w.h.p have different tasks. Could the authos elaborate more on this point?
[1] Romain Couillet and Zhenyu Liao. "Random matrix methods for machine learning."
Questions for authors
-
For the early stopping GD with i.i.d teacher, the optimal length of training () is could the author explain more on the intuition why the scaling is of this vanishing order?
-
Line 309-310 says "So far, our analysis characterizes how to optimally tune the parameters of early-stopped GD under the i.i.d. teacher setting, showing that the resulting generalization loss is asymptotically equivalent to that of optimally tuned isotropic regularization." However, I can not directly see the equivalence from the charactization of generalization error in Theorem 1 and Theorem 3, could the authors elaborate more on this point?
-
Theorem 4 characterize the benefit of generalized ridge. Would it be possible to characterize the optimal generalized ridge matrix and if so, does the optimal structure of give any insights? In particular, is it possible to construct such without knowing any prior information about the tasks?
3 comments
Part 2
Questions
Question 1 - For the early stopping GD with i.i.d teacher, the optimal length of training N is T/T could the author explain more on the intuition why the scaling is of this vanishing order?
Intuitively, as the number of tasks increases, we need to use a smaller step size to avoid interference between tasks and reduce the accumulation of errors. This allows the model to better preserve what it has learned from previous tasks while continuing to learn new ones. However, the exact rate at which the step size should decrease is not clear a priori, which motivates our theoretical analysis. Informally, our analysis reveals that the contribution of the initialization error, arising from the difference between the initial parameters and the teacher, decreases exponentially with the cumulative training length , whereas the accumulated effect of task-specific fluctuations is approximately linear in in the relevant asymptotic regime. Balancing these two effects yields the optimal scaling .
Question 2 - Line 309-310 says "So far, our analysis characterizes how to optimally tune the parameters of early-stopped GD under the i.i.d. teacher setting, showing that the resulting generalization loss is asymptotically equivalent to that of optimally tuned isotropic regularization." However, I can not directly see the equivalence from the characterization of generalization error in Theorem 1 and Theorem 3, could the authors elaborate more on this point?
In Theorem 3, lines 216-217 we write that if then, the resulting generalization loss of the early-stopped GD matches that of optimally tuned L2 regularization up to factor. In particular, it establishes that as then if .
Question 3a - Theorem 4 characterizes the benefit of generalized ridge. Would it be possible to characterize the optimal generalized ridge matrix and if so, does the optimal structure give any insights?
We thank the reviewer for this interesting question. In general, obtaining a closed-form expression for the optimal generalized ridge matrix is not possible. Indeed, minimizing the generalization loss characterized in Theorem 3 generally requires solving a polynomial equation of degree greater than four (see the equation defining ), which is provably impossible in the general case. However, our Theorem 3 can provide insights into the optimal structure of generalized ridge regularization. For example, when the squared distances between the teacher and initialization are equal across all coordinates, the optimal diagonal regularizer is isotropic, as established in Theorem 9.
Question 3b - In particular, is it possible to construct such without knowing any prior information about the tasks?
It is impossible to construct an optimal without knowing any prior information about the tasks. For example, consider two settings in which the teacher is respectively and with initialization . In the presence of noise, the first setting favors weaker regularization of the first coordinate and stronger regularization of the second, whereas the opposite holds in the second setting. Thus, the optimal regularization matrix necessarily depends on the underlying task distribution.
Thank you for the detailed clarification and explanation.
I think the rebuttal has resolved/clarified the technical issues I proposed in the review. Thus I would like to increase the score to 3.
Part 1
We thank the reviewer for their effort and for providing a constructive review to improve our paper.
We believe that the main insights extend qualitatively to more practical continual learning settings. Importantly, we provide empirical evidence supporting this in Section 7. In particular, our theory predicts that the optimal learning rate decreases approximately as γ⋆≍lnT/T under a single teacher or i.i.d. teachers. To investigate whether this prediction extends beyond linear models, we conduct experiments on MNIST using a two-layer ReLU neural network, under both single-teacher and i.i.d. teacher settings. As shown in Figure 3, we observe that the empirically optimal learning rate follows the same predicted scaling, despite the nonlinear architecture and the fact that the MNIST features do not satisfy our theoretical assumptions. This provides empirical evidence that the identified scaling is not restricted to linear regression. Additionally, following the reviewer’s question, we have extended our empirical evaluation to include more challenging experiments with CNN on CIFAR-10 based dataset. These results are now presented in Appendix B.2.
We disagree that our deterministic equivalent results for the test error easily follow from [1].
The results in [1] are established in a single-task setting, and extending them to multiple tasks is highly nontrivial. In continual learning, the predictor at each task depends on all previous tasks, leading to products of random matrices and resolvents across tasks, rather than the individual random matrices typically analyzed in single-task settings. Characterizing these products requires additional theoretical developments beyond the existing results in [1]. In particular, our analysis establishes novel results in the random matrix theory framework as detailed in Lemma 4 and Theorem 7.
The characterization of the optimal learning rate under the sinusoidal model uses Theorem 5, which provides an explicit expression for the asymptotic generalization loss under the sinusoidal model. We then numerically optimize this expression to obtain the optimal step size shown in Figure 2. Thus, the numerical results are based on our theoretical characterization rather than empirical simulations. Note that it suffices to verify whether there exists γ such that G∞(γ)<G∞(0). Indeed, we numerically verified that this condition holds for grid points whose optimal value is not attained at the smallest value of γ in the grid.
Note that the sinusoidal model does not necessarily contain only 1/f distinct teachers, as for irrational f, the sequence does not repeat exactly. Importantly, our purpose in considering the sinusoidal model is to provide an example where the optimal step size remains finite as T increases, demonstrating that the vanishing scaling established for i.i.d. teachers is not universal and can depend on the underlying task structure.