- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies the generalization error of a two layer quadratic-activated neural network trained via an adaptive optimizer in a student-teacher setting. The paper shows that the risk scaling matches that of an optimally tuned SGD. In addition, the analysis reveals two learning phases: one corresponding to noisy training and one corresponding to noiseless. Experiments validate the theoretical claims.
Critical strengths
- The paper derives an end-to-end risk bound for adaptive optimizers for a feature-learning network. Specifically, different from the majority of prior work, the theory is 1) for a feature-learning network, 2) adaptive optimizer, 3) jointly considers optimization and generalization. The relative novelty of the theoretical setting is a major strength of this paper.
Critical weaknesses
The major weaknessess are twofold:
-
The proposed adaptive optimizer is very specific: the results apply only to a particular optimizer, and not a broad family that may plausibly include practical optimizers. In particular, equation (ADP) seems to suggest that when the loss is smaller, the step sizes get larger (unless I am misunderstanding), which seems to be the opposite of (or at least not aligned with) how practical adaptive optimizers like Adam operate.
-
The connections with prior theory on adaptive optimizers is not clear. I understand that the approach of this work is to study the generalization error of a specific model on a specific task in a teacher-student setting. However, as the authors point out in the related work section, there has been extensive work on studying generalization/optimization of adaptive algorithms in other settings. It would be useful to compare how risk bounds derived from these methods compare to what the authors derive. For instance, the authors could consider a classical optimization bound for an adaptive algorithm on non-convex loss landscapes and combine it with a generic capacity-based generalization bound for the two layer network studied (something like the ERM bound referred to in Table 1) to provide an overall baseline risk bound.
Questions for authors
The questions relate to the two weaknesses above: 1a. Can the theory be extended to a broader/more flexible class of adaptive optimizers? 1b. How realistic is the adaptive optimizer analyzed? Are there practical optimizers with a similar form (not just similar risk/loss curves)? 2. How does the derived risk bound compare to prior theory on adaptive optimizers?
3 comments
Response to Reviewer rSyh (2/2): W2, Q2 and Q1a
W2, Q2. Connection with prior theory on adaptive optimizers. Classical guarantees for adaptive methods (Ward et al., 2018; Li & Orabona, 2019; Défossez et al., 2020; all cited in Section 1.2) hold for general non-convex objectives, but under assumptions our problem does not satisfy: global smoothness and bounded stochastic gradients or noise. Here is quartic in and the gradient noise is multiplicative, . Their conclusion is also weaker than a risk bound: they control a gradient norm, typically at rate , whereas our random initialization is already near a saddle, with a vanishing gradient and excess risk about .
By fixing the model we give up that generality, and in exchange we can go beyond these assumptions and show the benefit of adaptivity directly: an explicit risk curve with -dependent exponents, the learning-rate schedule that produces it, and a comparison with fixed-rate SGD (Table 1, Figs. 1–2). For instance, in the signal-dominated phase with the adaptive rate increases and the risk decays as , against for a fixed rate.
Closest high-dimensional analysis. Collins-Woodfin et al. (2024) analyze AdaGrad-Norm for least squares on horizons proportional to , and they observe transitions similar to ours, driven by a very similar mechanism. First, with label noise their learning rate decays as , as ours does in the noise-dominated phase. Second, without noise, their exponent (in their notation) plays the role of our : their decaying-rate regime gives our schedule and risk for . Their constant-rate regime corresponds to and gives the fixed-rate exponent ; the discount in our rule is what lets the rate increase there and reach . Our setting extends this mechanism to a non-convex feature-learning model, with time scales beyond . We highlighted this connection in Section 1.2 of the revised manuscript.
Q1a. Can the theory cover a broader class? In part it already does, since the mechanism is not tied to the specific rule, as discussed above. It has three ingredients. (i) For the squared loss, the squared gradient is the loss times a slowly varying factor, so accumulating squared gradients tracks the loss. (ii) A scalar learning rate acts as a change of time on the fixed-rate dynamics; this holds for every scalar rule. (iii) The inverse square root of the accumulated loss keeps the learning rate in a constant ratio to the signal-to-noise level of the feature being learned.
Within our model, the analysis therefore extends to any polynomial discount with , and to rules that accumulate squared gradient norms instead of the loss, up to the factor , which stays bounded for . For RMSprop and Adam, the computation in part 1/2 and Figure 2 indicate the same type of loss-controlled dynamics; a proof for per-parameter methods and momentum remains open (Limitation (iii)).
We expect the same mechanism beyond quadratic networks, since learning is incremental there as well. Our experiments with ReLU and absolute-value activations (Figure 2) already show similar scaling behavior. Extending the theory to general link functions is an interesting direction for future work (Limitation (i)).
Thank you for the detailed response. This largely resolves my concerns and I remain in favor of acceptance.
Response to Reviewer rSyh (1/2): W1 and Q1
Thank you for the review. Your questions helped us state more precisely what the algorithm is and how it relates to practical methods. W2, Q2 and Q1a are answered in part 2/2.
W1, Q1b. Is the optimizer realistic? Yes. In our model, a loss-based rule and a gradient-based rule are driven by the same quantity: the loss our rule accumulates is essentially what RMSprop and Adam accumulate. For the weight wai (coordinate a of neuron i), the squared per-sample gradient has mean approximately
rs28 P(Gt)(∥wi∥2+wai2),wai2≪∥wi∥2,
with Gt=WtWt⊤/rs. The second-moment estimate of RMSprop or Adam is therefore an aggregate of the past loss times a per-neuron factor ∥wi∥2, with almost no dependence on the coordinate a. Substituting it into their update gives, for neuron i, a drift proportional to
γt ∥wi∥(Ψ∗−Gt) wi,
where γt is the inverse root of this aggregated loss. These methods thus effectively reduce to the type of dynamics we analyze: a gradient method, normalized per neuron, whose step size is controlled by the aggregated loss. Our theory characterizes the optimal learning-rate schedule for such dynamics and shows that the aggregation rule of (ADP) attains it. The per-neuron normalization only changes constants: the squared neuron norms have mean Tr(Gt) and variance approximately 2∥Gt∥F2 across neurons, independently of the width. This is consistent with Figure 2, where RMSprop and Adam follow risk curves similar to those of our rule. We added this computation to the revised manuscript (Remark 1 in Appendix A).
W1. Is "smaller loss, larger step" opposite to Adam? No, it is the same direction. Adam and RMSprop divide by the root of the accumulated squared gradients, so smaller recent gradients give a larger effective step, and by the previous item squared gradients are proportional to the loss here.
Importantly, the step does not grow during training in general. νt is a weighted sum, not an average: it is roughly t times the recent average loss, and the rate is its inverse root. So the rate decays unless the loss falls faster than 1/t. There are two regimes.
Why should the learning rate increase for α>1? Because optimality requires it. SGD with a fixed learning rate attains at best the rate t(1−2α)/α in this setting (Table 1; Ren et al., 2025; Ben Arous et al., 2025), which for α>1 is slower than the noiseless ERM rate t1−2α. By the time change of Section 4, the risk at time t is the fixed-rate risk curve evaluated at F∗(t)=∫0tγ, so reaching the ERM exponent requires F∗(t)≍tα: any scalar schedule that is optimal in this regime must increase, like tα−1. The increase is thus not an artifact of our rule. We agree that this regime is of limited practical relevance, since data are typically noisy and the increase is then transient. We leave practical scheduling in this regime as interesting future work (Limitation (iv)).