acceptodds
Under review as a conference paper at ICLR 2027

Double descent in learning rate in the feature learning regime

Abstract

Double descent is the non-monotonic behavior of test error as a function of a single hyperparameter. It is well documented for model width, dataset size and training time. We show that it also happens along a different axis: the learning rate. We give a mechanistic account of how this happens and under what conditions it appears. We also separate it from other large-learning-rate effects. The Maximal Update Parametrization (µP) is widely thought to make the optimal learning rate stay the same as width changes, so it can be tuned on small models and reused on large ones. We show that on some datasets this transfer fails due to the learning rate double descent. As width increases, the optimal learning rate jumps from the low-learning-rate regime to the high-learning-rate regime. As a result, a learning rate tuned on a small proxy model can lead to clearly worse choices at scale.

Primary area:
learning theory
Keywords:
double descent, feature learning, hyperparameter tuning, learning rate, maximum update parameterisation, generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cmRqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper discovers and analyzes double descent in the learning rate curve (performance versus LR); it tries to provide an explanation for why this behavior occurs, and analyzes why MuP transfer can fail because of it.

Note that, I could not understand large parts of this submission.

Critical strengths

This paper tries to understand this new double descent phenomema. Instead of proposing a new method, this paper tries to explain a new observation.

The phenomema seems to be new and novel

The paper has a fair section regarding limitations of the analysis.

Critical weaknesses

Figure 1 was not referenced in the main body, which made me miss it on the first pass of reading. It is really essential... Make sure that each figure is referenced in the text. The introduction does mention figure 2, but I think Figure 1 is much more insightful...

Section 3 was, for me, pretty much unreadable. The jargon is pretty dense, and no clear references and definitions were given to help me, either. For example, K was not clearly defined anywhere. The informal Theorem is, for me, not readable. I cannot explain in lay terminology what the (i), (ii) and (iii) conditions mean, at all. It would be good at least that these conditions contain one sentence that is readable to a non-expert. Now I cannot get any take aways from this. Similarly, I could not understand section 4.2, even not an global or informal level. Section 4.1 I could grasp.

Similarly, Section 5 seems explained in an overly complicated way. I believe this can be explained in a much more concise and clear way, which should be pretty clear from Figure 1 already.

Questions for authors
  1. The paper assumes a fixed and constant learning rate schedule. What about the more realistic case of a learning rate that varies; e.g. a cosine learning rate schedule? Or early stopping? In that case, does this behavior still occur?

  2. Are there any realistic scenario's where the MuP transfer could fail because of this pheneomema? Can you characterize when that could be the case? For it it was quite hard to understand if such practically relevant cases exist from Section 5.

  3. I find the setting quite unusual; so line 99 describes that the labels are one-hot, but that Gaussian noise is added to the one-hot vectors? Which other works also study this setting, or did you come up with this setting yourself?

  4. What is a half squared error? A reference would be useful here.

  5. I am not familiar with the notation _{pop}; any pointers for that?

  6. I am not sure why or how in line 106 g was defined.

  7. It would be good to define what is "feature learning regime".

  8. I found the remark that GD is not broadly used very surprising. What are the authors here talking about? I dont think this is true at all.

1 comment

We thank the reviewer for the careful reading and the questions. The revised Pdf is uploaded. New text is in red.

W1

Fig. 1 is now cited in the introduction. It is also a new figure, which shows the two minima and the jump of the tuned learning rate directly.

W2

We now define the kernel K, the feature-learning regime and the other terms in the setup (Sec. 2.1) and the introduction, and each of the conditions (i)–(iii) starts with one sentence in plain words. Section 4.2 is rewritten as a four-step picture with a 2D illustration of the main idea (Fig. 5).

W3

Section 5 is rewritten and shorter. Its main statement (Proposition 5.1) is that µP can keep each of the two minima at its own learning rate as the width grows, but it does not fix which of them is lower, so the best rate can jump from one to the other (Fig. 1, Fig. 7).

Q1

Yes: in the teacher–student problem both minima remain under a cosine, step or warm-up schedule (App. F.4), and for ResNet-18 on CIFAR-10 under a cosine schedule (App. F.2). On clean CIFAR-10 augmentation removes the first minimum and leaves one optimum at a large rate.

Q2

The jump can happen when both minima exist and are close in depth, so that the part of the loss that still changes with width decides which one is lower (Sec. 5). We show it in a controlled teacher–student problem with clean or weakly noisy labels, where a rate tuned on a narrow network costs a few percent of test loss at every larger width, and on CIFAR-10 with heavy label noise and no augmentation, where the lower minimum also switches with width, in the other direction (App. F.6, F.8).

Q3

Our text was unclear here, sorry. Gaussian noise is added only to the regression targets of the teacher–student experiments. In classification the target is the one-hot label, and label noise means that a fraction of the training labels is replaced by another class. Both are standard in the double-descent literature (additive noise in regression and replaced labels in classification, e.g. Nakkiran et al. which the the original double descent paper https://arxiv.org/abs/1912.02292 ). We corrected it.

Q4

It is one half of the squared error, 12∥y^−y∥2\frac{1}{2}\|\hat y-y\|^2; the factor only removes a 2 from the gradient. This is the usual form, e.g. the sum of squares error in Bishop, Pattern Recognition and Machine Learning (2006, Sec. 1.1).

Q5

It stands for the population error: the mean squared error over fresh inputs drawn from the data distribution, as opposed to the average over the training set.

Q6

There g was only a placeholder for an arbitrary function of the input in the definition of the population norm, not a new object.

Q7

Sec. 2.1 now has a short paragraph. In the lazy regime the tangent kernel of the network stays at its initial value, so the network is a linear model on fixed features. In the feature-learning regime the kernel changes substantially during training.

Q8

We meant full-batch GD at a constant rate without momentum. This was imprecise. In practice most models are trained with adaptive methods such as Adam, and recently Muon. If the revision addresses the reviewer's questions and concerns, we kindly ask the reviewer to reconsider the score.

Reviewer_3ALrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies double descent phenomena with respect to learning rate of gradient descent. The paper first shows that as learning rate increases from 0, test error initially decreases, increases around the interpolation threshold, decreases again before finally increasing towards the region of instability. The second dip is shown empirically to be an artifact of discretization (it dissapears for gradient flow). The second dip is also characterized theoretically. Finally, it is shown that the optimal learning rate can change with network width (under mu parameterization).

Critical strengths
  1. The primary strength of this paper is its novelty: this is the first paper to present a careful theoretical and empirical analysis of double descent wrt learning rate in non-trivial neural networks. The phenomenon is particularly interesting because the shape of the error curves is somewhat different than other double descent curves: there are two dips in error, (in the student-teacher setting) a plateau region and a break in the curve when instability occurs. This makes it an interesting object of study, and the paper contributes a number of observations on the phenomenon.
Critical weaknesses
  1. The primary weakness is the lack of intuition provided in section 4 (the section explaining why a higher lr can lower generalization error). The theory appears sound, but the intuition behind why a higher learning rate reduces variance in the bias variance breakdown is not made clear. The decomposition of the kernel into a head component and a tail component has no explained intuition. Several terms are also not adequately explained such as "coherent mode" and "stability cap." The work of understanding intuition of the result is left entirely to the reader.
Questions for authors
  1. Intuitively, why does higher learning rate past the interpolation plateau lower variance (and thereby generalization error)?
2 comments

We thank the reviewer for the positive assessment. The revised PDF is uploaded. New text is in red.

W1

Thank you, we agree. We rewrote Section 4.2 around a four-step picture with a 2D illustration (Fig. 5). The head and tail of the kernel now come after the picture, introduced as the part that the training points can determine and the part that they cannot. The stability ceiling is defined in the introduction now.

Q1

A network that fits noisy labels has to store the noise somewhere in its features.With small steps most of it sits in features that many inputs share, and these carry it to test inputs. A large step is only stable if such shared features are small, so the noise has to move into features that respond to few training inputs and rarely to a test input: the labels are still fitted, but the prediction at a test input depends less on the noise draw. We prove this for SiLU networks (Theorem 4.1) and show it in a 2D example (Fig. 5) now. It covers the part of the variance that comes with the label noise.

Thank you for your response. I remain in favor of acceptance. However, as the other reviewers have also noted, the lack of clarity is a major issue with this paper and must be properly addressed.

Reviewer_jhF1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper shows that in the overparameterized regime, the test loss versus learning rate can have two minima (dips), separated by an order of magnitude in the learning rate value, in ResNet-18 CIFAR-10 image classification and synthetic teacher-student setups. The authors disentangle this phenomenon from existing phenomena, such as epoch-wise double descent, catapult dynamics, and the edge of stability. Next, they analyze the necessary conditions required for this phenomenon to occur (Section 3) and show it requires feature learning and the existence of separated curvature regions with different accuracies. In Section 4, they decompose the test loss into bias and variance terms at the two dips (Section 4.1), and analyze stylized theoretical setups (SiLU networks) to shed light on some aspects of the phenomenon (Section 4.2). Finally, they analyze the implications for learning rate transfer.

Critical strengths

Characterization of Double Descent wrt LR. The paper introduces a new phenomenon where the test loss exhibits double descent wrt learning rate and carefully characterizes it.

Thorough ablations. The authors perform several ablations to quantify the phenomenon. For example, the bias-variance decomposition of the test loss across the two dips and ablations to understand the role of catapults and EoS.

The paper is quite honest about its contributions and limitations.

Critical weaknesses
  • My understanding is that the paper operates in the overparameterization regime very close to the interpolation threshold, and the reported double descent disappears in the useful deep overparameterization regime (see Figure 6). As the authors also pointed out, their results do not apply to the underparameterized regime. Therefore, this double descent in learning rate is only observed in the overparameterization regime for a small width window, near the double descent peak wrt width. This regime is not of practical interest.

  • The paper’s three questions do not hold together. Question 2: Section 4.2 explains why Dip 2 beats the interpolation plateau. Question 3 instead depends on Dip 1 vs. Dip 2 across width, which the theory doesn't cover. Furthermore, Section 5 addresses it only with a fitted law (Eq. 6). Finally, Section 4.1 also finds different mechanisms in different settings: variance reduction in MLPs, bias improvement on CIFAR-10. Overall, its not clear what to take away from the paper apart from the existence of double descent wrt LR.

  • I personally found the paper really hard to read. It introduces several new terms that are not commonly used in the literature to explain the observed results. Furthermore, the paper is quite dense, with at times the text not connecting the sections well. I think it significantly falls short in terms of human readability and would significantly benefit from a careful rewrite.

Questions for authors
  1. Does the double dip in Fig. 2b appear in test accuracy as well, or only in cross-entropy? (CIFAR-10 setting only)
  2. How do you reconcile Fig. 2b with prior plain-GD/SGD results on CIFAR, where a second dip hasn’t been reported?
  3. Does Dip 2 survive data augmentation or regularization in the CIFAR-10 image classification experiment?
  4. Is label noise used in Fig. 1b? Neither the caption nor App. C.10 states it.
  5. How exactly does your jump differ from Kalra & Barkeshli? The main text says it differs but doesn't discuss it. By comparison, Appendix E frames your work as explaining the mechanism behind such switches.
  6. What happens to dip2 at large widths? From Fig 6 and 13, I think it disappears. Does that imply it's a finite-width window effect close to the interpolation threshold?
1 comment

We thank the reviewer for the thoughtful comments and address all points below. The revised PDF is uploaded, with a cleaner theory section and a new result on the µP failure (Proposition 5.1). New text is in red.

Weakness

W1

We agree that overparameterization is necessary. This is condition (iii) in Sec. 3. But the double dip is not confined to the neighbourhood of the interpolation threshold: with label noise the second dip is there at every width we tried, far beyond the threshold (App. F.5), and the width at which the best rate jumps also lies far inside the overparameterized regime. The submitted Fig. 6 shows which of the two dips is lower, not whether they exist. The two minima also appear for ResNet-18 on the full CIFAR-10 training set (Fig. 2), which we do not consider an uninteresting setting. as the referee claims.

W2

We agree that we do not have a general theory: finite-step GD is hard to control. Section 4 presents one mechanism, proved for SiLU networks with noisy labels, and states that the gain of the mean predictor on CIFAR-10 is outside it. Section 5 no longer rests on the fitted law, we added Proposition 5.1. It treats the comparison of Dip 1 and Dip 2 directly and shows that µP can keep each minimum in place without fixing which one is lower. Beyond existence, the takeaway is what creates the second dip: it is absent under gradient flow and for fixed kernels, and imposing the stability ceiling with stable steps keeps it (Sec. 3).

W3

We agree and rewrote the theoretical sections for readability. See the new pdf. Section 4.2 now explains in four steps, with a 2D illustration (Fig. 5), how a high learning rate changes which solution GD reaches and when this improves generalization.

Questions

Q1

Yes. With label noise both the test loss and the test error (accuracy) have two clear minima. With clean labels the two minima are pronounced in the loss, while in test error the first minimum is more shallow (App. F.1).

Q2

We see two reasons why it was not reported. The two minima lie several decades apart in the learning rate, further than a usual sweep covers, and standard recipes use augmentation, which remove the first minimum and leave a single optimum at a large rate (Sec. 3.2, App. F.2). The optimum that standard training finds is therefore our high-rate one, the first minimum shows only without augmentation and with a sweep that reaches very small rates.

Q3

Yes for regularization: with weight decay or a cosine schedule both minima remain (App. F.2). Augmentation removes the first minimum, not the second, so a single optimum at a large learning rate is left. With label noise every recipe we tried, including a standard one with augmentation, shows a first minimum and a second descent, although with augmentation that descent no longer fits the training set. We think this is very interesting as this was not reported before in the literature.

Q4

No, that figure uses clean labels.

Q5

They report that the best learning rate changes sharply when the width crosses the model-wise double-descent transition, so they found a related failure of transfer. Our jump is a different one: it happens between two widths far inside the overparameterized regime, because the learning-rate curve at a fixed width has two minima and the lower one switches. Sec. 5 gives the account of when the switch happens.

Q6

No: with label noise the second dip is present at every width we tried, far beyond the interpolation threshold (App. F.5). With clean labels both dips become shallower as width grows, but for a different reason: under µP the variance over initializations that they remove goes to zero, so there is less and less to gain. What is confined to a window of widths is which dip is lower, and therefore the jump of the best rate.

If the revision addresses the reviewer's questions and concerns, we kindly ask the reviewer to reconsider the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.