- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper discovers and analyzes double descent in the learning rate curve (performance versus LR); it tries to provide an explanation for why this behavior occurs, and analyzes why MuP transfer can fail because of it.
Note that, I could not understand large parts of this submission.
Critical strengths
This paper tries to understand this new double descent phenomema. Instead of proposing a new method, this paper tries to explain a new observation.
The phenomema seems to be new and novel
The paper has a fair section regarding limitations of the analysis.
Critical weaknesses
Figure 1 was not referenced in the main body, which made me miss it on the first pass of reading. It is really essential... Make sure that each figure is referenced in the text. The introduction does mention figure 2, but I think Figure 1 is much more insightful...
Section 3 was, for me, pretty much unreadable. The jargon is pretty dense, and no clear references and definitions were given to help me, either. For example, K was not clearly defined anywhere. The informal Theorem is, for me, not readable. I cannot explain in lay terminology what the (i), (ii) and (iii) conditions mean, at all. It would be good at least that these conditions contain one sentence that is readable to a non-expert. Now I cannot get any take aways from this. Similarly, I could not understand section 4.2, even not an global or informal level. Section 4.1 I could grasp.
Similarly, Section 5 seems explained in an overly complicated way. I believe this can be explained in a much more concise and clear way, which should be pretty clear from Figure 1 already.
Questions for authors
-
The paper assumes a fixed and constant learning rate schedule. What about the more realistic case of a learning rate that varies; e.g. a cosine learning rate schedule? Or early stopping? In that case, does this behavior still occur?
-
Are there any realistic scenario's where the MuP transfer could fail because of this pheneomema? Can you characterize when that could be the case? For it it was quite hard to understand if such practically relevant cases exist from Section 5.
-
I find the setting quite unusual; so line 99 describes that the labels are one-hot, but that Gaussian noise is added to the one-hot vectors? Which other works also study this setting, or did you come up with this setting yourself?
-
What is a half squared error? A reference would be useful here.
-
I am not familiar with the notation _{pop}; any pointers for that?
-
I am not sure why or how in line 106 g was defined.
-
It would be good to define what is "feature learning regime".
-
I found the remark that GD is not broadly used very surprising. What are the authors here talking about? I dont think this is true at all.
We thank the reviewer for the careful reading and the questions. The revised Pdf is uploaded. New text is in red.
W1
Fig. 1 is now cited in the introduction. It is also a new figure, which shows the two minima and the jump of the tuned learning rate directly.
W2
We now define the kernel K, the feature-learning regime and the other terms in the setup (Sec. 2.1) and the introduction, and each of the conditions (i)–(iii) starts with one sentence in plain words. Section 4.2 is rewritten as a four-step picture with a 2D illustration of the main idea (Fig. 5).
W3
Section 5 is rewritten and shorter. Its main statement (Proposition 5.1) is that µP can keep each of the two minima at its own learning rate as the width grows, but it does not fix which of them is lower, so the best rate can jump from one to the other (Fig. 1, Fig. 7).
Q1
Yes: in the teacher–student problem both minima remain under a cosine, step or warm-up schedule (App. F.4), and for ResNet-18 on CIFAR-10 under a cosine schedule (App. F.2). On clean CIFAR-10 augmentation removes the first minimum and leaves one optimum at a large rate.
Q2
The jump can happen when both minima exist and are close in depth, so that the part of the loss that still changes with width decides which one is lower (Sec. 5). We show it in a controlled teacher–student problem with clean or weakly noisy labels, where a rate tuned on a narrow network costs a few percent of test loss at every larger width, and on CIFAR-10 with heavy label noise and no augmentation, where the lower minimum also switches with width, in the other direction (App. F.6, F.8).
Q3
Our text was unclear here, sorry. Gaussian noise is added only to the regression targets of the teacher–student experiments. In classification the target is the one-hot label, and label noise means that a fraction of the training labels is replaced by another class. Both are standard in the double-descent literature (additive noise in regression and replaced labels in classification, e.g. Nakkiran et al. which the the original double descent paper https://arxiv.org/abs/1912.02292 ). We corrected it.
Q4
It is one half of the squared error, 21∥y^−y∥2; the factor only removes a 2 from the gradient. This is the usual form, e.g. the sum of squares error in Bishop, Pattern Recognition and Machine Learning (2006, Sec. 1.1).
Q5
It stands for the population error: the mean squared error over fresh inputs drawn from the data distribution, as opposed to the average over the training set.
Q6
There g was only a placeholder for an arbitrary function of the input in the definition of the population norm, not a new object.
Q7
Sec. 2.1 now has a short paragraph. In the lazy regime the tangent kernel of the network stays at its initial value, so the network is a linear model on fixed features. In the feature-learning regime the kernel changes substantially during training.
Q8
We meant full-batch GD at a constant rate without momentum. This was imprecise. In practice most models are trained with adaptive methods such as Adam, and recently Muon. If the revision addresses the reviewer's questions and concerns, we kindly ask the reviewer to reconsider the score.