- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes two separate expressions for the "LR gradient", that is, the first-derivative of an LR vs Final Loss sweep curve, which have previous only been characterised empirically. One formula is a theoretical prediction which relies on an Edge of Stability assumption, and the other is a variant of implicit differentiation approaches to hypergradients, utilising time averaging. The authors explain and demonstrate the explanatory power of both approaches, with their predictions widely matching empirically plotted LR gradient values. The EoS approach is particularly useful when used in conjunction with results from other EoS literature, whilst the implicit approach allows layerwise factorisation to understand layerwise effects.
Critical strengths
-
The characterisation of the "LR gradient" is, to my knowledge, novel, and a useful new introduction in deep learning theory. The authors clearly demonstrate the connections to existing theory, and how this line of enquiry is useful to the theory community.
-
The paper is very well written, explaining their arguments and results in a logical order, making it quite an easy paper to follow despite potentially technical content.
-
Notes:
- I followed the majority of the main text argument but did not check proofs in the appendix.
Critical weaknesses
-
I have no serious issues with the paper.
-
Non-critical weaknesses
- A few more details of the model setups for the various figures in the main text would be beneficial.
- I wasn't quite sure what the relevance of Figure 7 and its associated paragraph was to the rest of the work. Some minor writing changes might make this more obvious.
Questions for authors
-
I have no questions whose answers would materially affect my assessment of the paper.
-
Non-critical questions / suggestions
- How does Figure 7 link to the rest of the discussion about the LR gradients?
- Page 8 is very laggy. It may be worth checking whether any of the content on that page is causing it to load slowly.
- A potentially relevant related work that is not discussed is "Function Space Learning Rates" (Milsom et al 2025): https://arxiv.org/abs/2502.17405. In that paper, the authors consider an empirical method to estimate the contribution of a single weight update to the change in the network's logits (and then use this to automatically tune LRs to enable hyperparameter transfer). They also investigate layerwise effects (see their Figure 1), and I wonder if there is a relevant connection to the LR gradient work. The key aspects in which their work differs from LR gradient are:
- Their work revolves around estimation of impractically expensive first-order estimates by using random projections.
- They consider the magnitude of logit changes, instead of loss changes (though it would be trivial to change their work to focus on loss instead).
- They consider the effect of a single optimiser step, rather than the aggregate effect over the entire training run.
- In particular, I wonder if the techniques proposed in the LR gradient work may applied to study functional / logit changes instead of the final loss, or whether their work could be extended to consider entire trajectory effects of learning rates, as in the LR gradient paper.
2 comments
Thank you for your response. I have looked at the other reviews and still do not see any serious problems with the paper that would make me reconsider my score, so I maintain my rating of Clear Accept, and would like to emphasise this to the area chair. In particular, some other reviews seem to give very harsh scores for soundness despite not clearly explaining where serious errors in the analysis have been made.
We thank the reviewer for the positive assessment and for recognizing the novelty of our LR-gradient analysis.
Thank you for the suggestion. We have added key training details from Appendix D to the captions of the corresponding figures in the main text.
Figure 7 in the original submission (now Figure 8) aims to further test the importance of the output layer suggested by Figure 6. Our layer-wise FP-LR analysis shows that the final layer (classifier) makes a large contribution to LR sensitivity in ResNet.
To examine this finding from another perspective, we change the output multiplier, which rescales the final logits and is commonly used in muP (Appendix D.3). Since this multiplier is also multiplied to the parameter gradient through the differentiation, it effectively changes the LR of the output layer. Figure 8 shows that changing the output multiplier substantially affects accuracy and LR transfer across widths, while changing the other layer’s multiplier has little effect. These results further highlight the importance of the output layer in shaping LR transfer curves.
Thank you for pointing this out. We have fixed Figure 6 to reduce the PDF rendering load.
Thank you for pointing out this connection. Their work measures logit changes caused by single parameter updates, whereas our LR gradient measures how the final loss depends on the LR through the training process.
Our FP-LR formulation could also be used to estimate the sensitivity of logits to the LR. For the logits f(θ), we can write
dηdf(θˉ)=Jf(θˉ)dηdθˉ,
where θˉ=⟨θt⟩ denotes the time-averaged parameters and Jf is the Jacobian of the logits with respect to the parameters. The parameter sensitivity can then be approximated using our FP-LR formulation in Section 4. We have not tested this extension, but it would be an interesting direction for future work.
In contrast, our current EoS-LR formula cannot be directly applied to logits, since its derivation relies on the optimality conditions of constrained loss minimization.
We agree that extending function-space learning rates to account for the entire training trajectory would be an interesting direction. However, simply accumulating the amplitude of individual updates, as proposed by Milsom et al. (2025), may lead to large discrepancies from the true LR gradient of the final loss or logits. Moreover, using LR gradients to adjust the LR during training would introduce feedback, making the LR gradient calculation non-trivial.
Nevertheless, combining our approach with their function-space analysis, for example by studying how the final logits depend on the LR through the entire training trajectory, would be an interesting direction for future work. We hope the community will explore this connection further.
You have correctly recognized the significance of our work, particularly the novelty of the LR gradient perspective in deep learning, which several other reviewers seem to have overlooked. We would greatly appreciate it if you could help persuade the other reviewers of this point during the discussion phase.