acceptodds
Under review as a conference paper at ICLR 2027

Measuring the local shape of learning-rate transfer curves

Abstract

Understanding the learning-rate (LR) dependence of trained model performance is essential for better optimization of neural networks, but the appropriate size of the LR depends on learning dynamics, architecture, and their hidden biases, and thus our understanding of this complex dependence remains limited. The behavior of the loss curve as a function of the LR has recently been observed in extensive experiments on LR transfer across different settings, such as model sizes, and it is becoming more important to explore methodologies for quantitatively characterizing the shape of this LR curve. In this work, we provide two relational equations that give a good empirical explanation of the gradient of the LR transfer curve, i.e., the LR gradient. One is a theoretical formula that holds at the edge of stability, and the other is an approximate expression obtained by naively treating this gradient as a hyper-parameter gradient. Using these two relational equations in a complementary manner, we show a lower bound on the gradient of the curve and a zero gradient under specific conditions for normalization layers in theory. The approximate expression also applies to a wide range of experimental settings and allows us to decompose the LR gradient into layer-wise components and quantify the strong contribution of specific layers to the curve shape. Thus, this serves as a foundation for better understanding the internal mechanisms of LR dependence.

Primary area:
optimization
Keywords:
learning-rate transfer, hyper-parameter tuning, hypergradient, edge of stability, sharpness, science of deep learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_3tJxOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes two separate expressions for the "LR gradient", that is, the first-derivative of an LR vs Final Loss sweep curve, which have previous only been characterised empirically. One formula is a theoretical prediction which relies on an Edge of Stability assumption, and the other is a variant of implicit differentiation approaches to hypergradients, utilising time averaging. The authors explain and demonstrate the explanatory power of both approaches, with their predictions widely matching empirically plotted LR gradient values. The EoS approach is particularly useful when used in conjunction with results from other EoS literature, whilst the implicit approach allows layerwise factorisation to understand layerwise effects.

Critical strengths
  • The characterisation of the "LR gradient" is, to my knowledge, novel, and a useful new introduction in deep learning theory. The authors clearly demonstrate the connections to existing theory, and how this line of enquiry is useful to the theory community.

  • The paper is very well written, explaining their arguments and results in a logical order, making it quite an easy paper to follow despite potentially technical content.

  • Notes:

    • I followed the majority of the main text argument but did not check proofs in the appendix.
Critical weaknesses
  • I have no serious issues with the paper.

  • Non-critical weaknesses

    • A few more details of the model setups for the various figures in the main text would be beneficial.
    • I wasn't quite sure what the relevance of Figure 7 and its associated paragraph was to the rest of the work. Some minor writing changes might make this more obvious.
Questions for authors
  • I have no questions whose answers would materially affect my assessment of the paper.

  • Non-critical questions / suggestions

    • How does Figure 7 link to the rest of the discussion about the LR gradients?
    • Page 8 is very laggy. It may be worth checking whether any of the content on that page is causing it to load slowly.
    • A potentially relevant related work that is not discussed is "Function Space Learning Rates" (Milsom et al 2025): https://arxiv.org/abs/2502.17405. In that paper, the authors consider an empirical method to estimate the contribution of a single weight update to the change in the network's logits (and then use this to automatically tune LRs to enable hyperparameter transfer). They also investigate layerwise effects (see their Figure 1), and I wonder if there is a relevant connection to the LR gradient work. The key aspects in which their work differs from LR gradient are:
      • Their work revolves around estimation of impractically expensive first-order estimates by using random projections.
      • They consider the magnitude of logit changes, instead of loss changes (though it would be trivial to change their work to focus on loss instead).
      • They consider the effect of a single optimiser step, rather than the aggregate effect over the entire training run.
      • In particular, I wonder if the techniques proposed in the LR gradient work may applied to study functional / logit changes instead of the final loss, or whether their work could be extended to consider entire trajectory effects of learning rates, as in the LR gradient paper.
2 comments

We thank the reviewer for the positive assessment and for recognizing the novelty of our LR-gradient analysis.

W1 (Model settings)

A few more details of the model setups for the various figures in the main text would be beneficial.

Thank you for the suggestion. We have added key training details from Appendix D to the captions of the corresponding figures in the main text.

W2 / Q1 (Figure 7)

How does Figure 7 link to the rest of the discussion about the LR gradients?

Figure 7 in the original submission (now Figure 8) aims to further test the importance of the output layer suggested by Figure 6. Our layer-wise FP-LR analysis shows that the final layer (classifier) makes a large contribution to LR sensitivity in ResNet.

To examine this finding from another perspective, we change the output multiplier, which rescales the final logits and is commonly used in muP (Appendix D.3). Since this multiplier is also multiplied to the parameter gradient through the differentiation, it effectively changes the LR of the output layer. Figure 8 shows that changing the output multiplier substantially affects accuracy and LR transfer across widths, while changing the other layer’s multiplier has little effect. These results further highlight the importance of the output layer in shaping LR transfer curves.

Q2 (PDF rendering)

Page 8 is very laggy.

Thank you for pointing this out. We have fixed Figure 6 to reduce the PDF rendering load.

Q3 (Function-space LRs)

A potentially relevant related work is "Function Space Learning Rates" (Milsom et al., 2025).

Thank you for pointing out this connection. Their work measures logit changes caused by single parameter updates, whereas our LR gradient measures how the final loss depends on the LR through the training process.

Whether the techniques proposed in the LR gradient work may be applied to study functional / logit changes instead of the final loss.

Our FP-LR formulation could also be used to estimate the sensitivity of logits to the LR. For the logits f(θ)f(\theta), we can write

df(θˉ)dη=Jf(θˉ)dθˉdη,\frac{d f(\bar{\theta})}{d\eta} = J_f(\bar{\theta})\frac{d\bar{\theta}}{d\eta},

where θˉ=⟨θt⟩\bar{\theta}=\langle\theta_t\rangle denotes the time-averaged parameters and JfJ_f is the Jacobian of the logits with respect to the parameters. The parameter sensitivity can then be approximated using our FP-LR formulation in Section 4. We have not tested this extension, but it would be an interesting direction for future work.

In contrast, our current EoS-LR formula cannot be directly applied to logits, since its derivation relies on the optimality conditions of constrained loss minimization.

Whether their work could be extended to consider entire trajectory effects of learning rates.

We agree that extending function-space learning rates to account for the entire training trajectory would be an interesting direction. However, simply accumulating the amplitude of individual updates, as proposed by Milsom et al. (2025), may lead to large discrepancies from the true LR gradient of the final loss or logits. Moreover, using LR gradients to adjust the LR during training would introduce feedback, making the LR gradient calculation non-trivial.

Nevertheless, combining our approach with their function-space analysis, for example by studying how the final logits depend on the LR through the entire training trajectory, would be an interesting direction for future work. We hope the community will explore this connection further.


You have correctly recognized the significance of our work, particularly the novelty of the LR gradient perspective in deep learning, which several other reviewers seem to have overlooked. We would greatly appreciate it if you could help persuade the other reviewers of this point during the discussion phase.

Thank you for your response. I have looked at the other reviews and still do not see any serious problems with the paper that would make me reconsider my score, so I maintain my rating of Clear Accept, and would like to emphasise this to the area chair. In particular, some other reviews seem to give very harsh scores for soundness despite not clearly explaining where serious errors in the analysis have been made.

Reviewer_52J9OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the local shape of learning-rate transfer curves by analyzing how the final model loss changes with the learning rate. The authors propose two approaches to estimate the LR gradient: EoS-LR, which derives an expression for LR sensitivity using an edge-of-stability constraint and Lagrange multipliers, and FP-LR, which uses a time-averaged fixed-point approximation of the hypergradient. Experiments on CNNs, ResNets, and GPT models show that the proposed estimates often agree with finite-difference measurements and provide insights into the LR sensitivity of different architectures and layers.

Critical strengths

The paper moves beyond studying the optimal LR and instead focuses on the local gradient of the final loss with respect to LR. The EoS-LR formulation provides an interesting connection between training stability and LR sensitivity, while the FP-LR approach offers a complementary way to estimate this quantity. The analysis of normalization and the layer-wise FP-LR decomposition provide useful insights into why certain parameter groups are more sensitive to LR changes. The experimental results across several settings support connecting the proposed mathematical analysis to practical architectural choices.

Critical weaknesses

The connection between the proposed LR-gradient analysis and learning-rate transfer could be demonstrated more directly. Although the paper successfully analyzes local LR sensitivity, it is less clear how these estimates help explain or predict LR transfer across model widths beyond existing empirical observations. In particular, the layer-wise decomposition relies on a block-diagonal Hessian approximation, and its quantitative accuracy is not directly established against per-layer finite-difference measurements. Similarly, the observed scaling is interesting, but the theoretical analysis provides a conditional lower bound rather than establishing the observed scaling exponent.

The direct comparisons between the proposed LR-gradient estimates and finite differences are primarily conducted on relatively small models, while the GPT experiments focus more on scaling behavior and layer-wise sensitivity. It would be helpful to evaluate the estimation accuracy more directly on larger models, particularly for the layer-wise decomposition.

Questions for authors
  1. The EoS-LR derivation assumes that trained parameters solve the constrained optimization problem in Eq. (3). However, satisfying the EoS relation does not necessarily imply the first-order optimality condition used in the proof of Lemma 3.1. Could the authors directly evaluate whether the trained parameters satisfy this stationarity condition and whether deviations from it correlate with EoS-LR estimation errors?

  2. The proposed methods estimate local LR gradients, but it is not yet clear whether these estimates can predict LR-transfer behavior across model widths. Could the authors evaluate whether LR gradients estimated on smaller models can predict the LR sensitivity or transfer behavior of larger, previously unseen widths?

  3. The layer-wise decomposition relies on a block-diagonal Hessian approximation that ignores interactions between layers. Could the authors compare the estimated layer-wise LR gradients with finite-difference estimates obtained by changing the LR of each layer individually?

  4. The main EoS-LR experiments use relatively small models and datasets, including a CNN trained on 8,192 CIFAR-10 examples. Although the paper also includes GPT experiments on FineWeb, the accuracy of the proposed LR-gradient estimators is not directly validated across larger language-model training settings. Could the authors provide more direct comparisons between EoS-LR, FP-LR, and finite-difference estimates on larger datasets and models?

4 comments

We thank the reviewer for the kind comments and questions. We hope that our responses and revisions will resolve some confusions on the scope of our paper, and that additional experiments will address the concerns. We believe no critical flaws remain and would appreciate further discussion or reconsideration of the score.


First, let us answer to W1 and Q2, essentially the same concern on the transfer wrt the width. While our main contribution is to characterize the sensitivity of LR transfer curves rather than LR transfer across model widths, we hope the following response clarifies the meaning of our results.

W1 (LR transfer)

The connection between the proposed LR-gradient analysis and learning-rate transfer could be demonstrated more directly.

We theoretically succeeded to derive the specific condition under which dL/dη=0dL/d\eta = 0 holds using EoS-LR framework, and this result is independent of the width and provides a direct connection to LR transfer. The transfer curve is exactly flat under this condition, implying that the optimal LR remains unchanged across widths.

W1'

how these estimates help explain or predict LR transfer across model widths beyond existing empirical observations

Theorems 5.1–5.2 provide a theoretical connection to LR transfer by showing that the LR gradient vanishes, independently of model width. Theory of existing empirical observations seems a valuable contribution in its own right. Moreover, our results provide novel insight in the following points: (1) the key quantity is the homogeneity of the sharpness, caused by the combination of scale invariance and gradient. (2) Prior studies have empirically highlighted the importance of normalization layers [1], $ \eta \gamma = const. $ [2], and their combination for LR transfer [3]. However, they have not precisely characterized when and why LR transfer becomes easier. Our Theorem 5.2 and eqs. (40-42) clarify when and why this combination matters.

[1] Haas et. al. On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling. NeurIPS. 2025.
[2] Wang & Aitchison. How to set AdamW's weight decay as you scale model and dataset size. ICML. 2025.
[3] Kosson. et. al. Weight Decay may matter more than muP for Learning Rate Transfer in Practice. ICLR. 2026.

Additional remark on the significance of sensitivity analysis for W1: As discussed in Section 6, characterizing transfer across widths beyond local sensitivity requires analyzing higher-order derivatives. We believe that such a comprehensive analysis goes beyond what can reasonably be addressed in a single paper. The first-order sensitivity alone provides meaningful information about the shape of LR transfer curves. Specifically, if the magnitude of LR sensitivity is bounded by a small constant ss over an interval, then the LR transfer curve L(η)L(\eta) satisfies

∣L(η)−L(η′)∣≤s∣η−η′∣,|L(\eta) - L(\eta')| \leq s |\eta - \eta'|,

for any η\eta and η′\eta' in that interval. Thus, small sensitivity implies that the LR curve varies only slightly with the chage of LR (Note that the Lipschitzness is not particularly strong assumptions, compared with existing theoretical studies of LR transfer, such as the strong convexity of LL assumed in [4]).

Q2 (LR gradient at Unseen widths)

Could the authors evaluate whether LR gradients estimated on smaller models can predict the LR sensitivity or transfer behavior of larger, previously unseen widths?

We have also added an order analysis of FP-LR gradient in Appendix A.4. The FP-LR gradient scales as at least O(1)O(1) with width. Thus, understanding how well LR transfers requires a finer-grained analysis within this order-one regime. The suggestion of the current work is that layer-wise dependence matters in this regime and that FP-LR captures aspects of LR transfer that cannot be understood simply by increasing width and examining the resulting curves.

A complete characterization of LR transfer across model widths remains an important direction for future research, and we expect the research community to build on our findings in future work.

[4] Ghosh et. al. Understanding the Mechanisms of Fast Hyperparameter Transfer. ICLR 2026.

W2 (Scaling bound)

the observed scaling is interesting, but the theoretical analysis provides a conditional lower bound rather than establishing the observed scaling exponent.

To our knowledge, no prior work has explained the LR dependence of LR transfer curves through a scaling law. We therefore believe that establishing even a conditional lower bound, which reproduces the experimentally observed exponent of $ -3/2 $ under specific conditions, is already an interesting contribution as an initial step toward understanding the shape of the LR curve. We view proving the tightness of this bound as an important open problem, which we hope will motivate further theoretical work in the community.

Q1 (Stationarity)

satisfying the EoS relation does not necessarily imply the first-order optimality condition used in the proof of Lemma 3.1

If training remains at EoS and converges to a minimum of the loss, the solution must satisfies Eq. (3) and its corresponding first-order optimality condition. Indeed, as shown in Figure. 14, training loss can converge to the fixed point while satisfying the EoS condition. Thus, the parameter does converge to the first-order optimality condition.

Could the authors directly evaluate whether the trained parameters satisfy this stationarity condition and whether deviations from it correlate with EoS-LR estimation errors?

We have already directly examined the gradient alignment corresponding to the stationarity condition you pointed out, using a ResNet trained on the full CIFAR-10 dataset (Figure 11). The condition used in Lemma 3.1 is ∇L+λ∇F=0\nabla\mathcal L+\lambda\nabla F=0, which requires ∇L\nabla\mathcal L and −∇F-\nabla F to be parallel ( ∣cos⁡(∇L,∇F)∣=1|\cos(\nabla\mathcal L,\nabla F)|=1) . Figure 11 directly measures this cosine similarity.

Q3 (Per-layer validation)

compare the estimated layer-wise LR gradients with finite-difference estimates changing the LR of each layer individually

Thank you for your insightful suggestion. We have added new Figure 7, which compares layer-wise LR gradients with finite-difference obtained by varying the LR of each layer individually, for ResNet on CIFAR-10 and GPT on FineWeb. As shown in this figure, we found that the layer-wise LR gradients certainly agree closely with the finite-difference estimates and almost all estimates are within a factor of three of the corresponding finite differences. These results demonstrate that our layer-wise LR-gradient estimates capture the overall trends in LR sensitivity across layers in both ResNet and GPT.

Q4 (Larger models)

a CNN trained on 8,192 CIFAR-10 examples

In addition to the training with 8,192-example subset, we also report experiments using the full CIFAR-10 training set: EoS-LR versus finite differences for ResNet (Appendix C.1, Figure 10), and FP-LR versus finite differences for ResNet8 (Appendices C.2–C.3, Figures 18, 20, and 22).

Although the paper also includes GPT experiments on FineWeb, the accuracy of the proposed LR-gradient estimators is not directly validated across larger language-model training settings

As mentioned in our response to Q3, we have added new Figure 6, which compares layer-wise FP-LR gradients with finite-difference estimates for both ResNet on CIFAR-10 and GPT on FineWeb. Thus, the revised manuscript now has a direct comparison with finite differences for GPT as well.

We do not evaluate EoS-LR gradients for GPT because GPT are typically trained with AdamW using relatively large weight decay, a setting beyond the scope of current EoS theory in the literature. Although prior work has observed EoS behavior in Adam, as discussed in Section 6, its theoretical understanding remains incomplete for practical preconditioned optimizers involving moving averages and damping, particularly AdamW with large weight decay. We therefore refrain from applying EoS-LR to GPT. As noted in Section 6, establishing the exact EoS relation and optimality conditions for practical AdamW remains an important direction for future work in the research field of EoS.

I appreciate the authors for their detailed response, and I have raised my score to 2. However, I remain concerned about the experimental scale and settings. Although the additional results strengthen the empirical validation, the experiments are still conducted on relatively small models and under controlled training conditions. Given the substantial differences between these settings and modern large-scale language models, I am not yet convinced that the proposed analysis can provide meaningful guidance for recent models.

AuthorsOpenReview ↗

Response to the additional reply by Reviewer 52J9

Thank you for considering our response and additional experiments. We would like to further address your concern about the experimental scale and settings.

Experimental scale and settings

We would like to briefly revisit our experimental settings since you might missing them. Our experiments are not limited to artificially simplified conditions. For ResNet, we use Airbench [1], an established CIFAR-10 training baseline that achieves approximately 96% accuracy in just 34.7 seconds on a single A100 GPU, which is a very fast training. We validate EoS-LR and FP-LR against finite differences using this setup. For GPT, we use the nanoGPT training codebase with FineWeb. In the revision, we also added validation of layer-wise LR gradients on GPT. We therefore believe it would be misleading to characterize our experiments as merely being conducted under “controlled training conditions.”

More broadly, studies aimed at understanding phenomena such as EoS and μP commonly use relatively small models. For example, recent papers at top conferences use the following settings.

  • Noci et al. (NeurIPS 2024) [2]: CNNs on CIFAR-10 and a 2-layer Transformer on WikiText-2.
  • Atanasov et al. (ICLR 2025) [3]: Theoretical analysis of simple deep linear networks, along with experiments on MNIST, CIFAR, and TinyImageNet.
  • Kosson et al. (ICLR 2026) [4]: 20-layer LLaMA-style Transformers with widths ranging from 128 to 2048.
  • Ghosh et al. (ICLR 2026) [5]: Synthetic problems and 4-layer LLaMA-style Transformers.

We therefore believe that experimental settings should be judged in light of the research objectives, rather than differences in scale from modern LLMs as a major concern in themselves.

meaningful guidance for recent models

We want to clarify that it is important to distinguish scientific understanding from immediate practical utility. Our goal is to understand the mechanisms of LR sensitivity theoretically and quantitatively, rather than to propose a practical LR tuning method for modern LLMs.

Research on fundamental phenomena in neural network optimization often progresses incrementally toward more realistic settings through the collective efforts of the community. EoS research is a good example: The original work [6] focused on understanding a fundamental phenomenon in gradient descent, and subsequent studies extended this understanding to other more practical settings [7]. Such progress has been made incrementally by the research community, without requiring each paper to demonstrate direct practical utility for the largest models.

Likewise, our work aims to provide a foundation for future studies that may eventually lead to practical training guidance. We believe that direct validation on large-scale LLMs is a valuable direction for future research, but should not be required to recognize the present contribution.

We would appreciate it if you could also take this into account and reevaluate the score alongside the soundness.

References

[1] Keller Jordan. CIFAR-10 Airbench. https://github.com/KellerJordan/cifar10-airbench.
[2] Noci et al. Super Consistency of Neural Network Landscapes and Learning Rate Transfer. NeurIPS 2024.
[3] Atanasov et al. The Optimization Landscape of SGD Across the Feature Learning Strength. ICLR 2025.
[4] Kosson et al. Weight Decay may matter more than μP for Learning Rate Transfer in Practice. ICLR 2026.
[5] Ghosh et al. Understanding the Mechanisms of Fast Hyperparameter Transfer. ICLR 2026.
[6] Cohen et al. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. ICLR 2021.
[7] Cohen et al. Adaptive Gradient Methods at the Edge of Stability. NeurIPS 2022.

Reviewer_mQraOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The submission studies the derivative of training loss with respect to learning rate (LR) to characterization of LR-transfer curves. It introduces two estimators: an EoS-based metric derived by assuming EoS constraint constraint at the end of training, and a heuristic time-averaged fixed-point approximation to the LR hypergradient. The authors then use these metrics to analyze the effects of normalization and weight decay, and estimate layer-wise contributions to LR sensitivity.

Critical strengths

Understanding how the training loss varies with the learning rate is an important problem, with implications for both hyperparameter tuning and LR transfer across model sizes. The authors introduce two complementary estimators of the local slope of the loss–LR curve: one based on an edge-of-stability constraint and another based on a time-averaged fixed-point approximation to the LR hypergradient. These provide a quantitative perspective beyond simply inspecting LR sweeps.

The authors also use the proposed quantities to investigate architectural sources of sensitivity, including layer-wise learning rates and input and output scaling constants. Connecting overall LR sensitivity to individual parameter groups is a useful diagnostic direction, as it can help identify which components warrant closer investigation when tuning learning rates or studying transfer behavior.

Critical weaknesses

The practical utility and insight provided by the proposed metrics are not sufficiently demonstrated. Although estimating the local slope of a loss–LR curve is a reasonable objective, the paper does not convincingly establish what these estimates enable beyond conventional LR sweeps or existing qualitative understanding. For example, it would strengthen the contribution to show that the metrics reduce the computational cost of hyperparameter tuning or reliably predict which parameter groups should be tuned independently. The architectural experiments largely agree with known results, such as sensitivity to output-layer scaling and reduced sensitivity to input scaling in normalized layers. More compelling evidence would include non-obvious sensitivity results, or conditions under which established expectations break down. Without such evidence, the significance of the proposed measurements remains unclear.

Some notations are confusing: the authors use λ\lambda for the lagrange multipliers and the eigenvalues of the Hessian. Furthermore, the s

Questions for authors
  1. What does the gradient of loss wrt LR tells us? Prior work [1], for example, has shown that the curvature of loss wrt LR is related to reliability of transfer.
  2. What does the scaling of LR gradient tells us?

[1] https://arxiv.org/abs/2605.21486

2 comments

We thank the reviewer for taking the time to follow our paper and provide feedback.

Reply to Weaknesses

W1 (Practical utility)

the paper does not convincingly establish what these estimates enable beyond conventional LR sweeps or existing qualitative understanding

We would like to clarify the goal of our paper.
Our goal is to better understand the mechanisms behind LR sensitivity, rather than to propose a practical LR tuning method or to demonstrate a reduction in LR-tuning cost. Many studies in neural network research have similarly focused on theoretical or scientific understanding without proposing practical methods, including [1, 2, 3] cited in our paper.

Our analysis provides insights beyond conventional LR sweeps. For example, Eq. (6) relates the LR gradient to EoS and Theorems 5.1 and 5.2 identify conditions under which the LR gradient vanishes, including an optimizer-dependent condition in Theorem 5.2. The FP-LR gradient also allows us to quantify layer-wise contributions to LR sensitivity, with empirical comparisons against finite differences in Figure 7.

These results help explain when and why LR sensitivity arises or vanishes, and they clearly fo beyond the existing qualitative understanding.

[1] Barrett and Benoit Dherin. Implicit gradient regularization. In International Conference on Learning Representations, 2021.
[2] Smith. et. al. On the origin of implicit regularization in stochastic gradient descent. ICLR. 2021.
[3] Noci. et. al. Super consistency of neural network landscapes and learning rate transfer. NeurIPS, 2024.

W2 (Novelty)

The architectural experiments largely agree with known results …

More compelling evidence would include non-obvious sensitivity results, or conditions under which established expectations break down.

As is described in our answer to W1, our analysis identifies previously unrecognized theoretical conditions under which LR sensitivity vanishes. Let us re-emphasize in more detail three findings that go beyond existing observations.

(1) While concurrent work [4] has also highlighted the importance of embedding layers for LR transfer, we find that the classifier in ResNet and the output and MLP input layers in GPT also show large LR sensitivity (Figures 4 and 6). Figure 7 further shows that our layer-wise FP-LR estimates broadly agree with finite-difference measurements. These results provide a quantitative view of layer-wise sensitivity beyond previous studies of parameterization.

(2) Previous studies have mainly examined LR transfer through parameterization choices or layer-wise LR scaling across widths. In contrast, we sweep the LR of one selected layer while keeping the other LRs fixed (Figure 4). This directly measures how sensitive the loss is to each layer's LR. For example, we quantify how much more sensitive the ResNet classifier is than the other layers. Our layer-wise FP-LR estimates also broadly agree with finite differences (Figure 7).

(3) Prior studies have empirically highlighted the importance of normalization layers [5], ηγ=const.\eta\gamma=const. [6], and their combination for LR transfer [7]. However, they have not directly characterized when and why these factors reduce LR sensitivity. Our Theorem 5.2 shows that, the LR gradient vanishes when the EoS condition has homogeneity degree a=1a=1. Eqs. (40–42) further characterize the LR gradient without scale invariance. These results provide a theoretical explanation beyond the previous empirical observations.

Finally, even if some of our focuses overlap with known empirical observations, providing a theoretical explanation for such empirical phenomena is a significant contribution in its own right.

[4] Karla. et. al. Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate. 2026.
[5] Haas et. al. On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling. NeurIPS. 2025.
[6] Wang & Aitchison. How to set AdamW's weight decay as you scale model and dataset size. ICML. 2025.
[7] Kosson. et. al. Weight Decay may matter more than muP for Learning Rate Transfer in Practice. ICLR. 2026.

W3 (Notation)

the authors use λ for the lagrange multipliers and the eigenvalues of the Hessian

Thank you for pointing this out. We have changed the notation for the Lagrange multiplier from λ\lambda to ν\nu to avoid confusion with the Hessian eigenvalue λmax⁡\lambda_{\max}.

Reply to Questions

Q1 (LR gradient)

What does the gradient of loss wrt LR tells us? Prior work [1], for example, has shown that the curvature of loss wrt LR is related to reliability of transfer.

For instance, the LR gradients tells us
(1) Theorems 5.1 and 5.2 identify conditions under which the LR gradient vanishes, thereby clarifying when LR transfer can hold independently of width.
(2) Layer-wise LR gradients can also help identify which layers are most sensitive to changes in their LRs. For example, Figure 6 shows that the classifier in ResNet has a much larger layer-wise FP-LR gradient magnitude than the other layers, suggesting that its LR is worth tuning separately. Such profiles may also help identify which layers to examine when tuning LRs in new architectures.
(3) If you are interested in more general evaluation of LR gradient, the first-order sensitivity alone provides meaningful information about the shape of LR transfer curves. Specifically, if the magnitude of LR sensitivity is bounded by a small constant ss over an interval, then the LR transfer curve L(η)L(\eta) satisfies

∣L(η)−L(η′)∣≤s∣η−η′∣,|L(\eta) - L(\eta')| \leq s |\eta - \eta'|,

for any η\eta and η′\eta' in that interval. Thus, small sensitivity implies that the LR curve varies only slightly with the chage of LR (Note that the Lipschitzness is not particularly strong assumptions, compared with existing theoretical studies of LR transfer, such as the strong convexity of LL assumed in [8]).

We agree that the curvature of the LR transfer curve is also important for transfer robustness, as discussed in your reference [1]. Near an optimal LR, where the first derivative vanishes, curvature describes how quickly the loss increases as the LR moves away from the optimum. Our analysis complements this view by studying first-order sensitivity and its relation to EoS, normalization, and weight decay.

[8] Ghosh et. al. Understanding the Mechanisms of Fast Hyperparameter Transfer. ICLR 2026.

Q2 (Scaling)

What does the scaling of LR gradient tells us?

We reported the empirical scaling of the LR gradient in order to examine how well our two expressions capture the LR dependence.

In Section 5.2, we show that the EoS-LR gradient provides a more specific quantitative characterization of the observed η−3/2\eta^{-3/2} scaling than the FP-LR gradient. Under certain assumptions, EoS-LR yields a lower bound of Ω(η−3/2)\Omega(\eta^{-3/2}), consistent with the experimental results, whereas the bounds from FP-LR do not directly reproduce this scaling.

Moreover, the derivation relies in part on a gradient-norm scaling that has also been reported in studies of implicit gradient regularization. Both implicit gradient regularization and EoS are known to arise from the discreteness of the update rule. This suggests that it is important to pay attention to discrete-time learning dynamics for understanding the shape of LR transfer curves.

Reviewer_TeBcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tries to answer an important question: How does the final loss of a model change with respect to the learning rate? To answer this, the authors propose two approaches: EoS-LR, a formulation based on the edge of stability, and FP-LR, based on an approximation of the learning rate hypergradient. The authors validate their approaches across multiple architectures (GPT, CNN, ResNet) by comparing the estimated learning rate gradients against those obtained using their proposed framework. The authors also analyze interesting ablations on the scaling behavior of the learning rate gradient, as well as layer-wise sensitivity to the learning rate.

Critical strengths
  • The problem statement itself is very interesting and important. Understanding the learning rate curve can lead to better optimization
  • The empirical study and the insights provided in the paper are quite intensive and sufficiently provide evidence for the validity of the solution.
  • The layerwise ablation over the learning rate sensitivity is very interesting and offers an important perspective into different components' learning rate sensitivity.
Critical weaknesses
  • The paper presents the important algorithm but the authors rely on heavy assumption for both the methods. It would be good if the authors make assumption made and either provide more theoretical or empirical evidences into why the assumption holds
Questions for authors
  • In the following sections (Section 5.2, L410–416; Appendix A1, L710–717; Appendix B1.1, L810–812), some assumptions, such as the one in L810, imply conditional independence between the Hessian and the hypergradient, which requires further clarification or empirical evidence. The authors should provide more details about the assumptions and conditions required for the framework.
  • The authors discuss the previous method in [1]. However, it would be interesting to compare their proposed method with [1] in terms of the accuracy of gradient estimation and computational cost. Even a comparison on extremely small models would be useful to assess how well the gradient estimates compare between the two methods.
  • The paper presents layer-wise gradient estimation (Section 5.3) and shows interesting results on layer-wise sensitivity. However, the analysis relies on a diagonal approximation of the Hessian, which ignores inter-layer dependencies. Could the authors compare their hypergradient estimates with those obtained using the full Hessian and a block-diagonal approximation? Additionally, visualizing the Hessian eigenvectors on a smaller model could provide further validation
  • The paper primarily focuses on linear learning rate schedulers. However, it remains unclear whether the findings generalize to other commonly used schedulers, such as cosine decay. It would be interesting to evaluate FP-LR under different learning rate schedulers.
  • Finally, it would be helpful to provide more insight into how the proposed estimator performs near zero-crossing points. A finer-grained comparison with finite-difference estimates around these regions could provide further insight into the accuracy

References

  1. Maclaurin, Dougal, David Duvenaud, and Ryan Adams. "Gradient-based hyperparameter optimization through reversible learning." International conference on machine learning. PMLR, 2015.
2 comments

We thank the reviewer for the helpful comments and for recognizing the value of our layer-wise analysis.

Concerns on assumptions

W1 the authors rely on heavy assumption for both the methods

Q1 In the following sections (Section 5.2, L410–416; Appendix A1, L710–717; Appendix B1.1, L810–812), some assumptions, such as the one in L810, imply conditional independence between the Hessian and the hypergradient

(1) Reply for the case of FP-LR

The only strong assumption is the conditional independence between the Hessian and the hypergradient (Appendix B1.1, L810–812). However, this is not a weakness of this paper. Let us first explain why. FP-LR is proposed as a heuristic that closely matches experimental results. This empirical agreement is our main contribution, while the assumption is used only to provide an interpretation of the formula. Thus, FP-LR does not require the same level of mathematical rigor as EoS-LR, and how strong the assumption is does not affect our main contribution.

While we do not consider the assumption is essential to our contribution, it would be imformative to touch on the prior empirical work [1] that has reported that neural networks can be locally approximated as quadratic functions with a constant Hessian. Under this condition, the conditional independence between the Hessian and the hypergradient naturally holds, supporting the reasonableness of this assumption. We have referred to this work in B.1.1 in the main text.

[1] Meterez. et. al. A Defense of the Quadratic Model. arXiv: 2607. 21716.

(2) Reply for the case of EoS-LR

The assumptions underlying EoS-LR are substantially weaker than those in prior work.

(2-1). Assumptions in Section 5.2, L410–416

We assume that the Hessian is positive semidefinite and that implicit regularization holds (η∣∣∇L∣∣2=const\eta || \nabla L ||^2 = \mathrm{const}). Actually, both are supported by prior work. We have added the missing citations to this paragraph, following your advice.

The condition $\eta \gamma = \mathrm{const} $ has also received considerable attention in recent work and hypothesized as an important factor for the learning-rate transfer. It has been empirically argued not only by [2], but also by [3] and [4].

(2-2). Assumptions in Appendix A1, L710–717

These assumptions are weaker than those used by [2]. Specifically, assuming only bounded gradient norms and independent weight decay in the optimizer leads to the parameter-norm regularization in Eq. (25). By imposing this regularization as a condition, we can establish improved transfer without the strong orthogonality assumption additionally required by [2]. (L709). Moreover, unlike their intuitive argument, our EoS-LR analysis theoretically identifies the conditions under which transfer improves. This is a key contribution of our work. We have revised L710–717 to clarify this distinction from prior work.

[2] Kosson. et. al. Weight Decay may matter more than muP for Learning Rate Transfer in Practice. ICLR 2026.
[3] Wang & Aitchison. How to set AdamW's weight decay as you scale model and dataset size. ICML 2025.
[4] Wortsman. et. al. Small-scale proxies for large-scale Transformer training instabilities. ICLR 2024.

Q2 (Reversible learning). compare their proposed method with [1] in terms of the accuracy of gradient estimation and computational cost

We have added a figure in the Appendix (Figure.22) which evaluates the reversible training on a CNN with CIFAR-10. As shown in the figure, the LR-gradient estimated by reversible training diverge at large LRs, similarly to those with standard forward-mode differentiation, and become orders of magnitude larger than the finite-difference estimates. In contrast, the method provides stable LR-gradient estimates at LRs far below the optimal LR. (10 or 100 times lower than the optimal LR.)

Since our paper focuses on reliably measuring LR transfer curves in the large-LR regime of neural network training, such small-LR settings are outside the scope of our study.

Q3 (Hessian approximation) compare their hypergradient estimates with those obtained using the full Hessian and a block-diagonal approximation.

Thank you for this suggestion. We have added a Figure.31 for ResNet on CIFAR-10 comparing finite-difference (which corresponds to full Hessian; obtained by varying the LRs of all layers simultaneously) with the sum of the layer-wise FP-LR gradients using layer-wise approximations of the Hessian. As shown in the new figure, the summed layer-wise FP-LR gradients successfully predict the FP-LR without layerwise approximation. This demonstrate that the block-diagonal approximation can capture the overall LR sensitivity.

Moreover, following Q3 from Reviewer 52J9, we have added a comparison between the layer-wise LR gradients and finite-difference estimates obtained by varying each layer's LR individually in Figure 6. This provides further empirical validation of our layer-wise LR-gradient estimates.

Q4 (LR schedules) evaluate FP-LR under different learning rate schedulers

We empirically confirmed that our findings generalize to different schedulers. We use cosine annealing in the ResNet8 experiments (Appendix D.2). Comparisons between FP-LR and finite differences under this setting are shown in Figures 20, 23, and 25. We also want to clarify that we chose Constant + Linear decay because it is widely used in recent large-scale model training and is becoming even more popular [5].

[5] Wen et. al. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective. ICLR 2025.

Q5 (Zero crossings) it would be helpful to provide more insight into how the proposed estimator performs near zero-crossing points

Near the zero crossing, the LR transfer curve is nearly flat, and the sign of the LR gradient is unstable. Importantly, even finite differences become unstable in this region, so this is not an issue with our proposed methods. For example, in Fig. 1 at width 512, the loss is nearly identical for LRs between 0.25 and 0.5, and the LR transfer curve fluctuates. As a result, the variance across seeds is also very large. Our goal is not to obtain the precise value of optimal LR, but simply to capture the overall shape of the LR transfer curve. Therefore, we believe it is sufficient for the LR gradient to work correctly for LRs below 0.25 (Note that as discussed in Q2, unlike existing methods, our method accurately captures the LR transfer curve up to the point where it flattens near the optimal LR).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

Dear Reviewers and AC,

We thank the reviewers for their constructive feedback and for recognizing the paper’s strengths, including the novel perspective of LR gradients (3tJx, mQra), the connection between EoS and LR sensitivity and insights from layer-wise analysis (52J9, 3tJx), and the strong empirical evidence (TeBc).

We addressed the reviewers’ concerns through the following experiments and clarifications.

  • Layer-wise approximation (52J9, TeBc): We added direct comparisons with finite differences for layer-wise LR gradients on ResNet and GPT (Fig. 7). Almost all estimates are within a factor of three of the measurements. We also confirmed that their sum agrees with FP-LR without the layer-wise approximation (Fig. 31). These results address 52J9's concern about the lack of direct validation in more realistic settings, including GPT.

  • Clarification of contributions (mQra, 52J9): We clarified how our existing results go beyond LR sweeps and empirical observations. Theorems 5.1–5.2 identify conditions under which LR sensitivity vanishes, including the roles of normalization and weight decay, while our layer-wise analysis quantifies the sensitivity of individual layers in ResNet and GPT. These results explain when and why LR sensitivity arises or vanishes, rather than merely confirming known observations.

  • Assumptions and validity (TeBc, 52J9): We clarified the assumptions behind EoS-LR and FP-LR and added supporting references. Fig. 11 already examines the gradient alignment associated with the EoS-LR stationarity condition. We also clarified that our normalization and weight-decay analysis does not require the additional orthogonality assumption of prior work. These concerns do not indicate any flaw in our theory.

  • Other revisions (TeBc, 3tJx, mQra): We added a comparison with reversible learning, showing substantial deviations from finite differences at large LRs, resolved the notation ambiguity, and added training details to figure captions.

We believe these experiments and clarifications adequately address the reviewers’ concerns, with no issues remaining that undermine the paper’s score.

If our responses and revisions have resolved these concerns, we would appreciate reconsideration of the overall scores based on the revised manuscript. If serious concerns remain, we welcome further discussion of the specific issues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.