- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces a framework to estimate the "characteristic" learning rate of Adam to navigate the heterogenous loss landscape of neural networks through a statistical physics lens. This framework is cost-efficient requiring only a few gradient computations and a short power iteration at a fixed checkpoint. They validate their framework on a toy landscape and then comprehensive pretraining bechmarks such as CNN and a ResNet-28 on CIFAR-10 and LLaMA on OpenWebMath as well as finetuning benchmarks GPT/LLaMA/BERT on SST-2. Their framework empirically works on Adafactor and AdamW as well.
Critical strengths
-
Learning rate selection of Adam, the flagship optimizer in deep learning, is definitely an important problem to tackle with wide implications. Tuning learning rates is quite arduous and expensive and recently been made cheaper with complex frameworks like . This work takes a different approach that is extricably linked to a statistical physics theoretical framework. Existing alternatives either only bound where training becomes unstable (e.g., edge-of-stability thresholds) or change the optimizer itself (D-Adaptation, Prodigy, DoG). The paper offers a different angle: Adam's second-moment normalization damps noise-driven motion more in high-curvature directions than in low-curvature ones, and the best learning rate is where this contrast is largest. This yields a closed-form physically interpretable prediction which makes the work valuable even where the derivation relies on simplifying assumptions. Such assumptions are acceptable for the sake of tractability at the cost of some approximation errors so long as they are within performance bounds.
-
The method is cost-efficient, requiring only 31 gradient equivalent steps at the beginning of training. Both sensitivity and performance are tracked on a wide variety of relevant benchmarks ranging from image classification (CNN/ResNet28 on CIFAR-10) to transformer pretraining (LLaMA 1B on OpenWebMath) and finetuning (GPT2, BERT, LLaMA on SST-2). It is also clear that it performs better than related methods such as D-Adaptation, Prodigy, L-DoG. Moreover, it generalizes to AdamW and Adafactor which are the optimizers implemented in practice.
Critical weaknesses
-
The framework requires two key assumptions (both of which are addressed by the authors): approximately Gaussian mini-batch noise, and a second moment to which both noise and the deterministic gradient contribute. The authors acknowledge that the predictor holds only at intermediate batch sizes (due to Gaussianity requirement), which is a reasonable limitation for a tractable heuristic. However: a) The valid window is task-dependent (e.g., 4–32 for MNIST-MLP vs. 16–256 for QNLI). It is unclear how a user of this framework is supposed to define "intermediate" a priori which reduces its effectiveness as a pre-hoc metric. b) The Gaussianity check passes if in just 2 of 10 seeds (App. E). This seems to be dubious as I would expect this to be a consistently satisfied as specified in Section 6.
-
The theory requires a) quadratic expansion around a local minimum b) stationary fluctuations within a basin c) a quasi-stationary second moment. However, and are measured "at a fixed checkpoint well before progressive sharpening", with "no optimizer warm-up required". That means initialization for from-scratch models, and a pretrained backbone with a freshly initialized head for fine-tuning. Neither is a minimum of the task loss, and at that point Adam's does not yet exist. App. Fig. 7 shows rising steeply early in training. Since , a 100× change in moves by 10×. This seems to be a fundamental divergence of theory from practice.
-
The flat regime is described by which corresponds to a squared gradient negligible compared with the noise. This could correspond not only to low curvature but also near-stationary when noise dominates. So contains no curvature term and depends only on , and . Because the paper's central claim is a contrast between two regions: the experiments test the formula's output but not the proposed mechanism (see Q4).
-
The derivation works in the Hessian eigenbasis (App. A) but Adam's preconditioner is diagonal in parameter coordinates implying that Adam is not rotation invariant. Pairing per-coordinate and with the raw-Hessian hence requires either that its eigenvector be roughly axis-aligned or that be roughly uniform.This step links the one-dimensional model to Adam's actual update, the dependence is not derived from Adam's dynamics as written.
Questions for authors
Q1 [Clarification]: . How exactly are and estimated? Section 4 only mentions 20 mini-batches and 10 power iterations but I couldn't find the estimator in the appendix. In particular, is per-sample or per-batch, how are the per-coordinate combined into one number and which batch and checkpoint define the Hessian?
Q2 [Experiment]: The derivation assumes the dynamics are near a minimum yet and are measured at initialization or at a pretrained checkpoint. Why does the prediction still work there and how much of the agreement comes from Adam's broad plateau rather than the proposed mechanism? Reporting at a few checkpoints during trainig along with the accuracy each value gives would help settle this. The experiments do not have to be large ones. Moderate scale pretraining (e.g CNN on CIFAR-10) and a finetuning one would suffice.
Q3 [Ablation]: Since Adam is not rotation-invariant, what justifies pairing its per-coordinate statistics with the raw-Hessian ? You already compute 20 gradients, so you could estimate v from them and run the same power iteration on the preconditioned Hessian. More concretely: form from the same 20 gradients, compute the top eigenvalue of with using the same 10 HVPs, and compare the resulting prediction. How does change?
4 comments
W3: This is the purpose of toy model validation. The mobility-contrast mechanism is tested directly in the toy landscapes (Fig. 2 and Appendix Fig. 3), where the mobility contrast is measured and peaks at the predicted . On real networks, the test is the formula output against grid-search optima, across many different models and tasks. We have separated these two kinds of evidence in the Discussion of our revision and stated as “The mobility contrast acts as a probabilistic bias: raising the learning rate increases mobility everywhere, and is the point at which the mobility contrast is maximized. Because high-curvature regions are more sensitive to parameter and sampling perturbations, we highlight that keeping their mobility low relative to that of low-curvature regions increases the chance that training remains in or reaches perturbation-stable regions, and thereby favors generalization. Our experiments support this view through the agreement of with grid-search optima.”
Thank you for your time and effort in reviewing our work. We are really encouraged by your positive assessment, and your questions have helped us improved our clarification.
Thanks for addressing my concerns. I am largely satisfied with the responses and have thus increased soundness from 2 to 3.
Thank you for your positive feedback and for the increased soundness. We sincerely appreciate your time and effort.
Thank you for your very positive assessment. We appreciate your supportive review and for specifying the experiments precisely enough to run as written. All three were run.
Q1/W1: We have added Appendix Algorithm 1, which gives the pseudocode of our estimator, and we also provide the corresponding code in the supplemental. σ is per-sample: variance of mini-batch gradients across K=20 batches λmax is the top eigenvalue of the raw Hessian at the same parameters, from 10 power iterations. The checkpoint is initialization for from-scratch models, or the pretrained weights for fine-tuning. All these detailed information were added in our revision with blue highlighted.
Furthermore, we have tied the evaluation window to measured performance. To assess whether our modest criterion identifies reliable predictions, we compared accuracy gaps inside (IN) and outside (OUT, marked '-') the applicability window (Fig. 4c-d). Gaps are measured relative to the best grid-searched result at the same batch size and training budget. We found that accuracy gaps are larger for OUT than for IN batch sizes (paired t-test, t(19)=5.13, p=5.93×10−5). We also highlighted this diagnostic is optional. When it is used for additional assessment, its repeated noise measurements require 48×10=480 mini-batch gradient evaluations per candidate batch size (Appendix E). The common batch-size window observed across the tested vision and language tasks provides an empirically supported range for direct use of our predictor, which requires approximately 31 GE steps. We have added this information to our revision on Pages 9-10.
Q2/W2: This is a good point, and one we have also considered. In fact, our derivation needs less than a local minimum. The sharp-region analysis requires stationarity only along the dominant Hessian direction u, so that u⊤∇F≈λmaxu⊤(θ−θ⋆). The remaining gradient components enter only through ⟨v⟩, where they must be small relative to the mini-batch noise. We have stated this in Appendix A (also attached here: "A local minimum is sufficient but not necessary. In a narrow valley, the optimizer settles onto the valley floor within a few steps, long before it reaches the lowest point along the valley. Our analysis uses only the fact: it requires stationarity along the dominant eigenvector u, u⊤∇F(θ⋆)=0, so that u⊤∇F(θ)≈λmaxu⊤(θ−θ⋆). The remaining gradient components enter only through Adam's second moment, where they need to be small compared with the mini-batch noise, E[(∇iF)2]≪σi2/b. This is why αc can be measured at initialization or at a pretrained checkpoint, and the noise fraction along u (Appendix~E) checks the condition there"), and the noise-fraction diagnostic (Appendix E) checks the condition point.
Here σ is the mini-batch-induced noise. An important point is that the gradient-noise variance and the curvature co-vary along a direction, as reported in the inverse variance-flatness relation (PNAS, 2021). Thus, αc depends on both, and a change in λmax alone does not determine the change in αc.
For simulations, we have run the suggested experiment (Appendix Fig. 13): αc is re-estimated at checkpoints (training steps) t∈{0,5,10,15,20} for CNN/CIFAR-10 and Qwen3-1.7B/RTE, and training is continued from each checkpoint at αc(t), with 20 seeds each. Over these checkpoints, the estimate taken before training gives a final accuracy at least as high as any later estimate. This is why we can measure once at the start.
Q3/W4: We ran this ablation as specified (Appendix Fig. 14): P=diag(v^)−1/2 is formed from the same 20 gradients, and the top eigenvalue of P1/2HP1/2 is computed with the same 10 HVPs. The preconditioned prediction is lower by factors of 31.8 (CNN/CIFAR-10), 26.4 (1B Transformer), and 5.0 (Qwen3/RTE), placing it well below the grid optimum that the raw-Hessian prediction matches. The reason is that the closure already accounts for Adam's normalization through ⟨v⟩; preconditioning the curvature as well counts it twice. We have now highlighted this in Section 3 of our revision.