acceptodds
Under review as a conference paper at ICLR 2027

Predicting characteristic learning rate based on second-moment-shaped stochastic mobility

Abstract

Neural networks are trained using local gradients with finite step size, making the learning rate central to balancing exploration in the loss landscape and stability in training dynamics. To effectively navigate a highly heterogeneous loss landscape, adaptive training is widely used in which training steps are further adjusted adaptively using the second moment. However, the selection of learning rate remains empirical and often costly. Here we derive a statistical physics prediction for this parameter. We find that the competition between stochastic exploration and curvature-induced confinement in training can be quantitatively defined, so that there exists an optimal characteristic . We estimate from a few mini-batches at a cost of 31 gradient-equivalent steps, rather than selecting it by comparing final performance across multiple training runs. Across deep neural networks and large-scale language models, the predicted lies near the generalization optimum and closely matches the best learning rate found by grid search. Moreover, we provide a measurable criterion for identifying the predictor's applicability regime and show that it attains the highest development accuracy among seven inexpensive selection baselines. Therefore, learning rate selection can be transformed from a costly empirical tuning procedure into an intrinsic dynamical prediction.

Primary area:
optimization
Keywords:
loss landscape, science of optimization, neural network training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_q69mOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces a framework to estimate the "characteristic" learning rate of Adam to navigate the heterogenous loss landscape of neural networks through a statistical physics lens. This framework is cost-efficient requiring only a few gradient computations and a short power iteration at a fixed checkpoint. They validate their framework on a toy landscape and then comprehensive pretraining bechmarks such as CNN and a ResNet-28 on CIFAR-10 and LLaMA on OpenWebMath as well as finetuning benchmarks GPT/LLaMA/BERT on SST-2. Their framework empirically works on Adafactor and AdamW as well.

Critical strengths
  1. Learning rate selection of Adam, the flagship optimizer in deep learning, is definitely an important problem to tackle with wide implications. Tuning learning rates is quite arduous and expensive and recently been made cheaper with complex frameworks like μP\mu P. This work takes a different approach that is extricably linked to a statistical physics theoretical framework. Existing alternatives either only bound where training becomes unstable (e.g., edge-of-stability thresholds) or change the optimizer itself (D-Adaptation, Prodigy, DoG). The paper offers a different angle: Adam's second-moment normalization damps noise-driven motion more in high-curvature directions than in low-curvature ones, and the best learning rate is where this contrast is largest. This yields a closed-form physically interpretable prediction which makes the work valuable even where the derivation relies on simplifying assumptions. Such assumptions are acceptable for the sake of tractability at the cost of some approximation errors so long as they are within performance bounds.

  2. The method is cost-efficient, requiring only 31 gradient equivalent steps at the beginning of training. Both sensitivity and performance are tracked on a wide variety of relevant benchmarks ranging from image classification (CNN/ResNet28 on CIFAR-10) to transformer pretraining (LLaMA 1B on OpenWebMath) and finetuning (GPT2, BERT, LLaMA on SST-2). It is also clear that it performs better than related methods such as D-Adaptation, Prodigy, L-DoG. Moreover, it generalizes to AdamW and Adafactor which are the optimizers implemented in practice.

Critical weaknesses
  1. The framework requires two key assumptions (both of which are addressed by the authors): approximately Gaussian mini-batch noise, and a second moment to which both noise and the deterministic gradient contribute. The authors acknowledge that the predictor holds only at intermediate batch sizes (due to Gaussianity requirement), which is a reasonable limitation for a tractable heuristic. However: a) The valid window is task-dependent (e.g., 4–32 for MNIST-MLP vs. 16–256 for QNLI). It is unclear how a user of this framework is supposed to define "intermediate" a priori which reduces its effectiveness as a pre-hoc metric. b) The Gaussianity check passes if R2≥0.95R^2 \geq 0.95 in just 2 of 10 seeds (App. E). This seems to be dubious as I would expect this to be a consistently satisfied as specified in Section 6.

  2. The theory requires a) quadratic expansion around a local minimum b) stationary fluctuations within a basin c) a quasi-stationary second moment. However, σ\sigma and λmax\lambda_{\text{max}} are measured "at a fixed checkpoint well before progressive sharpening", with "no optimizer warm-up required". That means initialization for from-scratch models, and a pretrained backbone with a freshly initialized head for fine-tuning. Neither is a minimum of the task loss, and at that point Adam's vv does not yet exist. App. Fig. 7 shows λmax\lambda_{\text{max}} rising steeply early in training. Since αc∝λmax−1/2\alpha c \propto \lambda_{\text{max}}^{−1/2}, a 100× change in λmax\lambda_{\text{max}} moves αc\alpha c by 10×. This seems to be a fundamental divergence of theory from practice.

  3. The flat regime is described by <v>=σ2b<v> = \frac{\sigma^2}{b} which corresponds to a squared gradient negligible compared with the noise. This could correspond not only to low curvature but also near-stationary when noise dominates. So TflatT_{\text{flat}} contains no curvature term and αc\alpha c depends only on σ\sigma, bb and λmax\lambda_{\text{max}}. Because the paper's central claim is a contrast between two regions: the experiments test the formula's output but not the proposed mechanism (see Q4).

  4. The derivation works in the Hessian eigenbasis (App. A) but Adam's preconditioner is diagonal in parameter coordinates implying that Adam is not rotation invariant. Pairing per-coordinate σi\sigma_i and viv_i with the raw-Hessian λmax\lambda_{\text{max}} hence requires either that its eigenvector be roughly axis-aligned or that viv_i be roughly uniform.This step links the one-dimensional model to Adam's actual update, the λmax−1/2\lambda_{\text{max}}^{−1/2} dependence is not derived from Adam's dynamics as written.

Questions for authors

Q1 [Clarification]: . How exactly are σ\sigma and λmax\lambda_{\text{max}} estimated? Section 4 only mentions 20 mini-batches and 10 power iterations but I couldn't find the estimator in the appendix. In particular, is σ\sigma per-sample or per-batch, how are the per-coordinate σi\sigma_i combined into one number and which batch and checkpoint define the Hessian?

Q2 [Experiment]: The derivation assumes the dynamics are near a minimum yet σ\sigma and λmax\lambda_{\text{max}} are measured at initialization or at a pretrained checkpoint. Why does the prediction still work there and how much of the agreement comes from Adam's broad plateau rather than the proposed mechanism? Reporting αc\alpha_c at a few checkpoints during trainig along with the accuracy each value gives would help settle this. The experiments do not have to be large ones. Moderate scale pretraining (e.g CNN on CIFAR-10) and a finetuning one would suffice.

Q3 [Ablation]: Since Adam is not rotation-invariant, what justifies pairing its per-coordinate statistics with the raw-Hessian λmax\lambda_{\text{max}}? You already compute 20 gradients, so you could estimate v from them and run the same power iteration on the preconditioned Hessian. More concretely: form v^=diag(E[g²])\hat{v} = \text{diag}(E[g^²]) from the same 20 gradients, compute the top eigenvalue of P1/2HP1/2P^{1/2}HP^{1/2} with P=diag(1(v^+ϵ)P = \text{diag}(\frac{1}{(\sqrt{\hat{v}} + \epsilon}) using the same 10 HVPs, and compare the resulting prediction. How does αc\alpha_c change?

4 comments

Thank you for your very positive assessment. We appreciate your supportive review and for specifying the experiments precisely enough to run as written. All three were run.

Q1/W1: We have added Appendix Algorithm 1, which gives the pseudocode of our estimator, and we also provide the corresponding code in the supplemental. σ\sigma is per-sample: variance of mini-batch gradients across K=20K=20 batches λmax⁡\lambda_{\max} is the top eigenvalue of the raw Hessian at the same parameters, from 10 power iterations. The checkpoint is initialization for from-scratch models, or the pretrained weights for fine-tuning. All these detailed information were added in our revision with blue highlighted.

Furthermore, we have tied the evaluation window to measured performance. To assess whether our modest criterion identifies reliable predictions, we compared accuracy gaps inside (IN) and outside (OUT, marked '-') the applicability window (Fig. 4c-d). Gaps are measured relative to the best grid-searched result at the same batch size and training budget. We found that accuracy gaps are larger for OUT than for IN batch sizes (paired tt-test, t(19)=5.13t(19)=5.13, p=5.93×10−5p=5.93\times10^{-5}). We also highlighted this diagnostic is optional. When it is used for additional assessment, its repeated noise measurements require 48×10=48048\times10=480 mini-batch gradient evaluations per candidate batch size (Appendix E). The common batch-size window observed across the tested vision and language tasks provides an empirically supported range for direct use of our predictor, which requires approximately 31 GE steps. We have added this information to our revision on Pages 9-10.

Q2/W2: This is a good point, and one we have also considered. In fact, our derivation needs less than a local minimum. The sharp-region analysis requires stationarity only along the dominant Hessian direction uu, so that u⊤∇F≈λmax⁡u⊤(θ−θ⋆)u^\top\nabla F\approx\lambda_{\max}u^\top(\theta-\theta^\star). The remaining gradient components enter only through ⟨v⟩\langle v\rangle, where they must be small relative to the mini-batch noise. We have stated this in Appendix A (also attached here: "A local minimum is sufficient but not necessary. In a narrow valley, the optimizer settles onto the valley floor within a few steps, long before it reaches the lowest point along the valley. Our analysis uses only the fact: it requires stationarity along the dominant eigenvector uu, u⊤∇F(θ⋆)=0u^\top\nabla F(\theta^\star)=0, so that u⊤∇F(θ)≈λmax⁡ u⊤(θ−θ⋆)u^\top\nabla F(\theta)\approx\lambda_{\max}\,u^\top(\theta-\theta^\star). The remaining gradient components enter only through Adam's second moment, where they need to be small compared with the mini-batch noise, E[(∇iF)2]≪σi2/b\mathbb{E}[(\nabla_i F)^2]\ll\sigma_i^2/b. This is why αc\alpha_c can be measured at initialization or at a pretrained checkpoint, and the noise fraction along uu (Appendix~E) checks the condition there"), and the noise-fraction diagnostic (Appendix E) checks the condition point.

Here σ\sigma is the mini-batch-induced noise. An important point is that the gradient-noise variance and the curvature co-vary along a direction, as reported in the inverse variance-flatness relation (PNAS, 2021). Thus, αc\alpha_c depends on both, and a change in λmax⁡\lambda_{\max} alone does not determine the change in αc\alpha_c.

For simulations, we have run the suggested experiment (Appendix Fig. 13): αc\alpha_c is re-estimated at checkpoints (training steps) t∈{0,5,10,15,20}t\in\{0,5,10,15,20\} for CNN/CIFAR-10 and Qwen3-1.7B/RTE, and training is continued from each checkpoint at αc(t)\alpha_c(t), with 20 seeds each. Over these checkpoints, the estimate taken before training gives a final accuracy at least as high as any later estimate. This is why we can measure once at the start.

Q3/W4: We ran this ablation as specified (Appendix Fig. 14): P=diag(v^)−1/2P=\mathrm{diag}(\hat v)^{-1/2} is formed from the same 20 gradients, and the top eigenvalue of P1/2HP1/2P^{1/2}HP^{1/2} is computed with the same 10 HVPs. The preconditioned prediction is lower by factors of 31.8 (CNN/CIFAR-10), 26.4 (1B Transformer), and 5.0 (Qwen3/RTE), placing it well below the grid optimum that the raw-Hessian prediction matches. The reason is that the closure already accounts for Adam's normalization through ⟨v⟩\langle v\rangle; preconditioning the curvature as well counts it twice. We have now highlighted this in Section 3 of our revision.

W3: This is the purpose of toy model validation. The mobility-contrast mechanism is tested directly in the toy landscapes (Fig. 2 and Appendix Fig. 3), where the mobility contrast is measured and peaks at the predicted αc\alpha_c. On real networks, the test is the formula output against grid-search optima, across many different models and tasks. We have separated these two kinds of evidence in the Discussion of our revision and stated as “The mobility contrast acts as a probabilistic bias: raising the learning rate increases mobility everywhere, and αc\alpha_c is the point at which the mobility contrast is maximized. Because high-curvature regions are more sensitive to parameter and sampling perturbations, we highlight that keeping their mobility low relative to that of low-curvature regions increases the chance that training remains in or reaches perturbation-stable regions, and thereby favors generalization. Our experiments support this view through the agreement of αc\alpha_c with grid-search optima.”

Thank you for your time and effort in reviewing our work. We are really encouraged by your positive assessment, and your questions have helped us improved our clarification.

Thanks for addressing my concerns. I am largely satisfied with the responses and have thus increased soundness from 2 to 3.

Thank you for your positive feedback and for the increased soundness. We sincerely appreciate your time and effort.

Reviewer_jcxgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a low-cost learning-rate selector for Adam. A reduced effective-temperature model of second-moment normalization yields a sharp–flat mobility ratio whose maximization gives αc≈3.22,σbλmax⁡.\alpha_c \approx \frac{3.22,\sigma}{b\sqrt{\lambda_{\max}}}. Here, bb is batch size, σ\sigma a gradient-noise scale, and λmax⁡\lambda_{\max} dominant curvature. The predictor costs approximately 31 gradient-equivalent operations. Vision and language experiments report near-grid-optimal performance, with comparisons against inexpensive selectors and proposed applicability checks.

Critical strengths
  • A practically relevant objective and a concrete low-cost procedure. The predictor avoids validation-based tuning and optimizer modification, with a standalone cost of 31 gradient-equivalent steps. Section 4 provides explicit cost accounting and wall-clock measurements demonstrating practical savings.

  • Competitive empirical performance. Results across models and tasks approach grid-search optima. Table 2 compares seven inexpensive alternatives over ten seeds, showing competitive performance while appropriately acknowledging overlap with the strongest baselines rather than claiming significant superiority.

  • Useful mechanism and robustness checks. The toy experiment, optimizer and schedule comparisons, and applicability diagnostics provide complementary tests of the proposal (Figures 2 and 4; Table 3). The discussion also acknowledges limitations involving momentum, weight decay, and high-dimensional dynamics.

Critical weaknesses
  • Unresolved modeling assumptions behind the prediction. The theoretical derivation of the characteristic learning rate relies on a self-consistent approximation whose justification remains unclear. Specifically, Appendix A (Eqs. 18 and 26) establishes the approximate relations
\qquad T_i^{\mathrm{eff}}\approx\frac{C_{\mathrm{neq}}}{\langle v_i\rangle},$$ where $C_{\mathrm{neq}}=\alpha^2\sigma_i\sqrt{b}/2$. Meanwhile, Appendix B.2 (Eqs. 31–33) assumes that the squared gradient scales as $\mathbb{E}[(\nabla F)^2]\sim\lambda_{\max}T^{\mathrm{eff}}$. Combining these relations naturally suggests $$T^{\mathrm{eff}}\approx \frac{C_{\mathrm{neq}}} {\lambda_{\max}T^{\mathrm{eff}}+\sigma_i^2/b}.$$ However, Eq. (34) instead introduces a squared denominator, motivated by an additional feedback mechanism that is not explicitly derived. This distinction is important because the additional power changes the learning-rate dependence of the effective temperature and consequently the predicted characteristic learning rate in Eq. (5). Although Appendix Figure 11 provides empirical support for the squared closure, it does not fully explain its consistency with the preceding approximations. A clearer derivation or justification of this modeling assumption would substantially strengthen the theoretical contribution. Additionally, the effective time interval $\Delta t_{\mathrm{eff}}=1/b$ is introduced without sufficient justification, despite its direct influence on the batch-size scaling. Finally, the theoretical analysis concerns local mobility contrast rather than final basin selection or generalization performance. The connection between these quantities therefore remains largely heuristic and would benefit from further clarification. - The maximization in Eqs. (4)–(5) is reasonable conditional on the proposed temperature laws, but their justification remains incomplete. The preceding relations suggest $T\approx C_{\mathrm{neq}}/(\lambda_{\max}T+s)$, whereas Appendix B.2 introduces a squared denominator, changing the learning-rate exponent. Appendix Figure 11 provides empirical support but does not reconcile these relations or independently establish the learning-rate dependence. The choice $\Delta t_{\mathrm{eff}}=1/b$ also needs justification because it determines batch-size scaling. Finally, the toy experiment tests local mobility rather than final basin selection, leaving the proposed generalization mechanism a hypothesis rather than a demonstrated consequence. - **Incomplete specification and end-to-end cost accounting.** Section 4 should specify how measured noise variances yield the scalar $\sigma$, including mini-batch-to-per-sample conversion and the associated curvature direction. The relationship between the Adam-preconditioned operator discussed in Section 7 and the raw-Hessian direction used in Appendix E also needs clarification. Moreover, the applicability test accepts Gaussianity when only two of ten seeds pass, despite the main text requiring consistent satisfaction. Its 48 mini-batches per seed imply 480 batch-gradient evaluations per candidate batch size, additional to the standalone 31-GE estimate. Required versus optional diagnostics, total costs, and whether passing these checks predicts reliable learning-rate selection should be clarified. - **Competitive results, but limited support for broader claims.** The three-point Adam sweep in Section 6 excludes Table 2’s $3\times10^{-5}$ reference and smaller fine-tuning rates, making the comparison sensitive to the candidate range. A matched-budget sweep covering lower rates would strengthen the evidence. Direct inexpensive-selector comparisons cover only two model–task pairs with partial fine-tuning; the Llama experiment discussed under “Training from scratch” is actually continued pretraining for 1,200 updates, not training from random initialization. Finally, the nonsignificant tests in Section 6 do not establish equivalence, while broad performance plateaus support practical usefulness rather than the precise numerical coefficient. Paired-gap confidence intervals would strengthen the comparisons currently supported only by nonsignificant tests.
Questions for authors
  • Which preceding approximation does the squared closure replace? Please specify the fitting window and test its predicted dependence on learning rate, not only curvature. How is the inverse-batch effective time justified independently of the desired scaling?

-Please provide estimator pseudocode defining the scalar noise statistic, curvature operator, and checkpoint. Is the full applicability gate required for deployment, what is its total cost, and how does its pass/fail decision relate to observed prediction error?

  • Does the Table 2 comparison remain favorable when the same-budget Adam sweep includes the reference and lower fine-tuning rates? Please also reconcile the RTE training duration and distinguish continued pretraining from training from scratch.
1 comment

Thank you for highlighting our useful mechanism and competitive results. We have addressed the three groups of comments in turn.

Q1/W1-2: Eq. (34) in the Appendix replaces the one-power relation ⟨v⟩≈λTeff+s\langle v\rangle\approx\lambda T^{\mathrm{eff}}+s that follows from Eqs. (18), (26), (33). Here, it computes the compound effect of vtv_t in sharp directions: vtv_t shrinks the normalized step and thereby limits the displacement that feeds back into vtv_t. We have tested them in two independent ways.

  1. As shown in Appendix Fig. 12, with Adam dynamics providing the stationary ⟨v^⟩\langle\hat v\rangle, the log-log slope of Cneq/TeffC_{\mathrm{neq}}/T^{\mathrm{eff}} against λTeff+s\lambda T^{\mathrm{eff}}+s is 2.08 (95% CI [1.84, 2.32]). Slope of 2 gives R2=0.995R^2=0.995; slope 1 gives R2=0.73R^2=0.73. We have added the details in our revision:" the fitting window is fixed before fitting, that is E[(λθ)2]/s≥3E[(\lambda\theta)^2]/s\ge3, the deterministic gradient contributes at least 75% of ⟨v^⟩\langle\hat v\rangle."

  2. Moreover, we have added a comparison of the alternatives. The combination of the squared closure and the per-sample normalization is compared against the alternatives in Appendix Table 6. The squared, per-sample (1/b1/b) combination used in our work gives the slope closest to one, i.e., the best agreement with the empirically optimal learning rate. We attached Appendix Table 6 below.

Appendix Table 6: Comparison of four closure and time-normalization combinations.

Closure / time normalizationSlope κ\kappa [95% interval] (accuracy)CV error ↓ (accuracy)Slope κ\kappa [95% interval] (cross-entropy)CV error ↓ (cross-entropy)
Squared / per-sample (1/b1/b)1.051 [−1.251, 1.320]0.0640.954 [−0.303, 2.134]0.254
Squared / per-step2.125 [−2.523, 2.738]0.4701.913 [−0.592, 4.254]0.460
One-power / per-sample (1/b1/b)0.833 [−1.000, 1.064]0.1850.747 [−0.235, 1.674]0.371
One-power / per-step1.389 [−1.675, 1.810]0.2541.229 [−0.376, 2.756]0.292

Q2/W3: Thanks for the suggestion, we have added the pseudocode in Appendix Algorithm 1. Then, we have tied the evaluation window to measured performance. To assess whether our modest criterion identifies reliable predictions, we compared accuracy gaps inside (IN) and outside (OUT, marked '-') the applicability window (as shown in Fig. 4c-d). Gaps are measured relative to the best grid-searched result at the same batch size and training budget. We found that accuracy gaps are larger for OUT than for IN batch sizes (paired tt-test, t(19)=5.13t(19)=5.13, p=5.93×10−5p=5.93\times10^{-5}). We also highlight this diagnostic is optional. When it is used for additional assessment, its repeated noise measurements require 48×10=48048\times10=480 mini-batch gradient evaluations per candidate batch size (Appendix E). The common batch-size window observed across the tested vision and language tasks provides an empirically supported range for direct use of our predictor, which requires approximately 31 GE steps. We have added this information to our revision on Pages 9-10.

Q3/W4: To address your concern, we have changed the sweep candidates to lower fine-tuning learning rates, {10−6,10−5,10−4}\{10^{-6}, 10^{-5}, 10^{-4}\}, and our predictor remains favorable. Furthermore, following your suggestion, we have added a new experiment that trains a custom 1B Transformer from scratch [see Page 7: "We further train a custom 1.02B-parameter decoder-only Transformer from random initialization on OpenWebMath, updating all parameters. Our predictor selects (9.73±0.69)×10−5(9.73\pm0.69)\times10^{-5}, whereas the best rate in the fixed grid is 1.73×10−41.73\times10^{-4}. The corresponding validation losses are 6.0991±0.05186.0991\pm0.0518 and 6.0971±0.03896.0971\pm0.0389 (mean ±\pm s.d.), a relative difference of only 0.033%0.033\% (95% CI: [−0.018%,0.248%][-0.018\%, 0.248\%] in Appendix Table 4)"], and the continued pretraining of Llama-1.1B has been moved to the fine-tuning part.

We hope the revision addresses your concerns, and we are happy to clarify any remaining points. We see this work as an important step toward predicting optimization hyperparameters from a physical perspective, and your comments have helped us improve it. Thank you for your time in reading and reviewing our work!

Reviewer_httoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a prediction method based in statistical mechanics for determining the best learning rate to use when training neural networks. They validate their method on different models (MLP, CNN, ResNet) trained from scratch using several pretrained transformers. They claim the learning rate derived from their method is comparable to the best learning rate found through a dense grid search.

Critical strengths

Choosing global learning-rate schedules without extensive hyperparameter tuning or relying on dense grid search is an important practical problem in training neural networks and the authors do a great job of framing the problem

The paper is very well written and I found it easy to understand the problem being addressed by the paper and how their experiments and method supported progress on this problem

I felt that the idea of a mobility contrast measure was a novel concept and a nice way to think about moving through optimization landscapes with high and low curvature

Critical weaknesses

It’s not clear that the insights derived from this method will transfer to larger transformers and/or LLMs. The authors do test the method on pretrained transformers, however these models have already undergone a substantial amount of training and it is difficult to say whether this method will work in this situation

Results seem limited to a relatively narrow noise range

Questions for authors

Is there an experiment you can run to validate your choice of learning rate for LLMs that doesn’t rely on having a pretrained model?

When applying your two-basin prediction to real models, can you find datasets where your prediction fails? (Beyond small/large batch size). In general, how data dependent is the success of your prediction?

If we extend the toy model to more than two basins, does this change the prediction for optimal learning rate?

1 comment

Thank you for your positive assessment of the novelty, results and presentation of our work. To address your concerns, we have added new experiments, as detailed below.

Q1/W1: Yes. We now train a 1.02B-parameter decoder-only custom Transformer from scratch on OpenWebMath, updating all parameters (new results in Section 6 Training from scratch, and model details in Appendix Table 4). Our predictor gives (9.73±0.69)×10−5(9.73\pm0.69)\times10^{-5}. The validation loss at that rate is 6.0990±0.05256.0990\pm0.0525, versus 6.0971±0.03896.0971\pm0.0389 at the best point of a five-rate grid, a difference of 0.033% (95% CI [−0.018%,0.248%][-0.018\%, 0.248\%] over ten seeds). Prediction took 4.2 s, against 1,535 s for the grid. Moreover, the Llama-1.1B case on OpenWebMath from the original submission is now labeled as continued pretraining.

Q2/W2: The evaluation window is now tied to measured performance. The new Fig. 4d compares the accuracy gap to the grid optimum for batch sizes that pass (labeled as IN) and fail (labeled as OUT, i.e., '-' in Fig. 4c) the diagnostic across four model-task pairs. Gaps are larger for OUT (paired t(19)=5.13t(19)=5.13, p=5.9×10−5p=5.9\times10^{-5}), so the diagnostic identifies the failure cases before training. The location of the window is data dependent. As you may notice, QNLI labels are derived automatically from the Stanford Question Answering Dataset (SQuAD) and its sentence pairs are heterogeneous, so its per-sample gradient noise at a given batch size is larger than that of MNIST, consistent with its window lying at larger batch sizes. The smaller mean gradient of a pretrained model may contribute as well. Across our tested cases, the windows overlap at 16–32, which gives a common range for direct use without running the diagnostic.

We believe this range is acceptable in practice for two reasons. First, a large batch size (~100) already suppresses the stochastic noise in Adam training, making it nearly deterministic, which is somehow “not good” for exploring good solutions. Second, if machine memory is limited, gradients can be accumulated over several smaller batches to reach the window. For these reasons, we consider the window reasonable, and we believe that predicting hyperparameters from a physical perspective is a promising future direction. We hope these points and the novelty of our approach can outweigh this “limitation”.

Q3: With more than two basins, our prediction holds when it is computed from the sharpest and the flattest basins. We have added three- and four-basin landscapes, including one with shuffled basin order (see Appendix Fig. 3). The prediction computed from the sharpest and flattest basins matches the measured contrast peak as in the two-basin case, so intermediate basins do not enter αc\alpha_c.

Thank you for your time in reading and reviewing our work. We hope our new experiments address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.