acceptodds
Under review as a conference paper at ICLR 2027

How Does Local Landscape Geometry Evolve in Language Model Pre-Training?

Abstract

The scale and expense of pre-training language models make efficient hyperparameter tuning essential, yet a principled guidance is still missing. In this work, we analyze language model pre-training dynamics from a local landscape geometry perspective.Our study reveals two distinct phases. In Phase I, sharpness of the local landscape is initially high, leading to instability and loss plateaus under large learning rates (LRs). The landscape shifts from sharp to flatter regions early in training. This dynamic explains the necessity of LR warmup and further suggests that larger peak LRs require proportionally longer warmup periods. In Phase II, the local landscape is governed by the gradient noise scale. Our theory identifies a depth flatness trade-off: high noise from smaller batches widens the loss basin, whereas reduced noise from larger batches deepens it. This theory motivates a dynamic batch-size (BS) scheduler that begins with a small BS and increases it late in training. Together, we provide a unified view of loss landscape evolution, which translates into actionable tuning strategies for large-scale pre-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Pretraining, Learning Dynamics, Sharpness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UrtdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how local loss-landscape geometry evolves during language-model pre-training. It proposes a two-phase view: the landscape shifts from sharp to flat early in training, which the authors use to motivate learning-rate warmup. Later, basin selection is governed by gradient noise, with smaller batches favoring flatter basins and larger batches favoring deeper ones.

Critical strengths

The presentation of this paper is clear and easy to understand.

Critical weaknesses
  1. To my understanding, the experiments do not directly validate the proposed stability mechanism, as decribed below. Lemma 4.1 characterizes stability using the preconditioned curvature λmax⁡(M1/2HM1/2)\lambda_{\max}(M^{1/2} H M^{1/2}), whereas the main experiments use AdamW and Figure 3 tracks the raw Hessian eigenvalue λmax⁡(H)\lambda_{\max}(H). Since AdamW's effective preconditioner varies during training, the raw Hessian trend alone may not explain the observed warmup behavior.

  2. This paper lacks the section of AI Use Statement.

Questions for authors

Lemma 4.1 uses preconditioned curvature as the stability measure, while Figure 3 reports the raw Hessian's largest eigenvalue. Can the authors verify that the sharp-to-flat trend also holds for a curvature measure that accounts for AdamW's preconditioning?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Urtd

Thank you for your great efforts on the review of this paper. We will try our best to address your questions.

If you still have further concerns, or if you are not satisfied by the current responses, please let us know, so that we can update the response ASAP.

Q1: On AdamW preconditioned curvature (also W1). “Can the authors verify that the sharp-to-flat trend also holds for a curvature measure that accounts for AdamW's preconditioning?” “Since AdamW's effective preconditioner varies during training, the raw Hessian trend alone may not explain the observed warmup behavior.”

A1:

  • Preconditioned curvature. We are measuring it from actual AdamW optimizer states and comparing it with the raw Hessian at the same checkpoints. These measurements are still in progress due to limited compute.

  • One-dimensional landscapes. At a fixed checkpoint, with fixed data and perturbation direction, we evaluate L(θ+αδ)L(\theta+\alpha\delta) directly, without the optimizer update rule. Slices along the same direction show early sharp-to-flat changes. The optimizer determines which checkpoints training reaches; the slice measures the loss geometry there.

Q2: On the AI Use Statement (W2). “This paper lacks the section of AI Use Statement.”

A2:

  • Writing. We used AI tools to polish the language of the paper and responses.

  • Code. AI helped write some code; the authors wrote most of it manually.

  • Research. The authors developed the research ideas and all experimental designs. We will include these details in the AI Use Statement.

Reviewer_WBHWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates local loss geometry during language-model pre-training and proposes a two-phase account. In the early phase, decreasing sharpness is used to explain learning-rate warmup and motivate longer warmup for larger peak learning rates. In the later phase, a stationary SDE analysis of preconditioned SGD motivates a trade-off between basin depth and flatness, leading to a schedule that starts with a small batch and increases it late in training. The main experiments use 93M/170M LLaMA models on FineWeb-Edu, with appendix extensions to GPT-2, additional optimizers, and models up to 530M parameters.

Critical strengths
  1. Useful empirical investigation of batch-switch timing. Figures 6 and 7 examine when to increase batch size, including comparisons under a fixed token budget and improvements over a constant small-batch baseline. This is the most promising contribution: a reliable late-ramping rule could simplify pre-training schedules without requiring online CBS estimation.

  2. Meaningful experimental coverage. The appendix extends the observations or scheduling experiments to GPT-2, Adam-mini, Lion, and models up to 530M, and checks early spikes with Muon. These results suggest that the empirical observations are not confined to one small AdamW/LLaMA configuration, strengthening their potential usefulness.

Critical weaknesses
  1. The analysis and measurements do not establish the claimed mechanism. Phase I analyzes deterministic, momentum-free GD with a fixed preconditioner MM, whereas the experiments use AdamW with momentum, adaptive moments, weight decay, and clipping. The relevant theoretical curvature is M1/2HM1/2M^{1/2}HM^{1/2}, but Figure 3 mainly measures raw HH. Lemma 4.1 also lacks sufficient conditions for its time-varying stability claim: with M=1M=1, η=1\eta=1, and Sk=(k+2)−2S_k=(k+2)^{-2}, every step satisfies the stated bound, yet the error converges to half its initial value rather than zero. Lemma 4.2’s vanishing descent guarantee likewise does not imply vanishing actual progress. In Phase II, the Gibbs model requires the strong noise-matching assumption (η/B)MΣM=2τM(\eta/B)M\Sigma M=2\tau M, which is not validated for the experimental optimizer. Theorem 5.1 further states a Laplace approximation as an exact basin-probability formula. Even with these issues repaired, a stationary distribution does not determine when to increase batch size, and one-dimensional random slices do not validate the log-determinant mechanism. These gaps affect the central explanatory contribution, not merely its wording.

  2. The remaining empirical contribution is not sufficiently distinguished from prior work or strong baselines. Curvature-based explanations of warmup and Adam preconditioned sharpness are studied by Kalra and Barkeshli, Why Warmup the Learning Rate? (NeurIPS 2024). The cited Jastrzębski et al., Three Factors Influencing Minima in SGD, already derive a closely related depth/volume basin-probability relation in Section 2.3. Replacing LR decay with batch growth, including for Adam, is established by Smith et al., Don’t Decay the Learning Rate, Increase the Batch Size (ICLR 2018). This leaves the late-switching rule as the main potential increment. However, Figure 7’s approximately 4× token saving is against a fixed large-batch, constant post-warmup LR baseline, while Figure 8 shows comparable final loss with small-batch LR decay. The paper therefore does not establish a comparable advantage over tuned conventional annealing, and token savings do not demonstrate wall-clock speedup. The trigger “loss reduction becomes marginal” is also not specified operationally. Consequently, neither the distinctiveness nor the practical advantage of the proposed scheduler is established.

Questions for authors
  1. After correcting the time-varying stability statement and treating Theorem 5.1 as a conditional Laplace approximation, which specific warmup-duration or batch-switch timing claims still follow from the theory, rather than being empirical heuristics?

  2. What evidence from the existing runs connects the analysis to actual AdamW dynamics? In particular, can the authors assess optimizer-relevant curvature and deviations from the assumed noise matching, or explicitly limit the mechanistic interpretation?

  3. Under matched token and tuning budgets, does late batch ramping improve over tuned small-batch LR decay/WSD? Please also specify the exact switching criterion used in the reported runs, so the proposed scheduler can be distinguished from manually chosen switch times.

2 comments
AuthorsOpenReview ↗

Response to Reviewer WBHW (1/2)

Thank you for your great efforts on the review of this paper. We will try our best to address your questions.

If you still have further concerns, or if you are not satisfied by the current responses, please let us know, so that we can update the response ASAP.

Q1: On the corrected theory and its scheduling implications (also W1). “After correcting the time-varying stability statement and treating Theorem 5.1 as a conditional Laplace approximation, which specific warmup-duration or batch-switch timing claims still follow from the theory, rather than being empirical heuristics?” “Lemma 4.2’s vanishing descent guarantee likewise does not imply vanishing actual progress.”

A1:

  • Warmup and Lemma 4.1. We will add sufficient conditions for the stability statement. For a fixed positive definite preconditioner MM, let Sk=M1/2HkM1/2S_k=M^{1/2}H_kM^{1/2} and assume μI⪯Sk⪯ΛI\mu I\preceq S_k\preceq\Lambda I for all kk, with 0<μ≤Λ0<\mu\leq\Lambda. For 0<η<2/Λ0<\eta<2/\Lambda, the linearized perturbation zk=M−1/2ekz_k=M^{-1/2}e_k satisfies

    \qquad q=\max\{|1-\eta\mu|,|1-\eta\Lambda|\}<1.$$ This gives a curvature-dependent stable LR range: the local model allows a larger LR as curvature falls. To determine how long to warm up, we also need to know how curvature evolves. The proportional relationship in Figure 4 is an experimental finding within the tested LR range, not a duration formula from the lemma.
  • Lemma 4.2. The descent lower bound can vanish without the actual decrease vanishing. We will remove that inference.

  • BS switching and Theorem 5.1. We will express the basin probabilities as a small-temperature Laplace approximation. For the ideal SDE with a fixed preconditioner and matched noise, assume a normalizable Gibbs distribution and basin integrals dominated by isolated, nondegenerate minima. Then

    \frac{e^{-F_i(\tau)/\tau}}{\sum_j e^{-F_j(\tau)/\tau}}[1+o(1)], \qquad F_i(\tau)=L_i+\frac{\tau}{2}\log\det H_i, \quad \tau\to0.$$ Here $L_i$ and $H_i$ are the loss and Hessian at the minimum. This retains the trade-off between depth and local volume. In the leading-order expression, lower temperature increases the relative preference for lower loss. It explains the role of noise, without specifying when training reaches these regions. Our late-switching recommendation comes from the fixed-token experiments in Figure 6. It combines this interpretation with the observed training dynamics.
AuthorsOpenReview ↗

Response to Reviewer WBHW (2/2)

Q2: On the connection to actual AdamW dynamics (also W1). “What evidence from the existing runs connects the analysis to actual AdamW dynamics? In particular, can the authors assess optimizer-relevant curvature and deviations from the assumed noise matching, or explicitly limit the mechanistic interpretation?”

A2:

  • Preconditioned curvature. We are measuring it from actual AdamW optimizer states and comparing it with the raw Hessian at the same checkpoints. These measurements are still in progress due to limited compute.

  • One-dimensional landscapes. At a fixed checkpoint, with fixed data and perturbation direction, we evaluate L(θ+αδ)L(\theta+\alpha\delta) directly, without the optimizer update rule. Slices along the same direction show early sharp-to-flat changes. The optimizer determines which checkpoints training reaches; the slice measures the loss geometry there.

  • Noise matching. Section 5 explicitly states this condition. For a fixed preconditioner, matching the diffusion covariance is a standard Langevin setup for a Gibbs stationary distribution (Ma, Chen & Fox, 2015). Jastrzębski et al., 2017 also analyze a Gibbs distribution under isotropic noise.

Q3: On novelty, data efficiency, and switching rules (also W2). “Under matched token and tuning budgets, does late batch ramping improve over tuned small-batch LR decay/WSD?” “Please also specify the exact switching criterion used in the reported runs” “The remaining empirical contribution is not sufficiently distinguished from prior work or strong baselines.” “token savings do not demonstrate wall-clock speedup.”

A3:

  • Our contribution. We measure landscape evolution during LLM pre-training, observe loss-curve collapse across different BS switching times, and study when to switch under a fixed token budget.

  • Relation to LR scheduling. Our goal is to use a fixed token budget more efficiently. Smaller batches allow more updates; larger batches give more accurate gradients. BS scheduling lets us vary this trade-off during training. LR changes step size, but at a fixed BS it does not change the number of updates supported by the same data. The roughly 4-fold token saving in Figure 7 is relative to the fixed large-BS baseline with a constant LR after warmup. We will describe this gain as “token savings.”

  • Combining the schedules. In Figure 8 (left), a 16-fold BS increase and a 16-fold LR reduction produce similar curves. On the right, a 4-fold LR reduction plus a 4-fold BS increase gives a lower loss than the same LR reduction alone and a result close to a 16-fold LR reduction alone.

  • Switching rule. The paper uses preset switching points, marked in the figures. We also tested an automatic rule based on training-loss plateaus: average loss over about 67.1M tokens, then increase BS fourfold when the absolute relative change between consecutive windows stays below 0.2% for three consecutive comparisons. The automatic rule gave a lower final loss than the preset schedule in this comparison. The window, threshold, and consecutive count need to be chosen for the task.

Reviewer_kyBaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Language model pre-training is expensive and therefore requires optimal hyperparameter tuning for the most efficient use of compute. This paper studies the local geometry of the loss landscape during training, revealing an important dependence of landscape curvature on the batch size, and suggesting a dynamic batch size tuning method. The paper combines theory of locally quadratic loss landscapes with empirical phenomena from LLMs. The phenomena are surprising and convincing.

Critical strengths
  1. The paper reports a new phenomenon that refreshes my understanding of the local geometry of the loss landscape during pre-training. Different from EoS line of work, where the loss gets sharper until the sharpness saturates, this paper found that there seem to be two phases in pretraining. In the first phase, the landscape gets sharper. In the second phase, if the batch size is large, the landscape gets sharper. Otherwise, the landscape gets flatter again.

  2. The paper is well-organized and easy to follow, with all necessary experiments and analysis.

Critical weaknesses
  1. I am not fully convinced by the big-picture theory proposed by the paper (see Questions for details). The paper can potentially be very important because of the new phenomena. But the lack of a convincing explanation makes me hesitate to choose acceptance.

  2. In terms of implications or engineering, the proposed batch size scheduler, as far as I can see, has a similar or equivalent effect to what a learning rate scheduler can provide, but may be less practical.

Questions for authors
  1. By looking at your Figure 1, I feel there are three phases: in the left panel, the sharpness first decreases (your first phase), the increases (you did not mention), and then decreases again (your second phase); in the right panel, the sharpness decreases (your first phase), then increases (the later two phases both increase sharpness).

  2. Can you provide more evidence on phases of pre-training or to better define your two phases? For example, a clear definition in my mind is that phase 1 is when the sharpness decreases no matter what the batch size or learning rate is, and phase 2 is when the sharpness depends on the batch size, etc. With such a definition, it is operationally measurable and decidable which phase the training is in.

  3. Why can there be two phases? What leads to the initial flattening or sharpening? What are the mechanistic reasons for sharp directions and flat directions in the loss landscape?

  4. What determines the transition between different phases? When does the transition happen? Can you predict it or only observe it?

  5. Why is your phenomenon different from the EoS literature? In particular, A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs also measures the sharpness of the LLM loss landscape but observes a monotonically increasing sharpness (See their Figure 3). I am curious about a comparison/discussion of your Figure 3 with their Figure 3.

  6. I am curious about what you mean by Basin Selection, or how do you define a basin? One of my hypotheses about the global landscape is still a river valley. Or I think a river valley can be modified to explain what you see. River means flat directions, along which the loss decreases slowly. Valley means sharp directions, which determine the saturation loss at the end. By increasing the noise τ=η/B\tau = \eta/B, you have to move to flatter places of the valley, and also have a higher saturation loss (your Figure 5, upper left). In this picture, there is no basin.

  7. What matters, from your theory, is τ=η/B\tau = \eta/B (for Adam, it may be η/B\eta / \sqrt{B}). So, what is the unique benefit of batch size scheduling? Can your theory or empirical findings provide a clear suggestion for the shape of the schedulers?

4 comments
AuthorsOpenReview ↗

Response to Reviewer kyBa (1/2)

Thank you for your great efforts on the review of this paper. We will try our best to address your questions.

If you still have further concerns, or if you are not satisfied by the current responses, please let us know, so that we can update the response ASAP.

Q1: On the intermediate rise in sharpness. “I feel there are three phases”

A1:

  • Two phases. Good point. Figure 1 has an intermediate rise in sharpness. By two phases, we mean the overall early and late training behavior; sharpness need not change monotonically within either phase.

  • Local fluctuations. During early flattening, the trajectory may still pass through sharper regions. We will clarify that the two-phase description allows these local changes.

Q2: On the definition of the two phases. “Can you provide more evidence on phases of pre-training or to better define your two phases?” “With such a definition, it is operationally measurable and decidable which phase the training is in.”

A2:

  • Phase description. Our two-phase description follows the evolution of the local geometry. Early in training, the landscape becomes flatter overall. Later, the geometry reflects a noise-dependent depth–flatness trade-off: smaller BS favors wider regions, while larger BS favors deeper ones .

  • Transition time. The timing of this transition depends on the training setup.

Q3: On the causes of sharpening and flattening. “Why can there be two phases? What leads to the initial flattening or sharpening? What are the mechanistic reasons for sharp directions and flat directions in the loss landscape?”

A3:

  • Initialization and architecture. We suspect that early sharpness depends strongly on the initialization scale, parameterization, and architecture. These choices affect sensitivity to parameter perturbations and may make some directions much steeper than others.

  • The first update. In our experiments, sharpness is low at initialization, rises sharply after the first update, and then declines overall. Footnote 5 on page 5 notes this, and Figure 3 starts after the first update. The flattening we describe follows the first-step sharpening. The first step also depends on the LR and optimizer.

  • Normalization. In Kalra & Barkeshli (2024, Figure 17), a Pre-LN Transformer starts in a flat region and progressively sharpens. Removing the final LayerNorm gives much higher initial curvature, followed by early sharpness reduction. This suggests that both initialization and architecture affect early curvature.

Q4: On predicting the phase transition. “What determines the transition between different phases? When does the transition happen? Can you predict it or only observe it?”

A4:

  • Prediction. We currently observe the transition through curvature measurements, but cannot predict when it will happen.

  • Scheduling. We tested an automatic BS rule based on training-loss plateaus. We average the loss within each window and increase BS when the absolute relative change between consecutive windows stays below a threshold for several consecutive comparisons. The window size, threshold, and required number of comparisons can be adjusted for the task.

AuthorsOpenReview ↗

Response to Reviewer kyBa (2/2)

Q5: On the comparison with the EoS literature. “observes a monotonically increasing sharpness (See their Figure 3).”

A5: Thank you for the reference. Their Figure 3 studies MLPs on CIFAR-10. The LLM experiments are in Figures 4–5 and also show an early decrease in sharpness.

Q6: On basin selection and the river-valley interpretation. “how do you define a basin?” “Or I think a river valley can be modified to explain what you see.” “In this picture, there is no basin.”

A6:

  • Basin and river valley. We see these as compatible descriptions of a high-dimensional landscape. A river valley is flat along some directions and steep along others.

  • Our use of basin. We use sharp/flat basin to describe the overall behavior across most local directions: sharper when most are steep, and flatter when most are flat. This is a qualitative description, and curvature can still vary across directions.

  • Connectivity. Mode-connectivity studies find low-loss paths between trained solutions (Garipov et al., 2018; Draxler et al., 2018). These paths can coexist with high curvature in other directions, so calling a region a basin does not imply that it is globally disconnected from other regions.

Q7: On the benefit and shape of BS scheduling (also W2). “what is the unique benefit of batch size scheduling?” “Can your theory or empirical findings provide a clear suggestion for the shape of the schedulers?” “has a similar or equivalent effect to what a learning rate scheduler can provide, but may be less practical.”

A7:

  • Relation to LR scheduling. When high-quality tokens are limited, our aim is to achieve lower loss with the same token budget. BS scheduling starts with small batches to get more updates from the same tokens, then increases BS to reduce gradient noise. LR scheduling changes the step size, but cannot control how the token budget is spent across updates.

  • Combining the schedules. In Figure 8 (right), a 4-fold LR reduction plus a 4-fold BS increase gives a lower loss than the same LR reduction alone. Its result is also close to a 16-fold LR reduction alone. BS and LR scheduling can be used together.

  • Schedule shape. We suggest starting with a small BS and increasing it later. The number, timing, and size of the increases depend on the task and budget.

I thank the authors for the detailed reply!

Most of my questions are clarified, especially the comparison with previous literature. My concern about the lack of theory remains. A lot of the authors' arguments are likely, but not derived or strongly supported by the current empirical evidence. The reply focused more on framing and description, but it cannot address the concern. I tend to keep my score, but I am still open to increasing it during later discussions with AC.

AuthorsOpenReview ↗

Response to Reviewer kyBa

Thank you for taking the time. We are glad that you reply so quickly, and we appreciate your concerns on our theory part.

  • Role of the theory. Our main contribution is the empirical picture of local geometry evolution in LLM pre-training. The theory is mainly there to help explain what we observe. Although its explanatory power is limited, it still provides useful intuition.
  • What this picture tells us. For example, why does BS scheduling work, and why increase BS late? Our theory suggests that the late training regime is noise-dominated and increasing BS increases the sharpness and can help lower the loss. Why is warmup sometimes necessary? Early training often moves from sharp to flatter regions. Keeping the LR small while the landscape is still sharp can help avoid instability.
  • Our aim. We do not aim to explain every detail of training, and our current compute limits how far we can investigate the underlying mechanisms. Even so, connecting these insights with interesting methods and phenomena is, we believe, still meaningful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.