- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how local loss-landscape geometry evolves during language-model pre-training. It proposes a two-phase view: the landscape shifts from sharp to flat early in training, which the authors use to motivate learning-rate warmup. Later, basin selection is governed by gradient noise, with smaller batches favoring flatter basins and larger batches favoring deeper ones.
Critical strengths
The presentation of this paper is clear and easy to understand.
Critical weaknesses
-
To my understanding, the experiments do not directly validate the proposed stability mechanism, as decribed below. Lemma 4.1 characterizes stability using the preconditioned curvature , whereas the main experiments use AdamW and Figure 3 tracks the raw Hessian eigenvalue . Since AdamW's effective preconditioner varies during training, the raw Hessian trend alone may not explain the observed warmup behavior.
-
This paper lacks the section of AI Use Statement.
Questions for authors
Lemma 4.1 uses preconditioned curvature as the stability measure, while Figure 3 reports the raw Hessian's largest eigenvalue. Can the authors verify that the sharp-to-flat trend also holds for a curvature measure that accounts for AdamW's preconditioning?
Response to Reviewer Urtd
Thank you for your great efforts on the review of this paper. We will try our best to address your questions.
If you still have further concerns, or if you are not satisfied by the current responses, please let us know, so that we can update the response ASAP.
Q1: On AdamW preconditioned curvature (also W1). “Can the authors verify that the sharp-to-flat trend also holds for a curvature measure that accounts for AdamW's preconditioning?” “Since AdamW's effective preconditioner varies during training, the raw Hessian trend alone may not explain the observed warmup behavior.”
A1:
Preconditioned curvature. We are measuring it from actual AdamW optimizer states and comparing it with the raw Hessian at the same checkpoints. These measurements are still in progress due to limited compute.
One-dimensional landscapes. At a fixed checkpoint, with fixed data and perturbation direction, we evaluate L(θ+αδ) directly, without the optimizer update rule. Slices along the same direction show early sharp-to-flat changes. The optimizer determines which checkpoints training reaches; the slice measures the loss geometry there.
Q2: On the AI Use Statement (W2). “This paper lacks the section of AI Use Statement.”
A2:
Writing. We used AI tools to polish the language of the paper and responses.
Code. AI helped write some code; the authors wrote most of it manually.
Research. The authors developed the research ideas and all experimental designs. We will include these details in the AI Use Statement.