- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Previous work has identified an interesting behavior: when fine-tuning an LLM by cycling through a few documents one after the other, always in the same order, the loss on a single document will decrease before seeing the document, highlighting some anticipatory behavior. The paper reproduces this behavior in a much smaller setup, which enables more careful experimentations and studies, both empirically and theoretically, on which properties of optimization give raise to this behavior.
Critical strengths
The phenomenon studied here is intriguing and challenges our understanding of deep learning and stochastic optimization. The paper sheds some light into this behavior with well designed ablations; it feels like the paper is looking at the right quantities to understand this behavior, as varying them can yield significantly different dynamics.
Critical weaknesses
From my understanding, the paper falls short of some of its claims in different ways:
- the theoretical analysis holds for stochastic gradient descent; yet Table 1, as far as I understand it, seems to suggest that the behavior only occurs for AdamW. It is therefore unclear how the analysis explains the desired behavior.
- related to the previous point, there is no quantitative relationship linking T and Ring-related quantities to anticipation directly in the main text. It may be only a presentation problem, in which case I would encourage the authors to make this link clearer.
- the core experiments hold for contexts of length 8 (!), which is very far from anything standard in large language modeling, and anticipation goes away when increasing the context length. This suggests that the current setup might not be suited to explain the phenomenon observed at scale.
I am happy to increase my scores if the authors answer my concerns or if my remarks are unfair.
Minor: it would be useful to have a description of the data in the main text.
Questions for authors
What does AdamW (no momentum) mean, and how is it different from SGD?
3 comments
I thank the authors for the detailed rebuttal, it addresses most of my concerns; I believe it is an interesting and solid contribution that deserves being accepted to the conference. I have updated my score accordingly.
The fact that the behavior only appears for SGD for sufficiently large learning rates is interesting and it is not directly clear to me why this is the case from the theory. I would encourage the authors to emphasize this point more clearly in the main text. I am also wondering how much of Figure 8 can be explained by this insight.
Response to follow-up questions
Thank you for your continued interest in our work and for increasing your score! Here are our answers to your follow-up questions. We updated the manuscript to emphasise these points.
SGD learning rate vs theory. Our theory does predict dependence on the learning rate. The learning rate η appears in our theoretical results twice:
- Theorem 2: the alignment pressure of Ring with respect to Bursty is proportional to η
- Theorem 3: the successor advantage (SA) gained per update is η times the (effective) SGA
The second point is shown in Fig. 15 (SGD ablation). With the low learning rate, SGA still increases, but the realized effect (step size × SGA) is much smaller, so anticipation does not develop within the training horizon (step size is learning rate adjusted with the effect of momentum and grad clipping).
Is the context length result related? The same reasoning can be applied to why longer contexts need longer bursts (Fig 8). Both effects above act once per update, so their total effect within one burst grows with the learning rate times the number of updates in the burst. If this product is small, the model barely changes during a burst and hence the order of the bursts does not matter, and no anticipation happens.
We hypothesise that longer context means slower for class-specific learning. For long contexts, the model can identify the class from the prefix, and so each update changes the class information stored in the weights less. Hence more updates per burst are needed to get to anticipation. Consistent with this, Fig. 22 shows that at context length 128 and late in training, the per-token SA is largest at the early token positions.
Response to review
Thank you for your review! We are glad you found our paper’s topic intriguing, our analysis shedding light on this behaviour, and our ablations well-designed. We address your concerns below. All new experiments and results are added to our updated manuscript, and are marked in blue.
1. Theory holds for SGD but experiments find SGD does not anticipate: We break down our answer into multiple parts.
What our theory holds for: our theoretical results hold for SGD, and a simplified version of AdamW (fixed preconditioner), and we show in Figure 5 that this simplified approximation is consistent with AdamW’s anticipation. So the essence of the theory holds for both SGD and AdamW.
Clarifying our implicit bias argument: We derived implicit biases that can enable anticipation. When anticipation happens, our explicit regularization experiment shows that we can manipulate it, hence anticipation is linked to the implicit bias we derived. However, for anticipation to emerge, the implicit biases alone are not enough, we also need the right conditions for them to accumulate, such as long enough bursts, training horizon, etc.
Clarifying SGD: Since our SGD experiment did not show anticipation, during the rebuttal, we tuned the learning rate (lr=0.3) and found that SGD anticipates as well (successor advantage, SA ≈0.16). We also added new analysis (Figure 15) about why the original, lower learning rate failed: the realized effect of successor-gradient alignment (SGA) was much smaller due to low learning rate, so anticipation could not accumulate on the tested time horizon.
Improved narrative: Our new analysis makes our explanation of optimizer-dependence much cleaner: instead of having to compare T2 and T3, now effective SGA (SGA measured along the optimizer's update) is predictive of anticipation, in accordance with our Theorem C.3. We have also updated Figure 6 (right) to include an SA vs SGA plot, which confirms this relationship empirically for SGD, AdamW and Muon. This ties our theory and experimental results closer.
2. No quantitative relationship linking T and Ring-related quantities to anticipation directly in the main text: Thank you for your observation. We have edited section 4.1 to include our quantitative formula in Result 2: T2 increases SGA, and SGA in turn increases SA (Result 3). See also our response to 1: the new Figure 6 on page 7 (right) now quantitatively links SA (anticipation) to SGA. In addition, our explicit regularization analysis links T2 (what our regularizer estimates) to control on SA.
3. Context length 8
We appreciate the point that context length 8 is non-standard in language modeling. We have picked this setting due to computational constraints, as a simple setting that allows us to cheaply manipulate anticipation.
For the rebuttal, we have rerun our main experiment (see Figure 8) for context lengths of up to 1024, and found that anticipation is preserved, but requires longer bursts (e.g. 128).
This confirms that context length 8 is not the only setting where anticipation emerges (our theory also does not depend on context length), rather, it’s a cheap setting where anticipation can be reproduced on a short training horizon.
What does AdamW (no momentum) mean, and how is it different from SGD?
It is AdamW with β1=0. AdamW modifies SGD in multiple ways apart from adding momentum. It also scales the learning rate per-parameter, using an exponential moving average (EMA) of the squared past gradients (controlled by the so-called second moment β2). This normalizes step sizes across parameters that may have different gradient magnitudes. AdamW also adds weight decay, which is not automatically present in SGD.
We amended our manuscript to include a description of the data on page 3.
We also added third- and fourth-order experiments: models anticipate the successor class even when it depends on the last three or four bursts together (Figures 4, 13).
Thank you again! We are happy to address any remaining questions.