- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how regularization affects length generalization. It shows that extrapolation can deteriorate with longer training and that weight decay can accelerate this deterioration. The authors analyze the statistical mismatch introduced by dropout before normalization, propose placing dropout after normalization, and introduce VPAD to preserve marginal activation variance. Experiments on Transformers and Mamba2 show substantial extrapolation improvements.
Critical strengths
- The experiments cover training from scratch and continual pretraining, with evaluations on NIAH, RULER, and HELMET.
- The analysis of dropout placement is useful and motivates a simple modification with clear empirical benefits.
- Results across Transformers and Mamba2 suggest that the observations are not specific to one architecture.
Critical weaknesses
My main concern is that the paper does not sufficiently establish the underlying cause of length overfitting. The analysis explains dropout-induced statistical mismatch, but does not explain why extrapolation deteriorates as training proceeds. Successful regularization alone does not establish that this deterioration is independent of positional learning.
An alternative explanation is that positional biases in the data encourage models to learn implicit absolute-position representations when explicit relative offsets j−i are unavailable or inadequately represented. Dropout may suppress this positional shortcut without addressing its source.
I recommend probing absolute positions from hidden states, following (https://arxiv.org/abs/2203.16634), across layers and training checkpoints. Does absolute-position prediction improve as extrapolation deteriorates? Comparing regularization variants and an explicit relative-position control would help distinguish these explanations. Probing establishes association rather than causality, so the control is important.
Questions for authors
see weakness
2 comments
Thanks again for your review. If you have any further questions, we would be happy to respond during the remaining two days of the discussion period. If our response has adequately addressed your concerns, we would be grateful if you could update your assessment.
Thank you for your review. We are happy to address your comments below.
We see this as a limitation of scope rather than a critical weakness, and we would like to clarify our claims.
First, we do not claim that the deterioration is independent of positional learning, and our claims also do not require that. We do not have a mechanistic explanation to how the overfitting takes place. Our hypothesis is more modest: the deterioration behaves like overfitting: it grows with training and can be mitigated by proper regularization - and we test exactly this prediction. We agree that successful regularization alone does not identify the cause; this is why we state a hypothesis and test its predictions rather than make a causal claim.
Second, if the cause (or one of the causes) is an implicit positional shortcut, this would not invalidate our claims - it would be a concrete instance of them. Learning implicit absolute-position features that are valid only up to the training length is itself length overfitting, and a regularizer suppressing such a shortcut is consistent with our hypothesis. We agree that the experiment you suggest (thank you!) would be interesting to explore the mechanism(s) of length overfitting in future work.
Third, we believe there may be multiple mechanisms at play: this kind of overfitting was first shown for linear RNNs in [1]. There, it was attributed to unexplored recurrent states, and the proposed fix targets the states. Our remedy is different: plain regularization, which does not target states or positions. We show it helps in three settings: transformers with NAPE, pre-trained transformers with RoPE and SW attention, and Mamba2. Since different remedies, aimed at different mechanisms, both mitigate the problem, we believe there may be multiple causes, and identifying them is a separate research direction.
[1] Understanding and Improving Length Generalization in Recurrent Models. Ricardo Buitrago Ruiz, Albert Gu.