acceptodds
Under review as a conference paper at ICLR 2027

Length Generalization Needs Proper Regularization

Abstract

Length generalization is the ability of sequential models to perform well on context lengths unseen during training. In this work, we show that the challenge of achieving length generalization is related not only to architectural choices such as positional encoding and the attention mechanism but also to the training procedure itself. We study how regularization affects length generalization and find that weight decay can hinder extrapolation. In contrast, dropout improves extrapolation when its placement within the architecture is reconsidered. In that regard, we show that the standard placement of dropout before layer normalization introduces a systematic distributional mismatch, and that applying dropout just before the linear projection resolves this issue. For example, a modified SmolLM3 with sliding window attention, continually pre-trained with dropout, can extrapolate perfectly to 64 on Needle-in-a-Haystack and far beyond the pre-training context size on RULER and HELMET. Mamba2 also benefits from dropout, suggesting an architecture-agnostic nature of the problem. We further propose Variance-Preserving Affine Dropout (VPAD), a new dropout strategy that substantially reduces the resulting pre-activation variance mismatch, leading to further extrapolation improvements in transformers.

Primary area:
generative models
Keywords:
length generalization, length extrapolation, long context, regularization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LJEZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how regularization affects length generalization. It shows that extrapolation can deteriorate with longer training and that weight decay can accelerate this deterioration. The authors analyze the statistical mismatch introduced by dropout before normalization, propose placing dropout after normalization, and introduce VPAD to preserve marginal activation variance. Experiments on Transformers and Mamba2 show substantial extrapolation improvements.

Critical strengths
  1. The experiments cover training from scratch and continual pretraining, with evaluations on NIAH, RULER, and HELMET.
  2. The analysis of dropout placement is useful and motivates a simple modification with clear empirical benefits.
  3. Results across Transformers and Mamba2 suggest that the observations are not specific to one architecture.
Critical weaknesses

My main concern is that the paper does not sufficiently establish the underlying cause of length overfitting. The analysis explains dropout-induced statistical mismatch, but does not explain why extrapolation deteriorates as training proceeds. Successful regularization alone does not establish that this deterioration is independent of positional learning.

An alternative explanation is that positional biases in the data encourage models to learn implicit absolute-position representations when explicit relative offsets j−i are unavailable or inadequately represented. Dropout may suppress this positional shortcut without addressing its source.

I recommend probing absolute positions from hidden states, following (https://arxiv.org/abs/2203.16634), across layers and training checkpoints. Does absolute-position prediction improve as extrapolation deteriorates? Comparing regularization variants and an explicit relative-position control would help distinguish these explanations. Probing establishes association rather than causality, so the control is important.

Questions for authors

see weakness

2 comments

Thank you for your review. We are happy to address your comments below.

My main concern is that the paper does not sufficiently establish the underlying cause of length overfitting. The analysis explains dropout-induced statistical mismatch, but does not explain why extrapolation deteriorates as training proceeds. Successful regularization alone does not establish that this deterioration is independent of positional learning. [...]

I recommend probing absolute positions from hidden states, following (https://arxiv.org/abs/2203.16634), across layers and training checkpoints.

We see this as a limitation of scope rather than a critical weakness, and we would like to clarify our claims.

First, we do not claim that the deterioration is independent of positional learning, and our claims also do not require that. We do not have a mechanistic explanation to how the overfitting takes place. Our hypothesis is more modest: the deterioration behaves like overfitting: it grows with training and can be mitigated by proper regularization - and we test exactly this prediction. We agree that successful regularization alone does not identify the cause; this is why we state a hypothesis and test its predictions rather than make a causal claim.

Second, if the cause (or one of the causes) is an implicit positional shortcut, this would not invalidate our claims - it would be a concrete instance of them. Learning implicit absolute-position features that are valid only up to the training length is itself length overfitting, and a regularizer suppressing such a shortcut is consistent with our hypothesis. We agree that the experiment you suggest (thank you!) would be interesting to explore the mechanism(s) of length overfitting in future work.

Third, we believe there may be multiple mechanisms at play: this kind of overfitting was first shown for linear RNNs in [1]. There, it was attributed to unexplored recurrent states, and the proposed fix targets the states. Our remedy is different: plain regularization, which does not target states or positions. We show it helps in three settings: transformers with NAPE, pre-trained transformers with RoPE and SW attention, and Mamba2. Since different remedies, aimed at different mechanisms, both mitigate the problem, we believe there may be multiple causes, and identifying them is a separate research direction.

[1] Understanding and Improving Length Generalization in Recurrent Models. Ricardo Buitrago Ruiz, Albert Gu.

Thanks again for your review. If you have any further questions, we would be happy to respond during the remaining two days of the discussion period. If our response has adequately addressed your concerns, we would be grateful if you could update your assessment.

Reviewer_8FfyOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The main point of the paper is that length generalisation depends not only on architecture and positional encoding, but also on how regularisation is applied during training.

The paper studies how training and regularisation affect a model’s ability to handle sequences longer than those seen during training. The authors find that transformers can lose this ability as training goes on, and that weight decay can make this decline faster. Dropout helps, but applying it before normalisation creates a mismatch between training-time and inference-time activation statistics. Moving dropout after normalisation, directly before the linear projections, removes the mean mismatch. The authors also introduce variance preserving affine dropout (VPAD), which preserves the marginal variance and further reduces the mismatch.

The experiments are ran on transformers and mamba2, and they show improvements in length generalization, both when training from scratch and during continual pretraining. VPAD seems to provide additional gains for Transformers, even if its advantage over standard pre-linear dropout is less consistent in Mamba2.

Critical strengths
  • The paper shows that length generalisation depends on just the architecture but also on the training procedure. The authors tracked performance throughout training, and show that a model can lose the ability to handle longer sequences even as training continues. And I think this brings attention to an important point that sometimes studies on positional encoding or attention design may overlook.

  • I think that moving dropout after normalisation and directly before the linear projections is a simple idea, yet it improves extrapolation compared with the original place. The comparisons within each experimental setting make a convincing practical case that the location of dropout matters.

  • The paper also investigates how the method interacts with other choices rather than presenting it as a complete solution. The experiments show that dropout can complement attention scaling, while also documenting losses in in-distribution performance at stronger dropout rates and the mixed advantage of VPAD in mamba2.

Critical weaknesses
  • The paper shows that changing the place where to put the dropout helps, but it doesn't fully explain why it helps. I think that moving dropout after normalisation removes the mean mismatch described by the theory, but it also changes which signal actually gets the noise. So, it's possible that the improvement comes from both changes. A useful control could be to correct the mean mismatch while keeping dropout in its original position.

  • The theory makes assumptions about the input distribution, and the pre-norm result applies in the large-width limit. VPAD preserves marginal variances under these assumptions, but a covariance-dependent mismatch remains after the linear projection. The implementation also estimates the required statistics from each token. The experiments show that VPAD reduces the variance mismatch, but they do not show that it removes all train–test distribution shifts.

  • The results don't prove that weight decay generally harms length generalisation. I think the training curves show that extrapolation deteriorates earlier with weight decay in the tested setting. But it's unclear if the same pattern would hold with different decay strengths, learning rates, or training budgets.

  • The model achieves perfect recall at 64 times the training context on needle in the haystack, but it uses a modified SmolLM3 model with sliding-window attention and NoPE layers. But the performance on RULER still drops quite a bit as the context grows. So, I think it'd be important to say clearly that this impressive retrieval result doesn't mean that the model has a broader ability to solve different tasks reliably at those lengths as well.

Questions for authors

Moving dropout after normalisation removes the mean mismatch and it also changes which signal receives noise. Have you tested a control that corrects the mean mismatch while keeping dropout in its original location? This would help clarify if the improvement in length generalisation comes from matching the activation statistics or from changing how dropout regularises the model.

5 comments

Thank you for your thorough review. We are happy to address your comments below.

The paper shows that changing the place where to put the dropout helps, but it doesn't fully explain why it helps. I think that moving dropout after normalisation removes the mean mismatch described by the theory, but it also changes which signal actually gets the noise. So, it's possible that the improvement comes from both changes. A useful control could be to correct the mean mismatch while keeping dropout in its original position.

We ran the control experiment you suggested. We fed samples from the Pile to the "SW, PN-D p=0.3" checkpoint to estimate cϵc_{\epsilon} (Eq.10) and used it as a correction during inference. This indeed improved both NIAH and overall RULER scores. Our analysis uses several simplifying assumptions (e.g., i.i.d. entries), and the mismatch is proved only in expectation. Still, a single scalar correction taken directly from the theory recovers a large part of the lost performance (e.g., 6.2 → 49.6 on NIAH at 16K). We see this as strong evidence that the train-test mismatch is a real cause of the degradation.

NIAH-1

Model4K8K16K32K
PN-D p=0.3 w/o correction100.082.26.20.0
PN-D p=0.3 correction100.0100.049.65.2

RULER

Model4K8K16K32K
PN-D p=0.3 w/o correction76.122.52.10.9
PN-D p=0.3 correction81.630.76.51.7

The theory makes assumptions about the input distribution, and the pre-norm result applies in the large-width limit. VPAD preserves marginal variances under these assumptions, but a covariance-dependent mismatch remains after the linear projection. The implementation also estimates the required statistics from each token. The experiments show that VPAD reduces the variance mismatch, but they do not show that it removes all train–test distribution shifts.

As we mentioned in our general response, our goal is not to “fix everything”, but rather we show that well-designed regularization helps with length generalization. Reducing the variance mismatch is one step further in that direction. We demonstrate that indeed it can improve extrapolation, at least in transformers.

The results don't prove that weight decay generally harms length generalisation. I think the training curves show that extrapolation deteriorates earlier with weight decay in the tested setting. But it's unclear if the same pattern would hold with different decay strengths, learning rates, or training budgets.

Our observation is that weight decay accelerates length-based overfitting, and this pattern is robust across the configurations we varied:

  • Architectures. The effect is not specific to one model family: we observe it for transformers and for Mamba2.

  • Learning rates. We trained with several learning rates, and the phenomenon persists. We will add the full hyperparameter grid to the appendix.

  • Decay strength. We trained with different WD values: lower WD does not remove the deterioration, it only postpones it (see the table below). Note also that [1] uses WD = 0.01 while we use WD = 0.1 - the same pattern at two different strengths. More importantly, the overfitting also happens without weight decay (the no-WD curves in Section 2 degrade as well, only later): WD accelerates the deterioration, but it is not its only source.

Results for transformers trained from scratch with various decay strengths:

Model4K8K16K32K
WD = 0.012.272.562.823.04
WD = 0.052.634.094.685.01
WD = 0.13.325.536.36.72
  • Training budget. The budget is exactly the variable we study in Section 2: the deterioration grows as training proceeds. A larger budget therefore makes the effect stronger, not weaker, and a smaller budget only means the model has not yet reached the point where it degrades.

The model achieves perfect recall at 64 times the training context on needle in the haystack, but it uses a modified SmolLM3 model with sliding-window attention and NoPE layers. But the performance on RULER still drops quite a bit as the context grows. So, I think it'd be important to say clearly that this impressive retrieval result doesn't mean that the model has a broader ability to solve different tasks reliably at those lengths as well.

We agree that it is important to separate the two claims, and we believe the paper already does so, though, maybe, not explicitly enough. Perfect extrapolation is claimed only for NIAH-1: the abstract, contributions, and conclusion all say "extrapolates perfectly to 64x on Needle-in-a-Haystack and far beyond the pre-training context on RULER and HELMET" — for RULER and HELMET we claim only an improved range, and the tables report the decline at longer contexts in full.

The improvements also transfer to other tasks: Tables 2 and 3 report In-Context Learning, Retrieval-Augmented Generation, and LongQA results from HELMET. As for the degradation itself, it is common to all current sequence architectures: in extrapolation, performance drops as the sequence grows. Our claim is comparative: with proper regularization, the model degrades more gracefully as context length increases.

…but it uses a modified SmolLM3 model with sliding-window attention and NoPE layers.

SmolLM3 interleaves RoPE and NoPE layers by default, so NoPE is not our modification. Our only change is sliding-window attention in the RoPE layers, to avoid RoPE-related issues that are well-known to hurt extrapolation. Note that the SW baseline without dropout is exactly the same architecture and still fails to extrapolate even on NIAH-1 (Table 1), so the architecture alone does not explain the result — the gap comes from regularization.

Thank you for running the correction experiment. This provides useful evidence that the mean mismatch contributes to the degradation, since performance improves without changing the dropout placement used during training. The corrected model still stays well below pre-linear dropout, particularly on RULER, so I think I'd describe the mismatch as explaining part of the improvement rather than the full difference. If it's still possible, it'd be great if you could also clarify where the correction is applied and whether its value is shared across layers.

The additional weight-decay results strengthen the empirical finding, and I appreciate the clarification that the sliding-window baseline uses the same architecture as the dropout models. I agree that the perfect-extrapolation claim is explicitly limited to NIAH.

My remaining concern is mainly about the wording around VPAD: reducing the mismatch in the measured moments doesn't establish that all train-test distribution shifts are removed.

Reviewer_9Eh8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how regularization affects length generalization in Transformers and Mamba2. The authors find that extrapolation can deteriorate during training and that weight decay accelerates this deterioration. The authors compare conventional dropout on sublayer outputs before residual addition (Pre-Norm dropout) with dropout after RMSNorm, immediately before the input projections (Pre-Linear dropout). They analyze the resulting changes in pre-activation statistics and introduce Variance-Preserving Affine Dropout (VPAD). Experiments include training from scratch and continual pre-training, with evaluations on long-context benchmarks including NIAH, RULER, and HELMET. The results suggest that changing dropout placement improves extrapolation and VPAD could provide additional gains in some settings.

Critical strengths
  • The dropout-placement results are convincing. The paper includes WD, no-WD, PN-D, PL-D, and VPAD comparisons, and the gains from PL-D appear in both training from scratch and continual pre-training. The sliding-window baselines also help establish that the architecture change alone does not explain the gains.

  • The basic analysis is clear and the proposed method is practical. The mean shift from Pre-Norm dropout and the variance inflation from Pre-Linear dropout have straightforward derivations. The proposed methods can be applied during training without adding an extra inference module.

Critical weaknesses
  • My main concern is that the covariance formula in Proposition 3 does not cover the per-token implementation. The proof treats α\alpha and β\beta as constants computed from population moments. Section 4.4 instead computes them from the current token, making both coefficients dependent on the input. These shared random coefficients can introduce covariance that is absent from Equation (13).

    Concentration of the empirical moments does not resolve this by itself. Consider i.i.d inputs $x_i\in { -1,+1 } $ with equal probabilities. After the paper's RMSNorm, let hi=cxih_i=cx_i, where c=(1+ϵ)−1c=(1+\epsilon)^{-1}, and choose wi=1/dw_i=1/\sqrt d. Then ∥w∥2=1\lVert w\rVert_2=1, C(h,w)=0C(h,w)=0, and Var⁡(Ytest)=c2\operatorname{Var}(Y_{\mathrm{test}})=c^2. For the per-token implementation, write q=1−pq=1-p and m=d−1∑ihim=d^{-1}\sum_i h_i. Its scale satisfies

    αd2=c2−m2q(c2−qm2).\alpha_d^2=\frac{c^2-m^2}{q(c^2-qm^2)}.

    Conditional on the input, the output mean is YtestY_{\mathrm{test}}, while its variance is pqαd2c2pq\alpha_d^2c^2. The law of total variance therefore gives

    =1+pq\mathbb E[\alpha_d^2]\longrightarrow1+p.$$ Thus, a nonzero relative variance gap remains at arbitrarily large width. An analysis of the per-token estimator or additional assumptions on the projections would address this concern.
  • Section 4 analyzes a normalization layer followed by a linear projection, without a context-length variable. This explains a local train-test mismatch, but it does not yet explain why that mismatch should become especially harmful beyond the training length.

Questions for authors

See weakness.

3 comments

Thank you for your comments. We are happy to address them below.

My main concern is that the covariance formula in Proposition 3 does not cover the per-token implementation. The proof treats and as constants computed from population moments. Section 4.4 instead computes them from the current token, making both coefficients dependent on the input. These shared random coefficients can introduce covariance that is absent from Equation (13)....

Thank you for your careful analysis. Your example is indeed correct. We would like to highlight that Proposition 3 itself remains valid: it is proved for coefficients computed from the population moments. What your example reveals is that replacing population statistics with per-token ones introduces an additional variance shift. We will make this explicit in a new proposition in the appendix, and mention it in the main text.

More importantly, this does not change the overall picture. Figure 3 compares Pre-Linear dropout with VPAD, where VPAD is implemented with per-token statistics, and the variance mismatch is still reduced substantially. Mathematically, per-token VPAD always has a strictly smaller variance inflation than Pre-Linear dropout, for every weight vector, so it can be shown Var[YPL]>Var[YVPAD]\mathrm{Var}[Y^{\mathrm{PL}}] > \mathrm{Var}[Y^{\mathrm{VPAD}}]. Concretely, in your example Var[YPL]/Var[Ytest]=1+p/(1−p)\mathrm{Var}[Y^{\mathrm{PL}}]/\mathrm{Var}[Y^{\mathrm{test}}] = 1 + p/(1-p) exactly, at every width, which is larger than the 1+p1 + p attained by per-token VPAD for every p∈(0,1)p \in (0, 1).

Finally, as we mentioned in the general response, the scope of our work is not to fix dropout but to show that length extrapolation can be improved with proper regularization. The variance-mismatch analysis is additional supporting evidence for this hypothesis.

Section 4 analyzes a normalization layer followed by a linear projection, without a context-length variable. This explains a local train-test mismatch, but it does not yet explain why that mismatch should become especially harmful beyond the training length.

The last sentence of Section 4.2 gives our view on how the mismatch and the sequence length are connected: “Since the mean shifts, this can result in activation of completely different neurons, disrupting the subsequent layers, which in turn can propagate error to the next tokens by attention.”

In more detail: the mismatch appears at every token position and in every layer. During training, the entries of the query and key vectors are shrunk in expectation (Eq. 10), so at test time, with dropout off, the attention scores are distributed differently from what the model saw during training. The value vectors are shrunk in expectation in the same way, so the attention output of token tt also changes. Every token adds its own error, and the output of each token is read by all later tokens through attention. So the errors move forward and add up as the sequence grows.

Thanks again for your review. If you have any further questions, we would be happy to respond during the remaining two days of the discussion period. If our response has adequately addressed your concerns, we would be grateful if you could update your assessment.

Thank you for the clarification, which addresses my concerns. I will raise my score from 2 to 3.

Comments on the paper

2 comments

Dear AC and reviewers, We would like to highlight our contributions to resolve some confusions. We show that:

  • (1) length-based overfitting, previously demonstrated for linear RNNs in [1], also occurs in transformers (Section 2);
  • (2) weight decay accelerates this overfitting, in both transformers and Mamba2;
  • (3) the overfitting can be mitigated by proper regularization, again in both transformers and Mamba2.

The train-test mismatch analysis of dropout belongs to (3): it is our answer to what "proper" means - it explains which dropout variants help and why. VPAD takes this one step further: it shows that standard dropout does not remove the mismatch completely, and that a more careful procedure gives additional gains. We do not aim to "fix everything". The overall message of the mismatch analysis is that length generalization is sensitive to the choice of regularization - not every regularizer works out of the box. We will make this framing more explicit in the revised version.

[1] Understanding and Improving Length Generalization in Recurrent Models. Ricardo Buitrago Ruiz, Albert Gu.

Area_Chair_wYR2OpenReview ↗

Discussion encouraged in this period!

Hi all reviewers,

We’re about halfway through the discussion period. This is a friendly reminder to check the authors' replies. If the authors have answered questions or responded to critiques in the reviews, please leave a comment for the authors!

Thank you so much!

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.