acceptodds
Under review as a conference paper at ICLR 2027

Same Code, Different Reach: Learning Positional Comparisons for Length Extrapolation

Abstract

Length extrapolation in copying requires reliable source retrieval beyond the training range. We study how positional structure and training shape this learned computation through controlled experiments in small transformers. Across six harmonic frequency grids and twelve paired seeds, 25% frequency-plane dropout improves correct-history token accuracy at four times the training length by 34–41 percentage points on four grids; one shows no resolved gain and one worsens. All bands are restored at evaluation, so these gains arise with the positional code held fixed. To investigate the learned addressing computation, we distinguish context matching from selection among matching occurrences. Positional features help construct retrieval contexts as well as distinguish addresses. A task-derived contrast predicts mean spectral-growth peaks at four training lengths, but different learned band mixtures implement similar local comparisons with divergent distant responses. Removing distant attention competitors improves context selection and copying. Frozen short-range spectral forecasts locate consequential distant attention, although equal-mass controls show no resolved accuracy advantage for the exact selected addresses. Matched continuations and signed-coefficient transfers establish a contributing role for learned positional comparisons. Band interventions reveal useful alternatives, and a conditional subset-margin theorem gives sufficient conditions for distant separation in a reduced model after all bands are restored. Context discrimination improves, while ambiguity among matching occurrences remains. Together, these findings distinguish a positional code’s available comparisons from those learned through training, and identify support-dependent positional regularization as a promising approach to length extrapolation.

Primary area:
interpretability and explainable AI
Keywords:
length extrapolation, positional encoding, transformer copying, attention-based addressing, spectral structure, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SMi6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies length extrapolation in small transformers trained on synthetic copying with additive harmonic positional encodings. Its main intervention is “component dropout”: whole sine/cosine bands are randomly removed during training and all bands are restored at evaluation.

Critical strengths

The experimental controls are careful: paired seeds, seed-level bootstrap intervals, some predictions frozen before the runs, and failed cohorts reported rather than dropped.

The PE-gap probe (fixed tokens, shifted positional coordinates) cleanly separates positional effects from content effects.

Critical weaknesses

1. Limited novelty; key related work is not discussed. The central finding is that distant attention mass interferes with context construction and that restricting it restores extrapolation (Sec. 5, App. L). This overlaps substantially with prior work that the paper neither cites nor discusses:

  • Distant-attention restriction: LM-Infinite (Han et al., NAACL 2024) attributes length-generalization failure to attention-entropy growth and proposes a Λ-shaped mask; the paper's Local243 intervention (local window plus BOS/separator) is essentially this mask. StreamingLLM (Xiao et al., ICLR 2024) similarly shows that initial/marker tokens must be kept.
  • Attention dilution and scaling: Chiang & Cholak (ACL 2022), Scalable-Softmax (Nakanishi, 2025) and the attention temperature in YaRN (Peng et al., ICLR 2024).
  • Frequency-band-specific methods: PI (Chen et al., 2023), YaRN/NTK-by-parts, p-RoPE (Barbero et al., ICLR 2025; cited but not discussed in this respect) and DoPE (Xiong et al., 2025).
  • Positional schemes for length generalization: FIRE (Li et al., ICLR 2024). Randomized PE (Ruoss et al., 2023) is cited but not compared against.

The paper's own equal-mass control (App. N) finds no advantage for forecast-selected keys over diffusely removing the same distant mass. This supports a generic attention-dilution explanation rather than one specific to learned harmonic comparisons, and leaves band-level dropout as an incremental contribution.

2. The experimental scale is insufficient.

  • Setting: 2-layer, width-64 models (~10^5 parameters), trained at length ≤120 with vocabulary 4 and tested at length 480. This says little about length extrapolation in practice. The tiny vocabulary also makes repeated n-grams pervasive, which inflates exactly the ambiguity the paper analyzes.
  • Gains do not hold where it matters: they are concentrated in additive sinusoidal encodings. With RoPE (Table 3) they are negligible or negative, except +4.25 points at V=8, where absolute accuracy is only 17%. Under greedy decoding (Table 2), most grids gain only a few points.
  • Missing baseline: there is no working generic-dropout baseline; App. G acknowledges the comparator failed. The benefit is therefore not separated from ordinary regularization.
  • Minimum expected evaluation:
    • RoPE models of ≥100M parameters trained on natural text with ≥4K context.
    • Evaluation well beyond the training length (e.g., 16K–64K) on PPL and RULER/NIAH.
    • Baselines: generic dropout, Randomized PE, PI/YaRN, attention scaling and Λ-shaped masking.

Since the setting does not support the central claim, I do not evaluate the mechanistic analyses or the theorem in detail.

Questions for authors
  1. Does band dropout outperform standard dropout at matched short-range loss?

  2. Do the gains hold for RoPE models at ≥100M parameters with ≥4K training context on natural-language benchmarks?

  3. How does the method compare with Λ-shaped masking or attention logit scaling, which directly target the distant-interference mechanism the paper identifies?

Reviewer_qBz3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates length extrapolation in copying tasks. The authors demonstrate that even when Transformers use the same positional encoding and frequency components, their length extrapolation performance can differ substantially depending on the positional comparisons learned during training.

In particular, the authors argue that randomly dropping frequency components of Sinusoidal Positional Encoding during training can prevent the model from relying too heavily on specific frequencies, thereby improving copying performance at longer sequence lengths.

However, the experiments are conducted on a relatively small scale, and the paper does not evaluate standard length extrapolation benchmarks such as RULER or Needle-in-a-Haystack (NIAH). The authors primarily focus on length extrapolation in copying tasks, with most experiments conducted using small Transformers on synthetic copying tasks. Therefore, the findings do not establish that the proposed approach improves long-context understanding in general-purpose LLMs. It is also worth noting that the proposed approach is not primarily developed or evaluated using RoPE, which is a commonly used positional encoding baseline in modern LLMs.

Critical strengths
  • The authors demonstrate that the frequency configuration of positional encodings alone does not determine length extrapolation performance in copying tasks. I find this to be an interesting insight.
  • The paper analyzes how learned positional comparisons affect context retrieval in long-range copying. Since this aspect has not been extensively investigated in the context of copying tasks, I consider this analysis to be one of the contributions of the paper.
Critical weaknesses

I do not think the claim in the Abstract that "support-dependent positional regularization as a promising approach to length extrapolation" is sufficiently supported by the experiments presented in this paper. The paper primarily focuses on copying tasks, rather than standard benchmarks commonly used to evaluate length extrapolation. Therefore, I believe the experimental setting does not adequately support such a broad claim. If the authors wish to argue that their approach is promising for improving length extrapolation in general, substantially more experimental evidence is needed, particularly on standard length extrapolation benchmarks and larger models. I would find the claim more convincing if the authors narrowed its scope to copying tasks, for example, by stating that "support-dependent positional regularization is a promising approach to improving length extrapolation in copying tasks." However, even within the scope of copying tasks, I believe the experimental evaluation is still insufficient. The experiments are limited in terms of model sizes, evaluation settings, and comparison methods. More extensive experiments are needed to establish the effectiveness and generalizability of the proposed approach, even for copying tasks.

Questions for authors

Could you clarify the scope of your main claim? Are you claiming that your approach is "a promising approach to improving length extrapolation in copying tasks," or that it is "a promising approach to improving length extrapolation in general"?

Reviewer_1Z9aOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

As I have communicated to the AC, I do not have sufficient expertise to assess this submission reliably.

Critical strengths

As I have communicated to the AC, I do not have sufficient expertise to assess this submission reliably.

Critical weaknesses

As I have communicated to the AC, I do not have sufficient expertise to assess this submission reliably.

Questions for authors

As I have communicated to the AC, I do not have sufficient expertise to assess this submission reliably.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.