acceptodds
Under review as a conference paper at ICLR 2027

Mind the Gaps: Learning Positions That Generalise

Abstract

Transformers can solve algorithmic tasks nearly perfectly within the domain; however, they typically fail on out-of-distribution evaluation, specifically on longer input sequences than those seen during training. This lack of generalisation has been partially addressed in the literature by assigning token positions according to their computational roles. Removing this guidance is challenging: the decoder can adapt to learned positional assignments that support accurate predictions within the training range but fail to generalise to longer sequences. Here, we show that models can learn positional assignments that support length generalisation from next-token prediction alone. To make these assignments easier to identify, we train across randomly gapped coordinate systems. These systems preserve exact positional matches, whereas mismatches induce varying displacements that are harder for the decoder to compensate for consistently. Building on this principle, we introduce Reference-relative RoPE (ReRoPE), which learns a mixture of reference-relative RoPE transformations. For each token, ReRoPE scores the nearest visible occurrences of every token type as candidate references, with each candidate defining a transformation in the mixture. Across addition, copy, reverse, and parity tasks, ReRoPE generalises to lengths several times the training maximum, matching supervised position coupling. Further experiments on addition show that ReRoPE retains this performance in deeper models and exceeds position coupling in terms of the maximum length to which the model generalises.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Length generalization, algorithmic reasoning, transformers, positional encodings, learned positional encodings, rotary position embeddings, position coupling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_t2RzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Reference-relative RoPE (ReRoPE), which learns task-aligned positional encodings without position labels. For each token, reference tokens are selected from the K nearest occurrences of each token type, and the token's position is a mixture of RoPE rotations determined by its distances to those references. The results are promising: ReRoPE matches the closest baseline on addition, copy, reverse, and parity, and it reaches 99.7% exact-match accuracy on 1,000-digit addition after training only on lengths 1–30. However, the method builds in strong human priors, since it relies on grouping tokens by type and sorting the candidate set by distance. It is also evaluated only on synthetic arithmetic tasks.

Critical strengths
  1. Performance. ReRoPE achieves 99.7% exact-match accuracy on 1,000-digit operands after training only on lengths 1–30.
  2. Clarity. The method is well explained with simple examples in Section 3.1.
  3. Evaluation. The evaluation is rigorous and fair. Main results are reported across three model/data seed pairs: curves show the pointwise median with min–max shaded regions, and scalar results are reported as mean ± sample std (Section 4.3).
  4. Reproducibility. All training and evaluation details are disclosed, making the paper reproducible.
Critical weaknesses
  1. Major weakness. The claim that positional assignments are learned "from next-token prediction alone" is overstated. The method relies heavily on human priors: the candidate set is built by a human-defined procedure (visibility rule, grouping by token type, ranking by distance), and K, the number of retained candidates per type, is also a human-set constant.
  2. Major weakness. As acknowledged in the Limitations, it is unclear whether the proposed method improves performance on impactful applications such as natural language modelling and retrieval. The evaluation matches the design of the method, which is essentially an inductive bias for arithmetic. Main results are on toy tasks (addition, copy, reverse and parity), while decoder depth ablation is only on addition task. How well does the method perform on semantics-related tasks that also require extrapolation beyond the training length? Consider evaluating on non-arithmetic tasks such as max retrieval (https://arxiv.org/abs/2410.01104), dictionary lookup and argmax retrieval (e.g., https://arxiv.org/abs/2504.02827), and the broader algorithmic tasks in CLRS-Text (https://arxiv.org/abs/2406.04229). Max retrieval is a particularly natural test, since the paper already cites Veličković et al. (2025) in Section 3.2 to motivate bounding the number of softmax terms.
  3. Minor weakness. K, the hyperparameter for the number of terms in the softmax, is set manually.
  4. Minor weakness. There is no analysis of the method's time and memory overhead during training.
  5. Minor weakness. Why apply random gaps only during burn-in? An ablation on burn-in length and its impact on downstream task performance would be beneficial.
Questions for authors

Clarification request: Why apply random gaps only during burn-in? An ablation on burn-in length and its impact on downstream task performance is missing.

I am willing to reconsider my rating if the method indeed helps with natural language modelling and retrieval.

Reviewer_GYrnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Reference-relative RoPE, which learns positional representations from next-token prediction without explicit position labels. Each token selects a weighted set of reference occurrences and mixes rotary transformations based on its distances to those references. On addition, copy, reverse, and running parity, the proposed method matches supervised Position Coupling at the primary evaluation lengths. The extended addition experiment reports 99.67% mean exact-match accuracy on 1k digits operands after training on lengths 1–30.

Critical strengths
  1. Learning positional information without manually position labels addresses a real limitation of Position Coupling. Matching its primary results with matched training settings and budgets, is a positive result.

  2. The random-gap ablation is persuasive within the tested setting. Removing gaps preserves perfect accuracy at the training boundary but reduces accuracy to 2.15% at length 100 and zero at lengths 200. Global position scaling also fails. This provides a useful distinction between fitting short examples and learning a representation that transfers to longer cases.

Critical weaknesses
  1. Important baselines are missing.

The paper does not compare with NoPE, a relevant baseline for length generalisation. The 1k digits comparison also uses different positional representations: ReRoPE uses rotations, while Position Coupling uses learned absolute position embeddings. Without a matched Position Coupling + RoPE baseline and clearer position-table settings, it is difficult to separate the contribution of reference learning from that of the positional representation.

  1. The evaluation is limited to highly controlled formats.

The tasks use small vocabularies and fixed formats, including padding and reversed outputs for addition. Although the paper acknowledges these restrictions, it does not test robustness to format changes or provide a systematic analysis of the learned reference patterns. It is therefore unclear how well the learned alignments transfer beyond these settings.

  1. The additional cost is unclear.

Each token considers up to K references per visible token type, so the candidate count can grow substantially with vocabulary diversity. No runtime or memory results are reported. These costs need to be measured to assess the method's practical value.

Questions for authors
  1. Can you report matched NoPE and Position Coupling + RoPE results on the extended addition test?

  2. What reference patterns does the trained model learn? Does length generalisation still hold when training and evaluating with a simple format change, such as a different separator or no operand padding, under the same training budget?

  3. How much training time, prefill time, decoding time, and peak memory does ReRoPE overhead to standard RoPE at matched sequence lengths? How do these costs scale with input length?

Reviewer_PkqsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This work proposed ReRoPE, a novel positional encoding method that learns task-aligned positions for arithmetic tasks without direct supervision. They add a lightweight scorer that learns to assign positional scores to K nearest occurrences of each token type. Each token’s position is then represented as a mixture of RoPE rotations based on the relative distance to each top K token weighted by the score assigned by the scorer. This combined with randomized distances leads to a final model with very strong extrapolation performance. They demonstrate it matches Position Coupling which introduces explicit position supervision.

Critical strengths
  • Their method removes the need for hand-designed position rules that previous works demonstrated to be beneficial. Directly learning the rules is an intuitive and reasonable idea.
  • The randomized gap augmentation is simple and substantially boosts extrapolation performance.
Critical weaknesses
  • The authors never tested the gains from using RoPE in Position Coupling. Their own experiments showed that using learned positional embeddings with their scorer worked worse than RoPE but they never tested Position Coupling + RoPE. That experiment would strengthen their claim that learning the couplings is beneficial.
  • The paper lacks analysis of what patterns the scorer learns. Some analysis or discussion on that would improve understanding of why it works.
Questions for authors
  • Could the authors share the learned weights/distribution/pattern of their Coupler mechanism and compare it to existing coupling designs?
  • How does Position Coupling + RoPE perform? Do the randomized gaps improve Position Coupling too?

Nit

  • The name ReRoPE (rectified RoPE, Jianlin Su’s blogpost) already exists and is relatively well known.
1 comment

Thank you for the additional PC experiments and visualizing what your coupler learns. I think these strengthen your method w.r.t. the baselines. I will move my soundness from a 2 to a 3. At this time I will maintain my recommendation of 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.