- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Reference-relative RoPE (ReRoPE), which learns task-aligned positional encodings without position labels. For each token, reference tokens are selected from the K nearest occurrences of each token type, and the token's position is a mixture of RoPE rotations determined by its distances to those references. The results are promising: ReRoPE matches the closest baseline on addition, copy, reverse, and parity, and it reaches 99.7% exact-match accuracy on 1,000-digit addition after training only on lengths 1–30. However, the method builds in strong human priors, since it relies on grouping tokens by type and sorting the candidate set by distance. It is also evaluated only on synthetic arithmetic tasks.
Critical strengths
- Performance. ReRoPE achieves 99.7% exact-match accuracy on 1,000-digit operands after training only on lengths 1–30.
- Clarity. The method is well explained with simple examples in Section 3.1.
- Evaluation. The evaluation is rigorous and fair. Main results are reported across three model/data seed pairs: curves show the pointwise median with min–max shaded regions, and scalar results are reported as mean ± sample std (Section 4.3).
- Reproducibility. All training and evaluation details are disclosed, making the paper reproducible.
Critical weaknesses
- Major weakness. The claim that positional assignments are learned "from next-token prediction alone" is overstated. The method relies heavily on human priors: the candidate set is built by a human-defined procedure (visibility rule, grouping by token type, ranking by distance), and K, the number of retained candidates per type, is also a human-set constant.
- Major weakness. As acknowledged in the Limitations, it is unclear whether the proposed method improves performance on impactful applications such as natural language modelling and retrieval. The evaluation matches the design of the method, which is essentially an inductive bias for arithmetic. Main results are on toy tasks (addition, copy, reverse and parity), while decoder depth ablation is only on addition task. How well does the method perform on semantics-related tasks that also require extrapolation beyond the training length? Consider evaluating on non-arithmetic tasks such as max retrieval (https://arxiv.org/abs/2410.01104), dictionary lookup and argmax retrieval (e.g., https://arxiv.org/abs/2504.02827), and the broader algorithmic tasks in CLRS-Text (https://arxiv.org/abs/2406.04229). Max retrieval is a particularly natural test, since the paper already cites Veličković et al. (2025) in Section 3.2 to motivate bounding the number of softmax terms.
- Minor weakness. K, the hyperparameter for the number of terms in the softmax, is set manually.
- Minor weakness. There is no analysis of the method's time and memory overhead during training.
- Minor weakness. Why apply random gaps only during burn-in? An ablation on burn-in length and its impact on downstream task performance would be beneficial.
Questions for authors
Clarification request: Why apply random gaps only during burn-in? An ablation on burn-in length and its impact on downstream task performance is missing.
I am willing to reconsider my rating if the method indeed helps with natural language modelling and retrieval.
Thank you for the additional PC experiments and visualizing what your coupler learns. I think these strengthen your method w.r.t. the baselines. I will move my soundness from a 2 to a 3. At this time I will maintain my recommendation of 3.