- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies a synthetic positional retrieval task in which the target token's index is provided. It theoretically analyzes how the choice of RoPE frequencies affects the query–key norm required for reliable retrieval, particularly when the same frequencies must be reused across different sequence lengths and target positions. The authors also investigate robustness to frequency perturbations and provide supporting numerical experiments.
Critical strengths
The paper presents a mathematically precise formulation of positional retrieval and derives theoretical results concerning frequency reuse and robustness. However, given the presentation difficulties described above, I was unable to confidently assess the correctness and significance of these contributions.
Critical weaknesses
-
Poor presentation. The paper is poorly written, with many concepts/jargons used with little to no proper introduction/descriptions. I find it hard to read/understand. Since I believe the main purpose of publication is to communicate knowledge to each other, a poorly-written paper would has limited contribution to the community.
-
Practicality of the synthetic task. It's unclear to me what practical value the proposed theoretical results have. I find it hard to imagine a use case where the model is provided with ground truth token index to pay attention to.
Questions for authors
- Why must a known address be retrieved through RoPE attention instead of direct indexing?
- In what realistic architectures does this constraint arise?
- Do the resulting mathematical guarantees provide useful insights for ordinary Transformers?
2 comments
Response to Reviewer ZUqq
Thank you for your follow-up and constructive feedback. We understand your remaining concerns regarding practical utility, real-world transfer, and presentation clarity.
We would like to emphasize that the primary goal of this work is to establish a theoretical understanding of reliable positional retrieval under RoPE, rather than to propose an immediately deployable long-context solution. Specifically, we characterize the cost of sharing a fixed positional representation across different sequence lengths and target positions, prove that exactly one additional rotation block is necessary and sufficient for constant-factor competitiveness with task-specific designs, and establish the critical frequency-error tolerance of . We further show that adapting a single frequency can restore optimal-order robustness and develop convex calibration methods to quantify the associated retrieval costs. Our finite experiments provide additional evidence supporting these theoretical findings.
Our theoretical findings are directly relevant to RoPE-based language models and may inform the design of more reliable positional representations. However, extending these guarantees to full-scale language models requires additional empirical validation, which we are currently completing this experiment and commit to including the results in a future version. However, we would like to reiterate that our core contributions are primarily: we characterize the cost of sharing RoPE spectra across different sequence lengths and target positions for reliable retrieval, establish the necessity of an additional rotation block and the robustness benefits of single-frequency adaptation, and develop a convex calibration framework to quantify the associated retrieval costs.
We appreciate your time and thoughtful evaluation. We sincerely hope for your support.
Thank you for your responses. While my core concerns regarding the practical utility and real-world transfer gap remain, I appreciate the author's clarification. The paper still requires significant effort to follow. I will thus keep my score for now.