- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes -RoPE, a parameter-free generalization of Rotary Position Embedding (RoPE) that replaces the relative positional offset in the original formulation with an arbitrary pairwise function, making it applicable to non-Euclidean structures. In particular, the method uses shortest-path distance (SPD) as the pairwise function for graph-structured data. Across eight graph benchmarks, -RoPE improves mean performance over standard attention in the standalone setting and achieves notable improvements on several long-range benchmarks when integrated into GraphGPS.
Critical strengths
S1. This paper is well structured and easy to read.
S2. The proposed method shows promising empirical performance, particularly on the long range graph benchmark.
Critical weaknesses
W1. Limited methodological novelty and insufficient justification of the proposed generalization.
The main methodological contribution is to replace the relative positional offset in the original RoPE formulation with an arbitrary pairwise function, which appears to be a relatively straightforward extension. More importantly, the paper provides limited discussion of how the proposed distance-dependent rotations interact with the attention mechanism and how the resulting attention patterns reflect the underlying graph structure. In particular, it remains unclear what structural inductive biases are introduced by this formulation and why it is an appropriate way to capture graph relationships. These concerns raise questions about both the methodological novelty and the justification for the proposed approach.
W2. Limited discussion of alternative pairwise relations.
Although the proposed framework supports arbitrary pairwise functions, the paper only considers shortest-path distance (SPD), with limited discussion of alternative choices. Other pairwise measures, such as effective resistance, diffusion distance, or random-walk-based relations, capture different aspects of graph structure beyond SPD. Given the generality claimed by the proposed framework, a more thorough discussion of these alternatives, including their potential advantages and limitations, would help clarify the scope and applicability of the proposed approach.
W3. Insufficient comparisons with existing graph positional encodings.
The experimental evaluation lacks sufficiently controlled comparisons with existing graph positional encodings. In Table 2, -RoPE is incorporated into GraphGPS while retaining existing positional encodings (e.g., LapPE, RWSE, and ESLapPE). While this demonstrates its additional benefits, the paper does not sufficiently investigate how -RoPE interacts with these encodings or whether it can serve as an effective alternative to them. Furthermore, WIRE, a closely related approach that also extends RoPE to graphs, is included only with results obtained under a different backbone and training configuration, making a direct comparison difficult. More controlled comparisons with existing relative positional encodings, particularly WIRE, and ablations with and without existing encodings would help clarify the distinct contribution of -RoPE.
Questions for authors
Q1. Could the authors further clarify what structural inductive biases are introduced by the proposed rotary formulation and how the distance-dependent rotations affect attention in relation to the underlying graph structure?
Q2 Could the authors discuss one or two representative alternatives to SPD, such as effective resistance, diffusion distance, or random-walk-based relation, and their potential advantages and limitations within the -RoPE framework?
Q3. Could the authors provide ablation studies with and without existing positional encodings (e.g., LapPE, RWSE, and ESLapPE) to clarify how -RoPE interacts with or complements them?
Q4. Could the authors provide more controlled comparisons with existing relative positional encodings, particularly WIRE, under comparable settings?
2 comments
Thank you for the response. The additional comparisons are helpful and address some of my concerns. However, I still think the methodological novelty is limited. Specifically, I find the justification for replacing RoPE’s positional offset with an arbitrary scalar-valued graph relation insufficient. The response clarifies how this extension works, but does not sufficiently explain why the resulting inductive bias is particularly appropriate for capturing graph relationships. Due to remaining concern, I maintain my score.
Response to Reviewer 1BTL
We thank the reviewer for the thoughtful feedback.
W1 / Q1 (novelty and inductive bias). Why RoPE works so well in LLMs is still not fully understood. Barbero et al. ("Round and Round We Go! What makes Rotary Positional Encodings useful?", ICLR 2025) show that the multi-frequency structure lets the model organize itself: in a trained Gemma 7B, the highest frequencies are used to build robust positional attention patterns (e.g., previous-token heads), while the lowest frequencies, which barely rotate over the context, are used as nearly position-independent channels. δ-RoPE provides the same multi-frequency structure, with the graph distance in place of the token offset. On graphs, attention depends on the structure only through pairwise distances, so it is permutation-equivariant and imposes no locality: all node pairs remain reachable, and the model can learn which distances matter. Unlike an additive distance bias such as Graphormer's, which shifts the score of all pairs at a given distance by the same amount, the rotation modulates the query–key interaction itself, so the effect of distance depends on the content of the two nodes.
W2 / Q2 (alternative pairwise relations). We chose the shortest-path distance because it is the direct graph analogue of what RoPE encodes on sequences, the distance between two tokens (on a path graph, the two coincide). The formulation is, however, not tied to it, and we ran δ-RoPE with the effective resistance in place of the shortest-path distance, inside GraphGPS, with the same configuration and base b=20 (seed 0):
Effective resistance performs within seed variance of the shortest-path distance (slightly above on CIFAR10 and Peptides-func, slightly below on MNIST and PCQM-Contact), which shows that δ-RoPE can be used with other scalar pairwise relations without any change. We agree that diffusion distances and random-walk relations are interesting as well; they are, however, naturally a vector over diffusion times or walk lengths rather than a single scalar, unless a time scale is fixed. Extending δ-RoPE to such vector-valued relations, e.g., by assigning different groups of frequencies to different time scales, is left for future work.
W3 / Q3 (interaction with absolute PEs). We added an ablation of the absolute PE inside GraphGPS (one seed), comparing standard attention and δ-RoPE with LapPE and without any absolute PE. On Peptides-func, δ-RoPE without any absolute PE (69.84 AP) still outperforms standard attention with LapPE (64.71), and removing LapPE costs δ-RoPE less than one point; on PCQM-Contact LapPE has little effect on δ-RoPE (47.83 vs. 47.77) and helps standard attention slightly more (47.07 vs. 46.71), and on MNIST it helps both attentions by a similar amount. δ-RoPE is therefore complementary to absolute encodings, and on Peptides-func it can largely replace them.
W3 / Q4 (controlled comparison with WIRE). WIRE is now trained by us under the same settings as δ-RoPE, both in the standalone setting (Table 1) and inside GraphGPS (Table 2), with 4 seeds (see general comment).