Generalized Rotary Position Encoding for Graphs via Arbitrary Pairwise Distances

Rotary Position Embedding (RoPE) has become a key building block of modern large language models. In this work, we investigate how RoPE can be generalized from sequential offsets to arbitrary pairwise relative distances. Because the original formulation of RoPE is intrinsically 1D, and its usual multi-axis extensions apply this 1D scheme to different slices of the hidden space, it cannot naturally capture relations beyond a small number of independent axes. We introduce $\delta$-RoPE, a parameter-free generalized position encoding (PE) that directly conditions attention on an arbitrary pairwise function, making it naturally applicable to non-Euclidean structures. We explore this generalization on graphs, where the pairwise relation can naturally be instantiated as the shortest-path distance between nodes. This yields a unified formulation of position encoding in which sequential RoPE is recovered as a special case. We first evaluate $\delta$-RoPE in a standalone transformer setting, where it consistently improves over standard attention. We then evaluate it within strong graph-learning settings, including the widely used General Powerful Scalable Graph Transformer (GraphGPS) framework, using standard graph-learning benchmarks. $\delta$-RoPE yields substantial gains on established long-range graph benchmarks, where modeling non-local dependencies is particularly important. Code is available at https://anonymous.4open.science/r/delta_rope-FDFF/