acceptodds
Under review as a conference paper at ICLR 2027

Generalized Rotary Position Encoding for Graphs via Arbitrary Pairwise Distances

Abstract

Rotary Position Embedding (RoPE) has become a key building block of modern large language models. In this work, we investigate how RoPE can be generalized from sequential offsets to arbitrary pairwise relative distances. Because the original formulation of RoPE is intrinsically 1D, and its usual multi-axis extensions apply this 1D scheme to different slices of the hidden space, it cannot naturally capture relations beyond a small number of independent axes. We introduce -RoPE, a parameter-free generalized position encoding (PE) that directly conditions attention on an arbitrary pairwise function, making it naturally applicable to non-Euclidean structures. We explore this generalization on graphs, where the pairwise relation can naturally be instantiated as the shortest-path distance between nodes. This yields a unified formulation of position encoding in which sequential RoPE is recovered as a special case. We first evaluate -RoPE in a standalone transformer setting, where it consistently improves over standard attention. We then evaluate it within strong graph-learning settings, including the widely used General Powerful Scalable Graph Transformer (GraphGPS) framework, using standard graph-learning benchmarks. -RoPE yields substantial gains on established long-range graph benchmarks, where modeling non-local dependencies is particularly important. Code is available at https://anonymous.4open.science/r/delta_rope-FDFF/

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Rotary Position Embedding, Graph Transformer, long-range interactions, LRGB

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1BTLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes δ\delta-RoPE, a parameter-free generalization of Rotary Position Embedding (RoPE) that replaces the relative positional offset in the original formulation with an arbitrary pairwise function, making it applicable to non-Euclidean structures. In particular, the method uses shortest-path distance (SPD) as the pairwise function for graph-structured data. Across eight graph benchmarks, δ\delta-RoPE improves mean performance over standard attention in the standalone setting and achieves notable improvements on several long-range benchmarks when integrated into GraphGPS.

Critical strengths

S1. This paper is well structured and easy to read.

S2. The proposed method shows promising empirical performance, particularly on the long range graph benchmark.

Critical weaknesses

W1. Limited methodological novelty and insufficient justification of the proposed generalization.

The main methodological contribution is to replace the relative positional offset in the original RoPE formulation with an arbitrary pairwise function, which appears to be a relatively straightforward extension. More importantly, the paper provides limited discussion of how the proposed distance-dependent rotations interact with the attention mechanism and how the resulting attention patterns reflect the underlying graph structure. In particular, it remains unclear what structural inductive biases are introduced by this formulation and why it is an appropriate way to capture graph relationships. These concerns raise questions about both the methodological novelty and the justification for the proposed approach.

W2. Limited discussion of alternative pairwise relations.

Although the proposed framework supports arbitrary pairwise functions, the paper only considers shortest-path distance (SPD), with limited discussion of alternative choices. Other pairwise measures, such as effective resistance, diffusion distance, or random-walk-based relations, capture different aspects of graph structure beyond SPD. Given the generality claimed by the proposed framework, a more thorough discussion of these alternatives, including their potential advantages and limitations, would help clarify the scope and applicability of the proposed approach.

W3. Insufficient comparisons with existing graph positional encodings.

The experimental evaluation lacks sufficiently controlled comparisons with existing graph positional encodings. In Table 2, δ\delta-RoPE is incorporated into GraphGPS while retaining existing positional encodings (e.g., LapPE, RWSE, and ESLapPE). While this demonstrates its additional benefits, the paper does not sufficiently investigate how δ\delta-RoPE interacts with these encodings or whether it can serve as an effective alternative to them. Furthermore, WIRE, a closely related approach that also extends RoPE to graphs, is included only with results obtained under a different backbone and training configuration, making a direct comparison difficult. More controlled comparisons with existing relative positional encodings, particularly WIRE, and ablations with and without existing encodings would help clarify the distinct contribution of δ\delta-RoPE.

Questions for authors

Q1. Could the authors further clarify what structural inductive biases are introduced by the proposed rotary formulation and how the distance-dependent rotations affect attention in relation to the underlying graph structure?

Q2 Could the authors discuss one or two representative alternatives to SPD, such as effective resistance, diffusion distance, or random-walk-based relation, and their potential advantages and limitations within the δ\delta-RoPE framework?

Q3. Could the authors provide ablation studies with and without existing positional encodings (e.g., LapPE, RWSE, and ESLapPE) to clarify how δ\delta-RoPE interacts with or complements them?

Q4. Could the authors provide more controlled comparisons with existing relative positional encodings, particularly WIRE, under comparable settings?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1BTL

We thank the reviewer for the thoughtful feedback.

W1 / Q1 (novelty and inductive bias). Why RoPE works so well in LLMs is still not fully understood. Barbero et al. ("Round and Round We Go! What makes Rotary Positional Encodings useful?", ICLR 2025) show that the multi-frequency structure lets the model organize itself: in a trained Gemma 7B, the highest frequencies are used to build robust positional attention patterns (e.g., previous-token heads), while the lowest frequencies, which barely rotate over the context, are used as nearly position-independent channels. δ\delta-RoPE provides the same multi-frequency structure, with the graph distance in place of the token offset. On graphs, attention depends on the structure only through pairwise distances, so it is permutation-equivariant and imposes no locality: all node pairs remain reachable, and the model can learn which distances matter. Unlike an additive distance bias such as Graphormer's, which shifts the score of all pairs at a given distance by the same amount, the rotation modulates the query–key interaction itself, so the effect of distance depends on the content of the two nodes.

W2 / Q2 (alternative pairwise relations). We chose the shortest-path distance because it is the direct graph analogue of what RoPE encodes on sequences, the distance between two tokens (on a path graph, the two coincide). The formulation is, however, not tied to it, and we ran δ\delta-RoPE with the effective resistance in place of the shortest-path distance, inside GraphGPS, with the same configuration and base b=20b = 20 (seed 0):

DatasetSPD (seed 0)SPD (4 seeds)Effective resistance (seed 0)
MNIST (Acc.)98.2298.24±0.0598.24 \pm 0.0598.11
CIFAR10 (Acc.)71.5572.22±0.6772.22 \pm 0.6772.73
Peptides-func (AP)70.7670.60±0.4870.60 \pm 0.4871.11
PCQM-Contact (MRR filt.)47.8347.64±0.1947.64 \pm 0.1947.43

Effective resistance performs within seed variance of the shortest-path distance (slightly above on CIFAR10 and Peptides-func, slightly below on MNIST and PCQM-Contact), which shows that δ\delta-RoPE can be used with other scalar pairwise relations without any change. We agree that diffusion distances and random-walk relations are interesting as well; they are, however, naturally a vector over diffusion times or walk lengths rather than a single scalar, unless a time scale is fixed. Extending δ\delta-RoPE to such vector-valued relations, e.g., by assigning different groups of frequencies to different time scales, is left for future work.

W3 / Q3 (interaction with absolute PEs). We added an ablation of the absolute PE inside GraphGPS (one seed), comparing standard attention and δ\delta-RoPE with LapPE and without any absolute PE. On Peptides-func, δ\delta-RoPE without any absolute PE (69.8469.84 AP) still outperforms standard attention with LapPE (64.7164.71), and removing LapPE costs δ\delta-RoPE less than one point; on PCQM-Contact LapPE has little effect on δ\delta-RoPE (47.8347.83 vs. 47.7747.77) and helps standard attention slightly more (47.0747.07 vs. 46.7146.71), and on MNIST it helps both attentions by a similar amount. δ\delta-RoPE is therefore complementary to absolute encodings, and on Peptides-func it can largely replace them.

W3 / Q4 (controlled comparison with WIRE). WIRE is now trained by us under the same settings as δ\delta-RoPE, both in the standalone setting (Table 1) and inside GraphGPS (Table 2), with 4 seeds (see general comment).

Thank you for the response. The additional comparisons are helpful and address some of my concerns. However, I still think the methodological novelty is limited. Specifically, I find the justification for replacing RoPE’s positional offset with an arbitrary scalar-valued graph relation insufficient. The response clarifies how this extension works, but does not sufficiently explain why the resulting inductive bias is particularly appropriate for capturing graph relationships. Due to remaining concern, I maintain my score.

Reviewer_DLuzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

δ-RoPE replaces the sequential offset in RoPE's relative attention score with an arbitrary pairwise relation, instantiated on graphs as the shortest-path distance. It is parameter-free and reduces to standard RoPE on sequences. Evaluated as a standalone transformer against plain attention, and as GraphGPS's attention branch against published baselines, it beats plain attention everywhere and improves GraphGPS clearly on the two Peptides tasks, slightly on PCQM-Contact, and matches it elsewhere.

Critical strengths
  • Clean, drop-in formulation. The "two readings of RoPE" argument shows why only a pairwise relation is needed, so no node coordinates or eigendecomposition are required (unlike WIRE). No new weights; honest about within-variance results.
  • Real gain where long-range structure matters. The Peptides improvements are well beyond seed noise with no per-dataset tuning.
Critical weaknesses
  • The claimed novelty, the rotary form of distance conditioning, is never isolated. The standalone baseline gets no distance information at all, so Table 1 only shows that distance beats nothing. Missing controls: a Graphormer-style learned distance bias, a fixed sinusoidal bias, hop-limited attention, and the RRWP encoding removed from the inherited GRIT config. Graphormer is discussed as the closest method but appears in no table. GraphGPS numbers are copied rather than rerun, although the timing study shows the authors ran it.
  • The clear improvements come from one graph family (Peptides). The "long-range" explanation is untested: no attention-vs-distance analysis, and the single-seed base ablation covers only the two smallest-diameter datasets. Training is several times slower, and the two largest datasets could not be timed. The "unified across modalities" framing is unsupported: with a general relation the score no longer factors into per-token rotations, losing cached keys, fused kernels and linear-attention compatibility.
Questions for authors
  • Does the best base grow with diameter? A multi-seed sweep on Peptides and an attention-vs-distance plot would test the long-range claim.
  • What distance is used for disconnected node pairs?
2 comments
AuthorsOpenReview ↗

Response to Reviewer DLuz (1/2)

We thank the reviewer for the constructive and precise feedback.

W1 (isolating the rotary form). Table 1 now compares δ\delta-RoPE with a Graphormer-style learned distance bias (SPDBias) and with WIRE under the same configuration (see general comment). Graphormer and GRPE, evaluated with their original architecture as proposed by their authors, now also appear in Table 2, where they match δ\delta-RoPE on PCQM-Contact but remain behind on all other datasets. Note that the plain-attention baseline of Table 1 was not blind to the graph: all models of Table 1 use the same absolute position encoding (LapPE, same eigenvectors and encoder). Regarding a fixed sinusoidal bias: such encodings are designed to express a linear offset between positions, and it is not straightforward to see how they would handle arbitrary pairwise distances. Regarding GraphGPS, we ran it ourselves with the benchmark configurations on all eight datasets as part of our initial experiments (one seed per dataset, to save compute) and obtained results consistent with the published ones, e.g. ZINC 0.07770.0777 (published 0.074±0.0060.074 \pm 0.006), MNIST 98.2398.23 (98.18±0.1298.18 \pm 0.12) and Peptides-struct 0.25140.2514 (0.2497±0.00240.2497 \pm 0.0024); on Peptides-func, PascalVOC-SP and COCO-SP our single run is below the published mean, so reporting the published numbers is the conservative choice for our comparison. We do not use GRIT's RRWP encoding because it is itself a relative (pairwise) encoding and stacking it with δ\delta-RoPE would blur the comparison of the attention mechanism. All models of Table 1 therefore use the same absolute encoding (LapPE).

W2 (gains beyond Peptides). Within GraphGPS, we agree that the clear gains are on the molecular LRGB datasets: in addition to Peptides, δ\delta-RoPE also improves PCQM-Contact beyond seed variance, while on COCO-SP, ZINC and MNIST it obtains a better mean score than GraphGPS, within seed variance. In the standalone setting, δ\delta-RoPE performs better than all baselines on 3 of the 4 tested benchmarks.

AuthorsOpenReview ↗

Response to Reviewer DLuz (2/2)

W2 / Q1 (long-range explanation, base sweep). We used the term "long-range" because the gains of δ\delta-RoPE appear on several benchmarks of the Long Range Graph Benchmark, while on the canonical benchmarks of Table 8 it is on par with GraphGPS. We agree, however, that this does not fully establish that the gains come from long-range interactions, since plain attention already attends globally, and we have adapted the phrasing of our submission accordingly.

Following the reviewer's suggestion, we compared how attention is distributed over shortest-path distances in the trained GraphGPS models with plain attention and with δ\delta-RoPE (validation set, seed 0). For each head hh, we compute the mean shortest-path distance of the attended nodes, weighted by attention, δˉh=1∣V∣∑m∑nsm,nhδ(m,n)\bar{\delta}_h = \frac{1}{|\mathcal{V}|} \sum_{m} \sum_{n} s^h_{m,n} \delta(m,n), where sm,nhs^h_{m,n} are the attention weights of head hh and the outer sum runs over all query nodes mm:

Peptides-funcZINC
Mean distance between node pairs29.85.2
Plain attention: range over heads26.2 – 29.25.0 – 5.7
δ\delta-RoPE: range over heads17.7 – 32.34.2 – 7.6

With plain attention, all heads attend to nearly the same mix of distances, whereas δ\delta-RoPE heads are distance-selective, from local to global ones, and the trained model organises its layers from local to global. This supports our interpretation that δ\delta-RoPE lets the model separate short- and long-range interactions.

We also repeated the base sweep inside GraphGPS on MNIST, CIFAR10 and Peptides-func, the dataset with the largest diameter (b∈{5,…,10000}b \in \{5, \dots, 10000\}, one seed, given time and compute constraints).

W2 (training cost, untimed datasets). We added a scaling study on synthetic graphs up to 2142^{14} nodes (see general comment). δ\delta-RoPE is clearly slower than plain attention, but its cost is close to that of similar distance-based attentions. On the largest graphs, during training, its time is within 15%15\% of the learned SPD bias and between 0.4×0.4\times and 2.6×2.6\times that of WIRE, with 1.4×1.4\times the memory of the SPD bias; at inference, its time is within 5%5\% of the SPD bias and between 0.17×0.17\times and 2.4×2.4\times that of WIRE, with the same memory as the SPD bias. We also completed Table 3 with the previously untimed datasets.

W2 ("unified across modalities"). We agree that for a general relation the score no longer factors into per-token rotations, so δ\delta-RoPE loses part of the efficiency of standard RoPE. However, for graph-structured inputs, keys can still be cached un-rotated, the rotation by δ(m,n)\delta(m,n) being applied when the score is computed. Our claim is thus a unified formulation rather than a unified efficient implementation.

Q2 (disconnected pairs). In our setup, node pairs in different connected components are assigned a distance equal to the number of nodes, which is larger than any real distance. This only concerns the Peptides datasets, where about 1%1\% of graphs have more than one component (0.3%0.3\% of node pairs); all graphs of the other benchmarks are connected. Other choices, such as masking attention between components (infinite distance), are left for future work.

Reviewer_wHsZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends Rotary Position Embedding (RoPE) from sequential offsets to arbitrary pairwise relations. For graph learning, the proposed δ-RoPE replaces the signed token offset in RoPE with the shortest-path distance between two nodes. The authors evaluate δ-RoPE in a standalone Transformer with an edge-to-node transformation and as the global attention module in GraphGPS. Experiments are conducted on eight graph benchmarks. The reported results show that δ-RoPE consistently outperforms plain attention in the standalone setting and improves GraphGPS on several long-range graph benchmarks, especially PCQM-Contact and the two Peptides datasets.

Critical strengths
  1. The paper is well-written and easy to follow. The connection between standard RoPE and the proposed pairwise formulation is clearly explained.

  2. The method is simple and parameter-free. It can be inserted into an existing attention module without introducing additional learnable parameters. The controlled comparison against plain attention in Table 1 also shows that shortest-path information can be useful on these benchmarks.

Critical weaknesses
  1. The technical novelty is limited. The main design directly replaces the one-dimensional offset in RoPE with shortest-path distance. Pairwise graph relations and shortest-path-distance attention have already been extensively studied in Graphormer, GRPE, and recent path-based positional encodings. WIRE also directly extends rotary position encoding to graphs. The paper does not clearly establish what new capability is obtained beyond using an existing graph distance inside the RoPE score function. Important related work, especially GRPE and recent path structural encodings, is also missing.

  2. The experimental evidence does not show that the new design is better than existing positional encodings. Table 1 only compares δ-RoPE with pure attention, so it mainly shows that adding graph positional information is better than using no relative positional information. A fair evaluation should compare δ-RoPE with learned shortest-path bias, GRPE, WIRE, Laplacian/RW encodings, and other pairwise positional encodings under the same backbone and training setup. In Table 2, the WIRE numbers are taken from a different backbone and training configuration, which the paper itself acknowledges is not a fair comparison. Table 8 further shows that δ-RoPE is not better than several existing methods on CIFAR10, ZINC, and MNIST. Therefore, the current experiments do not demonstrate that the specific rotary formulation is useful beyond simply providing shortest-path distance.

  3. All evaluated graphs are small. The largest datasets contain only about 477–479 nodes per graph on average, while most molecular graphs contain fewer than 151 nodes. Meanwhile, δ-RoPE requires pairwise shortest-path information and dense pairwise attention. The paper even omits some timing experiments because the SPD matrices are difficult to cache. It is therefore unclear whether this method is useful or practical for large graphs.

Questions for authors
  1. Can the authors compare δ-RoPE with a learnable shortest-path-distance attention bias and other graph positional encodings under exactly the same GraphGPS backbone, parameter budget, splits, and seeds? This would clarify whether the gain comes from the rotary formulation or simply from access to shortest-path distance.

  2. Can the authors provide a fair comparison with WIRE using the same backbone and training protocol? The current cross-paper comparison is not sufficient to conclude that δ-RoPE is better.

  3. Why is shortest-path distance used as an unsigned scalar? On undirected graphs, node pairs with the same distance but very different structural roles receive the same positional rotation. Can the authors discuss or empirically evaluate this limitation against richer path encodings?

  4. Can the authors evaluate the method on substantially larger graphs and report preprocessing time, peak memory, and runtime as the number of nodes increases? A scaling experiment would materially affect my assessment of the practical contribution.

1 comment
AuthorsOpenReview ↗

Response to Reviewer wHsZ

We thank the reviewer for the detailed feedback.

W1 (novelty, related work). As shown by the controlled comparisons (general comment), δ\delta-RoPE matches or outperforms learned alternatives such as SPDBias, GRPE, Graphormer and WIRE on most benchmarks, without adding any parameter. We added GRPE, path-based structural encodings (SPSE) and the comparison of absolute and relative encodings of Black et al. (2024) to the related work.

More broadly, while this paper focuses on graphs, we see δ\delta-RoPE as a step towards a unified relative position encoding across modalities: since it reduces exactly to RoPE on sequences, the same attention could natively handle both text and graph-structured inputs, as is done today for images. Only the pairs within the graph would then require the pairwise rotation, while all other tokens keep the standard, efficient RoPE.

W2 / Q1 / Q2 (controlled comparisons). We agree that Table 1 previously only compared against attention without relative distance information. It now includes a learned shortest-path-distance bias (SPDBias, Graphormer's spatial encoding) and WIRE, under exactly the same configuration, absolute PE, splits and 4 seeds (see general comment): δ\delta-RoPE is best on MNIST, CIFAR10 and Peptides-func by 3.63.6 to 6.46.4 points, while SPDBias is slightly ahead on PCQM-Contact. In Table 2, WIRE is now trained by us inside GraphGPS with the same configurations as δ\delta-RoPE (4 seeds): δ\delta-RoPE is ahead on Peptides-func (70.60±0.4870.60 \pm 0.48 vs. 65.09±1.5665.09 \pm 1.56 AP), PCQM-Contact (47.64±0.1947.64 \pm 0.19 vs. 47.12±0.0647.12 \pm 0.06) and Peptides-struct (0.2450±0.00080.2450 \pm 0.0008 vs. 0.2469±0.00130.2469 \pm 0.0013), and the two are on par on COCO-SP (39.14±0.4039.14 \pm 0.40 vs. 38.80±0.2738.80 \pm 0.27) and PascalVOC-SP (45.17±0.7545.17 \pm 0.75 vs. 45.80±0.6045.80 \pm 0.60). We also added GRPE and Graphormer with the network size of their papers and our training configuration: they match δ\delta-RoPE on PCQM-Contact but remain behind on all other LRGB datasets. The final version will report 4 seeds for every model and dataset.

Q3 (unsigned SPD and richer encodings). On both directed and undirected graphs, the shortest-path distance is a non-negative integer (the number of hops of the shortest path), which is why we use it unsigned. On undirected graphs, a sign could still be introduced by fixing an arbitrary (e.g. random) order of the nodes and setting δ(n,m)=−δ(m,n)\delta(n,m) = -\delta(m,n), as in sequence RoPE, so that the rotation distinguishes the two directions of a pair. Richer relations are also possible: the hidden dimension can be split into groups of rotary frequencies, each rotated by a distance computed on a different subgraph (e.g. one per edge type), so that two pairs at the same distance but connected through different edges are rotated differently. Evaluating these variants is left to future work.

W3 / Q4 (larger graphs). We added the requested scaling experiment (see general comment): synthetic graphs up to 2142^{14} nodes, reporting training and inference time and peak memory as the number of nodes grows. Shortest-path distances, and Laplacian eigenvectors for WIRE, are computed on the fly at every step and included in the timings, as caching them is not possible for large graphs or large datasets. δ\delta-RoPE is clearly slower than plain attention, but its cost is close to that of similar distance-based attentions. On the largest graphs (2142^{14} nodes), during training, its time is within 15%15\% of the learned SPD bias and between 0.4×0.4\times (small diameter) and 2.6×2.6\times (large diameter) that of WIRE, with a peak memory of 3333 GB (vs. 2323 GB for the SPD bias and 1414 GB for WIRE); at inference, its time is within 5%5\% of the SPD bias and between 0.17×0.17\times and 2.4×2.4\times that of WIRE, with the same memory as the SPD bias (2020 GB). We also completed Table 3 with the previously untimed datasets.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful reading and address shared concerns here. Unless stated otherwise, new results use the same configurations, splits and seeds as in the main paper.

Controlled comparisons with other methods. In the standalone setting (Table 1: MNIST, CIFAR10, Peptides-func, PCQM-Contact), we now compare δ\delta-RoPE with plain attention, with SPDBias, a Graphormer-style learned bias (an embedding of each shortest-path distance into one additive logit bias per head), and with WIRE. In Table 1, plain attention is not blind to the graph structure: like all other models, it receives the same absolute position encoding (LapPE). δ\delta-RoPE obtains the best results on MNIST, CIFAR10 and Peptides-func (+3.6+3.6 to +6.4+6.4 points over the best baseline), and SPDBias the best result on PCQM-Contact (46.0346.03 vs. 45.4845.48 MRR for δ\delta-RoPE). In Table 2 (LRGB), we add GRPE and Graphormer as implemented by their authors, using their reference implementations and the architecture and network size proposed in their papers (with a parameter count comparable to the other models) and our optimization configuration. The attention mechanism itself is isolated in Table 1, where SPDBias, Graphormer's spatial encoding, is compared with δ\delta-RoPE under exactly the same backbone. GRPE and Graphormer match δ\delta-RoPE on PCQM-Contact but remain behind on all other LRGB datasets. For WIRE, which was itself proposed inside GraphGPS, we use exactly our GraphGPS configuration with WIRE as the attention, and keep its default hyper-parameters (10 Laplacian eigenvectors) untuned, as we do for our base b=20b = 20. In this setting, δ\delta-RoPE is ahead of WIRE on Peptides-func (+5.5+5.5 AP), Peptides-struct and PCQM-Contact, and on par on COCO-SP and PascalVOC-SP. The final version will report 4 seeds for every model and dataset.

Scaling to large graphs. We added a scaling study on synthetic graphs (Barabási–Albert and random geometric graphs, up to 2142^{14} nodes), with shortest-path distances, and Laplacian eigenvectors for WIRE computed on the fly at every step, as required for large datasets and for inference. We report training and inference time and peak memory (new appendix). δ\delta-RoPE is clearly slower than plain attention, but its cost is close to that of similar distance-based attentions. During training, it takes between 0.4×0.4\times and 2.6×2.6\times the time of WIRE and 1.4×1.4\times the memory of the learned SPD bias. During inference, it takes between 0.17×0.17\times and 2.4×2.4\times the time of WIRE and the same memory as the SPD bias. We acknowledge that the edge-to-node lifting used in the standalone model adds a compute overhead, as each edge becomes an input token. The goal of this first experiment is not a practical architecture, but a simple way of integrating edge features into a pure attention model, so that the attention mechanism can be compared in isolation.

Distance conditioning effects. We added a positional-encoding ablation (with/without LapPE, new appendix), repeated the RoPE-base sweep inside GraphGPS on MNIST, CIFAR10 and Peptides-func (the dataset with the largest diameter), and measured how attention is distributed over distances (see our answer to Reviewer DLuz).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.