acceptodds
Under review as a conference paper at ICLR 2027

Let the Heads Talk: Beyond Diagonal Graph Attention

Abstract

Sheaf Neural Networks generalize scalar-weighted message passing by replacing scalar edge weights with linear transport maps between local feature spaces. Yet the role of this matrix-valued transport is entangled with the broader sheaf-diffusion construction. We isolate the transport primitive through quiver representations and establish a direct connection with multi-head attention. Treating attention heads as coordinates of a local transport space reveals that standard multi-head attention implements diagonal edge maps: along each directed interaction, a source head can contribute only to the corresponding receiver head. Allowing off-diagonal entries instead enables edge-conditioned communication across heads before neighborhood aggregation. We show that this operation cannot, in general, be absorbed into a single shared linear map applied after aggregation. Building on this characterization, we introduce Topological Attention (Top-A), a multi-head attention that learns edge-dependent off-diagonal routes while preserving the original same-head paths and exactly recovering vanilla attention when the additional routing vanishes. We evaluate Top-A on relational reasoning, heterogeneous graph learning, and algorithmic reasoning, including out-of-distribution generalization, with heterophilic node classification as a contrast setting. The results show that cross-head transport is most useful when the task benefits from interaction-dependent transformations, while heterophily alone provides no systematic advantage. These findings identify edge-conditioned cross-head communication as a distinct computational primitive of matrix-valued transport.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Attention, Sheaf, Graphs

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vLNhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Using quiver representations, the paper characterizes standard multi-head attention as diagonal transport on its native head space. It introduces Topological Attention (Top-A), which uses Te=De(IH+λΩe)T_e=D_e(I_H+\lambda\Omega_e) to add edge-dependent off-diagonal routing across value heads before neighborhood aggregation while preserving the original same-head attention paths. For fixed attention and transport operators, the paper establishes a necessary and sufficient condition under which the routed aggregation can be reproduced by a shared post-aggregation linear map. Experiments with GATv2 and Graph Transformer (GT) backbones show improvements on STaR, MovieLens-100K, and CLRS.

Critical strengths

1. A simple and coherent design that facilitates controlled architectural comparisons. Top-A preserves the original same-head attention paths and exactly recovers the corresponding baseline function at zero initialization, while allowing the routing generator to receive gradients. This design facilitates controlled comparisons and direct integration with different attention backbones.

2. The main empirical gains are supported. The CLRS experiments cover all 30 tasks and 20 paired seeds for both attention backbones. The paper also reports task-specific regressions and the absence of systematic gains on heterophilic node classification, providing a balanced account of the proposed inductive bias and its empirical scope.

Critical weaknesses

1. The core mechanism is not sufficiently isolated to attribute the gains specifically to edge-dependent cross-head routing. The main experiments primarily compare each vanilla backbone with the full Top-A model, without key controls such as globally shared routing, relation-type-conditioned routing, or alternative value transformations of comparable capacity. Matching initialization controls the starting point but does not disentangle the contributions of additional capacity, multiplicative transformations, and the edge-dependent cross-head structure.

2. The distinct contribution and advantages over the closest existing mechanisms remain insufficiently established. Copresheaf attention [1] already applies learned edge-dependent matrices to value representations before aggregation, while Compositional Attention [2] allows context-dependent composition of search and value-retrieval mechanisms; the latter is not discussed in this paper. However, the main experiments demonstrating gains compare only against vanilla GATv2 and GT.


[1] Copresheaf Topological Neural Networks: A Generalized Deep Learning Framework, NeurIPS 2025

[2] Compositional Attention: Disentangling Search and Retrieval, ICLR 2022

Questions for authors

See Weaknesses.

5 comments

Dear Reviewer vLNh,

Thank you again for your careful review and valuable suggestions.

We hope that our responses have helped clarify the contribution of Top-A, the scope of our experimental controls, and its relationship to existing attention mechanisms.

We would appreciate knowing whether our responses have addressed your main concerns, and we welcome any further questions or suggestions.

Best regards,
The Authors

Reviewer_vLNhOpenReview ↗

Thank you for your response

Thank you for your detailed response and the clarifications. I have carefully read the rebuttal and the other reviewers' comments. After consideration, I have decided to maintain my original score, as it remains unclear whether Top-A can be integrated with more advanced and powerful GNNs and further enhance their performance; therefore, it is not yet possible to determine Top-A’s general applicability and usefulness in this field.

Dear Reviewer vLNh,

Thank you for reconsidering our work and for your additional feedback.

We would like to clarify that Top-A is designed as a general augmentation of multi-head attention mechanisms, rather than of arbitrary GNN architectures. Its construction is independent of the specific attention-scoring function, as it introduces edge-conditioned cross-head transport before aggregation while preserving the underlying attention mechanism.

To demonstrate this flexibility, we evaluated Top-A on two distinct and widely used attention-based GNN families, GATv2 and Graph Transformers, observing improvements across multiple tasks. We therefore believe that the architectural applicability of Top-A extends beyond a particular backbone, although we agree that evaluating additional, more advanced architectures would further strengthen the empirical evidence.

As the rebuttal period is approaching its end, we would particularly appreciate your guidance on which architectures or additional comparisons you would consider most informative for establishing the broader applicability of Top-A. We would be happy to address this concern as concretely as possible.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_RRrZOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to increase the expressivity of graph attention networks (GAT) by allowing transport between different heads during the message passing operation. It differentiates itself from classical GATs where only heads of the same type communicate, and from cross-head communication at the same node. The authors draw parallel to sheaf neural networks (SNN) and show that some special case of Top-A GAT can be realized by sheaves.

The paper gives a clear definition of the proposed Top-A graph attention addition, and provides proofs of what simpler solutions the proposed method extend.

Comparison with GAT and Graph Transformer shows that the Top-A attention in some cases can increase accuracy significantly. The paper aims also to evaluate whether this extended attention mechanism manages to achieve higher accuracy for heterophilious datasets, to support the parallel to SNN-s, but these evaluations do not lead to real conclusion.

To summarize, the paper proposes a somewhat straightforward extension of multi-head attention networks, and shows that this extension is very powerful for some tasks and datasets.

Critical strengths

The paper proposes and theoretically motivates a graph attention network extension that allows non-diagonal transport between heads at message passing. They relate the proposed method to existing solutions, showing for example that the symmetric case can be implemented by sheaves. The added flexibility seems to address well some cases where classical graph attention did not help to increase accuracy. The gain achieved for some tasks is very high, up to 23% in accuracy, which suggests that the proposed capability has transformative effect for some problems.

Critical weaknesses

The paper does not present a theoretical motivation for the proposed architecture (compared to for example SNNs where such motivation exists). It neither presents the challenge that the proposed solution should address. The conclusion the paper has “The results show that cross-head transport is most useful when the task benefits from interaction-dependent transformations” is self explanatory, the property is derived from the results, it is not a property of the input dataset. While the authors find cases where the proposed method is very efficient, they do not try to find out what characterizes these cases.

One of the goals of the paper is to evaluate whether cross-head attention may address heterophily. I do not see that this capability is motivated by the design, and such motivation is not given in the paper. What is presented in the paper are parallels to sheaves. However, what would be needed is a proof that the proposed method extends the capabilities of the sheaves, but it is not given in the paper.
The evaluation of efficiency under heterophily is not easy as recently it was shown that standard datasets may not be representative. There are works on sheaves using synthetic heterophilious datasets, the authors may try something similar to strengthen this part of the paper.

Questions for authors

The paper aims to compare different architectures. These should be presented with update equations that are possible to relate to each other, so that the differences can be easily pointed out.

Please provide training convergence results for a reasonably large example. How does training convergence compare to that of GAT and GT?

I am wondering about the reasoning for the depth generalization results. The decreasing accuracy seems to be some kind of combination of the high accuracy of learned depth and the accuracy of the random guess.

3 comments

Dear Reviewer RRrZ,

Thank you again for your constructive feedback.

We hope that our responses and the additional material in the revised manuscript, particularly the unified update equations and convergence analysis, have addressed your concerns.

We would appreciate hearing whether you find these clarifications satisfactory, and we remain happy to address any further questions.

Best regards,
The Authors

Dear Reviewer RRrZ,

We have made substantial efforts to address all the concerns raised in your reviews, providing additional clarifications and evidence. Specifically:

  • Motivation and theoretical contribution: We clarified the central research question of our work, explaining how Top-A isolates the computational inductive bias of matrix-valued transport and generalizes sheaf-based approaches within the quiver framework.
  • Distinction from heterophily: We clarified that edge-dependent cross-head routing addresses a different property from heterophily, discussing our negative results on heterophilic benchmarks and their consistency with recent literature.
  • Benchmark selection and generalization: We expanded the motivation for our benchmarks, explaining how they isolate edge-dependent transport and evaluate generalization to unseen path lengths and larger graph instances.
  • Architectural comparisons: We added Table 4 in Appendix B.4, providing a unified formulation of GATv2, GT, and their Top-A extensions, making the architectural differences and the role of edge-dependent cross-head routing explicit.
  • Training convergence: We added training-loss curves for MovieLens-100K, showing faster initial convergence for GATv2 with Top-A and comparable convergence for GT with and without Top-A.
  • Depth generalization: We clarified that Figure 3 reports accuracy at each exact unseen path length, rather than cumulative accuracy, strengthening the interpretation of the generalization results.

We believe that constructive dialogue is essential to the rebuttal process and would welcome the opportunity to discuss any remaining concerns before the discussion period closes. We hope that our responses have addressed your concerns and that, in light of the additional evidence and clarifications provided, you might consider supporting the acceptance of our work.

Thank you for your time and consideration.

Best regards,
The Authors

Thank you for the clarifications and the additional results. The clarifications in the answer and also the more detailed discussion of the data sets in the article made tha contribution more focused. Therefore, I increased my rate in the original review. I still find the connection to sheaves farfetched, and the lerning challenge unclear. Therefore, I think the idea would need further investigations to clarify these.

Reviewer_LKT5OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors tackle the problem of learning on graphs with Graph Neural Networks (GNNs). They propose to consider a connection between multi-head attention and quiver representations as generalisations of Sheaf Neural Network. The main idea is that heads define coordinates for edge-wise weighting of messages from neighbours. According to this view, the standard attention mechanisms in GNNs amount to having diagonal maps that transform incoming messages, thus only covering a precise, strict subspace of all quiver-like representations, which admit, instead, full maps.

Building upon this observation, the authors propose to extend the standard attention mechanisms with an additional cross-head communication term (the "transport term") which merges, in a linear but learnable way, the information coming form different heads before aggregating across neighbours/nodes. The authors show that this operation is equivalent to a simpler post aggregation transformation only under certain conditions (when incoming edges induce the same effective transformation).

The authors test their method on three main classes of tasks: (i) a reasoning benchmark, (ii) a standard link prediction dataset, and (iii) over a neural algorithmic task suit. They compare their proposed enhancements against the vanilla attention based GNN parameterisations. They report better average performance on (i) and (ii) and generally task-dependent outperformance on (iii).

Critical strengths
  • (S1) The paper is generally written well and easy to follow. Along the manuscript, the authors make efforts in explaining their ideas and their consequences, making the contribution accessible. Figure are clear and helpful.

  • (S2) The idea of connecting attention mechanisms in GNNs with per-edge transport maps is somehow interesting (but see weaknesses).

Critical weaknesses
  • (W1) While somewhat interesting, I do not find the connection with the topological angle particularly useful per se. To the best of my understanding, the authors are ultimately proposing an attention mechanism with cross-head linear interactions, but then, what has this to do with Sheafs and Quivers? One could argue that, if anything, the connection with quivers enlighten about this possible architectural design. But, to the best of my understanding, this is not particularly novel (see W2) and, practically speaking, it mainly amounts to having an attention parameterisation of Quiver-based Graph Neural Networks. The authors do not discuss other existing quiver parameterisations making it difficult to contextualise their contribution from this angle. On top of this, they do not seem to experiment on benchmarks warranting such formalism, that is, that of having node-specific vector spaces and maps between them over edges (see W3).

  • (W2) I am struggling to see the proposed parameterisation as particularly novel or otherwise needed. The cross-head communication appear to the reviewer to be simply what is implemented in Transformer models by the Output transformation WOW_O. It is known that one can write the Transformer equations foregrounding a summation over heads where each value is transformed by a corresponding slice of such matrix, in the form of: ∑h=1HZhWOh=∑h=1HAhXWVhWOh\sum_{h=1}^{H} Z_h W_O^h = \sum_{h=1}^{H} A_h X W_V^h W_O^h. This already captures linear head interactions similarly to what proposed in the paper. The only remaining difference could be in the case where the head communication components are edge-specific (see, e.g., lines 319-321). However, it is not clear when this can really (practically) occur. Even in the case where edges have distinct features, wouldn't it be possible to simply account for those in the calculation of the values, that is ZhZ_h's? In which sense isn't the contribution simply an edge-feature aware attention mechanism?

  • (W3) It is not clear why the proposed benchmarks serve the purpose of demonstrating the value of the proposed architectural design. Why should we expect attentive quiver-based representations to be helpful in reasoning tasks and why specifically at unseen reasoning depths? Why would they be useful in heterogeneous graph tasks or algorithmic reasoning tasks and in generalisation to larger instances thereof? What is there that cannot simply be accounted for by injecting edge features in the attention mechanism (refer to lines 345-354)?

  • (W4) The choice of baselines is truly insufficient. The authors only compare with the base backbones and no other method. Is not even clear if these baselines are at least exposed to the same edge-related information at all. And how do their Top-A method compare with the state of the art? Are results obtained by simple backbone baselines and their methods reasonable at all? This is not discussed and is hard to understand if authors do not report any other method in comparison. If anything, they should report results by other quiver/sheaf based method or standard GNNs on these benchmarks.

Questions for authors

See Weaknesses.

Comments on the paper

1 comment

We thank the reviewer for engaging in the discussion and for raising their score.

On parameter sharing and reliance on local features

The reviewer is correct that Top-A uses shared parameters to compute its edge-dependent cross-head routing using local features. However, we believe that it is still easily adaptable to producing different maps for different edges even when the local features are the same via, for example, positional encoding. Positional encoding would indeed distinguish the node features surrounding each edge, resulting in a different assignment.

On edge-conditioned message-passing

We note that other architectures that realise edge transport adopt similar parameter-sharing techniques that further rely on local features, e.g. neural sheaf diffusion, or even CNNs (Copresheaf Topological Neural Networks: A Generalized Deep Learning Framework, Hajij et al., 2025), so we believe that parameter-sharing and reliance on local features are not incompatible with the realisation of edge transport.

Under this view, also sheaf neural networks (SNNs) fall within the broader family of edge-conditioned message-passing schemes, as the message passing function of SNNs can be written as ψ(xv,xu,xe)\psi(x_v,x_u,x_e) (Sheaf-MPNN: A Library for Sheaf Neural Networks as Message Passing, Borgi et al., 2026), since the restriction maps Fv→eF_{v\to e} are also computed from local features and can be absorbed in ψ\psi.

We thank the reviewer for giving us the opportunity to make this clarification.

On isolating edge-dependent cross-head routing

On this point, we would like to reiterate on the fact that all baseline backbones also have access to the very same features (including edge features) of their Top-A augmented counterparts. Moreover, GT represents a baseline that has both (i) access to edge features and (ii) realizes edge-independent cross-head routing via WOW_O, making the comparison GT vs. GT+Top-A particularly suited to isolate the contribution of edge-dependent cross-head routing.

We would have been grateful if the reviewer would have pointed out specific methods in the literature, as we would have done our best to accommodate their request. Nevertheless, we still welcome these suggestions to be later added to the paper.


We hope that these additional clarifications will positively address the reviewer's concerns.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.