acceptodds
Under review as a conference paper at ICLR 2027

SheaFormer: Sheaf Diffusion as a Structured Multi-Head Operator

Abstract

Sheaf Neural Networks (SNNs) enrich graph diffusion with vector-valued stalks connected by learned restriction maps, and have been extended to directed hypergraphs by encoding orientation through a complex phase. We show that sheaf diffusion admits an exact stream-major decomposition whose diagonal specialization has the additive structure of the Transformer's multi-head attention: stalk coordinates act as heads, sheaf-induced node operators take the place of softmax attention matrices, and stalk and channel maps become structured value and output maps. We introduce SheaFormer, an SNN whose multi-head structure is inherited from the sheaf itself: it places sheaf propagation inside a residual block with a complex whitening of the propagated branch and generates role-aware restriction maps from the head and the tail of each hyperedge as bilinear node–context compatibilities. Whereas the phase encoding of the Directional Sheaf Hypergraph Network (DSHN) discards orientation at zero charge, the role-aware restrictions retain it and, in a single layer, distinguish orientations of a graph that no single DSHN layer with diagonal restriction maps distinguishes for any charge. Across twelve directed hypergraph benchmarks, the gain of the residual block grows as homophily decreases, while the role-aware restrictions improve eleven of the twelve benchmarks. SheaFormer attains the highest average accuracy, 2.03 points above the strongest baseline, and the highest F1 on two real-world molecular-reaction hypergraphs.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Sheaf Neural Netowrks, Hypergraphs, Multi-Head-Attention

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BYDBOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The author proposes SheaFormer, which reframes sheaf diffusion as a structured multi-head operator. Specifically, the method consists of residual propagation + role-aware bilinear restriction maps for directed hypergraphs.

Critical strengths
  1. The overall idea that role-aware restrictions can distinguish orientations that DSHN cannot is interesting.

  2. Theoretical claims (stream-major decomposition) are intuitive.

  3. Experimental results are competitive compared to the SOTA methods.

Critical weaknesses
  1. The novelty seems incremental, as a sheaf-attention mechanism already exists. Theorem 1 can be viewed as just a new interpretation.

  2. The ablation also changes the normalization placement and dropout. Is the residual block's contribution properly isolated? The ablation results seem inconsistent with Table 1.

  3. Table 1 separates the gains. Since each variant is independently hyperparameter-tuned, how do the authors rule out that part of these differences comes from different selected hyperparameters?

  4. Code is not provided.

Questions for authors

Please see the above weaknesses

4 comments
AuthorsOpenReview ↗

Reply to Reviewer BYDB (1/4)

We thank the reviewer for the detailed review and for highlighting that role-aware restrictions distinguish orientations that DSHN cannot, which is the core novelty of the paper. We address the novelty, ablation, tuning and code concerns below, with new experiments where the reviewer asked for evidence.

Weakness 1. The novelty seems incremental, as a sheaf-attention mechanism already exists. Theorem 1 can be viewed as just a new interpretation.

W1 (novelty vs. sheaf attention; Theorem 1 as an interpretation). Unlike the common trend in the literature, where an attention block is added on top of an existing architecture, with our work we establish (Theorem 1) an exact identity showing that sheaf diffusion with diagonal restriction maps already features the additive multi-head structure of Transformer attention without the need for introducing additional attention weights. Let us look at works which add a Transformer-based block on top of an existing network. SheaFormer differs from the existing Sheaf Attention Networks (SANs) both in its construction of the propagation operator and in the structural properties it preserves. SANs multiply the sheaf-derived adjacency coefficients with GAT-style scalar attention coefficients. However, the resulting attention-weighted propagation operators are not generally standard sheaf operators. In particular, they need not preserve Hermitian symmetry or admit the positive-semidefinite factorization underlying sheaf diffusion. In contrast, SheaFormer does not introduce a separate attentional mechanism on top of sheaf propagation. Instead, it learns the restriction maps themselves through role-aware bilinear node-hyperedge compatibilities, with separate head and tail contexts, preserving the Hermitian positive-semidefinite structure of the standard sheaf operators.

Theorem 1 provides the structural motivation for this construction. It establishes an exact decomposition of the standard sheaf layer into stream-wise node operators with structured value and output maps. For diagonal sheaves, the decomposition takes the additive multi-head form of Transformer attention, without introducing additional attention weights. This identity is what the design rests on, not an interpretation added afterwards: since the multi-head structure is already in the sheaf layer, SheaFormer learns its attention-like operators through the restriction maps instead of adding attention.

The novelty also goes beyond this decomposition:

  • Theorem 2: Beyond the stalk-mixing mechanism studied by [1] in Proposition 17, complex whitening introduces two additional mechanisms through which a harmonic propagated update can leave ker⁡L\ker L: centering and the conjugate-linear component of whitening. Appendix H.2 provides empirical evidence of these mechanisms in trained SheaFormer models.
  • Theorem 3: with hyperedge features that do not depend on orientation, DSHN at q=0q=0 cannot distinguish two orientations of a hypergraph at any depth, for any parameters, whereas one SheaFormer layer at q=0q=0 can.
  • Theorem 4: on simple directed graphs with identical node and edge features, the operators of a diagonal DSHN stream are contained in those of a scalar SheaFormer stream on every graph, and strictly on some; some orientations cannot be separated by one DSHN layer for any charge, whereas one SheaFormer layer at q=0q=0 separates them.

Theorems 3 and 4 establish formal representational advantages of the proposed restriction construction, while Theorem 2 characterizes additional mechanisms governing the behavior of sheaf diffusion. These contributions go beyond alternative interpretations of existing operators.

Empirically, the role-aware bilinear restriction generator improves accuracy by +1.92+1.92 points on average over the DSHN generator, on 11 of 12 benchmarks.

[1] Bodnar, Cristian, et al. "Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in gnns." Advances in Neural Information Processing Systems 35 (2022).

AuthorsOpenReview ↗

Reply to Reviewer BYDB (2/4)

Weakness 2. The ablation also changes the normalization placement and dropout. Is the residual block's contribution properly isolated? The ablation results seem inconsistent with Table 1.

W2 (residual isolation; consistency with Table 1). Yes: we now remove only the residual term, leaving the rest of the layer unchanged, and the results are consistent with Table 1, since Δarch\Delta_{\mathrm{arch}} and Δres\Delta_{\mathrm{res}} measure different changes. Table 1 measures the complete SheaFormer block rather than the residual connection alone: SheaFormer (DSHN restr.) replaces DSHN's layer with the SheaFormer block, which changes the residual formulation together with the placement of normalization and dropout. This is what Δarch\Delta_{\mathrm{arch}} measures: +1.08+1.08 and +0.11+0.11 points relative to DSHN and DSHN-Light.

To isolate the residual connection, we removed only the residual term, so that the layer becomes

X^(t+1)=σ(Drop⁡[Norm⁡(∑p=1dQNpp(X^(t))X^(t)WvalpWoutp)]),\widehat X^{(t+1)}=\sigma\Big(\operatorname{Drop}\Big[\operatorname{Norm}\Big(\sum_{p=1}^{d}Q_N^{pp}\big(\widehat X^{(t)}\big)\widehat X^{(t)}W_{\mathrm{val}}^pW_{\mathrm{out}}^p\Big)\Big]\Big),

with normalization, dropout and the role-aware bilinear restrictions unchanged, and re-tuned the resulting model with the same search space and budget as the SheaFormer variants of Table 1 (150 Optuna trials, 10 splits). The residual connection improves accuracy on 10 of 12 benchmarks, by +2.72+2.72 points on average:

DatasetCEHomSheaFormerSheaFormer w/o residualΔres\Delta_{\mathrm{res}}
Roman-empire0.05020.050291.07±0.4291.07 \pm 0.4291.97±0.4991.97 \pm 0.49−0.90-0.90
Squirrel0.18890.188945.22±2.2945.22 \pm 2.2941.98±1.0341.98 \pm 1.03+3.24+3.24
Chameleon0.25740.257447.46±4.1047.46 \pm 4.1042.91±4.3242.91 \pm 4.32+4.55+4.55
email-EU0.25810.258181.54±2.4281.54 \pm 2.4280.85±2.1780.85 \pm 2.17+0.69+0.69
Telegram0.27750.277584.42±4.7584.42 \pm 4.7588.46±3.2788.46 \pm 3.27−4.04-4.04
Cornell0.31390.313977.30±4.0777.30 \pm 4.0767.84±5.6267.84 \pm 5.62+9.46+9.46
email-Enron0.31730.317372.97±6.2472.97 \pm 6.2472.16±4.2472.16 \pm 4.24+0.81+0.81
Wisconsin0.34230.342386.67±2.8986.67 \pm 2.8976.47±3.4676.47 \pm 3.46+10.20+10.20
Amazon-ratings0.37110.371149.16±0.6249.16 \pm 0.6248.64±0.7148.64 \pm 0.71+0.52+0.52
Texas0.39490.394982.97±5.4182.97 \pm 5.4177.03±2.9277.03 \pm 2.92+5.94+5.94
Citeseer0.71550.715576.17±1.1776.17 \pm 1.1775.30±1.4375.30 \pm 1.43+0.87+0.87
Cora0.78100.781086.82±0.5886.82 \pm 0.5885.58±1.4585.58 \pm 1.45+1.24+1.24
Mean—73.4873.4870.7770.77+2.72+2.72

This resolves the apparent inconsistency: Δarch\Delta_{\mathrm{arch}} measures a complete layer replacement, whereas Δres\Delta_{\mathrm{res}} measures the residual connection alone. The two are also measured with different restriction generators, DSHN's for Δarch\Delta_{\mathrm{arch}} and SheaFormer's for Δres\Delta_{\mathrm{res}}, so they are not additive steps of a single chain. Table 1 labels this row's controlled change "Residual block", whereas the Q1 discussion attributes Δarch\Delta_{\mathrm{arch}} to the SheaFormer layer as a whole; the revision relabels the row "SheaFormer block", carries this attribution into the abstract, the contributions, the Q1 question and the conclusions, and reports Δres\Delta_{\mathrm{res}} separately.

AuthorsOpenReview ↗

Reply to Reviewer BYDB (3/4)

Weakness 3. Table 1 separates the gains. Since each variant is independently hyperparameter-tuned, how do the authors rule out that part of these differences comes from different selected hyperparameters?

W3 (independent tuning). Independent tuning under a common protocol is what makes the comparison fair, and the differences do not hinge on the selected configurations: averaged over the neighborhoods of their selected configurations, the four variants keep the order of Table 1. Our intention is to compare the performance achievable by each architecture under equal tuning effort. No variant receives more tuning than another. The four SheaFormer variants of Table 1, and the model without residual, are tuned over the same search space (dd, cc, LL, learning rate, weight decay, both dropouts, qq; Appendix E), with the same budget of 150 Optuna trials, the same ten splits and the same selection rule (mean validation accuracy).

The alternative is to fix the hyperparameters and compare performance across variants. However, no single configuration is necessarily neutral, and a configuration selected by tuning one variant may favor that variant over the others. Our residual ablation shows this bias directly. At SheaFormer's selected configuration, on the same ten splits, removing the residual term lowers accuracy by 4.724.72 points on average, against 2.722.72 when the model without residual is re-tuned under the same protocol: on average, the model without residual is 2.002.00 points less accurate at SheaFormer's configuration than at its own (68.7768.77 against 70.7770.77), so the fixed comparison overstates the contribution of the residual connection by 2.002.00 points. On Roman-empire it even reverses the sign, from +2.72+2.72 at SheaFormer's configuration to −0.90-0.90 after re-tuning. Thus, while fixing hyperparameters isolates the effect of an architectural change at a particular configuration, it may not fairly reflect the performance achievable by each architecture after optimization. Table 1 therefore compares test accuracy at independently validation-selected configurations under identical search protocols. Here, the selected hyperparameters are an outcome of optimizing each architecture, and comparability is ensured through the same search space, tuning budget and selection rule. In particular, every configuration searched or selected for one SheaFormer variant is admissible for all the others. This follows the established practice of comparing architectures after independent hyperparameter optimization under a common protocol [2–4].

Moreover, to compare the performance of the variants on similar hyperparameter configurations and to show that the selected configurations are not isolated optima, we conduct a further analysis. Following the sensitivity check of [2] (Section 7.3), for each of the 48 tunings of the four variants we examined the trials within one step of the search space of the selected configuration on every hyperparameter (dd and cc within a factor of 22, LL within one layer, weight decay within one grid value, qq within 0.050.05, dropout rates within 0.10.1), with a factor of 22 for the continuous learning rate. These neighborhoods hold 5 to 60 trials per tuning (median 28.5), with 90% of all 1,392 neighboring trials reaching a validation accuracy within 2 points of the selected one. Averaging the test accuracy over these neighborhoods, instead of taking the selected trial, gives SheaFormer (DSHN restr.) 71.5371.53, SheaFormer (MLP role-blind) 71.5871.58, SheaFormer (MLP role-aware) 72.7272.72 and SheaFormer 72.9672.96: Δrestr=+1.43\Delta_{\mathrm{restr}}=+1.43 (Table 1: +1.92+1.92), of which generator capacity accounts for +0.05+0.05 (+0.63+0.63), role awareness for +1.14+1.14 (+0.72+0.72) and the bilinear compatibility for +0.24+0.24 (+0.56+0.56).

[2] Melis, Gábor, Chris Dyer, and Phil Blunsom. “On the State of the Art of Evaluation in Neural Language Models.” International Conference on Learning Representations (2018).

[3] Shchur, Oleksandr, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. “Pitfalls of graph neural network evaluation.” arXiv preprint arXiv:1811.05868 (2018).

[4] Errica, Federico, Marco Podda, Davide Bacciu, and Alessio Micheli. “A Fair Comparison of Graph Neural Networks for Graph Classification.” International Conference on Learning Representations (2020).

AuthorsOpenReview ↗

Reply to Reviewer BYDB (4/4)

Weakness 4. Code is not provided.

W4 (code). The code is now provided in the supplementary material, and will be made public upon acceptance.

Reviewer_FTJGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Motivated by transformer multi-head attention, the paper proposes SheaFormer, a sheaf neural network with role-aware bilinear restriction maps, rather than restriction maps parameterized by MLPs, as in previous work.

Critical strengths
  • The proposed model achieves strong results on the tested benchmarks.
  • The motivation behind the model is sound, and the connection between sheaf diffusion and the additive structure of the transformer’s multi-head attention is appealing.
Critical weaknesses
  • Although well-motivated, the core idea seems too incremental: learn the sheaf using a transformer-inspired operator and perform sheaf diffusion with an augmented residual block.
Questions for authors
  • Can’t your model be used with orthogonal or general restriction maps? The motivation for using diagonal restriction maps to design the sheaf-learning process is understandable, but it seems that everything else – the residual block for diffusion and the learning process – can be used within any sheaf neural network framework.
  • The cost of sheaf-based architectures is generally high. Is it possible to run your model on the Trivago and Yelp benchmarks from [1]? I don’t expect you to achieve state-of-the-art or competitive results on these benchmarks during the short rebuttal phase, but it would be helpful to know if Sheaformer can be used with larger datasets.

[1] DHG-Bench: A Comprehensive Benchmark for Deep Hypergraph Learning.

5 comments
AuthorsOpenReview ↗

Reply to Reviewer FTJG (1/3)

We thank the reviewer for recognizing the soundness of the motivation, the appeal of the connection between sheaf diffusion and multi-head attention, and the strength of the results. We address the incrementality concern and the two questions below.

Weakness 1. Although well-motivated, the core idea seems too incremental: learn the sheaf using a transformer-inspired operator and perform sheaf diffusion with an augmented residual block.

Incrementality. SheaFormer does not combine Transformer-inspired attention with sheaf diffusion: Theorem 1 establishes an exact identity showing that sheaf diffusion with diagonal restriction maps already features the additive multi-head structure of Transformer attention without the need for introducing additional attention weights. This precise correspondence is what motivated our stream-wise construction, in which role-aware bilinear compatibilities induce the restriction maps themselves, rather than augmenting sheaf propagation by adding an external attention mechanism on top of it (which is what most of the literature does).

Beyond this structural correspondence, SheaFormer introduces new representational capabilities:

  1. New expressivity results. Theorem 3 proves that, at zero charge and with hyperedge features that do not depend on orientation, DSHN cannot separate two orientations at any depth, whereas one SheaFormer layer can. Theorem 4 proves, on simple directed graphs with identical node and edge features, that the operators of a diagonal DSHN stream are contained in those of a scalar SheaFormer stream on every graph, and strictly on some, and that some orientations cannot be separated by one DSHN layer for any choice of the charge parameter.
  2. New harmonic-space analysis. Beyond the stalk-mixing mechanism studied by [2] in Proposition 17, Theorem 2 establishes two additional mechanisms through which complex whitening can move a harmonic-propagated update outside ker⁡L\ker L: centering and the conjugate-linear component of whitening. Appendix H.2 further provides empirical evidence of these mechanisms in trained SheaFormer models.
  3. Measured contributions of the components. The role-aware restriction generator improves accuracy by +1.92+1.92 points on average over the DSHN generator, on 11 of 12 benchmarks. The new isolated ablation of the residual connection, which removes only the residual term and re-tunes the resulting model with the same search space and budget, shows a +2.72+2.72-point average benefit, on 10 of 12 benchmarks (general response), showing the practical impact of the proposal.

[2] Bodnar, Cristian, et al. "Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in gnns." Advances in Neural Information Processing Systems 35 (2022).

AuthorsOpenReview ↗

Reply to Reviewer FTJG (2/3)

Question 1. Can’t your model be used with orthogonal or general restriction maps? The motivation for using diagonal restriction maps to design the sheaf-learning process is understandable, but it seems that everything else – the residual block for diffusion and the learning process – can be used within any sheaf neural network framework.

Q1 (general and orthogonal restriction maps). Yes, SheaFormer naturally extends to both general and orthogonal restriction maps. Theorem 1 holds for arbitrary restriction maps, and neither the residual block nor the role-aware generator depends on the diagonal form:

  • Residual block. It acts on the propagated branch and does not depend on how the restriction maps are produced. Table 1 shows this directly: SheaFormer (DSHN restr.) places DSHN's own restriction generator inside the SheaFormer block, and the same block is used with three further generators.
  • Role-aware generator. The head and tail contexts are inputs to the generator and do not depend on the restriction family. For general maps, the generator would output one bilinear compatibility per entry (r,s)(r,s) of Fi◃eF_{i\triangleleft e}, i.e., d2d^2 scores per incidence instead of dd; for orthogonal maps, its output could parametrize Fi◃eF_{i\triangleleft e} through Householder reflections, as in NSD (Appendix B).
  • The layer with general maps. The block extends directly: its branch becomes Z^=∑p=1d∑s=1dΩpsX^WvalsWoutp\widehat Z=\sum_{p=1}^{d}\sum_{s=1}^{d}\Omega^{ps}\widehat X W_{\mathrm{val}}^sW_{\mathrm{out}}^p, in which the cross-stream operators Ωps\Omega^{ps}, p≠sp\neq s, couple the heads, and the dd node operators of the diagonal case become d2d^2.

The revision adds this remark to Section 4 (paragraph "Beyond diagonal restriction maps").

AuthorsOpenReview ↗

Reply to Reviewer FTJG (3/3)

Question 2. The cost of sheaf-based architectures is generally high. Is it possible to run your model on the Trivago and Yelp benchmarks from [1]? I don’t expect you to achieve state-of-the-art or competitive results on these benchmarks during the short rebuttal phase, but it would be helpful to know if Sheaformer can be used with larger datasets.

[1] DHG-Bench: A Comprehensive Benchmark for Deep Hypergraph Learning.

Q2 (Trivago and Yelp). Yes. We ran SheaFormer on both datasets on DHG-Bench's 50/25/2550/25/25 splits, with 5 runs as in DHG-Bench, and tuned it with 150 hyperparameter trials per dataset:

DatasetNodes / hyperedgesSheaFormerBest in DHG-Bench [1]
Trivago172,738 / 233,20278.47±0.6878.47 \pm 0.68TF-HNN 90.79±0.7990.79 \pm 0.79
Yelp50,758 / 679,30238.10±0.3738.10 \pm 0.37TF-HNN 35.16±0.5435.16 \pm 0.54

Selected configurations, on one A100 GPU:

DatasetConfigurationParametersSeconds per epochWall time, 5 runsPeak memory (allocated)
Trivagod=4d=4, c=64c=64, L=5L=5536,240536{,}2400.900.9037.337.3 min44.044.0 GB
Yelpd=1d=1, c=64c=64, L=5L=5175,502175{,}5020.640.6426.726.7 min23.723.7 GB
  • Accuracy. On Trivago, SheaFormer reaches 78.47±0.6878.47 \pm 0.68, behind only TF-HNN and HyperND; on Yelp, it reaches 38.10±0.3738.10 \pm 0.37, above every accuracy reported in DHG-Bench, by +2.94+2.94 points over the strongest, TF-HNN. The comparison uses the accuracies reported in the DHG-Bench paper. Yelp is heterophilic and Trivago homophilic, with CEHom 0.22330.2233 and 0.97440.9744 (Appendix E): SheaFormer leads on the former, in line with Section 5, where its improvement over DSHN-Light grows as CEHom decreases.
  • Memory. On an A100, the selected configurations (536,240 and 175,502 parameters) peak at 44.0 GB and 23.7 GB of allocated GPU memory.
  • Time. An epoch takes 0.90 s on Trivago and 0.64 s on Yelp; five full runs of 500 epochs take 37.3 and 26.7 minutes.

These results complement the scalability evidence in the paper. SheaFormer runs on Molecular-1 (100,523 nodes, 50,016 hyperedges), where DSHN runs out of memory even on a 64 GB A100. Notice that SheaFormer never forms the sheaf operator explicitly: its propagation costs νdc\nu dc per layer, with ν=∑e∣e∣\nu=\sum_e|e| the number of incidences, against ν~2dc\tilde\nu_2 dc for the DSHN implementation, with ν~2\tilde\nu_2 the number of distinct ordered node pairs sharing a hyperedge; DSHN also evaluates dν2d\nu_2 pairwise products per layer to assemble the operator, with ν2=∑e∣e∣2\nu_2=\sum_e|e|^2 (Appendix G.1, Table 12). SheaFormer's restriction generator costs (n+m)(dc)2(n+m)(dc)^2 per layer; the selected configurations keep dcdc at 256256 on Trivago and 6464 on Yelp.

The revision adds the Trivago and Yelp results to Section 5 (Table 4) and Appendix F (Table 11).

Appendix G.2 of the revision also reports, on both datasets, the floating-point operations, time and peak memory of SheaFormer, DSHN and DSHN-Light (Tables 13–17): the training step of DSHN exceeds the memory of the A100 at every measured configuration. We hope these answers clarify both the generality and the practical value of SheaFormer.

[1] F. Li, X. Wang, W. Zhang, Y. Zhang, X. Lin. DHG-Bench: A Comprehensive Benchmark for Deep Hypergraph Learning. ICLR 2026.

Reviewer_dWJTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SheaFormer, a sheaf neural network for directed hypergraphs motivated by a multi-head decomposition of sheaf diffusion. The model combines residual sheaf propagation with role-aware restriction maps that distinguish head and tail contexts. Experiments on node-classification benchmarks and molecular-reaction hypergraphs evaluate the proposed components and compare against recent hypergraph and sheaf baselines.

Critical strengths

1.The connection between sheaf diffusion and multi-head operators is technically interesting and clearly presented. The paper is careful about the scope of the correspondence and identifies the additional structure imposed by the sheaf operator.

2.The empirical analysis is well controlled.

Critical weaknesses
  1. The effect attributed to the residual block is not isolated from the other architectural changes. The corresponding ablation also changes the placement of normalization and dropout, so the reported gain is more accurately attributed to the SheaFormer layer as a whole.

  2. The directed-hypergraph evaluation is somewhat narrower than the benchmark count suggests. The twelve node-classification datasets are constructed from directed graphs by converting outgoing neighborhoods into hyperedges, while only the two molecular-reaction datasets are native directed hypergraphs.

Questions for authors
  1. Can the effect of the residual connection be separated from the accompanying changes in normalization and dropout placement?

  2. How well do the conclusions from the graph-derived benchmarks transfer to naturally occurring directed hypergraphs beyond the two molecular-reaction datasets?

  3. Can the authors isolate the residual connection from the accompanying changes in normalization and dropout placement? The current �0��0� measures the SheaFormer layer as a whole rather than the residual connection alone.

  4. How much of the observed gain depends on genuinely higher-order directed structure? The twelve node-classification benchmarks are obtained by lifting directed graphs to hypergraphs, while only the two molecular datasets directly provide higher-order directed hyperedges.

  5. For molecular-reaction classification, the final hyperedge readout sums all incident node representations without separately pooling the head and tail sets. Is the directional information learned by the role-aware restrictions preserved reliably under this role-blind readout?

7 comments
AuthorsOpenReview ↗

Reply to Reviewer dWJT (1/4)

We thank the reviewer for recognizing the theoretical contribution and the careful experimental design. We address the questions in turn.

Weakness 1. The effect attributed to the residual block is not isolated from the other architectural changes. The corresponding ablation also changes the placement of normalization and dropout, so the reported gain is more accurately attributed to the SheaFormer layer as a whole.

Question 1. Can the effect of the residual connection be separated from the accompanying changes in normalization and dropout placement?

Question 3. Can the authors isolate the residual connection from the accompanying changes in normalization and dropout placement? The current (Δarch) measures the SheaFormer layer as a whole rather than the residual connection alone.

W1 / Q1 / Q3 (residual vs. normalization and dropout). Yes. We now isolate the residual connection: we remove only the residual term, keeping normalization, dropout and the restriction maps, so that the layer becomes

X^(t+1)=σ(Drop⁡[Norm⁡(∑p=1dQNpp(X^(t)) X^(t) WvalpWoutp)]),\widehat X^{(t+1)}=\sigma\Big(\operatorname{Drop}\Big[\operatorname{Norm}\Big(\sum_{p=1}^{d}Q_N^{pp}\big(\widehat X^{(t)}\big)\,\widehat X^{(t)}\,W_{\mathrm{val}}^pW_{\mathrm{out}}^p\Big)\Big]\Big),

and we independently re-tune the resulting model using the same hyperparameter search space and budget as the SheaFormer variants of Table 1 (150 Optuna trials, 10 splits). The residual connection improves accuracy on 10 of 12 benchmarks, by +2.72+2.72 points on average and by up to +10.20+10.20 points (Wisconsin):

DatasetCEHomSheaFormerSheaFormer w/o residualΔres\Delta_{\mathrm{res}}
Roman-empire0.05020.050291.07±0.4291.07 \pm 0.4291.97±0.4991.97 \pm 0.49−0.90-0.90
Squirrel0.18890.188945.22±2.2945.22 \pm 2.2941.98±1.0341.98 \pm 1.03+3.24+3.24
Chameleon0.25740.257447.46±4.1047.46 \pm 4.1042.91±4.3242.91 \pm 4.32+4.55+4.55
email-EU0.25810.258181.54±2.4281.54 \pm 2.4280.85±2.1780.85 \pm 2.17+0.69+0.69
Telegram0.27750.277584.42±4.7584.42 \pm 4.7588.46±3.2788.46 \pm 3.27−4.04-4.04
Cornell0.31390.313977.30±4.0777.30 \pm 4.0767.84±5.6267.84 \pm 5.62+9.46+9.46
email-Enron0.31730.317372.97±6.2472.97 \pm 6.2472.16±4.2472.16 \pm 4.24+0.81+0.81
Wisconsin0.34230.342386.67±2.8986.67 \pm 2.8976.47±3.4676.47 \pm 3.46+10.20+10.20
Amazon-ratings0.37110.371149.16±0.6249.16 \pm 0.6248.64±0.7148.64 \pm 0.71+0.52+0.52
Texas0.39490.394982.97±5.4182.97 \pm 5.4177.03±2.9277.03 \pm 2.92+5.94+5.94
Citeseer0.71550.715576.17±1.1776.17 \pm 1.1775.30±1.4375.30 \pm 1.43+0.87+0.87
Cora0.78100.781086.82±0.5886.82 \pm 0.5885.58±1.4585.58 \pm 1.45+1.24+1.24
Mean—73.4873.4870.7770.77+2.72+2.72

The effect is largest on the three WebKB benchmarks and Chameleon; the two exceptions are Telegram and Roman-empire, where accuracy without the residual connection is higher by 4.044.04 and 0.900.90 points.

This experiment, run on the full SheaFormer, separates the effect of the residual connection from the full architectural change of Table 1. The revision reports Δres\Delta_{\mathrm{res}}, which isolates the residual connection, separately from Δarch\Delta_{\mathrm{arch}}, which measures replacing DSHN's layer with the complete SheaFormer block, normalization and dropout placement included; Table 1 now attributes Δarch\Delta_{\mathrm{arch}} to the SheaFormer block, as the Q1 discussion already does.

AuthorsOpenReview ↗

Reply to Reviewer dWJT (2/4)

Weakness 2. The directed-hypergraph evaluation is somewhat narrower than the benchmark count suggests. The twelve node-classification datasets are constructed from directed graphs by converting outgoing neighborhoods into hyperedges, while only the two molecular-reaction datasets are native directed hypergraphs.

W2 (benchmark composition). Our evaluation covers both kinds of directed hypergraph used in the literature: the twelve benchmarks built from directed graphs by the protocol of prior directed hypergraph networks, and the two natively directed molecular-reaction datasets, on which SheaFormer attains the highest F1. Learning on directed hypergraphs is a relatively young research area, and labeled benchmarks of natively directed hypergraphs remain scarce. Building hypergraphs from graphs is common practice in hypergraph learning: HGNN [4] forms one hyperedge per vertex from its graph neighbors, HGNN+ [5] generates hyperedges from pairwise edges and k-hop neighborhoods, TopoBench [6] lifts graphs to hypergraphs through k-hop neighborhoods, and the co-citation benchmarks Cora, Citeseer and Pubmed [7], standard for AllSet, ED-HNN and SheafHyperGNN, connect all documents cited by a document through a hyperedge, the undirected counterpart of the construction we use. For node classification, the protocol of prior directed hypergraph neural networks (Tran & Tran [2]; GeDi-HNN [3], DSHN [1]) constructs directed hypergraphs from directed graphs. We follow this protocol and use the twelve benchmarks of DSHN unchanged, so that SheaFormer is compared with DSHN and its baselines on identical data. Our evaluation complements these benchmarks with the two molecular-reaction datasets of DSHN, which are natively directed hypergraphs (Q2 / Q4).

[1] E. Mule, S. Fiorini, A. Purificato, F. Siciliano, S. Coniglio, F. Silvestri. Directional Sheaf Hypergraph Networks: Unifying Learning on Directed and Undirected Hypergraphs. ICLR 2026.

[2] L. H. Tran, L. H. Tran. Directed Hypergraph Neural Network. arXiv:2008.03626.

[3] S. Fiorini, S. Coniglio, M. Ciavotta, A. Del Bue. Let There Be Direction in Hypergraph Neural Networks. TMLR 2024.

[4] Y. Feng, H. You, Z. Zhang, R. Ji, Y. Gao. Hypergraph Neural Networks. AAAI 2019.

[5] Y. Gao, Y. Feng, S. Ji, R. Ji. HGNN+: General Hypergraph Neural Networks. IEEE TPAMI 2023.

[6] L. Telyatnikov, G. Bernárdez, M. Montagna, et al. TopoBench: A Framework for Benchmarking Topological Deep Learning. Journal of Data-centric Machine Learning Research 2025.

[7] N. Yadati, M. Nimishakavi, P. Yadav, V. Nitin, A. Louis, P. Talukdar. HyperGCN: A New Method of Training Graph Convolutional Networks on Hypergraphs. NeurIPS 2019.

AuthorsOpenReview ↗

Reply to Reviewer dWJT (3/4)

Question 2. How well do the conclusions from the graph-derived benchmarks transfer to naturally occurring directed hypergraphs beyond the two molecular-reaction datasets?

Question 4. How much of the observed gain depends on genuinely higher-order directed structure? The twelve node-classification benchmarks are obtained by lifting directed graphs to hypergraphs, while only the two molecular datasets directly provide higher-order directed hyperedges.

Q2 / Q4 (higher-order directed structure). Labeled benchmarks of natively directed hypergraphs are scarce, and the two molecular-reaction datasets are, to our knowledge, the only ones used by prior directed hypergraph networks. On average, the directed roles account for +0.72+0.72 of the +1.92+1.92 points of the restriction maps, and for up to +4.50+4.50 points (email-EU); on email-EU and email-Enron, the whole gain of the restriction maps, +5.67+5.67 and +6.48+6.48, is obtained at q=0q=0, without the phase.

The molecular-reaction datasets are naturally occurring higher-order directed hypergraphs: each reaction is a hyperedge from its set of reagents to its set of products. SheaFormer's gains carry over to them: on both, it attains the highest F1 of all compared methods, exceeding the strongest baseline by +4.10+4.10 F1 points on Molecular-1 (DSHN-Light) and +1.69+1.69 on Molecular-2 (DSHN).

The lifted hyperedges of the twelve benchmarks induce genuinely higher-order propagation: each hyperedge groups a node with its entire out-neighborhood, with average size between 2.72.7 and 49.749.7 and maximum size up to 294294 (Table 5), and the sheaf operator couples every pair of members of a hyperedge, including out-neighbors of the same node, which need not be adjacent in the original graph, with restriction coefficients computed from the whole hyperedge.

On these benchmarks, the part of the gain that depends on directed roles is measured by Δrole\Delta_{\mathrm{role}} in Table 1. The role-aware and role-blind MLP generators have the same input width, parameters, layer and tuning protocol, and differ only in whether the head and tail members of each hyperedge are pooled separately. Separating the roles adds +0.72+0.72 points on average, with the largest gains on email-EU (+4.50+4.50), Telegram (+2.50+2.50) and email-Enron (+1.89+1.89). Both generators use the charge-dependent phase, so this gain comes from the roles entering the restriction coefficients, beyond the phase alone. On email-EU and email-Enron, both SheaFormer and SheaFormer (DSHN restr.) select q=0q=0, at which the phase carries no orientation, and SheaFormer's restriction maps still improve accuracy over DSHN's by +5.67+5.67 and +6.48+6.48 points (Δrestr\Delta_{\mathrm{restr}}, Table 6), in agreement with Theorem 3; neither dataset has node features (the node degree is the input), so the hypergraph structure is their only source of information.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Dear Area Chairs and all Reviewers,

We thank the three reviewers for their careful reading and their insightful comments. We appreciate their recognition of the connection between sheaf diffusion and multi-head attention as technically interesting and clearly presented (dWJT), appealing (FTJG) and intuitive (BYDB); of the empirical analysis as well controlled (dWJT); of the results as strong (FTJG) and competitive with the state of the art (BYDB); and of the ability of role-aware restrictions to distinguish orientations that DSHN cannot as interesting (BYDB).

Our central point is that attention need not be added to a sheaf network: the sheaf layer already has the multi-head structure of attention (Theorem 1), and SheaFormer learns it through role-aware restriction maps, keeping the operator a sheaf operator.

The comments led to three main additions:

  1. The residual connection, isolated (dWJT, BYDB). We remove only the residual term from SheaFormer, keeping normalization, dropout and the role-aware bilinear restrictions unchanged, and re-tune the resulting model with the search space and budget of the SheaFormer variants of Table 1 (150 Optuna trials, 10 splits). Removing the residual connection lowers accuracy on 10 of 12 benchmarks, by 2.722.72 points on average and by up to 10.2010.20 (Wisconsin). The revision reports this Δres\Delta_{\mathrm{res}} separately from Δarch\Delta_{\mathrm{arch}}, which measures replacing DSHN's layer with the whole SheaFormer block, normalization and dropout placement included. The per-dataset results are in our replies to dWJT and BYDB.
  2. Large real-world hypergraphs (FTJG). On Trivago and Yelp from DHG-Bench [1], with up to 172,738 nodes and 679,302 hyperedges, SheaFormer reaches 38.10±0.3738.10 \pm 0.37 on Yelp, above every accuracy reported in DHG-Bench (+2.94+2.94 over TF-HNN), and 78.47±0.6878.47 \pm 0.68 on Trivago, behind only TF-HNN and HyperND. It trains on one A100 at a peak of 44.044.0 and 23.723.7 GB of allocated memory, in under one second per epoch, whereas the training step of DSHN exceeds the memory of the A100 at every measured configuration. The details are in our reply to FTJG.
  3. Code (BYDB). The code is provided in the supplementary material.

[1] F. Li, X. Wang, W. Zhang, Y. Zhang, X. Lin. DHG-Bench: A Comprehensive Benchmark for Deep Hypergraph Learning. ICLR 2026.

Summary of changes (marked in blue in the revised PDF):

ChangeWhere in the revisionReviewer
Residual connection isolated: SheaFormer w/o residual, re-tuned, and Δres\Delta_{\mathrm{res}}Table 1, Section 5, conclusions, Appendix F (Table 9)dWJT, BYDB
"Residual block" relabeled "SheaFormer block": Δarch\Delta_{\mathrm{arch}} attributed to the block as a wholeAbstract, Section 1, Table 1, Section 5 (Q1), conclusionsdWJT, BYDB
Benchmark construction attributed to prior directed hypergraph networksSection 5 (setup)dWJT
Readout kept identical to DSHN's for every modelAppendix FdWJT
General and orthogonal restriction mapsSection 4FTJG
Trivago and Yelp: accuracy against DHG-Bench, and CEHomSection 5 (Table 4), conclusions, Appendix F (Table 11)FTJG
Trivago and Yelp: FLOPs, time and memory of SheaFormer, DSHN and DSHN-LightAppendix G.2 (Tables 13–17)FTJG
Same search space and budget for all variants, with referencesSection 5 (setup)BYDB
CodeSupplementary material; reproducibility statementBYDB
AI use statementAfter the reproducibility statement—

We hope that these additions and our individual responses address the reviewers' concerns, and we would be grateful if the reviewers would consider updating their scores accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.