- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The author proposes SheaFormer, which reframes sheaf diffusion as a structured multi-head operator. Specifically, the method consists of residual propagation + role-aware bilinear restriction maps for directed hypergraphs.
Critical strengths
-
The overall idea that role-aware restrictions can distinguish orientations that DSHN cannot is interesting.
-
Theoretical claims (stream-major decomposition) are intuitive.
-
Experimental results are competitive compared to the SOTA methods.
Critical weaknesses
-
The novelty seems incremental, as a sheaf-attention mechanism already exists. Theorem 1 can be viewed as just a new interpretation.
-
The ablation also changes the normalization placement and dropout. Is the residual block's contribution properly isolated? The ablation results seem inconsistent with Table 1.
-
Table 1 separates the gains. Since each variant is independently hyperparameter-tuned, how do the authors rule out that part of these differences comes from different selected hyperparameters?
-
Code is not provided.
Questions for authors
Please see the above weaknesses
4 comments
Reply to Reviewer BYDB (2/4)
Weakness 2. The ablation also changes the normalization placement and dropout. Is the residual block's contribution properly isolated? The ablation results seem inconsistent with Table 1.
W2 (residual isolation; consistency with Table 1). Yes: we now remove only the residual term, leaving the rest of the layer unchanged, and the results are consistent with Table 1, since and measure different changes. Table 1 measures the complete SheaFormer block rather than the residual connection alone: SheaFormer (DSHN restr.) replaces DSHN's layer with the SheaFormer block, which changes the residual formulation together with the placement of normalization and dropout. This is what measures: and points relative to DSHN and DSHN-Light.
To isolate the residual connection, we removed only the residual term, so that the layer becomes
with normalization, dropout and the role-aware bilinear restrictions unchanged, and re-tuned the resulting model with the same search space and budget as the SheaFormer variants of Table 1 (150 Optuna trials, 10 splits). The residual connection improves accuracy on 10 of 12 benchmarks, by points on average:
| Dataset | CEHom | SheaFormer | SheaFormer w/o residual | |
|---|---|---|---|---|
| Roman-empire | ||||
| Squirrel | ||||
| Chameleon | ||||
| email-EU | ||||
| Telegram | ||||
| Cornell | ||||
| email-Enron | ||||
| Wisconsin | ||||
| Amazon-ratings | ||||
| Texas | ||||
| Citeseer | ||||
| Cora | ||||
| Mean | — |
This resolves the apparent inconsistency: measures a complete layer replacement, whereas measures the residual connection alone. The two are also measured with different restriction generators, DSHN's for and SheaFormer's for , so they are not additive steps of a single chain. Table 1 labels this row's controlled change "Residual block", whereas the Q1 discussion attributes to the SheaFormer layer as a whole; the revision relabels the row "SheaFormer block", carries this attribution into the abstract, the contributions, the Q1 question and the conclusions, and reports separately.
Reply to Reviewer BYDB (3/4)
Weakness 3. Table 1 separates the gains. Since each variant is independently hyperparameter-tuned, how do the authors rule out that part of these differences comes from different selected hyperparameters?
W3 (independent tuning). Independent tuning under a common protocol is what makes the comparison fair, and the differences do not hinge on the selected configurations: averaged over the neighborhoods of their selected configurations, the four variants keep the order of Table 1. Our intention is to compare the performance achievable by each architecture under equal tuning effort. No variant receives more tuning than another. The four SheaFormer variants of Table 1, and the model without residual, are tuned over the same search space (, , , learning rate, weight decay, both dropouts, ; Appendix E), with the same budget of 150 Optuna trials, the same ten splits and the same selection rule (mean validation accuracy).
The alternative is to fix the hyperparameters and compare performance across variants. However, no single configuration is necessarily neutral, and a configuration selected by tuning one variant may favor that variant over the others. Our residual ablation shows this bias directly. At SheaFormer's selected configuration, on the same ten splits, removing the residual term lowers accuracy by points on average, against when the model without residual is re-tuned under the same protocol: on average, the model without residual is points less accurate at SheaFormer's configuration than at its own ( against ), so the fixed comparison overstates the contribution of the residual connection by points. On Roman-empire it even reverses the sign, from at SheaFormer's configuration to after re-tuning. Thus, while fixing hyperparameters isolates the effect of an architectural change at a particular configuration, it may not fairly reflect the performance achievable by each architecture after optimization. Table 1 therefore compares test accuracy at independently validation-selected configurations under identical search protocols. Here, the selected hyperparameters are an outcome of optimizing each architecture, and comparability is ensured through the same search space, tuning budget and selection rule. In particular, every configuration searched or selected for one SheaFormer variant is admissible for all the others. This follows the established practice of comparing architectures after independent hyperparameter optimization under a common protocol [2–4].
Moreover, to compare the performance of the variants on similar hyperparameter configurations and to show that the selected configurations are not isolated optima, we conduct a further analysis. Following the sensitivity check of [2] (Section 7.3), for each of the 48 tunings of the four variants we examined the trials within one step of the search space of the selected configuration on every hyperparameter ( and within a factor of , within one layer, weight decay within one grid value, within , dropout rates within ), with a factor of for the continuous learning rate. These neighborhoods hold 5 to 60 trials per tuning (median 28.5), with 90% of all 1,392 neighboring trials reaching a validation accuracy within 2 points of the selected one. Averaging the test accuracy over these neighborhoods, instead of taking the selected trial, gives SheaFormer (DSHN restr.) , SheaFormer (MLP role-blind) , SheaFormer (MLP role-aware) and SheaFormer : (Table 1: ), of which generator capacity accounts for (), role awareness for () and the bilinear compatibility for ().
[2] Melis, Gábor, Chris Dyer, and Phil Blunsom. “On the State of the Art of Evaluation in Neural Language Models.” International Conference on Learning Representations (2018).
[3] Shchur, Oleksandr, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. “Pitfalls of graph neural network evaluation.” arXiv preprint arXiv:1811.05868 (2018).
[4] Errica, Federico, Marco Podda, Davide Bacciu, and Alessio Micheli. “A Fair Comparison of Graph Neural Networks for Graph Classification.” International Conference on Learning Representations (2020).
Reply to Reviewer BYDB (4/4)
Weakness 4. Code is not provided.
W4 (code). The code is now provided in the supplementary material, and will be made public upon acceptance.
Reply to Reviewer BYDB (1/4)
We thank the reviewer for the detailed review and for highlighting that role-aware restrictions distinguish orientations that DSHN cannot, which is the core novelty of the paper. We address the novelty, ablation, tuning and code concerns below, with new experiments where the reviewer asked for evidence.
W1 (novelty vs. sheaf attention; Theorem 1 as an interpretation). Unlike the common trend in the literature, where an attention block is added on top of an existing architecture, with our work we establish (Theorem 1) an exact identity showing that sheaf diffusion with diagonal restriction maps already features the additive multi-head structure of Transformer attention without the need for introducing additional attention weights. Let us look at works which add a Transformer-based block on top of an existing network. SheaFormer differs from the existing Sheaf Attention Networks (SANs) both in its construction of the propagation operator and in the structural properties it preserves. SANs multiply the sheaf-derived adjacency coefficients with GAT-style scalar attention coefficients. However, the resulting attention-weighted propagation operators are not generally standard sheaf operators. In particular, they need not preserve Hermitian symmetry or admit the positive-semidefinite factorization underlying sheaf diffusion. In contrast, SheaFormer does not introduce a separate attentional mechanism on top of sheaf propagation. Instead, it learns the restriction maps themselves through role-aware bilinear node-hyperedge compatibilities, with separate head and tail contexts, preserving the Hermitian positive-semidefinite structure of the standard sheaf operators.
Theorem 1 provides the structural motivation for this construction. It establishes an exact decomposition of the standard sheaf layer into stream-wise node operators with structured value and output maps. For diagonal sheaves, the decomposition takes the additive multi-head form of Transformer attention, without introducing additional attention weights. This identity is what the design rests on, not an interpretation added afterwards: since the multi-head structure is already in the sheaf layer, SheaFormer learns its attention-like operators through the restriction maps instead of adding attention.
The novelty also goes beyond this decomposition:
Theorems 3 and 4 establish formal representational advantages of the proposed restriction construction, while Theorem 2 characterizes additional mechanisms governing the behavior of sheaf diffusion. These contributions go beyond alternative interpretations of existing operators.
Empirically, the role-aware bilinear restriction generator improves accuracy by +1.92 points on average over the DSHN generator, on 11 of 12 benchmarks.
[1] Bodnar, Cristian, et al. "Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in gnns." Advances in Neural Information Processing Systems 35 (2022).