- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that standard hypergraph benchmarks may not actually distinguish the aggregation mechanisms that different hypergraph neural networks are designed around. It shows that benchmark accuracy is often driven much more by whether a node’s own features can directly reach its prediction than by the choice of aggregation operator. The main contribution is therefore a diagnosis of what current benchmarks measure, together with HYPERCSP, a controlled benchmark suite designed to test higher-order constraint reasoning where different aggregation mechanisms should behave differently.
The validation is extensive and uses ablations, controlled synthetic tasks, standard hypergraph benchmarks, and a real error-correcting-code decoding problem. Additive aggregators fail on the modular constraint tasks that the proposed product-based aggregator can often solve, but the product-based method gives no improvement on seven standard benchmarks, suggesting that these benchmarks do not exercise this capability. The authors further validate this interpretation with positive controls, exact non-neural references, careful evaluation protocols, and LDPC decoding, where the expected advantage of the product-based aggregation reappears.
Critical strengths
The controlled ablations show that direct access to a node’s own features can influence performance substantially more than the choice of aggregation operator, which changes how small performance differences between HNN architectures should be interpreted.
The authors show that the proposed product-based aggregation succeeds on controlled modular constraints, injected parity structure, and LDPC decoding while providing no advantage on the standard benchmarks
Critical weaknesses
- The empirical scope is relatively narrow for a paper making conclusions about what hypergraph benchmarks measure, particularly when broader published evaluations now compare many more architectures, datasets, and tasks under unified protocols [1]. This concern is more consequential because prior work has already questioned whether commonly used hypergraph datasets meaningfully exercise hypergraph-specific structure. So, the paper’s strongest additional claim depends on showing that its feature-path diagnosis generalises beyond the limited set of released implementations [2].
- The paper identifies one specific form of higher-order interaction that standard benchmarks appear not to reward, but this does not establish that the benchmarks generally lack useful higher-order structure [3]. Other published architectures model joint higher-order interactions or factor-level dependencies in substantially different ways, so the negative result for the proposed product-based probe cannot by itself rule out higher-order information that would be exploitable by other model families [3,4].
References
[1] DHG-Bench: A Comprehensive Benchmark for Deep Hypergraph Learning, ICLR'26.
[2] Hypergraph Neural Networks Through the Lens of Message Passing: A Common Perspective to Homophily and Architecture Design, TMLR'25.
[3] T-HyperGNNs: Hypergraph Neural Networks via Tensor Representations, IEEE TNNLS'25.
[4] Factor Graph Neural Networks, NeurIPS'20.
Questions for authors
- How broadly do the authors intend the central benchmark conclusion to be interpreted? In particular, is the intended claim only that the seven examined benchmarks fail to reveal the modular, receiver-specific interaction detected by PRODSET, or is the paper making the stronger claim that aggregation mechanisms or higher-order information are generally not being tested by these benchmarks? A clear narrowing of the claim would alleviate my main concern about overgeneralization.
- Beyond ED-HNN, is there evidence that the feature-path attribution persists in the released implementations and published configurations of the other major HNN architectures considered? The paper states that such models are otherwise represented by simplified reimplementations. If important preprocessing, residual, normalisation, or optimisation behavior differs from the published implementations, that would materially affect how confidently the diagnosis can be generalised to the HNN literature.