acceptodds
Under review as a conference paper at ICLR 2027

What Do Hypergraph Benchmarks Measure? Feature Paths, Evaluation Protocols, and Modular Constraints

Abstract

Hypergraph neural networks (HNNs) are compared by node-classification accuracy on a fixed set of benchmarks, and the differences are read as evidence about their aggregation operators. We test that reading from two sides. First, we ask what the benchmarks measure. Ablating the released ED-HNN code at its published configurations, and a factorial over datasets, hyperedge constructions and architectures, show that accuracy depends mainly on whether a node's own features reach its output — often through a single preprocessing line — while the aggregation operator moves it by under two points. Second, we ask what the benchmarks cannot measure. We build HyperCSP, constraint hypergraphs with known answers, a majority-class floor and a non-neural reference, evaluated so that input labels cannot be copied and no test label touches model selection. On modular constraints, additive aggregators with linear messages solve none of 720 runs, while an exclusive-product aggregator (ProdSet) solves 149 of 240. Controls locate the difference in a receiver-specific interaction among the other members: the same product computed from additive power sums performs like ProdSet, averaging inside the same pipeline fails, a receiver-exclusive Deep Sets learns the binary constraint but rarely the ternary one, and every additive arm learns a cardinality constraint. A naive evaluation protocol hides this picture, overstating ProdSet and erasing the cardinality result. Carried back to seven benchmarks, ProdSet is never better than mean aggregation, even under a protocol in which it recovers parity constraints injected into a real hypergraph; on LDPC decoding, where such constraints exist, it recovers much of belief propagation's gain and additive arms none. Under the probes and protocols we test, the capability that separates these operators is one the benchmarks do not exercise. We release HyperCSP, the protocol and all code.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
hypergraph neural networks, benchmark evaluation, higher-order reasoning, message passing, belief propagation, aggregation operators, constraint satisfaction, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Ljs3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that standard hypergraph benchmarks may not actually distinguish the aggregation mechanisms that different hypergraph neural networks are designed around. It shows that benchmark accuracy is often driven much more by whether a node’s own features can directly reach its prediction than by the choice of aggregation operator. The main contribution is therefore a diagnosis of what current benchmarks measure, together with HYPERCSP, a controlled benchmark suite designed to test higher-order constraint reasoning where different aggregation mechanisms should behave differently.

The validation is extensive and uses ablations, controlled synthetic tasks, standard hypergraph benchmarks, and a real error-correcting-code decoding problem. Additive aggregators fail on the modular constraint tasks that the proposed product-based aggregator can often solve, but the product-based method gives no improvement on seven standard benchmarks, suggesting that these benchmarks do not exercise this capability. The authors further validate this interpretation with positive controls, exact non-neural references, careful evaluation protocols, and LDPC decoding, where the expected advantage of the product-based aggregation reappears.

Critical strengths

The controlled ablations show that direct access to a node’s own features can influence performance substantially more than the choice of aggregation operator, which changes how small performance differences between HNN architectures should be interpreted.

The authors show that the proposed product-based aggregation succeeds on controlled modular constraints, injected parity structure, and LDPC decoding while providing no advantage on the standard benchmarks

Critical weaknesses
  1. The empirical scope is relatively narrow for a paper making conclusions about what hypergraph benchmarks measure, particularly when broader published evaluations now compare many more architectures, datasets, and tasks under unified protocols [1]. This concern is more consequential because prior work has already questioned whether commonly used hypergraph datasets meaningfully exercise hypergraph-specific structure. So, the paper’s strongest additional claim depends on showing that its feature-path diagnosis generalises beyond the limited set of released implementations [2].
  2. The paper identifies one specific form of higher-order interaction that standard benchmarks appear not to reward, but this does not establish that the benchmarks generally lack useful higher-order structure [3]. Other published architectures model joint higher-order interactions or factor-level dependencies in substantially different ways, so the negative result for the proposed product-based probe cannot by itself rule out higher-order information that would be exploitable by other model families [3,4].

References

[1] DHG-Bench: A Comprehensive Benchmark for Deep Hypergraph Learning, ICLR'26.

[2] Hypergraph Neural Networks Through the Lens of Message Passing: A Common Perspective to Homophily and Architecture Design, TMLR'25.

[3] T-HyperGNNs: Hypergraph Neural Networks via Tensor Representations, IEEE TNNLS'25.

[4] Factor Graph Neural Networks, NeurIPS'20.

Questions for authors
  1. How broadly do the authors intend the central benchmark conclusion to be interpreted? In particular, is the intended claim only that the seven examined benchmarks fail to reveal the modular, receiver-specific interaction detected by PRODSET, or is the paper making the stronger claim that aggregation mechanisms or higher-order information are generally not being tested by these benchmarks? A clear narrowing of the claim would alleviate my main concern about overgeneralization.
  2. Beyond ED-HNN, is there evidence that the feature-path attribution persists in the released implementations and published configurations of the other major HNN architectures considered? The paper states that such models are otherwise represented by simplified reimplementations. If important preprocessing, residual, normalisation, or optimisation behavior differs from the published implementations, that would materially affect how confidently the diagnosis can be generalised to the HNN literature.
Reviewer_4sJ6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors claim that existing hypergraph neural network (HNN) benchmarks are "to a first approximation, comparisons of feature-path strength". They conclude this by showing how ablating HNN feature paths reduces their performance substantially. Thus, they propose a new synthetic HNN benchmark (hyperSCP) and a neighbor aggregation method (ProdSet) that improves HNN performance on the proposed benchmark.

Critical strengths
  • Motivation: The problem that the authors aim to address is important. Hypergraph learning literature has a distinct focus on high-order reasoning, and how existing benchmarks measure such capability has not been clearly established.
Critical weaknesses

Issue 1. Lack of soundness:

  • One of the key proposed contributions is ill-founded ('on most datasets published HNN comparisons are, to a first approximation, comparisons of feature-path strength'). I find Table 1 to be particularly problematic. I think ablating a black-box model's hyperparameters/architectures to conclude the role of the ablated component is too naive. By the authors' method, I could also (1) ablate normalization layers from a transformer, (2) test its performance, and (3) conclude the NLP benchmarks actually are tests of normalization. The fairness of the experimental setup (e.g., non-optimized hyperparameters for the ablated models) to make this conclusion is a separate issue that needs to be addressed.

Issue 2. Lack of novelty and significance:

  • I do not think the proposed method (prodset) and evaluation protocol (hyperscp) are novel or significant.
  • For example, both the practical and theoretical implications of the HyperSCP task are unclear. In practical applications, what advantages can we expect from an HNN that better solves HyperSCP tasks?
  • The design rationales behind the proposed method, prodset, are weak. The proposed benchmark task is synthetic and (arguably) non-practical, so I would expect formal analysis on why prodset improves HNNs in the proposed task. Without it, I think this paper provides little insight into the solution.
Questions for authors

What does HyperSCP stand for?

Reviewer_1G9qOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether performance differences among hypergraph neural networks (HNNs) genuinely reflect their aggregation capabilities. Through controlled experiments, the authors show that preprocessing choices largely dominate performance on existing benchmarks. To better evaluate aggregation operators, they introduce HyperCSP, a synthetic benchmark with known constraint structures and solutions. Experiments reveal that additive aggregators outperform multiplicative ones on certain modular constraints, while ProdSET struggles with cardinality constraints. Further evaluations on real-world hypergraphs suggest that standard benchmarks fail to distinguish aggregation capabilities effectively.

Critical strengths

S1. The empirical analysis is insightful. The paper identifies preprocessing as a major confounding factor in existing HNN evaluations, challenging the common attribution of performance gains to architectural improvements.

S2. The proposed benchmark provides a useful evaluation setting. HyperCSP introduces controlled constraints and known solutions, enabling more targeted analysis of aggregation capabilities beyond conventional node classification benchmarks.

Critical weaknesses

W1. The practical relevance of HyperCSP needs further justification. The proposed benchmark relies on synthetic constraint satisfaction problems, but it remains unclear how closely these constraints reflect higher-order relationships encountered in real-world hypergraph learning tasks.

W2. The evaluation of aggregation operators is relatively limited. Although the paper considers several additive and multiplicative aggregators, the findings are primarily based on specific configurations and constraint types. It is unclear whether the conclusions generalize to broader HNN architectures and aggregation mechanisms.

W3. The analysis of real-world benchmarks could be more comprehensive. The experiments focus on showing that ProdSET offers limited benefits on seven benchmarks, but additional evaluations across different datasets, tasks, and hypergraph characteristics would help establish the generality of this conclusion.

W4. The implications for future HNN design remain unclear. While the paper demonstrates that standard benchmarks may fail to distinguish aggregation capabilities, it provides limited guidance on which aggregation properties are important for practical tasks and how these findings should inform the development of more expressive HNNs.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.