Operator-Relative Energy: A Unified Framework for Understanding and Mitigating Over-smoothing in Hypergraph Neural Networks

Repeated message passing in hypergraph neural networks (HGNNs) can suppress distinctions between node representations, a phenomenon known as over-smoothing that limits the benefits of deeper architectures. The resulting form of representational collapse depends on the underlying propagation rule. However, classical measures of over-smoothing, such as the Dirichlet energy, can fail to detect collapse when applied across different propagation rules. To address this limitation, we introduce an operator-relative energy that measures representation variation relative to the collapse behavior of a given propagation rule, while recovering classical Dirichlet energies as special cases. Using the proposed energy, we show that representation variation can decay exponentially with network depth across a broad class of hypergraph propagation operators, leading to over-smoothing, whereas initial residuals can prevent complete collapse by preserving a nonzero level of variation. Motivated by these theoretical results, we incorporate initial residuals and identity mappings (II) into five representative HGNN architectures to mitigate over-smoothing. Controlled propagation experiments on six real-world datasets demonstrate that the classical Dirichlet energy can fail to detect over-smoothing, while our operator-relative energy correctly detects representation collapse and confirms that initial residuals prevent the energy from vanishing. Node-classification experiments further show that the proposed II variants consistently improve predictive accuracy over their corresponding baseline architectures and sustain their performance as network depth increases.