acceptodds
Under review as a conference paper at ICLR 2027

Hodge Spectral Concentration in Simplicial Complex Neural Networks

Abstract

Simplicial neural networks extend learning on graphs to higher-order topological structures, but feature propagation in these networks remains less well understood. We develop a spectral framework and characterize these dynamics through the Hodge decomposition. Rather than summarizing feature evolution with a single energy, as used to characterize over-smoothing and -sharpening in graph neural networks, we track the feature-normalized lower and upper Hodge Rayleigh quotients separately. This decouples spectral concentration from changes in feature magnitude and retains information about the gradient and curl subspaces, allowing us to distinguish three forms of spectral concentration: harmonic-dominant, gradient-dominant, and curl-dominant dynamics. For linear simplicial convolutions with independent, symmetric lower- and upper-channel mixing weights, we show that the dynamics are a gradient flow of a learnable energy and derive three competing spectral growth rates associated with the harmonic and highest-frequency gradient and curl eigenspaces. We then show that these rates determine the asymptotic regime for almost every input. Since they are computed from the learned weights and Hodge frequencies, these rates provide a post-training diagnostic of which eigenspace the normalized features concentrate in. We instantiate our framework with normalized propagation choices, including an adjacency-like form that recovers graph convolution at rank zero. Finally, experiments on synthetic edge classification and ocean-drifter trajectory prediction tasks validate the spectral predictions and illustrate how supervision and the choice of propagation operator select among the qualitatively distinct spectral concentration regimes.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
geometric deep learning, topological deep learning, simplicial complex neural networks, over-smoothing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rBTsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how features evolve through deep simplicial neural networks using the Hodge decomposition. It shows that features can eventually concentrate in one of three types of subspaces: harmonic, gradient, or curl, and provides simple spectral conditions that predict which regime will occur. Experiments on synthetic edge classification and ocean drifter prediction support these theoretical predictions.

Critical strengths

S1. A key strength is that the proposed spectral criterion is both interpretable and practical: after training, it can be computed directly from the learned weights and the relevant Hodge eigenvalues to identify whether the network amplifies harmonic, gradient, or curl components. The theoretical predictions are also supported by some experiments.

S2. The presentation of the paper is clear and well organised, and the figures are generally well designed and help illustrate the main ideas and results.

Critical weaknesses

W1. The novelty is meaningful but somewhat narrow. The harmonic/gradient/curl spectral-concentration framework is a clean extension of existing graph and simplicial oversmoothing analyses, but much of the underlying machinery builds on established Hodge spectral tools. The main conceptual novelty lies in the paired feature-normalised lower and upper Hodge Rayleigh quotients, the resulting three-regime characterisation, and the spectral growth-rate analysis describing when each regime arises. The related-work discussion could be broadened to better position these contributions within existing Hodge-based learning on simplicial complexes. In particular, prior work has already explored Hodge-aware spectral representations and learning architectures, with some works also studying feature propagation and oversmoothing-related behaviour in this setting [1 to 4]. Discussing these works would help clarify more precisely what is new in the proposed harmonic/gradient/curl spectral-concentration framework.

[1] Hodgelets: Localized Spectral Representations of Flows on Simplicial Complexes by Roddenberry and al. (2022)

[2] Topological Signal Processing over Weighted Simplicial Complexes by Battiloro and al. (2023)

[3] Hodge-Aware Contrastive Learning by Möllers and al. (2024)

[4] Graph and Simplicial Complex Prediction Gaussian Process via Hodgelet Representations by Alain and al. (2025)

[5] Binarized Simplicial Convolutional Neural Networks by Yan and al. (2025)

W2. The theoretical analysis is elegant but its strongest predictive results apply to a restricted class of linear, symmetric, depth-shared SNNs, while the conclusions concern asymptotic depth and may therefore not directly characterise the finite depths used in practice. Moreover, the main synthetic experiment defines its labels directly from harmonic, high-frequency gradient, and high-frequency curl eigenspaces, closely aligning the task with the decomposition underlying the theory and making this validation somewhat controlled. The empirical evidence beyond this setting is limited to a single small ocean-drifter dataset. It therefore remains unclear how informative the proposed spectral trichotomy will be for more realistic finite-depth SNNs with nonlinear propagation.

Questions for authors

I would be willing to raise my score if the questions below are addressed and the points raised in the weaknesses section are also adequately answered. Similarly, if they are not addressed, I would consider lowering my score.

Q1. Can the authors provide any finite-depth guarantee relating the predicted asymptotic regime to the Rayleigh quotients observed at the depths used in practice?

Q2. Could the authors include a synthetic task whose labels are not constructed directly from harmonic, gradient, or curl eigenspaces?

Q3. Can the proposed post-training diagnostic be validated on a broader set of real-world datasets or architectures?

3 comments

We thank the reviewer for their thoughtful suggestions. In particular, we appreciate the reviewer's recognition of the theoretical novelty and the interpretability of the Hodge Rayleigh Quotients as a spectral criterion. Below, we address the reviewer's concerns and questions related to the positioning of the contribution, finite depth behavior, and the breadth of empirical evidence.

W2(a) / Q1, Finite-depth guarantees We agree that characterizing finite-depth behavior is a useful extension of the asymptotic theory. We now address this by extending the spectral argument underlying Proposition 3.5 to give an explicit finite-depth bound on the "error" between a Rayleigh quotient at some layer and its limit. We now include this result in Section 3.2 (Eq. (15), the linear case of Proposition 3.7), with its proof in Appendix H.

Under the assumptions of Theorem 3.6, suppose the Euler step size satisfies 2+τ(μ∗+μmin⁡)>02+\tau(\mu_*+\mu_{\min})>0 and the input has nonzero projection onto the leading generator eigenspace. Here μ∗\mu_* and μmin⁡\mu_{\min} are the largest and smallest generator eigenvalues. For either Hodge Rayleigh quotient qq, its deviation from the endpoint q∞q_\infty predicted by Theorem 3.6 satisfies

\leq \nu_{\max}\frac{C\kappa^{2m}}{1+C\kappa^{2m}}, \qquad m\geq1,$$ where $$C = \frac{\Vert (I-P_{\ast})F_0 \Vert_F^2} {\Vert P_{\ast}F_0 \Vert_F^2}, \qquad \kappa = \max_{\mu_i < \mu_{\ast}} \frac{|1+\tau\mu_i|} {|1+\tau\mu_{\ast}|} < 1.$$ Here $P_*$ projects onto the leading generator eigenspace, $\mu_i$ range over the generator eigenvalues, and $\nu_{\max}$ is $\nu_g^{\max}$ or $\nu_c^{\max}$ for the lower or upper quotient, respectively. The bound identifies two factors controlling finite-depth concentration: (1) the initial mass outside the leading eigenspace relative to the mass inside it ($C$), and (2) the strongest relative amplification per layer outside that eigenspace ($\kappa$). A larger initial leading component and a smaller $\kappa$ give a tighter guarantee. For a fixed trained model and input, this provides an explicit error bound at any specified depth and a sufficient depth to reach a prescribed tolerance around the predicted endpoint. **W2(b) / Q2, Additional synthetic experiment** Regarding a synthetic task with labels not directly derived from H, G, and C eigenspaces, we include an additional experiment in Figure 3 and Table 1 of the manuscript with labels that correspond to the geometric "region" of each edge, i.e. which disk it belongs to, effectively a segmentation task. The learned endpoint rates predict harmonic dominance, and both Rayleigh quotients approach zero under extended propagation, in agreement with this prediction. This provides additional validation of the post-training diagnostic on a task whose labels are not defined by the spectral decomposition underlying our theory. We can tentatively interpret this result as highlighting the importance of harmonic information (i.e. global topological holes) for this particular complex and labeling. | Task | Macro-F1 | Predicted regime | $\mathrm{RQ}_{\mathrm{down}}$ | $\mathrm{RQ}_{\mathrm{up}}$ | |:--|:-:|:-:|:-:|:-:| | Harmonic | 0.952 | Harmonic | $2.5\times10^{-7}$ | $9.6\times10^{-8}$ | | Upper-gradient | 0.919 | Gradient | $0.7412$ | $\approx 0$ | | Upper-curl | 0.902 | Curl | $\approx 0$ | $0.9833$ | | Region | 0.883 | Harmonic | $6.6\times10^{-5}$ | $1.1\times10^{-3}$ | We continue our response in the following comment.

W2(c) / Q3, Other datasets and architectures We agree that validating the post-training spectral diagnostic and theoretical results across a broader set of datasets and architectures is important and now include additional experiments in Appendix J. We extend the evaluation to two additional real-world datasets, Forex and Last.fm (listener 1), and to nonlinear propagation and published simplicial architectures (Appendix J.2). On both additional datasets, the linear adjacency flow’s Rayleigh quotients approach the endpoints predicted from its trained weights (Figure 7). We also evaluate tanh\mathrm{tanh} variants of our flows (Figures 5 and 7), alongside SCoNe, SCNN and SCCNN architectures (Roddenberry et al., 2021; Yang et al., 2022a; Yang & Isufi, 2023), reporting task performance and layerwise Rayleigh quotients (Table 2 and Figure 7). Some qualitative concentration patterns persist under tanh\mathrm{tanh}, although the trajectories can retain mixed spectral content and depart from the linear endpoints. These experiments further validate the linear predictor and demonstrate the utility of the paired Rayleigh quotients to characterize finite-depth spectral evolution in more general SNNs, despite not guaranteeing an endpoint. See also Table A in response to reviewer GQFc for additional results on the ocean drifter dataset.

Table B: Forex and Last.fm

DatasetModelTest R2R^2Predicted regime (splits)Dominant regime at depth 1,024RQ/ν (down, up) at depth 1,024
ForexAdjacency flow, linear0.92G 8, C 2G 8, C 2(0.80, 0.20)
Adjacency flow, tanh0.89G 10G 7, C 3(0.62, 0.38)
SCoNe, tanh0.91G 5, C 5——
SCNN (order 2), tanh0.92G 6, C 4——
Last.fm (listener 1)Adjacency flow, linear0.30H 10H 10(0.00, 0.00)
Adjacency flow, tanh0.30H 10H 10(0.07, 0.07)
SCoNe, tanh0.14C 10——
SCNN (order 2), tanh0.03C 9, G 1——

W1, Additional related works and positioning

Finally, we agree that the established Hodge spectral machinery and prior Hodge-aware learning methods should be acknowledged more explicitly. We are grateful for these suggestions and expand our related works section accordingly, as it leads to a stronger final manuscript. We position our contribution as a theory of Hodge spectral concentration that complements existing Hodge-aware architectures and methods as a model-agnostic framework and diagnostic. Under certain conditions, the paired feature-normalised Rayleigh quotients give us the precise results with explicit growth-rate conditions taken directly from trained weights.

Closing We thank the reviewer again for the feedback and the clear pointers for addressing them. We hope that we have addressed all of the reviewer’s concerns with new theory and experiments and would be grateful if they would consider raising their score. We are available throughout the discussion period to clarify any concerns that remain after these responses.

Dear Reviewer rBTs,

With the discussion period closing on November 18 (AoE), we wanted to follow up on the finite-depth bound (W2(a) / Q1), the synthetic edge segmentation task with region-based labels (W2(b) / Q2), and the additional Forex and Last.fm experiments for linear and nonlinear models (W2(c) / Q3). Following discussion with Reviewer YHy1, we have also extended the finite-depth bound (Section 3.2, Proposition 3.7) to elementwise nonlinearities such as tanh⁡\tanh under certain conditions, and noted that it holds at every simplicial rank (Remark 3.8). If you've had a chance to review the new results, we would appreciate knowing whether you have any remaining concerns or questions, and we are happy to discuss further.

Reviewer_YHy1OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper develops a framework for understanding the feature propagation behavior in simplicial complex neural networks of a certain design, which encompasses several rank 1 and degree 1 simplicial convolutional networks. The main observation is that if there is no activation function, i.e., in the linear regime, and if the learnable weight matrices at all layers are symmetric and identical, then every layer implements an Euler discretization step of a particular dynamic. By studying this dynamic, the paper shows that depending on the spectral characteristics of the weight matrices, the dynamic exhibits mutually exclusive regimes: harmonic dominant (Rayleigh quotient →0\to 0), gradient dominant (down Rayleigh quotient converges to its upper bound), and curl-dominant (up Rayleigh quotient converges to its upper bound). Two experiments are presented, providing evidence for the theoretical results and demonstrates these behaviors on an established dataset.

Critical strengths
  • The paper is well-written. Although I didn't read the proofs carefully, the math ideas and some proof details that I read are sound.
  • The idea to track the Hodge Rayleigh quotients to gain more information from the dynamics is sound, and the results obtained support the claim that the dynamic can be understood naturally through the Hodge decomposition.
  • The classification of the dynamic into three mutually exclusive regime is clean and interesting from a dynamical system perspective.
Critical weaknesses
  • The main motivation for this paper is to develop better theoretical understanding of deep higher order networks. It can be argued that this is not of much interest to even theory-minded members of the community, such as I am.
  • Even discounting the previous weakness, the theory doesn't seem to have any implication for practitioners, and doesn't answer any question raised by other researchers. Its closest match in the literature is probably [Di Giovanni et al., 2023] (Understanding convolution on graphs via energies). That paper establishes several interesting observations, including that contrary to what was commonly believe, GCN is more than just a low past filter. This paper doesn't have any equivalent result.
  • The set of assumptions for the main theorems are restrictive: that there is no non-linearity, and the same layer with the same weights are applied again and again.
  • The results are very clean, but the techniques used to obtain them are common limiting arguments. Given this, I expect more (and more interesting) results than what the paper currently presents.
  • Overall, the paper's substance is good, but there's just not enough of it.
Questions for authors
  • Regarding my first weakness, my impression is that even in the GNN community where deep networks have been studied for a long time, they have not demonstrated the same gain in performance as those in other applications/domains. Given this, can the authors give me a reason as to why I should be interested in problems such as over-smoothing or over-sharpening in simplicial networks?
  • Aside from the dynamical behavior of a particular fixed weight, linear layer type, how does this paper improve our theoretical understanding of simplicial networks?
  • Can the authors comment on whether the assumptions can be relaxed?
  • This is a suggestion more than a question, but it would be very valuable to see some practical/theoretical implication/prediction from the theory, rather than just demonstration that the theory is correct.
7 comments

We thank the reviewer for the thoughtful review. We appreciate their positive assessment of the mathematical soundness, presentation, and the Hodge Rayleigh-quotient framework. Below, we address their concerns about motivation, theoretical and practical implications, and the scope of the assumptions.

W1 / Q1, Motivation for studying over-smoothing and over-sharpening

We respectfully note a discrepancy between the positive assessments of our paper’s technical quality (3,2,3) and the overall score (1). As stated in the first weakness and question, the primary driver of this rating is a broad skepticism toward the subfield of deep GNNs and higher-order learning as a whole ("not of much interest to even theory-minded members of the community", or "can the authors give me a reason as to why I should be interested in problems such as over-smoothing or over-sharpening in simplicial networks"). While every researcher is entitled to their perspective on which subfields are most promising, ICLR review guidelines require that a paper be evaluated on the scientific validity, originality, and execution of its contributions within its scope, which in this case is an active research area with many open questions and researchers investigating them.

Firstly, a mathematically sound and practically relevant theory remains largely lacking for deep learning in general, let alone graphs and their higher-order counterparts. A principled theory of learning directly informs architecture selection, hyperparameter tuning, and training dynamics, substantially reducing the costly trial-and-error that practitioners otherwise face. Ultimately, such theory-informed design translates directly into stronger and more reliable empirical performance.

Historically, the reason deep GNNs and SNNs have struggled to scale in depth is precisely the lack of theoretical understanding of their feature propagation dynamics. In standard GNNs, early architectures (like vanilla GCN) suffered from performance degradation at depth until theoretical works diagnosed over-smoothing and over-sharpening (e.g., Di Giovanni et al., 2023), which directly led to principled fixes (residual connections, continuous-time flows, and weight constraints) and further ongoing research (GNNs are still an active area of investigation). Charaterizing these is exactly the way forward to make more performant graph learning machines.

Strictly generalizing Di Giovanni et al., our work provides this exact foundational diagnostic before higher-order architectures proliferate blindly. More importantly, we show that higher-order depth dynamics are fundamentally different from graph depth dynamics:

  • On a connected graph (rank 0), low-frequency dominance (EDir→0E_{\text{Dir}} \to 0) collapses features to a trivial degree-scaled constant vector. Hence it is universally termed "over-smoothing".
  • On a simplicial complex (rank k≥1k \ge 1), driving the total Dirichlet energy to zero (Harmonic Dominance, HD) does not collapse features to a trivial constant. Instead, it projects signals onto the βk\beta_k-dimensional harmonic subspace (ker⁡Lk\ker \mathcal{L}_k), which encodes global topological invariants (e.g., circulation around holes).
  • Thus, depth in SNNs is not merely a pathology to be avoided. It acts as a topological filter that can isolate global homology from local divergence (gradient) and local circulation (curl).

These are surprising results, uncovering several phenomena that cannot exist on graphs. Note that, moving beyond pairwise graphs to higher-order structures (simplicial and cell complexes) is increasingly vital in domains where group-wise and multi-body interactions are fundamental such as molecular and protein modeling, 3D geometry/CAD processing, physical flow modeling, and multi-agent coordination, making our contribution an important milestone for future AI models.

Finally, we respectfully disagree that the paper lacks sufficient substance. Our work establishes a comprehensive theoretical framework for analyzing feature propagation in simplicial complex networks. Once again, we kindly ask that our work be evaluated objectively on its scientific merits, rather than through a broad dismissal of the graph and topological deep learning communities. Accordingly, we invite the reviewer to reconsider their overall assessment in light of our detailed responses which follow.

We continue our response in the following comment.

W2 / Q2 / Q4, Implications and predictions

We would like to emphasize that the theory yields predictions from the trained model that allow better interpretability of learned structure. Under the assumptions of Theorem 3.6, the spectra of the trained channel weights together with the extreme Hodge frequencies determine which Hodge sector will dominate, without propagating the network to large depth.

We agree that the low-/high frequency dynamic studied by Di Giovanni et al. represents useful observations and would say that we do that same at rank 1: our framework both allows us to recover those dynamics at rank 0 and observe distinct behavior at rank 1, i.e. the harmonic, top-gradient, and top-curl concentration trichotomy. In particular, unlike low-frequency concentration in the graph case, harmonic dominance is not collapse to a constant representation because it retains information on the global topological holes.

We strengthen these implications in the revision in several ways. We now provide an explicit finite-depth guarantee; see our response to Reviewer rBTs (W2(a)/Q1) and Section 3.3. We also report ocean-drifter prediction accuracy, where the adjacency-like flow consistently selects harmonic dominance and outperforms the pure-Laplacian flow (see our response to Reviewer GQFc, W1(c)/W1(d), Table A), and add experiments on additional real datasets and nonlinear architectures (see our response to Reviewer rBTs, W2(c)/Q3, Table B).

W3 / Q3, Relaxing the assumptions

Regarding the restrictiveness of our assumptions: such linearized, depth-homogeneous conditions are standard in theoretical analyses of deep message-passing dynamics, including the graph setting (e.g., Di Giovanni et al., 2023), where they enable tractable spectral characterizations of feature propagation. They can be relaxed to different extents, as discussed in Appendix C.3. The Hodge Rayleigh-quotient framework itself is model-agnostic and applies to arbitrary edge-feature dynamics, including nonlinear ones. For linear Hodge-polynomial propagation, the decomposition into independent Hodge-frequency modes extends beyond the model of Theorem 3.6, including higher-degree filters and, at the mode-decoupling level, non-symmetric or depth-dependent channel weights.

A pointwise nonlinearity generally breaks Hodge-subspace invariance, so the exact endpoint theorem does not extend directly. However, an important intermediate case is given by odd nonlinearities such as tanh⁡\tanh, which are commonly used in SNNs because they preserve orientation equivariance. This is also important for our diagnostic, since the measured Rayleigh quotients should not depend on an arbitrary orientation choice.

Moreover, tanh⁡(x)=x+O(x3)\tanh(x)=x+O(x^3) near the origin, so when its inputs remain sufficiently small the nonlinear propagation is locally close to the linear dynamics analyzed in our theory. This motivates new experimental directions to observe whether linear regime predictions remain informative for nonlinear SNNs. In Appendix J.2.1 we test this directly: under tanh⁡\tanh, the learned regime selection and qualitative Rayleigh-quotient trajectories are largely preserved, although the trajectories need not converge exactly to the linear endpoints (see also our response to Reviewer GQFc, W2/Q1(b)). Quantifying the deviation between the linear and nonlinear dynamics is an interesting direction for future work.

W4 / W5, Techniques and scope of results

We would say that the contribution is not the limiting argument in isolation, but the non-obvious higher-order structure it exposes. The revision further adds finite-depth guarantees, a clearer analysis of how the assumptions can be relaxed, additional real-data experiments, and nonlinear-architecture evaluations, broadening both the theoretical and empirical scope.

Closing

Thank you again for the feedback. We would be glad to clarify any remaining concerns.

Reviewer_YHy1OpenReview ↗

Comment

There was no discrepancy between my positive assessment (presentation 3, informativeness 2, soundness 3) and the overall low score. The latter was my own assessment of the original submission's value for the ICLR community before the rebuttal process, and it did not solely rely on the former. I value theoretical works, even when they don't have immediate applications. However, for works that don't have such applications, I expect a higher bar for the theoretical results to justify publication at ICLR.

I do not have a broad skepticism towards the subfield of deep GNNs and higher-order learning, and this is not the primary driver of this rating. W1 stems from my own observation of the progress in the field of deep GNNs over the years, as explained in more detail in Q1. My first question was meant as an invitation for the authors to provide more justifications for their research. But I agree that this shouldn't be considered a 'critical weakness'.

I appreciate the comparison with Di Giovanni et al., as it puts this work into a better context. It would be great if this comparison could be incorporated into the paper. This somewhat resolves my W2, but more implications from this would still be nice.

I appreciate the new finite-depth result, real-data experiments, and extra experiment with the tanh activation function.

After the authors' response, my W3, W4, and W5 remains. This work only has 3 main results (3.4, 3.5, 3.6), proved under restrictive assumptions (symmetric, no non-linearity, tied weight, rank 1) and with known techniques. Appendix C.3 mentions but doesn't give any proof for some extensions of the theory, but these extensions are still restrictive. The final sentence in this appendix mentions that a pointwise nonlinearity would break these results. The experiment with the tanh activation function helps, but it only slightly extends the scope for one activation function in an experimental setting. As such, I disagree with the assessment that this work 'establishes a comprehensive theoretical framework for analyzing feature propagation in simplicial complex networks'.

Reviewer_GQFcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper characterizes the evolution of feature representations in simplicial neural networks (SNNs), using Hodge decomposition, which separates the signals into three orthogonal gradient, curl and harmonic components. The paper introduces two Hodge Rayleigh quotients to measure how the spectral energy concentrates in the gradient subspace and the curl subspace under asymptotic infinite depth conditions, and defines corresponding gradient dominant (GD), curl dominant (CD) and harmonic dominant (HD) regimes. This extends the previous line of work on oversmoothing and oversharpening in GNNs to higher order networks. For a simplified setting of linear SNNs, with symmetric weights shared across layers, the authors model feature propagations as a continuous time ODE and prove that the normalized features concentrate in the eigenspaces with largest spectral growth rate. These three rates can be computed form the trained weight matrices and top gradient and curl eigenvalues of the simplicial complex and can be used to predict which Hodge subspace will be dominant at infinite depth without actually running the networks. The experiments confirm the theoretical predictions.

Critical strengths

The paper is well written. For higher order structures, the Hodge aware splitting of spectral energy and measuring the spectral concentration is a natural and intuitive way to extend the analysis on GNNs. The fact that oversharpening can be driven by distinct gradient or curl based dynamics is interesting. Also interesting is that harmonic dominance is not the same as oversmoothing on graphs and that it can retain $ \beta_1 $ topological information. It is also useful that at least in the simplified setting, you can get away with solving only three eigenproblems, and not have to compute the full eigendecomposition of the Hodge laplacian. The GNN analysis setting being recovered at rank 0 is a useful sanity check.

Critical weaknesses

On the experimental side of things, in Figure 4, the ocean drifters model is trained at depth 16 and then extended depth 1024. However, we see that the Rayleigh quotients barely budge till depth 16 and only separate out afterwards. Firstly, this separation in practice perhaps needs to be empirically verified by training to larger depths and computing the coefficients. This would check whether the same behavior occurs when the models are optimised at these depths. Secondly, what does this tell us about the practical applicability of these insights? Since the separation happens at considerably larger depths, representations at more realistic depths will not have this clear cut separation. The paper also does not report next vertex prediction accuracy on the ocean drifter experiments, so we cannot use that to estimate practical applicability. Also the end of section 4.3 claims " harmonic information may help for this topology", but I don't see any experimental evidence in support.

On the theory side of things, it is unclear what holds outside of the simplified setting. The motivating models in lines 149-150, use either non-linearities or different weights across layers, which violates these assumptions. I do not however consider this a critical weakness. That being said, it is perhaps useful to train an actual (lets say SCoNe) network and observe the drift in Rayleigh quotients, and see to what extent the simplified setting serves as a useful approximation.

Questions for authors

In general, I like how this paper has been written and presented. I found the theory interesting and, as far as I could tell, technically correct. I have raised my concerns regarding its practical applicability in the weaknesses section, and I believe addressing these would considerably strengthen the paper. In particular, to what extent do these asymptotic regime insights characterize the behaviour of trained representations in practical settings with fewer layers, and whether similar behavior is observed in existing SNN architectures which do not adhere to the simplified setting.

2 comments

We thank the reviewer for their thoughtful review and suggestions. We appreciate their positive assessment of the presentation and theory, and the observation of the distinct gradient, curl, and harmonic regimes being an intuitive way to look at oversmoothing/sharpening in higher-order networks. Below, we address their concerns about finite-depth applicability, optimization at larger depths, existing SNN architectures, and the practical evidence from the ocean-drifter experiments.

W1 / Q1(a), Implications for finite depth

We address the implications for finite depth in a couple of ways. First, see the finite-depth guarantee presented in our response to Reviewer rBTs (W2(a)/Q1), now included in Section 3.2 with its derivation in Appendix H. This bounds the deviation of either Hodge Rayleigh quotient from its predicted endpoint at any specified depth, under the assumptions of our theory.

W1(a), Optimization at larger depths

Second, to further address finite depth questions, we also now include a comparison of spectral predictions across training depths in Appendix J.2.1, Figure 6. The predicted limiting Rayleigh quotients remain largely consistent across the tested depths, supporting the stability of the regime predictions when models are optimized at larger depths.

W1(b), When separation happens

Regarding the reviewer's observation on the "slowness" of Hodge separation, we would like to clarify that convergence speed depends on the learned weights, the Hodge spectrum and the input, so training can lead to either fast or slow concentration. Faster convergence is not necessarily preferable. For example, at finite depth, slower concentration can retain substantial contributions from several Hodge subspaces, which may be useful when the task depends on multiple components.

W2 / Q1(b), Existing SNN architecture experiments

In order to broaden our evaluation on the ocean drifters dataset, we now include experiments with SCoNe (Roddenberry et al., 2021), order-2 SCNN (Yang et al., 2022a), and order-1 SCCNN (Yang & Isufi, 2023), using tanh\mathrm{tanh} activations and our normalized operators (Appendix J.2.1, Table 2). We report next-vertex accuracy and track both Rayleigh quotients across their three trained layers. All three architectures show increasing lower Rayleigh quotients while substantial upper Rayleigh quotients remain, showing spectral redistribution at practical depths. We also evaluate tanh\mathrm{tanh} variants of our flows (Figure 5) and see some qualitative concentration patterns persist. We focus in particular on tanh\mathrm{tanh} because odd nonlinearities preserve the orientation equivariance required of edge-feature SNNs, and its local linearity provides a natural setting in which to test how informative the linear analysis remains.

Table A: Ocean drifters

ModelTest accuracyPredicted regime (checkpoints)RQ at trained depthRQ at depth 1,024Limit
Adjacency flow, linear0.365 ± 0.110Harmonic (30)(0.036, 0.337)(0.002, 0.001)(0, 0)
Laplacian flow, linear0.242 ± 0.071Gradient (26)(0.183, 0.423)(0.748, 0.045)(0.789, 0)
Curl (4)(0.177, 0.479)(0.000, 0.997)(0, 1.000)
SCoNe, tanh0.298 ± 0.100—(0.260, 0.359)——
SCNN (order 2), tanh0.327 ± 0.075—(0.287, 0.451)——
SCCNN (order 1), tanh0.343 ± 0.089—(0.296, 0.393)——

W1(c) / W1(d), Ocean-drifter accuracy and harmonic information

We now report next-vertex prediction accuracy on ocean drifters in Appendix J.2.1, Table 2 in the manuscript and Table A above. At training depth 1616, the adjacency-like flow achieves 0.365±0.1100.365\pm0.110, compared with 0.242±0.0710.242\pm0.071 for the pure-Laplacian flow, reporting the mean and standard deviation across ten splits, each averaged over three seeds. All 30 adjacency-like models select harmonic dominance, whereas the pure-Laplacian models select gradient dominance in 26 cases and curl dominance in four. This association between harmonic selection and higher mean prediction accuracy provides suggestive evidence that harmonic information may be useful on this topology.

Closing

We thank the reviewer again for their helpful feedback. We would be glad to clarify any concerns or questions that remain after these responses.

Dear Reviewer GQFc,

As the discussion period ends on November 18 (AoE), we would be grateful to hear whether the new finite-depth bound (W1 / Q1(a)), the ocean-drifter results across training depths and the reported next-vertex prediction accuracy (W1), and the finite-depth experiments with SCoNe, SCNN, and SCCNN (W2 / Q1(b)) address your concerns about practical applicability. Following discussion with Reviewer YHy1, we have also extended the finite-depth bound (Section 3.2, Proposition 3.7) to elementwise nonlinearities such as tanh⁡\tanh under certain conditions, and noted that it holds at every simplicial rank (Remark 3.8). This brings the theory closer to the nonlinear architectures in W2. We’re happy to clarify any remaining points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.