acceptodds
Under review as a conference paper at ICLR 2027

Geometric Supercharger: Data-Informed Hyperbolic Initialization of MLP-based Sparse Neural Networks

Abstract

Sparse neural networks can substantially reduce computational and memory costs, yet their performance remains highly sensitive to the choice of sparse topology, particularly at extreme sparsity, where they struggle to outperform fully connected models. Existing sparse initialization methods are predominantly structure-driven and rarely conditioned on the data, motivating a central question: how can sparse connectivity be organized in a principled way that reflects the structure of the data rather than relying on predefined or heuristic topologies? Network science has shown that complex network topology can be characterized by an underlying latent geometry, helping to explain processes such as network routing, navigation, and dismantling. This suggests that, rather than imposing a predefined sparse topology, one can extract the latent geometry of the data and use it to organize network connectivity. Motivated by this connection, we introduce **Geometric Supercharger (GS)**, a data-informed framework for hyperbolic initialization of MLP-based sparse neural networks. GS extracts pairwise feature relationships using Pearson correlation and constructs a macro-scale geometric embedding through shortest-path-based pre-weighting and dimensionality reduction. The resulting geometry is then reused to reweight the network and refine the embedding, thereby *supercharging* it. This supercharged embedding is followed by meso-scale refinement based on community detection and angular reordering. The final angular coordinates encode data-informed node similarity and are used to generate a sparse hyperbolic topology. Building on previous findings that arbitrary, data-agnostic hyperbolic hierarchies can hinder sparse neural network training, we show that making the hierarchy data-informed turns the same geometric prior into a performance advantage. We evaluate Geometric Supercharger across four diverse tasks: image classification, physics-informed neural networks, reinforcement learning, and stock-market prediction. GS achieves state-of-the-art performance among all evaluated static sparse initialization methods in the ultra-sparse 99% regime and outperforms fully connected networks in the majority of experiments at 90% and 99% sparsity. These findings establish data-informed latent geometry as an important organizing principle for sparse MLP initialization and motivate a broader class of geometry-driven sparse neural networks.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
sparse neural networks, sparse training, sparse initialization, network geometry, geometry-informed learning, network science, complex systems

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_yn4wOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

GS uses feature correlations and repeated geometric embedding to initialize sparse MLP connectivity. Experiments cover image classification, physics-informed learning, stock prediction, and reinforcement learning.

Critical strengths

Table 1 reports consistent, substantial gains over several sparse baselines across three image datasets. Component ablations help assess the proposed pipeline.

Critical weaknesses

The method is insufficiently specified: correlation-to-distance transformations and disconnected-graph handling are unclear. The “supercharger” analogy does not substantiate the claimed suppression of spurious links. Outside image classification, comparisons are only against dense networks, so the results do not isolate the benefit of GS from sparsity itself.

Questions for authors
  1. What are the exact distance formulas and disconnected-graph rules? Are angular distances converted into affinities before Louvain?
  2. Which methods received hyperparameter searches? Page 7 says only the dense baseline.
  3. How does GS compare with random sparse initialization and CSTI on the non-classification tasks?
4 comments
AuthorsOpenReview ↗

Answer to Reviewer yn4w 1

We thank the reviewer for recognizing the “consistent, substantial gains” reported across the three image-classification datasets and the value of the component ablations.

1. The method is insufficiently specified: correlation-to-distance transformations and disconnected-graph handling are unclear. What are the exact distance formulas and disconnected-graph rules? Are angular distances converted into affinities before Louvain?

We agree that the distance transformations and disconnected-graph handling could be specified more explicitly. We have expanded Section 3 of the main manuscript to clarify each step of the embedding pipeline, including the exact distance and affinity transformations, connectivity repair, and the use of angular affinities in meso-scale community detection.

The GS embedding pipeline proceeds as follows. First, we compute pairwise Pearson correlations between input features and retain the strongest absolute correlations at the target density, defining edge strengths sij=∣rij∣s_{ij}=|r_{ij}|. Before embedding, disconnected components are joined using C−1C-1 bridge edges, connecting the first node of each consecutive component. Each bridge receives the mean positive edge weight of the original graph. The resulting connected graph is then converted into a distance-weighted network using ℓij=smin⁡+smax⁡−sij\ell_{ij}=s_{\min}+s_{\max}-s_{ij}, where smin⁡s_{\min} and smax⁡s_{\max} are the minimum and maximum positive edge weights after connectivity repair. Thus, stronger correlations correspond to shorter distances. At the macro scale, shortest-path Pearson pre-weighting (SPP) first computes the weighted all-pairs shortest-path matrix PP. Each node is represented by its shortest-path profile Pi,:P_{i,:}, and structural similarity is captured using the Pearson distance Kij=1−corr⁡(Pi,:,Pj,:)K_{ij}=1-\operatorname{corr}(P_{i,:},P_{j,:}). Zero off-diagonal distances are replaced by the smallest strictly positive kernel value. These distances are assigned only to existing edges The resulting distances are embedded through ncMCE [1] (ref), which computes curvilinear distances along a minimum spanning tree and converts the two-dimensional embedding into angular coordinates. These angular coordinates are subsequently used to reweight the existing network using the circular angular distance between all connected pairs of nodes, followed by a second SPP–ncMCE pass to obtain the supercharged embedding.

At the meso scale, circular angular distances are computed as Δθij=π−∣π−∣θi−θj∣∣\Delta\theta_{ij}=\pi-|\pi-|\theta_i-\theta_j|| and explicitly converted into affinities aij=1/(1+Δθij)a_{ij}=1/(1+\Delta\theta_{ij}). These affinities are assigned only to existing edges and used as weights in Louvain community detection. The detected communities are then organized using CLOVE's travelling-salesman-problem-based ordering, placing strongly related communities closer in angular space. The resulting refined angular coordinates are passed to nBPSO to generate the final hierarchical sparse bipartite topology.

[1] Cannistraci, C. V., Alanis-Lobato, G., & Ravasi, T. (2013). Minimum curvilinearity to enhance topological prediction of protein interactions by network embedding. Bioinformatics, 29(13), i199-i209.

2. The “supercharger” analogy does not substantiate the claimed suppression of spurious links.

We agree that the mechanical analogy does not itself substantiate the claim about spurious links. We have made the manuscript wording specific to MST spurious links and added the following ablations, Appendix subsection, and visualization to better explain our use of spurious links, which we have reworded to MST spurious links. In summary, we show that adding the MST links present in the first embedding but not in the second (supercharged) embedding hurts the performance, despite the inference sparsity now being higher. This is explained more in depth below.

Effect of the second embedding on MST links. The first SPP–ncMCE embedding constructs an MST using SPP-weighted relationships; the second pass instead reweights the feature graph using the angular distances from the first embedding and consequently produces a different, which we call ‘supercharged’, MST and ncMCE embedding. We refer to edges retained by the first MST but excluded by the second as candidate MST spurious links. We tested the effect on performance of adding the MST spurious links coming from the first MST on top of the links present in the (second) 'supercharger' MST. The results are discussed below.

AuthorsOpenReview ↗

Answer to Reviewer yn4w 2

On CIFAR-10, removing the second embedding reduces accuracy from 65.39 ± 0.22 (GS) to 64.78 ± 0.20. We then compare the refined GS MST (3,071 links in the representative visualization) with an augmented graph that restores 1,475 first-pass-only MST links (4,546 links total). These configurations correspond to 3.3% (GS MST) and 4.8% (3.3% GS MST + 1.5% MST spurious links) of the candidate feature-graph edges respectively and can be visualized in Figure 5 (https://anonymous.4open.science/r/gs_rebuttal-D2F2/fig_5.png). Restoring these links yields 65.07 ± 0.27, below GS; with this difference being statistically significant. The visualization shows that many restored links do not follow the refined angular organization. Thus, retaining the extra first-pass links does not improve performance, and reduces it, even though the network used by ncMCE is actually more dense. This supports the value of the second-pass refinement and the claim that the second pass removes those MST spurious links.

CIFAR-10Inference Sparsity
GS - w/o supercharger64.78 ± 0.203.3%
GS - w/ MST spurious links65.07 ± 0.274.8%
GS65.39 ± 0.223.3%

N: number of nodes, E: number of edges in the sparsified pearson distance network, MST: Minimum Spanning Tree (for which the number of edges is N-1), S: MST spurious links (links present in the first embedding MST but not in the second embedding MST). The asterisk (*) indicates a statistically significant difference between GS (6 seeds) and GS+spurious (6 seeds), according to a one-sided permutation test (p<0.05p < 0.05).

3. Which methods received hyperparameter searches? Page 7 says only the dense baseline.

Indeed, as stated in Table 1 in the manuscript a batch size and learning rate search was conducted only on the dense baseline. We have reworded the paragraph on page 18 to make this more clear and added the paragraph below in the Appendix to justify this choice. Additionally, we also run some extra tests on a few selected sparse baselines to confirm that our choice of parameters was correct and fair to evaluate all sparse baselines.

All sparse initialization methods use the same training hyperparameters: a learning rate of 0.025 and a batch size of 32. Only the fully connected (FC) baseline received a separate search, over learning rates 0.025,0.01,0.001{0.025, 0.01, 0.001} and batch sizes 32,64,128{32, 64, 128}. This was done because optimization settings can vary substantially with sparsity, as discussed by Dey et al. (2024) [1]. To assess whether the common sparse settings disadvantaged other initializers, we subsequently evaluated the same combinations of learning rates and batch sizes for GS, CSTI, and Uniform Random initialization. For all three methods, the originally selected pair (learning rate 0.025, batch size 32) gave the best result in these additional checks, and thus the main results do not change for these sparse methods.

[1] Dey, N., Bergsma, S., & Hestness, J. (2024). Sparse maximal update parameterization: A holistic approach to sparse training dynamics. Advances in Neural Information Processing Systems, 37, 33836-33862.

4. Outside image classification, comparisons are only against dense networks, so the results do not isolate the benefit of GS from sparsity itself. How does GS compare with random sparse initialization and CSTI on the non-classification tasks?

We agree that the original comparisons could benefit from adding other sparse baselines. We have added Uniform Random and CSTI initializations at matched architectures and sparsities to the stock prediction, physics-informed neural networks, and reinforcement-learning results in the main manuscript. The new results paragraph and tables read as follows, with the advantage of GS over sparse and dense baselines still significant.

Stock prediction. Across the 17 securities, GS has the lowest mean test error on 10, compared with three for the fully connected model, two for CSTI, and two for uniform random initialization. These results show that GS provides benefits beyond using sparse connectivity alone on the majority of the tested stocks.

AuthorsOpenReview ↗

Answer to Reviewer yn4w 3

StockFCGSCSTIUniform Random
AA2.28 ± 0.141.57 ± 0.22*3.58 ± 0.252.08 ± 0.81*
ARNC2.33 ± 0.082.03 ± 0.44*2.97 ± 0.082.78 ± 1.09
BA0.27 ± 0.030.26 ± 0.05*0.33 ± 0.030.46 ± 0.09
CNP0.56 ± 0.020.79 ± 0.221.37 ± 0.351.03 ± 0.47
DD0.66 ± 0.000.60 ± 0.09*0.91 ± 0.180.86 ± 0.33
DIS1.27 ± 0.140.37 ± 0.21*0.94 ± 0.29*0.27 ± 0.08*
DXC0.41 ± 0.010.32 ± 0.05*0.55 ± 0.060.60 ± 0.26
F1.80 ± 0.091.62 ± 0.44*3.70 ± 0.922.13 ± 1.16
GE3.81 ± 0.951.56 ± 0.67*3.12 ± 0.96*4.90 ± 3.21
HON0.34 ± 0.010.18 ± 0.02*0.39 ± 0.070.28 ± 0.09*
HPQ2.59 ± 0.402.84 ± 1.864.65 ± 2.011.69 ± 0.68*
IP1.36 ± 0.130.65 ± 0.17*2.07 ± 1.291.82 ± 1.41
JNJ2.15 ± 0.300.46 ± 0.14*1.69 ± 0.63*0.76 ± 0.27*
MCD0.40 ± 0.080.41 ± 0.130.32 ± 0.03*0.44 ± 0.08
MRO1.20 ± 0.141.71 ± 0.665.85 ± 0.031.98 ± 1.51
XLP3.25 ± 0.125.75 ± 0.163.59 ± 0.275.56 ± 1.21
YRCW1.20 ± 0.082.29 ± 0.661.02 ± 0.02*1.48 ± 0.30

Physics-informed neural networks. GS achieves the lowest mean error on both the Helmholtz and inverse Brinkman tasks after adding the matched sparse initializers.

HelmholtzInverse Brinkman
FC0.622 ± 0.0361.190 ± 0.369
Uniform Random0.536 ± 0.175*3.816 ± 0.766
CSTI0.169 ± 0.141*0.313 ± 0.120*
GS0.020 ± 0.012*0.284 ± 0.189*

Reinforcement learning. We add Uniform Random and CSTI comparisons at hidden widths h = 64 and h = 128, and an additional width h = 256. GS has the highest mean return across all three reported widths.

We further clarify that, throughout the paper, GS uses the default temperature (T=0), without temperature tuning). The exception is the reinforcement-learning experiments, where (T=0) sometimes resulted in lower performance than uniform random initialization. We therefore increased the temperature to (T=0.6) to soften the dependence of the initialized topology on the specific data samples used to construct it, thereby reducing potential overfitting to these samples. See Figure 7 (https://anonymous.4open.science/r/gs_rebuttal-D2F2/fig_7.png) for the plotted performance curves with temperature. This adjustment improved performance in the reinforcement-learning setting and illustrates that temperature can serve as an additional tunable parameter, although we did not systematically optimize it in our other experiments.

h=64h=128h=256
FC106.3 ± 32.9140.1 ± 43.3262.6 ± 64.7
Uniform Random252.5 ± 68.0*330.1 ± 55.7*232.2 ± 57.5
CSTI159.3 ± 52.5*128.1 ± 18.3184.6 ± 50.6
GS (T=0)192.4 ±52.1*262.3 ±50.1*227.3 ± 50.2
GS (T=0.6)307.9± 59.6*330.5 ±60.5*353.9 ± 56.9*

We hope our responses and additional experimental results have addressed the reviewer's questions and concerns. We remain happy to provide further clarification if needed.

AuthorsOpenReview ↗

Follow up to our response

We would like to kindly follow up on our previous response and additional experiments. We hope that our clarifications have addressed your concerns.

If there are any remaining questions or concerns, we would be happy to clarify them during the discussion period.

Thank you again for your time and constructive feedback.

Reviewer_cb17OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Geometric Supercharger (GS), a method that uses relationships in the training data to decide which connections a sparse neural network should have. It first constructs a graph in which nodes represent input features, such as image pixels, and edges connect strongly correlated features. It then uses paths through this graph to build a geometric layout that captures how features relate to one another. This layout is used to reweight the graph and compute an improved embedding. Community detection further organizes related features into groups, and a hyperbolic graph model uses the resulting layout to generate sparse connections between neural-network layers. The network is then trained normally: the connection pattern remains fixed, while the retained weights are updated. Across several tasks, the authors report that networks with 90−9990-99\\% sparsity in selected layers can outperform fully connected MLPs. The idea is that a sparse network can learn more effectively when its wiring reflects the structure of the training data.

Critical strengths
  1. The method uses relationships in the training data to organize the network’s wiring. It considers both the overall structure of the feature graph and groups of related features, going beyond selecting individual connections from pairwise correlations. The wiring is computed once before training, so the resulting network can use ordinary weight optimization with a fixed sparse structure.

  2. At 9999\\% sparsity in the targeted layers, the method outperforms all evaluated sparse initialization methods and the dense MLP baseline on three image-classification datasets. For example, on CIFAR-100, it achieves 38.1838.18\\% accuracy, compared with 34.6234.62\\% for the strongest competing sparse method and 30.6830.68\\% for the dense MLP. Experiments on physics-informed learning, stock prediction, and reinforcement learning provide additional evidence that the approach can work across different tasks.

Critical weaknesses
  1. The experiments do not clearly separate the benefit of hierarchy from the benefit of connecting related features. On images, feature correlations may recover spatial neighborhoods, so part of the method's improvement could come from giving neurons local receptive fields. This is a plausible explanation that the authors do not directly test. The temperature experiment changes how strongly connections follow the geometry; it does not isolate hierarchy. Likewise, random rewiring disrupts both the degree distribution and connections between related features. The radial hierarchy comes from the network generator, while the data determines the angular similarity coordinates. Moreover, the weakest hierarchy tested gives the best results. Although locality-based baselines are included, a more decisive control would preserve Geometric Supercharger's feature arrangement while giving neurons equal degrees. Comparing Geometric Supercharger with a network that connects neurons to nearby pixels, using the same number of connections, would help show whether the method improves on simple local wiring.

  2. The non-image experiments compare GS only with dense networks, so the improvements could come from sparsity itself rather than the proposed wiring. Could the authors add two sparse baselines with the same architecture and number of connections: one that chooses connections randomly, and another that selects connections directly from strong feature correlations (such as CSTI -- the paper uses it as a baseline for image classification, and extending that comparison to the other tasks seems appropriate)? The random baseline would test whether sparsity alone produces similar improvements, while the correlation-based baseline would test whether a simpler way of using the data works equally well without GS's geometric processing.

Questions for authors

Please refer to weakness section

4 comments
AuthorsOpenReview ↗

Answer to Reviewer cb17 1

We thank the reviewer for recognizing that GS incorporates both global feature relationships and community organization, beyond selecting individual correlated connections, resulting in good results under extreme sparsity.

1. The experiments do not clearly separate the benefit of hierarchy from the benefit of connecting related features.[...] Although locality-based baselines are included, a more decisive control would preserve Geometric Supercharger's feature arrangement while giving neurons equal degrees.

We agree that the existing ablations and experiments could isolate the radial hierarchy even more explicitly from the angular arrangement of related features. We have added the following angular-only control and the resulting ablation table to the Appendix.

We construct an angular-only GS control that keeps the same data-informed angular coordinates but removes radial coordinates and radial growth from edge selection. Connections are selected using angular proximity alone, at the same total number of edges as the full GS topology. On CIFAR-10, angular-only GS reaches 63.42 ± 0.10, whereas full GS reaches 65.39 ± 0.22, a difference of 1.97 percentage points. Thus, the combination of data-informed angular similarity and radial hierarchical generation improves performance relative to using similarity alone.

CIFAR-10
GS - angular63.42 ± 0.10
GS65.39 ± 0.22

Below is an additional paragraph added to the Appendix detailing exactly how we implemented the angular-only GS method:

In the full GS model, connectivity is determined by the hyperbolic distance between nodes, incorporating both radial coordinates (hierarchy) and angular coordinates (similarity):

dij=arccosh⁡(cosh⁡(ri)cosh⁡(rj)−sinh⁡(ri)sinh⁡(rj)cos⁡(Δθij)),d_{ij} = \operatorname{arccosh}\left(\cosh(r_i)\cosh(r_j)-\sinh(r_i)\sinh(r_j)\cos(\Delta\theta_{ij})\right),

where Δθij=π−∣π−∣θi−θj∣∣\Delta\theta_{ij}=\pi-|\pi-|\theta_i-\theta_j|| is the circular angular separation. The connection probability is defined as

p(i,j)=11+exp⁡(dij−Ri2T),0<T<1.p(i,j)=\frac{1}{1+\exp\left(\frac{d_{ij}-R_i}{2T}\right)}, \qquad 0<T<1. Here, RiR_i is the current radius of the hyperbolic disk governing the target node set during the nBPSO growth process, serving as the distance threshold for connection formation.

In the angular-only control, we replace the hyperbolic distance with the normalized angular distance:

dijang=Δθijπd_{ij}^{\text{ang}} = \frac{\Delta \theta_{ij}}{\pi}

The connection probability retains the same functional form, replacing dijd_{ij} with dijangd_{ij}^{\text{ang}} and RiR_i with RABangR_{AB}^{\text{ang}}. Unlike the growth-dependent threshold RiR_i, RABangR_{AB}^{\text{ang}} is defined as the KABK_{AB}-th smallest angular distance among all candidate neuron pairs between layers AA and BB, where KABK_{AB} is the target number of edges. This adapts the distance threshold to the angular-only setting while preserving the target edge count.

The edge-selection framework remains unchanged. At T=0T=0, both methods select the closest neuron pairs according to their respective distances. Thus, this control removes radial coordinates, radial growth, and their associated hierarchical organization from edge selection while preserving data-informed angular similarities, allowing us to evaluate the contribution of radial hierarchy to performance.

2. The non-image experiments compare GS only with dense networks, so the improvements could come from sparsity itself rather than the proposed wiring. Could the authors add two sparse baselines with the same architecture and number of connections [...] The random baseline would test whether sparsity alone produces similar improvements, while the correlation-based baseline would test whether a simpler way of using the data works equally well without GS's geometric processing.

We agree that the original comparisons could benefit from adding other sparse baselines. We have added Uniform Random and CSTI initializations at matched architectures and sparsities to the stock prediction, physics-informed neural networks, and reinforcement-learning results in the main manuscript. The new results paragraph and tables read as follows, with the advantage of GS over sparse and dense baselines is still significant.

AuthorsOpenReview ↗

Answer to Reviewer cb17 2

  • Stock prediction. Across the 17 securities, GS has the lowest mean test error on 10, compared with three for the fully connected model, two for CSTI, and two for uniform random initialization. These results show that GS provides benefits beyond using sparse connectivity alone on the majority of the tested stocks.
StockFCGSCSTIUniform Random
AA2.28 ± 0.141.57 ± 0.22*3.58 ± 0.252.08 ± 0.81*
ARNC2.33 ± 0.082.03 ± 0.44*2.97 ± 0.082.78 ± 1.09
BA0.27 ± 0.030.26 ± 0.05*0.33 ± 0.030.46 ± 0.09
CNP0.56 ± 0.020.79 ± 0.221.37 ± 0.351.03 ± 0.47
DD0.66 ± 0.000.60 ± 0.09*0.91 ± 0.180.86 ± 0.33
DIS1.27 ± 0.140.37 ± 0.21*0.94 ± 0.29*0.27 ± 0.08*
DXC0.41 ± 0.010.32 ± 0.05*0.55 ± 0.060.60 ± 0.26
F1.80 ± 0.091.62 ± 0.44*3.70 ± 0.922.13 ± 1.16
GE3.81 ± 0.951.56 ± 0.67*3.12 ± 0.96*4.90 ± 3.21
HON0.34 ± 0.010.18 ± 0.02*0.39 ± 0.070.28 ± 0.09*
HPQ2.59 ± 0.402.84 ± 1.864.65 ± 2.011.69 ± 0.68*
IP1.36 ± 0.130.65 ± 0.17*2.07 ± 1.291.82 ± 1.41
JNJ2.15 ± 0.300.46 ± 0.14*1.69 ± 0.63*0.76 ± 0.27*
MCD0.40 ± 0.080.41 ± 0.130.32 ± 0.03*0.44 ± 0.08
MRO1.20 ± 0.141.71 ± 0.665.85 ± 0.031.98 ± 1.51
XLP3.25 ± 0.125.75 ± 0.163.59 ± 0.275.56 ± 1.21
YRCW1.20 ± 0.082.29 ± 0.661.02 ± 0.02*1.48 ± 0.30
  • Physics-informed neural networks. GS achieves the lowest mean error on both the Helmholtz and inverse Brinkman tasks after adding the matched sparse initializers.
HelmholtzInverse Brinkman
FC0.622 ± 0.0361.190 ± 0.369
Uniform Random0.536 ± 0.175*3.816 ± 0.766
CSTI0.169 ± 0.141*0.313 ± 0.120*
GS0.020 ± 0.012*0.284 ± 0.189*
  • Reinforcement learning. We add Uniform Random and CSTI comparisons at hidden widths h = 64 and h = 128, and an additional width h = 256. GS has the highest mean return across all three reported widths.

We further clarify that, throughout the paper, GS uses the default temperature (T=0), without temperature tuning). The exception is the reinforcement-learning experiments, where (T=0) sometimes resulted in lower performance than uniform random initialization. We therefore increased the temperature to (T=0.6) to soften the dependence of the initialized topology on the specific data samples used to construct it, thereby reducing potential overfitting to these samples. See Figure 7 (https://anonymous.4open.science/r/gs_rebuttal-D2F2/fig_7.png) for the plotted performance curves with temperature. This adjustment improved performance in the reinforcement-learning setting and illustrates that temperature can serve as an additional tunable parameter, although we did not systematically optimize it in our other experiments.

h=64h=128h=256
FC106.3 ± 32.9140.1 ± 43.3262.6 ± 64.7
Uniform Random252.5 ± 68.0*330.1 ± 55.7*232.2 ± 57.5
CSTI159.3 ± 52.5*128.1 ± 18.3184.6 ± 50.6
GS (T=0)192.4 ±52.1*262.3 ±50.1*227.3 ± 50.2
GS (T=0.6)307.9± 59.6*330.5 ±60.5*353.9 ± 56.9*

We hope our responses and additional experimental results have addressed the reviewer's questions and concerns. We remain happy to provide further clarification if needed.

AuthorsOpenReview ↗

Follow up to our response

We would like to kindly follow up on our previous response and additional experiments. We hope that our clarifications have addressed your concerns.

If there are any remaining questions or concerns, we would be happy to clarify them during the discussion period.

Thank you again for your time and constructive feedback.

Dear authors, I appreciate your effort in clearing up my concerns. I have accordingly raised my score.

Reviewer_UFewOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Geometric Supercharger, a method for initializing sparse neural networks using a geometric representation inferred from the input data. The authors construct a feature network using pairwise correlations, recover its geometric organization through shortest-path-based embedding and community refinement, and use the resulting coordinates to generate sparse connectivity through a hyperbolic network model.

The central claim is that sparse connectivity should reflect the geometry of the data rather than being determined by a predefined topology. The authors evaluate this idea on image classification, physics-informed neural networks, stock prediction, and reinforcement learning, reporting improvements over several sparse initialization methods and fully connected baselines.

Critical strengths

The main strength of the paper is the idea that the connectivity of a neural network can be informed by the structure of the data before training. I find this direction interesting because it raises a more fundamental question about the relationship between the organization of a neural architecture and the geometry of the functions it is intended to learn.

Another strength is the observation that imposing a geometric structure is not necessarily beneficial unless that structure reflects relationships present in the data. The experiments provide evidence supporting this distinction, and the evaluation across different learning problems suggests that the idea may have applications beyond image classification.

Critical weaknesses

My main concern is that the paper does not sufficiently establish why hyperbolic geometry is the appropriate organizing principle for the proposed initialization. The method combines feature correlations, shortest-path distances, dimensionality reduction, community detection, and a hyperbolic graph generator. While the experiments show that their combination can be effective, it remains unclear which geometric properties are actually responsible for the improvements. In particular, is negative curvature essential, or would a sufficiently structured Euclidean or nongeometric initialization provide similar benefits? I think this distinction is fundamental to the central claim of the paper.

A second concern is the relationship between the inferred geometry of the input features and the organization of hidden neurons. The geometry is initially constructed from correlations between observed features, but the same geometric information is then used to organize connectivity across multiple neural layers. It is not clear why relationships between input coordinates should remain meaningful for hidden representations that are learned during training. This also raises questions about the generality of the construction when the dimensions of successive layers differ.

The computational motivation is also not fully established. Although the resulting network is sparse, the geometric initialization requires relatively expensive global computations. The paper would benefit from a clearer distinction between parameter sparsity and actual computational efficiency, particularly when considering larger architectures.

Questions for authors

What properties of the data justify the use of hyperbolic geometry rather than another geometric space? Is there evidence that the inferred feature network has an intrinsically hyperbolic organization, or is hyperbolic geometry primarily serving as a convenient mechanism for generating hierarchical connectivity?

How should we understand the relationship between the geometry of the input features and the geometry of hidden representations? In particular, why should the same geometric organization remain appropriate across layers whose representations change during training?

Is the improvement primarily a consequence of aligning connectivity with the data, or does the hierarchical degree distribution play an independent role? Could the authors disentangle these effects using comparisons that preserve degree distributions while modifying geometric relationships?

Since the geometry is inferred from input correlations without using the target labels, to what extent does it capture the structure relevant to the learning task rather than simply the statistical organization of the observations?

How does the total computational cost compare with alternative sparse training approaches when the initialization procedure is included? Does the advantage remain meaningful as the number of features increases?

7 comments
AuthorsOpenReview ↗

Answer to Reviewer UFew 1

We thank the reviewer for highlighting the value of informing neural-network connectivity with data, for recognizing that a geometric structure is useful only when it reflects relationships present in the data, and for the application of our Geometric Supercharger to different tasks.

1. My main concern is that the paper does not sufficiently establish why hyperbolic geometry is the appropriate organizing principle for the proposed initialization. [...] In particular, is negative curvature essential, or would a sufficiently structured Euclidean or nongeometric initialization provide similar benefits? [...] Is there evidence that the inferred feature network has an intrinsically hyperbolic organization, or is hyperbolic geometry primarily serving as a convenient mechanism for generating hierarchical connectivity?

We agree that the role of hyperbolic geometry and our choice of it requires a more clear explanation. Thus, we have added the following paragraph to the Appendix.

Hyperbolic space has negative curvature and an exponential growth of available space with radius. This makes it suitable for representing two properties frequently observed in complex networks: hierarchy, in which high-degree hubs organize connectivity and support short paths, and clustering, in which more similar nodes preferentially connect [1, 2]. GS uses nBPSO to generate hierarchical sparse connectivity, while its data-informed angular coordinates organize similarity and communities within that hierarchy.

The comparison with data-informed but non-hyperbolic initialization clarifies this distinction. CSTI reaches 61.94 ± 0.08 and SNIP also uses data information without the GS hyperbolic hierarchy, while GS reaches 65.39 ± 0.22 on CIFAR-10. Conversely, a data-agnostic nBPSO hierarchy alone reaches only 57.63 ± 0.22. As shown in Figure 6 (https://anonymous.4open.science/r/gs_rebuttal-D2F2/fig_6.png), imposing a stronger geometric constraint harms nBPSO but benefits GS when the geometry has been inferred from the data.

[1] Krioukov, D., Papadopoulos, F., Kitsak, M., Vahdat, A., & Boguná, M. (2010). Hyperbolic geometry of complex networks. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 82(3), 036106. [2] Boguna, M., Bonamassa, I., De Domenico, M., Havlin, S., Krioukov, D., & Serrano, M. Á. (2021). Network geometry. Nature Reviews Physics, 3(2), 114-135.

AuthorsOpenReview ↗

Answer to Reviewer UFew 2

2. A second concern is the relationship between the inferred geometry of the input features and the organization of hidden neurons. [...] Why should the same geometric organization remain appropriate across layers whose representations change during training?

We agree that the relationship between input-feature geometry and learned hidden representations could be justified empirically. We have added the following cross-layer ablation, topological metrics, and paragraph to the Appendix.

We compare full GS with a control that uses GS to initialize the first layer and Uniform Random connectivity for later hidden layers. Retaining the data-informed hierarchical organization throughout the network improves CIFAR-10 accuracy from 62.61 ± 0.02 to 65.39 ± 0.22. It also preserves a more heterogeneous degree distribution (fitted power-law exponent 3.22 versus 7.19) and stronger community structure (modularity 0.83 versus 0.38). Thus, the initial organization remains useful when extended across hidden layers.

CIFAR-10Power-law γModularityhub-CAP ratio
GS - first layer62.61 ± 0.027.190.380.12
GS65.39 ± 0.223.220.830.76

To examine the role of preserving this hierarchy across layers, we compute the hub–credit-assignment-path (hub-CAP) ratio [1,2]. Credit assignment paths (CAPs) represent potential information-flow paths from input to output, while the hub-CAP ratio measures the proportion of these paths that pass through the selected high-degree hidden neurons. Among all hidden neurons, we select the 80 with the highest total incoming and outgoing degree. These are the nodes which are more at the center of the hyperbolic disk in Figure 1 (https://anonymous.4open.science/r/gs_rebuttal-D2F2/fig_1.png). We adopt a cutoff of 80 nodes based on the CHT study [1], which identified 80 nodes that became clearly more central during the evolution of MLP connectivity. Unlike the evolved network in that study, however, our initialized network exhibits a less pronounced distinction between central and peripheral nodes, which is why we borrow the 80 node cutoff from that study. CAP is the total number of complete paths from input to output; hub-CAP is the number of these paths that pass through at least one of the selected hubs, with each path counted only once. The hub-CAP ratio is hub-CAP / CAP. It is 0.76 for full GS but only 0.12 when later layers are initialized randomly, showing that the full GS organization concentrates a much larger fraction of complete input-to-output paths through highly connected hidden neurons. This result is consistent with the importance of hub-connected credit-assignment paths for performance as well as the community-structured hierarchies found in networks trained with Cannistraci-Hebb Training, the current SOTA Dynamic Sparse Training method [1].

[1] Zhang, Y., Zhao, J., Wu, W., Muscoloni, A., & Cannistraci, C. V. (2024, May). Epitopological learning and cannistraci-hebb network shape intelligence brain-inspired theory for ultra-sparse advantage in deep learning. In International Conference on Learning Representations (Vol. 2024, pp. 46211-46239).

[2] Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural networks, 61, 85-117.

3. This also raises questions about the generality of the construction when the dimensions of successive layers differ.

We agree that different layer dimensions and sizes should be tested. We have added the following ablation and explanation to the Appendix.

We vary the number of hidden layers L ∈ {2, 3, 4} and hidden width relative to the input feature count d ∈ {0.5, 0.75, 1}; the original configuration is L = 3 and d = 1. GS outperforms the other tested sparse initialization methods in all nine settings and the fully connected network in eight of nine. The exception is L = 4, d = 0.5, where FC obtains 61.31 ± 0.08 and GS obtains 60.20 ± 0.31; even there, GS outperforms CSTI and uniform random initialization. This supports the applicability of GS across different MLP depths and widths.

L = 2L = 3L = 4
d = 0.5d = 0.75d = 1d = 0.5d = 0.75d = 1d = 0.5d = 0.75d = 1
FC58.76 ± 0.3159.04 ± 0.8059.65 ± 0.8161.20 ± 0.5061.42 ± 0.2760.76 ± 0.3161.31 ± 0.0861.36 ± 0.4461.55 ± 0.08
Uniform Random56.63 ± 0.3157.79 ± 0.1257.91 ± 0.4556.02 ± 0.5756.97 ± 0.2657.65 ± 0.1155.97 ± 0.6256.97 ± 0.0357.79 ± 0.15
CSTI60.38 ± 0.42*61.88 ± 0.13*63.18 ± 0.92*58.03 ± 0.6060.11 ± 0.1861.94 ± 0.08*56.37 ± 1.0358.43 ± 0.3160.60 ± 0.54
GS63.10 ± 0.47*64.71 ± 0.24*66.01 ± 0.16*61.36 ± 0.57*63.54 ± 0.51*65.39 ± 0.22*60.20 ± 0.3162.46 ± 0.25*62.97 ± 0.29*
AuthorsOpenReview ↗

Answer to Reviewer UFew 3

4. The computational motivation is also not fully established. Although the resulting network is sparse, the geometric initialization requires relatively expensive global computations. The paper would benefit from a clearer distinction between parameter sparsity and actual computational efficiency, particularly when considering larger architectures.

Indeed, the reviewer is right for highlighting the distinction between parameter sparsity and computational efficiency. The methods compared target the same 99% sparsity and use the same training protocol after initialization, with a fixed topology: they only differ in how the initial connections are selected. As a result, the only difference is in how this sparse topology is selected before training. Below, we have included an in depth analysis of the runtime and time and memory complexity of our method and how it compares to other initialization methods, as well as comparing it to the total training time.

5. How does the total computational cost compare with alternative sparse training approaches when the initialization procedure is included?

To fully answer this question we have reorganized Appendix B with the following sections, that we summarize below.

a. Runtime comparison with other methods and total training time

First, we include an analysis of our method’s runtime on a single NVIDIA RTX A6000 GPU and compare it to other tested methods and the total training time. While GS is more expensive to initialize than other methods, its estimated cost is still extremely small relative to the total training time (1.72%). This confirms that the initialization step is not the bottleneck for the training of sparse neural networks.

InitializerInitialisation Time (s), mean ± SEMInitialization / training time
GS35.488 ± 1.103≈1.72%
CSTI1.646 ± 0.174≈0.08%
BSW8.690 ± 0.098≈0.42%
BRF2.659 ± 0.047≈0.13%
SNIP1.446 ± 0.483≈0.07%
Uniform Random0.051 ± 0.021≈0.002%
Radix0.261 ± 0.197≈0.01%
DNM0.723 ± 0.018≈0.03%

b. Runtime scaling with input feature size and hidden layers’ dimensions

Second, we show how our model’s time runtime scales with input feature size and hidden layers’ dimensions (mean over three seeds). In Figure 2 (https://anonymous.4open.science/r/gs_rebuttal-D2F2/fig_2.png), we scale input features and hidden widths jointly. In Figure 3 (https://anonymous.4open.science/r/gs_rebuttal-D2F2/fig_3.png), we fix the input features at 3,072 and scale only the hidden widths. Taken together, these sweeps demonstrate that runtime scales primarily with the input feature dimension V, while hidden-layer width has only a marginal impact (~4.1 s across a 5× increase in width). However, even at the highest number of input features (3’072), our method still finishes in an acceptable time, with slightly more than half a minute needed, representing under 2% of the total training time.

  • Input feature size. In Figure 2, when the size of input feature count increases from 10% to 100% of standard CIFAR-10, mean initialization time increases from 3.36 s to 34.67 s. For this sweep, the hidden-layer widths also match the input feature count.

  • Hidden-layer width. In Figure 3, with the standard 3,072-feature input fixed, increasing each hidden layer from 614 to 3,072 neurons slightly increases mean initialization time from 31.37 s to 35.49 s.

This smaller increase indicates that, over the tested range, the size of the input features is the main part of initialization cost, whereas the size of hidden layers has a minor impact.

Reviewer_pfrpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Geometric Supercharger (GS), which is a data-informed framework for hyperbolic initialization of MLP-based sparse neural networks. GS extracts feature relationships from the training data and uses them to generate sparse network topologies. A macro-scale embedding combines shortest-path Pearson pre-weighting, ncMCE, and a second geometry-informed supercharging pass, while a meso-scale refinement incorporates community structure. The authors evaluate GS across several tasks and report promising performance. Overall, the proposed framework presents an interesting approach to data-informed sparse topology initialization. However, several aspects of the methodological motivation, the contributions of individual components, and the experimental evaluation require further clarification.

Critical strengths

The paper explores an interesting direction by incorporating data-informed geometric representations into sparse neural network initialization, rather than relying solely on predefined or random connectivity patterns. The experimental evaluation covers multiple application domains, demonstrating the potential applicability of the proposed framework beyond image classification. The reported results are promising, particularly under extreme sparsity, where GS outperforms the evaluated sparse initialization baselines and the fully connected baseline on several image-classification benchmarks.

Critical weaknesses
  1. The methodological design lacks sufficient justification. The motivation for combining the different components of GS could be explained more clearly. While Appendix A reports ablation results, a more thorough analysis of how the different components interact would strengthen the methodological justification.
  2. The attribution of performance improvements is not clearly defined. It remains unclear whether the performance improvements from the information extracted from the training data, the geometric embedding process, or the nBPSO-based topology generation mechanism. The current experiments fail to fully decouple the contributions of these factors, making it difficult to determine which specific parts of the proposed framework are primarily responsible for the observed performance gains.
  3. The assessment of scalability and computational cost is insufficient. Although the initialization process is performed only once, it still incurs additional computational and memory overhead. Since this initialization relies on computations involving a large amount of data, is its complexity dependent on the size and number of the samples? Furthermore, since the experiments focus primarily on architectures based on MLPs, the scalability and applicability of the framework to more complex architectures remain unclear.
Questions for authors
  1. Is there a theoretical basis for the effectiveness of the "macro-scale embedding"? Given that the initial embedding already captures global features, why is a second embedding step necessary?
  2. Regarding complexity analysis, the initialization process itself consumes computational resources. Furthermore, since this initialization relies on computations involving a large amount of data, is its complexity dependent on the size and number of the samples? 3.Is the proposed method transferable to more complex network architectures? 4.Since the data-informed nature of this approach involves learning extensive data correlations, which allows the network to implicitly acquire certain data features. It would strengthen the comparative analysis to more clearly distinguish between the advantages derived from data-informed initialization and those of the proposed geometric embedding and topology generation mechanisms. Specifically, to what extent can the performance gains reported in the paper be attributed to information extracted from the training data, as opposed to the specific geometric embedding or the hyperbolic network generation process? 5.Are the topological structures resulting from the initialization similar across different parameter settings? The experiments should demonstrate the extent of the differences in topological structure produced by various methods, particularly when they yield comparable performance.
8 comments
AuthorsOpenReview ↗

Answer to Reviewer pfrp 1

We thank the reviewer for recognizing the value of “incorporating data-informed geometric representations into sparse neural network initialization,” the breadth of evaluation across applications, and the promising results at high sparsity.

1. The methodological design lacks sufficient justification. The motivation for combining the different components of GS could be explained more clearly. [...] Is there a theoretical basis for the effectiveness of the "macro-scale embedding"? Given that the initial embedding already captures global features, why is a second embedding step necessary?

We agree that the motivation for combining these components and the role of the second meso-scale embedding could be further clarified. We have added the following explanation and visualization to the Appendix.

GS combines two complementary forms of hierarchical organization: degree-based hierarchy and community hierarchical organization. The radial growth mechanism of nBPSO generates heterogeneous node degrees, while the macro-scale embedding supplies data-informed angular coordinates that align this hierarchy with the relationships between input features. The meso-scale refinement then orders related communities in angular space, helping keep related feature groups within coherent branches of the hierarchical network rather than mixing unrelated groups.

The new comparison in Figure 1 (https://anonymous.4open.science/r/gs_rebuttal-D2F2/fig_1.png) illustrates these differences at the network level. Uniform Random connectivity achieves 56.65 ± 0.11, the macro-scale-only SPP–ncMCE–AD achieves 64.32 ± 0.39, and full GS (macro and meso) achieves 65.39 ± 0.22 on CIFAR-10. The difference in performance between the macro-scale only and full GS is statistically significant (using 6 seeds, using a one-sided permutation test (p<0.05p < 0.05)). The hyperbolic and representative bipartite-layer views show how the macro-scale embedding identifies degree-based hierarchy, while the meso-scale embedding refinement organizes the rest of the network in coherent communities.

In addition, in response to Reviewer UFew, we conducted a complementary ablation comparing the full GS initialization with a variant in which only the first layer is initialized using GS, while the remaining layers are initialized uniformly at random. The full GS initialization exhibits substantially higher performance as well as higher modularity, further highlighting the importance of preserving the meso-scale community organization identified by GS across the network.

2. The attribution of performance improvements is not clearly defined. [...] It would strengthen the comparative analysis to more clearly distinguish between the advantages derived from data-informed initialization and those of the proposed geometric embedding and topology generation mechanisms. Specifically, to what extent can the performance gains reported in the paper be attributed to information extracted from the training data, as opposed to the specific geometric embedding or the hyperbolic network generation process?

We agree that the separate contributions could be presented more explicitly. We have added the following paragraph and summary table to the Appendix.

The comparison below separates the contribution of hierarchy, data information, and the two embedding scales. nBPSO provides a data-agnostic hyperbolic hierarchy, but obtains only 57.63 ± 0.22 accuracy on CIFAR-10. CSTI uses data correlations without hyperbolic hierarchical generation and reaches 61.94 ± 0.08. The combination of data-informed hyperbolic hierarchy through the macro-scale embedding and network generation improves accuracy to 64.32 ± 0.39 (SPP–ncMCE–AD); the meso-scale community refinement further increases performance to 65.39 ± 0.22 (GS). Thus, neither an arbitrary hierarchy nor data correlations alone account for the full gain; it is the data-informed geometric organization of hierarchical connectivity that matters.

CIFAR-10HierarchyData-informedMacro-scale embeddingMeso-scale embedding
nBPSO57.63 ± 0.22YesNoNoNo
CSTI61.94 ± 0.08NoYesNoNo
SPP-ncMCE-AD64.32 ± 0.39YesYesYesNo
GS65.39 ± 0.22YesYesYesYes

We have also added the following stock prediction ablations to the Appendix to show the relative effects of the macro- and meso-scale embeddings on a second task, other than CIFAR-10, exhibiting the same results. We choose the first stock of our dataset for simplicity.

AA
CSTI3.58 ± 0.25
SPP-ncMCE-AD (macro)3.77 ± 1.66
GS (macro+meso)1.57 ± 0.22
AuthorsOpenReview ↗

Answer to Reviewer pfrp 2

3. The assessment of scalability and computational cost is insufficient. Although the initialization process is performed only once, it still incurs additional computational and memory overhead. [...] Furthermore, since this initialization relies on computations involving a large amount of data, is its complexity dependent on the size and number of the samples?

We agree that a one-time initialization still incurs computational and memory costs, and that its dependence on the calibration data and network architecture requires explicit discussion.

We have reorganized Appendix B with the following sections, that are summarized below.

a. Runtime comparison with other methods and total training time

First, we include an analysis of our method’s runtime on a single NVIDIA RTX A6000 GPU and compare it to other tested methods and the total training time. While GS is more expensive to initialize than other methods, its estimated cost is still extremely small relative to the total training time (1.72%). This confirms that the initialization step is not the bottleneck for the training of sparse neural networks.

InitializerInitialization Time (s), mean ± SEMInitialization / training time
GS35.488 ± 1.103≈1.72%
CSTI1.646 ± 0.174≈0.08%
BSW8.690 ± 0.098≈0.42%
BRF2.659 ± 0.047≈0.13%
SNIP1.446 ± 0.483≈0.07%
Uniform Random0.051 ± 0.021≈0.002%
Radix0.261 ± 0.197≈0.01%
DNM0.723 ± 0.018≈0.03%

b. Runtime scaling with input feature size and hidden layers’ dimensions

Second, we show how our model’s time runtime scales with input feature size and hidden layers’ dimensions (mean over three seeds). In Figure 2 (https://anonymous.4open.science/r/gs_rebuttal-D2F2/fig_2.png), we scale input features and hidden widths jointly. In Figure 3 (https://anonymous.4open.science/r/gs_rebuttal-D2F2/fig_3.png), we fix the input features at 3,072 and scale only the hidden widths. Taken together, these sweeps demonstrate that runtime scales primarily with the input feature dimension V, while hidden-layer width has only a marginal impact (~4.1 s across a 5× increase in width). However, even at the highest number of input features (3’072), our method still finishes in an acceptable time, with slightly more than half a minute needed, representing under 2% of the total training time.

  • Input feature size. In Figure 2, when the size of input feature count increases from 10% to 100% of standard CIFAR-10, mean initialization time increases from 3.36 s to 34.67 s. For this sweep, the hidden-layer widths also match the input feature count.

  • Hidden-layer width. In Figure 3, with the standard 3,072-feature input fixed, increasing each hidden layer from 614 to 3,072 neurons slightly increases mean initialization time from 31.37 s to 35.49 s.

This smaller increase indicates that, over the tested range, the size of the input features is the main part of initialization cost, whereas the size of hidden layers has a minor impact.

c. Time and memory complexity of all methods

Third, we compare the initialization time complexity and peak working-memory complexity for the methods tested. Here, (N) is the number of calibration examples, (B) the SNIP batch size. Memory includes the explicitly allocated masks and intermediate arrays; these are dense-shaped in the current implementation, despite the sparse connectivity.

InitializerTime complexityMemory complexity
GSO(NV2+V3)O(NV^2+V^3)O(NV+V2)O(NV+V^2)
CSTIO(NV2+V2log⁡V)O(NV^2+V^2\log V)O(NV+V2)O(NV+V^2)
BSWO(V2)O(V^2)O(V2)O(V^2)
BRFO(V2)O(V^2)O(V2)O(V^2)
SNIPO(BV2+V2log⁡V)O(BV^2+V^2\log V)O(V2+BV)O(V^2+BV)
Uniform RandomO(V2)O(V^2)O(V2)O(V^2)
RadiX-NetO(V2)O(V^2)O(V2)O(V^2)
DNMO(V2)O(V^2)O(V2)O(V^2)
RamanujanO(V3)O(V^3)O(V2)O(V^2)

All evaluated methods require at least O(V2)O(V^2) time and memory to materialize the dense mask tensors, and GS matches CSTI in peak memory complexity (O(NV+V2)O(NV + V^2)). Regarding time complexity, it is true that GS includes an O(V3)O(V^3) term dictated by the SPP correlation distance calculation; however, as shown in the stage-wise table in part d., in practice this step takes less than a second (0.54 s) thanks to its allocation on the GPU. Consequently, the empirical runtime of GS is governed by the O(V2)O(V^2) and O(V2log⁡V)O(V^2\log V) graph construction stages rather than the cubic term. GS requires (O(NV+V2)O(NV + V^2)) memory for the calibration data and dense embedding matrices.

AuthorsOpenReview ↗

Answer to Reviewer pfrp 3

d. Time complexity per-component breakdown of GS

Finally, we show the runtime and time complexity analysis for each component of our method. The number of calibration examples is N and the number of embedded input features is V. The table below reports a single diagnostic run for the standard 3,072-feature, 3,072-hidden-unit configuration.

StageTimeShareSimplified time complexity
Calibration loading2.02 s5.83%O(NV)O(NV)
GPU Pearson correlation1.97 s5.68%O(NV2)O(NV^2)
Correlation thresholding/sanitization0.46 s1.33%O(V2)O(V^2)
SPP all-pairs shortest paths7.74 s22.31%O(VElog⁡V)O(VE\log V)
GPU SPP correlation distances0.54 s1.55%O(V3)O(V^3)
SPP kernel preparation/repair/checks2.79 s8.03%O(V2)O(V^2)
ncMCE graph, MST and tree distances2.83 s8.14%O(V2log⁡V)O(V^2\log V)
GPU-backed rank-2 PROPACK SVD0.04 s0.12%O(V2)O(V^2)
Other CE affinities and AD2.23 s6.42%O(V2)O(V^2)
CLOVE community ordering8.01 s23.09%O(V2)O(V^2)
nPSO topology generation3.87 s11.15%O(V2)O(V^2)
Mask assembly and other2.21 s6.36%O(V2)O(V^2)
Overall34.71 s100%O(NV2+VElog⁡V+V3)O(NV^2+VE\log V+V^3)
Under the Sparse Graph Assumption (E=O(V)E=O(V)), the shortest-path complexity becomes O(V2log⁡V)O(V^2\log V), and the overall time complexity becomes O(NV2+V3)O(NV^2 + V^3). Interestingly, the SPP correlation distance is the component with the highest theoretical complexity (O(V3)O(V^3)), but requires less than a second to run thanks to the exploitation of GPU on that stage.
4. Is the proposed method transferable to more complex network architectures?
We have added the following clarification about the scope and transfer conditions of our method to the manuscript discussion.
GS is designed and evaluated for MLP-based sparse neural networks. Its essential prerequisite is a meaningful input feature space from which pairwise relationships can be extracted and mapped onto sparse weight-matrix connectivity. The principle may extend to other architectures when such a feature space and a corresponding sparse connectivity mechanism can be defined. However, the current construction is not directly transferable to Transformers without adapting the meaning of input features and the layers to which the generated topology is applied.
5. Are the topological structures resulting from the initialization similar across different parameter settings? The experiments should demonstrate the extent of the differences in topological structure produced by various methods, particularly when they yield comparable performance.
We agree that an explicit comparison of the generated topologies is useful. We have added the following analysis and interpretation to the Appendix.

The measured topological properties show topological differences among sparse initializers. GS has global efficiency 0.29, compared with 0.14 for both CSTI and SNIP; this is closer to the efficiencies of many data-agnostic initializers. Its assortativity is nearly neutral (−0.02), whereas CSTI is strongly assortative (0.59) and SNIP is disassortative (−0.41). GS and CSTI have relatively high average local clustering coefficients (0.12 and 0.11), while GS additionally has high modularity (0.83), consistent with the community organization enforced by the meso-scale refinement. These results further indicate that imposing specific topological properties alone is insufficient to achieve strong performance; the topology must also be aligned with the data. Indeed, several other initialization methods exhibit topological properties similar to those of GS but do not achieve comparable performance.

MethodGlobal EfficiencyAssortativityLoc. Clustering CoefficientPower-law γModularity
nBPSO0.32-0.050.062.930.79
Uniform Random0.33-0.400.018.920.19
BSW0.33-0.420.028.410.41
BRF0.19-0.480.122.630.88
Ramanujan0.33-0.500.012.870.17
RadiX-Nets0.22-0.500.053.590.57
dANN-R0.26-0.660.043.660.47
DNM0.31-0.140.035.370.47
SNIP0.14-0.410.033.520.47
CSTI0.140.590.113.240.73
GS0.29-0.020.123.220.83

In addition, we would like to clarify that we kept all parameters, notably batch size (32) and learning rate (0.025) the same for all sparse initialization methods. As detailed in the response to Reviewer yn4w, only the fully connected baseline was given an additional parameter search, to account for potentially different optimal parameters for a different sparsity level, as explained by Dey et al. (2024) [1].

[1] Dey, N., Bergsma, S., & Hestness, J. (2024). Sparse maximal update parameterization: A holistic approach to sparse training dynamics. Advances in Neural Information Processing Systems, 37.

We hope our responses and additional experimental results have addressed the reviewer's questions and concerns. We remain happy to provide further clarification if needed.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response for All Reviewers and Area Chair 1/2

We sincerely thank all reviewers for their detailed and constructive feedback, which has helped us strengthen the manuscript.

We are pleased that the reviewers recognized the strengths of our Geometric Supercharger (GS) framework. Reviewer pfrp highlighted the interesting direction of “incorporating data-informed geometric representations into sparse neural network initialization”, the broad experimental evaluation, and promising results at extreme sparsity. Reviewer UFew mentioned our idea to be “interesting because it raises a more fundamental question about the relationship between the organization of a neural architecture and the geometry of the functions it is intended to learn”, while recognizing our experimental results and its potential in different tasks. Reviewer cb17 recognized that GS captures both global feature relationships and community organization and performs well at high sparsity levels. Reviewer yn4w acknowledged the “consistent, substantial gains” across our experiments and the value of the component ablations.

We summarize below the experiments and clarifications provided during the rebuttal period. All referenced figures are available at the following link, and the manuscript PDF has been updated with all the mentioned changes.

https://anonymous.4open.science/r/gs_rebuttal-D2F2

Additional Experiments and Ablations

  • Sparse baselines for stock prediction, physics-informed neural networks, and reinforcement learning: Added Uniform Random and CSTI at matched sparsity, showing that GS is still the best performing method. (cb17, yn4w)

  • Performance across sparsity levels: Evaluated GS, data-agnostic Uniform Random, and data-informed SNIP at 60%–99% sparsity on CIFAR-10 and CIFAR-100, showing that GS consistently outperforms the three baselines at all tested sparsity levels (Figure 8). (pfrp)

  • Network depth and width: Evaluated nine MLP architectures with 2–4 hidden layers and three relative widths, showing that GS outperforms both sparse baselines in all nine configurations and FC in eight of nine. (UFew)

  • Macro- versus meso-scale embedding: Compared nBPSO, CSTI, SPP–ncMCE–AD, and full GS, showing a statistically significant improvement from macro-scale-only embedding to full GS, with additional stock-prediction ablations (Figure 1). (pfrp, UFew)

  • Angular-only ablation: Introduced a control preserving data-informed angular coordinates and edge count while removing radial coordinates and radial growth, confirming the improvement when radial hierarchical connectivity is included, validating the GS pipeline. (cb17, UFew, pfrp)

  • First-layer-only GS: Compared GS across all layers against GS applied only to the first layer, showing a reduction in accuracy, alongside reduced modularity, degree heterogeneity, and efficient information flow (hub-CAP ratio). (UFew, pfrp)

  • Supercharging and MST spurious links: Demonstrated that restoring first-pass-only MST links reduces accuracy, supporting the second embedding's refinement of MST spurious links (Figure 5). (yn4w, pfrp)

  • Static versus dynamic sparse training: Compared Random-static, GS-static, and GS with CHTs, showing that gains from using the information from the data are much larger than those using information from the task (Figure 4). (UFew)

  • Initialization runtime comparison: Benchmarked GS against other sparse initializers, showing that its initialization represents approximately 1.72% of total training time. (pfrp, UFew)

  • Runtime scaling: Measured initialization costs across input-feature dimensions and hidden-layer widths, showing that input-feature dimensionality dominates runtime while hidden width has a smaller impact (Figures 2 and 3). (pfrp, UFew)

  • Topological comparisons: Added topological measures for all sparse initialization methods, showing that individual topological properties alone do not explain GS's superior performance. (pfrp, UFew)

  • Hyperparameter fairness: Conducted additional learning-rate and batch-size checks for GS, CSTI, and Uniform Random, confirming that the original shared training settings were the best among the tested configurations. (yn4w)

AuthorsOpenReview ↗

General Response for All Reviewers and Area Chair 2/2

Clarifications

  • Multi-stage geometric pipeline: Added a dedicated main-text subsection, Motivation for the Multi-Stage Geometric Pipeline, explaining the complementary roles of degree-based hierarchy, data-informed macro-scale angular similarity, and meso-scale community organization. (pfrp, UFew)

  • GS pipeline clarification: Explicitly specified in more depth each step of the GS pipeline in the manuscript and the Appendix. (pfrp, cb17, yn4w)

  • Hyperbolic geometry: Explained the motivation for using hyperbolic geometry, as it models networks with clustering and hierarchy. (UFew, cb17)

  • Computational complexity: Expanded the Appendix with time and memory complexity comparisons across methods, calibration-data dependence, and a component-level runtime breakdown. (pfrp, UFew)

  • Hyperparameter selection: Clarified the unclear description of parameter searches, clarifying that all sparse methods share the same training settings while FC received a separate, more thorough, search. (yn4w)

Reviewer Discussion Outcomes

  • Reviewer pfrp: The reviewer acknowledged that we had addressed their concerns, then requested further justification for the multi-stage pipeline and performance at moderate sparsity, which we adressed.

  • Reviewer UFew: We addressed the questions concerning hyperbolic geometry, the respective roles of data alignment and hierarchy, cross-layer organization, different MLP architectures, task-informed versus data-informed structure, and computational costs.

  • Reviewer cb17: Both concerns, isolating radial hierarchy from data-informed similarity and adding matched sparse baselines beyond image classification, were addressed with new experiments, as the reviewer confirmed.

  • Reviewer yn4w: We addressed the requests for clarification about our method, notably disconnected-graph handling, angular-affinity conversion, as well as an empirical validation of the second embedding, hyperparameter-search clarification, and sparse baselines beyond classification.

We sincerely appreciate the reviewers' constructive engagement, particularly the explicit positive follow-up from cb17 and pfrp. We believe the additional experiments and manuscript revisions have strengthened the paper and we hope the Area Chair will consider the full scope of these responses and revisions in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.