- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
GS uses feature correlations and repeated geometric embedding to initialize sparse MLP connectivity. Experiments cover image classification, physics-informed learning, stock prediction, and reinforcement learning.
Critical strengths
Table 1 reports consistent, substantial gains over several sparse baselines across three image datasets. Component ablations help assess the proposed pipeline.
Critical weaknesses
The method is insufficiently specified: correlation-to-distance transformations and disconnected-graph handling are unclear. The “supercharger” analogy does not substantiate the claimed suppression of spurious links. Outside image classification, comparisons are only against dense networks, so the results do not isolate the benefit of GS from sparsity itself.
Questions for authors
- What are the exact distance formulas and disconnected-graph rules? Are angular distances converted into affinities before Louvain?
- Which methods received hyperparameter searches? Page 7 says only the dense baseline.
- How does GS compare with random sparse initialization and CSTI on the non-classification tasks?
4 comments
Answer to Reviewer yn4w 2
On CIFAR-10, removing the second embedding reduces accuracy from 65.39 ± 0.22 (GS) to 64.78 ± 0.20. We then compare the refined GS MST (3,071 links in the representative visualization) with an augmented graph that restores 1,475 first-pass-only MST links (4,546 links total). These configurations correspond to 3.3% (GS MST) and 4.8% (3.3% GS MST + 1.5% MST spurious links) of the candidate feature-graph edges respectively and can be visualized in Figure 5 (https://anonymous.4open.science/r/gs_rebuttal-D2F2/fig_5.png). Restoring these links yields 65.07 ± 0.27, below GS; with this difference being statistically significant. The visualization shows that many restored links do not follow the refined angular organization. Thus, retaining the extra first-pass links does not improve performance, and reduces it, even though the network used by ncMCE is actually more dense. This supports the value of the second-pass refinement and the claim that the second pass removes those MST spurious links.
| CIFAR-10 | Inference Sparsity | |
|---|---|---|
| GS - w/o supercharger | 64.78 ± 0.20 | 3.3% |
| GS - w/ MST spurious links | 65.07 ± 0.27 | 4.8% |
| GS | 65.39 ± 0.22 | 3.3% |
N: number of nodes, E: number of edges in the sparsified pearson distance network, MST: Minimum Spanning Tree (for which the number of edges is N-1), S: MST spurious links (links present in the first embedding MST but not in the second embedding MST). The asterisk (*) indicates a statistically significant difference between GS (6 seeds) and GS+spurious (6 seeds), according to a one-sided permutation test ().
3. Which methods received hyperparameter searches? Page 7 says only the dense baseline.
Indeed, as stated in Table 1 in the manuscript a batch size and learning rate search was conducted only on the dense baseline. We have reworded the paragraph on page 18 to make this more clear and added the paragraph below in the Appendix to justify this choice. Additionally, we also run some extra tests on a few selected sparse baselines to confirm that our choice of parameters was correct and fair to evaluate all sparse baselines.
All sparse initialization methods use the same training hyperparameters: a learning rate of 0.025 and a batch size of 32. Only the fully connected (FC) baseline received a separate search, over learning rates and batch sizes . This was done because optimization settings can vary substantially with sparsity, as discussed by Dey et al. (2024) [1]. To assess whether the common sparse settings disadvantaged other initializers, we subsequently evaluated the same combinations of learning rates and batch sizes for GS, CSTI, and Uniform Random initialization. For all three methods, the originally selected pair (learning rate 0.025, batch size 32) gave the best result in these additional checks, and thus the main results do not change for these sparse methods.
[1] Dey, N., Bergsma, S., & Hestness, J. (2024). Sparse maximal update parameterization: A holistic approach to sparse training dynamics. Advances in Neural Information Processing Systems, 37, 33836-33862.
4. Outside image classification, comparisons are only against dense networks, so the results do not isolate the benefit of GS from sparsity itself. How does GS compare with random sparse initialization and CSTI on the non-classification tasks?
We agree that the original comparisons could benefit from adding other sparse baselines. We have added Uniform Random and CSTI initializations at matched architectures and sparsities to the stock prediction, physics-informed neural networks, and reinforcement-learning results in the main manuscript. The new results paragraph and tables read as follows, with the advantage of GS over sparse and dense baselines still significant.
Stock prediction. Across the 17 securities, GS has the lowest mean test error on 10, compared with three for the fully connected model, two for CSTI, and two for uniform random initialization. These results show that GS provides benefits beyond using sparse connectivity alone on the majority of the tested stocks.
Answer to Reviewer yn4w 3
| Stock | FC | GS | CSTI | Uniform Random |
|---|---|---|---|---|
| AA | 2.28 ± 0.14 | 1.57 ± 0.22* | 3.58 ± 0.25 | 2.08 ± 0.81* |
| ARNC | 2.33 ± 0.08 | 2.03 ± 0.44* | 2.97 ± 0.08 | 2.78 ± 1.09 |
| BA | 0.27 ± 0.03 | 0.26 ± 0.05* | 0.33 ± 0.03 | 0.46 ± 0.09 |
| CNP | 0.56 ± 0.02 | 0.79 ± 0.22 | 1.37 ± 0.35 | 1.03 ± 0.47 |
| DD | 0.66 ± 0.00 | 0.60 ± 0.09* | 0.91 ± 0.18 | 0.86 ± 0.33 |
| DIS | 1.27 ± 0.14 | 0.37 ± 0.21* | 0.94 ± 0.29* | 0.27 ± 0.08* |
| DXC | 0.41 ± 0.01 | 0.32 ± 0.05* | 0.55 ± 0.06 | 0.60 ± 0.26 |
| F | 1.80 ± 0.09 | 1.62 ± 0.44* | 3.70 ± 0.92 | 2.13 ± 1.16 |
| GE | 3.81 ± 0.95 | 1.56 ± 0.67* | 3.12 ± 0.96* | 4.90 ± 3.21 |
| HON | 0.34 ± 0.01 | 0.18 ± 0.02* | 0.39 ± 0.07 | 0.28 ± 0.09* |
| HPQ | 2.59 ± 0.40 | 2.84 ± 1.86 | 4.65 ± 2.01 | 1.69 ± 0.68* |
| IP | 1.36 ± 0.13 | 0.65 ± 0.17* | 2.07 ± 1.29 | 1.82 ± 1.41 |
| JNJ | 2.15 ± 0.30 | 0.46 ± 0.14* | 1.69 ± 0.63* | 0.76 ± 0.27* |
| MCD | 0.40 ± 0.08 | 0.41 ± 0.13 | 0.32 ± 0.03* | 0.44 ± 0.08 |
| MRO | 1.20 ± 0.14 | 1.71 ± 0.66 | 5.85 ± 0.03 | 1.98 ± 1.51 |
| XLP | 3.25 ± 0.12 | 5.75 ± 0.16 | 3.59 ± 0.27 | 5.56 ± 1.21 |
| YRCW | 1.20 ± 0.08 | 2.29 ± 0.66 | 1.02 ± 0.02* | 1.48 ± 0.30 |
Physics-informed neural networks. GS achieves the lowest mean error on both the Helmholtz and inverse Brinkman tasks after adding the matched sparse initializers.
| Helmholtz | Inverse Brinkman | |
|---|---|---|
| FC | 0.622 ± 0.036 | 1.190 ± 0.369 |
| Uniform Random | 0.536 ± 0.175* | 3.816 ± 0.766 |
| CSTI | 0.169 ± 0.141* | 0.313 ± 0.120* |
| GS | 0.020 ± 0.012* | 0.284 ± 0.189* |
Reinforcement learning. We add Uniform Random and CSTI comparisons at hidden widths h = 64 and h = 128, and an additional width h = 256. GS has the highest mean return across all three reported widths.
We further clarify that, throughout the paper, GS uses the default temperature (T=0), without temperature tuning). The exception is the reinforcement-learning experiments, where (T=0) sometimes resulted in lower performance than uniform random initialization. We therefore increased the temperature to (T=0.6) to soften the dependence of the initialized topology on the specific data samples used to construct it, thereby reducing potential overfitting to these samples. See Figure 7 (https://anonymous.4open.science/r/gs_rebuttal-D2F2/fig_7.png) for the plotted performance curves with temperature. This adjustment improved performance in the reinforcement-learning setting and illustrates that temperature can serve as an additional tunable parameter, although we did not systematically optimize it in our other experiments.
| h=64 | h=128 | h=256 | |
|---|---|---|---|
| FC | 106.3 ± 32.9 | 140.1 ± 43.3 | 262.6 ± 64.7 |
| Uniform Random | 252.5 ± 68.0* | 330.1 ± 55.7* | 232.2 ± 57.5 |
| CSTI | 159.3 ± 52.5* | 128.1 ± 18.3 | 184.6 ± 50.6 |
| GS (T=0) | 192.4 ±52.1* | 262.3 ±50.1* | 227.3 ± 50.2 |
| GS (T=0.6) | 307.9± 59.6* | 330.5 ±60.5* | 353.9 ± 56.9* |
We hope our responses and additional experimental results have addressed the reviewer's questions and concerns. We remain happy to provide further clarification if needed.
Follow up to our response
We would like to kindly follow up on our previous response and additional experiments. We hope that our clarifications have addressed your concerns.
If there are any remaining questions or concerns, we would be happy to clarify them during the discussion period.
Thank you again for your time and constructive feedback.
Answer to Reviewer yn4w 1
We thank the reviewer for recognizing the “consistent, substantial gains” reported across the three image-classification datasets and the value of the component ablations.
1. The method is insufficiently specified: correlation-to-distance transformations and disconnected-graph handling are unclear. What are the exact distance formulas and disconnected-graph rules? Are angular distances converted into affinities before Louvain?
We agree that the distance transformations and disconnected-graph handling could be specified more explicitly. We have expanded Section 3 of the main manuscript to clarify each step of the embedding pipeline, including the exact distance and affinity transformations, connectivity repair, and the use of angular affinities in meso-scale community detection.
The GS embedding pipeline proceeds as follows. First, we compute pairwise Pearson correlations between input features and retain the strongest absolute correlations at the target density, defining edge strengths sij=∣rij∣. Before embedding, disconnected components are joined using C−1 bridge edges, connecting the first node of each consecutive component. Each bridge receives the mean positive edge weight of the original graph. The resulting connected graph is then converted into a distance-weighted network using ℓij=smin+smax−sij, where smin and smax are the minimum and maximum positive edge weights after connectivity repair. Thus, stronger correlations correspond to shorter distances. At the macro scale, shortest-path Pearson pre-weighting (SPP) first computes the weighted all-pairs shortest-path matrix P. Each node is represented by its shortest-path profile Pi,:, and structural similarity is captured using the Pearson distance Kij=1−corr(Pi,:,Pj,:). Zero off-diagonal distances are replaced by the smallest strictly positive kernel value. These distances are assigned only to existing edges The resulting distances are embedded through ncMCE [1] (ref), which computes curvilinear distances along a minimum spanning tree and converts the two-dimensional embedding into angular coordinates. These angular coordinates are subsequently used to reweight the existing network using the circular angular distance between all connected pairs of nodes, followed by a second SPP–ncMCE pass to obtain the supercharged embedding.
At the meso scale, circular angular distances are computed as Δθij=π−∣π−∣θi−θj∣∣ and explicitly converted into affinities aij=1/(1+Δθij). These affinities are assigned only to existing edges and used as weights in Louvain community detection. The detected communities are then organized using CLOVE's travelling-salesman-problem-based ordering, placing strongly related communities closer in angular space. The resulting refined angular coordinates are passed to nBPSO to generate the final hierarchical sparse bipartite topology.
[1] Cannistraci, C. V., Alanis-Lobato, G., & Ravasi, T. (2013). Minimum curvilinearity to enhance topological prediction of protein interactions by network embedding. Bioinformatics, 29(13), i199-i209.
2. The “supercharger” analogy does not substantiate the claimed suppression of spurious links.
We agree that the mechanical analogy does not itself substantiate the claim about spurious links. We have made the manuscript wording specific to MST spurious links and added the following ablations, Appendix subsection, and visualization to better explain our use of spurious links, which we have reworded to MST spurious links. In summary, we show that adding the MST links present in the first embedding but not in the second (supercharged) embedding hurts the performance, despite the inference sparsity now being higher. This is explained more in depth below.
Effect of the second embedding on MST links. The first SPP–ncMCE embedding constructs an MST using SPP-weighted relationships; the second pass instead reweights the feature graph using the angular distances from the first embedding and consequently produces a different, which we call ‘supercharged’, MST and ncMCE embedding. We refer to edges retained by the first MST but excluded by the second as candidate MST spurious links. We tested the effect on performance of adding the MST spurious links coming from the first MST on top of the links present in the (second) 'supercharger' MST. The results are discussed below.