- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper constructs undirected functional graphs over attention heads using similarities between question-level, mean-pooled query activations. It uses a Fisher-z sum of SVCCA correlations, retains the strongest edges at a fixed density, and measures small-world organization relative to degree-preserving randomized graphs. SWI increases alongside decreasing DRE-Bench reference-answer NLL across seven Pythia checkpoints and Qwen/Llama models. Community-based core and bridge scores are then used in Small-World Allocation (SWA), which assigns layer and head sparsity budgets for Wanda and SparseGPT. Experiments report improved WikiText perplexity, average zero-shot accuracy, and ARAOC output-shape preservation, with hierarchical allocation and graph-source ablations.
Critical strengths
-
The paper offers a clear way to summarize relationships among attention heads using activation-based functional graphs. The association between the small-world index (SWI) and DRE-Bench performance is examined both across model families and across checkpoints of the same model.
-
The authors also turn their observation into a pruning heuristic and report improvements over several sparsity-allocation baselines.
Critical weaknesses
-
Limited mechanistic insight. The paper applies a small-world metric to a graph built from attention-head activation similarities, but the resulting SWI–performance correlation does not explain how this structure supports reasoning. It could also reflect broader improvements in model quality during training.
-
Limited evidence for practical impact on reasoning. SWA improves pruning results, but the main evaluations emphasize WikiText perplexity and general zero-shot performance. The abstract-reasoning evaluation measures only output-grid shape correctness. These results provide limited evidence that the proposed principle preserves reasoning ability, and do not show that it can improve the reasoning ability of an unpruned model.
Questions for authors
-
Does SWI predict reasoning performance beyond model size, training progress, and general language-modeling quality? Is the association specific to reasoning tasks?
-
How does SWA compare with allocating sparsity directly according to measured attention-head importance? What additional benefit do the graph-based scores provide?
-
Can the authors report held-out reasoning accuracy after pruning? Perplexity and output-grid shape correctness provide limited evidence that reasoning ability is preserved.
6 comments
To examine how each community affects the model’s final hidden representation, we first examine the six largest communities identified by applying Louvain community detection to our functional graph. We perturb each community separately by multiplying the outputs of all its attention heads by 0.5, following [8, 9]. For each question, we subtract the original representation from the intervened representation at the output of the final Transformer block and normalize the difference to obtain an intervention direction:
u_C(q)=\frac{\Delta r_C(q)}{\|\Delta r_C(q)\|_2}.$$ We compare these directions across communities for the same held-out questions. The off-diagonal cosine similarities range from −0.20 to 0.28, indicating limited alignment across communities (Figure 4 in Appendix A.6 of the revised manuscript). The corresponding PCA visualization shows largely separate groups, with particularly compact groups for C206, C207, and C208 (Figure 5 in Appendix A.6 of the revised manuscript). These results indicate that the identified communities have distinguishable effects on the model’s final hidden representation. Additionally, to assess the validity of the communities, we first examine whether each cluster captures distinct semantics in the output space. Specifically, perturbing a valid cluster should induce consistent directional changes in the outputs across different inputs, while perturbations to different clusters should produce sufficiently distinct directions. To examine these properties, we first estimate a representative semantic direction for each community using training questions. Specifically, we compute the changes in the representations caused by perturbing each community, then average the normalized changes across training questions to obtain a fixed, community-specific semantic direction: $$d_C=\operatorname{normalize}\left( \frac{1}{|\mathcal T|}\sum_{q\in\mathcal T}u_C(q) \right).$$ As a case study, we focus on community C125 and examine whether its semantic direction induces consistent changes across different counting questions. Specifically, we add a scaled version of this direction to the original representation at the same location, while keeping the input unchanged: $$r'(q)=r_0(q)+\alpha d_C.$$ To quantify the resulting changes in numerical predictions, we compute a probability-weighted count score based on the predicted probabilities of digits 1–9: $$E_\alpha(q)=\sum_{n=1}^{9}n\,P_\alpha(n\mid q).$$ By comparing changes in this score across held-out questions, we assess whether shifting representations along the same community-specific semantic direction produces consistent effects on numerical predictions. As shown in Figure 6 in Appendix A.6 of the revised manuscript, increasing the scaling coefficient along the fixed C125 semantic direction consistently increases the expected count across both input conditions.These results suggest that C125 is associated with majority-related semantic information. Overall, these findings support our view that the functional graph captures task-dependent aspects of a trained model's internal organization. The distinguishable effects of different communities on hidden representations, together with the C125 case study linking a specific semantic direction to the notion of majority, provide further evidence that the identified communities exhibit meaningful functional differences and can capture interpretable, task-specific semantics. ### Additionally reasoning accuracy (q3) To evaluate whether our SWA better preserves reasoning ability after pruning, we conduct additional experiments on held-out GSM8K and Prealgebra. As shown in Table 2, SWA consistently achieves higher reasoning accuracy than the SparseGPT baseline across all three models and both sparsity levels. For example, on Meta-Llama-3-8B at 50% sparsity, SWA improves GSM8K accuracy from 16.20% to 21.00% and Prealgebra accuracy from 11.80% to 13.00%. These results provide direct evidence that SWA better preserves reasoning performance after pruning. Table 2: Reasoning Accuracy (↑) on GSM8K and Prealgebra After Pruning at 50% and 70% Sparsity. | Model | Sparsity | GSM8K | | Prealgebra | | | :--- | :---: | :---: | :---: | :---: | :---: | | | | SparseGPT | + SWA (Ours) | SparseGPT | + SWA (Ours) | | Llama-3.2-1B | 0.5 | 2.10 ± 0.12 | **2.67 ± 0.12** | 3.80 ± 0.20 | **3.93 ± 0.23** | | Llama-3.2-1B | 0.7 | 1.33 ± 0.12 | **1.73 ± 0.23** | 1.80 ± 0.20 | **2.07 ± 0.12** | | Llama-3.2-3B | 0.5 | 6.87 ± 0.12 | **7.50 ± 0.12** | 6.40 ± 0.20 | **7.27 ± 0.12** | | Llama-3.2-3B | 0.7 | 2.60 ± 0.20 | **2.87 ± 0.20** | 3.00 ± 0.20 | **3.20 ± 0.23** | | Meta-Llama-3-8B | 0.5 | 16.20 ± 0.20 | **21.00 ± 0.20** | 11.80 ± 0.20 | **13.00 ± 0.16** | | Meta-Llama-3-8B | 0.7 | 1.27 ± 0.12 | **1.73 ± 0.12** | 3.47 ± 0.12 | **3.73 ± 0.31** |Comparison with Attention-Head Importance-Based Sparsity Allocation (q2)
To assess whether SWA outperforms sparsity allocation based directly on measured attention-head importance, we compare SWA with a head-importance-based allocation baseline using Wanda at 50% sparsity. As shown in Table 3, SWA consistently achieves lower WikiText perplexity across all three models. These results suggest that SWA's structural information provides a more effective signal for sparsity allocation than attention-head importance alone.
Table 3: Comparison of WikiText perplexity (↓) between attention-head importance-based sparsity allocation and SWA using Wanda at 50% sparsity.
| Model | Sparsity | Head Importance + Wanda | SWA + Wanda (Ours) |
|---|---|---|---|
| Llama-3.2-1B | 0.5 | 27.24 ± 0.50 | 21.92 ± 0.05 |
| Llama-3.2-3B | 0.5 | 17.41 ± 0.17 | 13.32 ± 0.12 |
| Meta-Llama-3-8B | 0.5 | 11.12 ± 0.20 | 10.14 ± 0.03 |
Justification for the Functional Graph (w1)
We would like to explain the physical meaning of our graph. For a given input, attention heads in a neural network exhibit correlations, and our graph captures these dependencies. Thus, the graph constructed in this way captures differences in the physical angles between the query subspaces of different attention heads for the same input.
By representing statistical dependencies among attention heads, our graph serves a role analogous to that of functional brain graphs, which capture dependencies between signals from different brain regions. In neuroscience, co-activation may arise from physical interactions, whereas in our setting, these dependencies emerge through neural network training. Although their underlying mechanisms may differ, both forms of dependency can be represented and studied using graphs.
To characterize these dependencies theoretically, we show that the similarity used to construct our graph measures the alignment between the query representation subspaces retained by PCA. This gives the graph’s edges a geometric interpretation. We outline the proof below and provide the full derivation in Appendix A.7 of the revised manuscript.
Proof sketch. Consider any two attention heads evaluated on the same set of questions. For each head , let be its centered query matrix after PCA, where is the number of questions and is the number of retained dimensions. Assuming full column rank, define
The columns of form an orthonormal basis for the column space of . Whitening the sample cross-covariance of heads and gives
=U_i^\top U_j.$$ The singular values of this matrix are both the canonical correlations between the retained query representations and the cosines of the principal angles between their column spaces. Therefore, $$\rho_k(i,j)=\sigma_k(U_i^\top U_j) =\cos\theta_k(\mathcal U_i,\mathcal U_j), \qquad 1\leq k\leq\min(p_i,p_j).$$ Here, $\rho_k(i,j)$ and $\sigma_k$ denote the $k$-th largest canonical correlation and singular value, respectively, and $\theta_k$ is the $k$-th smallest principal angle. By aggregating the these canonical correlations into edge weights, our graph captures dependencies among attention heads through the alignment of their query representation subspaces. Regarding the functional roles of the clusters identified in our graph, we have provided explanations in the “Clarification on function graph” section of our rebuttal. Please let us know whether any additional clarification or explanation would help address your concerns about the graph’s validity. ## References [1] Evaluating Mathematical Reasoning Beyond Accuracy. Proceedings of the AAAI Conference on Artificial Intelligence, 39(26), 27723–27730.
Additional Evidence for Preserving Reasoning Performance (w2, q3)
To directly assess whether SWA preserves reasoning capabilities beyond general language modeling performance, we use ReasonEval [1] to evaluate the validity and redundancy of intermediate reasoning steps, alongside final-answer accuracy on GSM8K. As shown in Table 1, SWA consistently achieves the highest reasoning accuracy and validity, as well as the lowest redundancy, across both models and pruning methods.
Specifically, we evaluate Qwen3-8B and Meta-Llama-3-8B on GSM8K, comparing Wanda, SparseGPT, ATP, and our SWA method at 50% sparsity. Validity measures the correctness of reasoning steps, while redundancy measures unnecessary but logically valid steps. Both metrics are evaluated on shared correct responses. All comparisons use the same base model and sparsity level, controlling for model size and training progress. These results provide additional evidence that SWA helps preserve both reasoning accuracy and the quality of intermediate reasoning steps.
Table 1: Comparison of reasoning performance on GSM8K using ReasonEval at 50% sparsity. We report accuracy (ACC ↑), validity (↑), and redundancy (↓). Best results for each model and pruning method are highlighted in bold.
Mechanistic insight from community interventions (w1, q1)
To provide mechanistic insight into the communities identified by our functional graph, we examine how community interventions affect the model’s representations and predictions on counting tasks. We first find that different communities induce distinct directional changes in the final hidden representation, as shown by cosine similarity analysis (Figure 4 in Appendix A.6 of the revised manuscript) and PCA visualization (Figure 5 in Appendix A.6 of the revised manuscript). We then use a counting case study to show that a specific community-associated direction is related to object count and can influence the model’s numerical predictions (Figure 6 in Appendix A.6 of the revised manuscript), providing a task-specific semantic interpretation.