- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates and empirically validates a phenomenon termed Neuron Collaboration. Through semantic alignment and task-related co-activation experiments, it shows that some MLP neurons share a task-related co-activation pattern. Because this co-activation structure is task related, it may help identify model parameters that should be prioritized for updating. Based on this neuron co-activation structure, the authors propose CollabMask, a gradient-mask-based adaptation method that aims to improve the task adaptation performance of LLMs and mitigate catastrophic forgetting during continual learning.
Critical strengths
The paper tries to understand the internal representations of LLMs from the perspective of neuron co-activation structure, and applies this structure to model adaptation and continual learning. This research direction has some value. By building a neuron collaboration graph, the authors give a fairly clear description of the statistical associations between neurons, and provide corresponding empirical evidence through semantic alignment and task-related co-activation experiments. In addition, the overall design of CollabMask is reasonable to some extent, and using a task-related internal structure to guide selective parameter update is a direction worth exploring. The paper runs experiments on several models and compares with a number of gradient-masking and continual learning methods, and obtains meaningful performance gains on some tasks.
Critical weaknesses
Neuron co-activation and its functional association have already been discussed in related studies on neural networks and language models. Therefore, the Neuron Collaboration identified in this paper is closer to a further analysis and verification of an existing phenomenon on specific LLM tasks. The paper still lacks a sufficiently deep discussion on what unique contribution this phenomenon makes compared with existing neuron-association analyses.
The paper has not fully explained the relationship between the identified neuron collaboration structure and the performance improvement. The semantic alignment and task-related co-activation experiments verify that the collaboration graph has some statistical meaning, but these results do not directly show that a specific collaboration relation is necessary for guiding gradient update. Although the authors carry out component ablation and compare with random gradient masks and other methods, a direct controlled experiment on the collaboration graph itself is still missing. The stability of the experimental results and the conclusions on continual learning also need further verification. In Table 1, the performance varies noticeably across models and mask ratios, so ruling out randomness is necessary in the final version if the paper is accepted.
Questions for authors
- Could the authors further explain what new understanding the proposed Neuron Collaboration provides compared with existing studies? In particular, besides reflecting the statistical associations between neurons, does the identified collaboration structure have other properties worth noting?
- The core starting point of CollabMask is to use the neuron collaboration graph to guide gradient update, but the current experiments have not fully proved that the specific connections in the collaboration graph are an important reason for the performance gain. Could the authors add controlled experiments targeting the graph structure itself, for example, randomly shuffling the connections in the collaboration graph, or building gradient masks with a random graph that has a similar node-degree distribution? These comparisons would help show how much of the gain comes from the identified neuron collaboration structure rather than from a general sparse gradient update strategy.
- The paper takes mitigating catastrophic forgetting as one of the main application goals of CollabMask, but the results in Table 2 and Table 3 do not show a consistent advantage in preserving old tasks. In Table 3, in particular, CollabMask drops the GSM8K accuracy on Mistral-7B from 59.0% to 49.0%, a larger drop than standard SFT. Could the authors further explain this phenomenon?
2 comments
I appreciate the first four sections of the paper. Representing task-conditioned co-activation patterns as weighted edges offers a useful perspective, and the experiments reveal interesting co-activation phenomena. The additional results in the revised version further strengthen the empirical analysis. However, I do not think the authors have successfully translated this observation into a well-motivated method. Although the revised version has toned down its claim about mitigating catastrophic forgetting, the rationale for using Neuron Collaboration to guide gradient updates remains unclear, both intuitively and empirically. In particular, it is unclear why stronger collaboration should lead to higher update priority. The low modularity reported in rebuttal of the collaboration graph also leaves its potential benefits for multi-task adaptation insufficiently explained.
Overall, I find the empirical discovery interesting, but the method and its practical implications remain underdeveloped. I think the paper is between 2–3. I will maintain my current score. Nevertheless, I would consider acceptance reasonable.
We thank the reviewer for distinguishing the graph’s interpretability contribution from its use in gradient masking.
The primary contribution of this paper is the neuron collaboration graph: neurons in an MLP layer are represented as nodes, and task-conditioned co-activation relations as weighted edges. Semantic alignment with independent neuron descriptions and held-out task-token co-activation validate this structure. CollabMask then serves as a functional test of whether these relations affect adaptation and as one example of applying the graph to post-training.
1. Relation to existing neuron-association work
Our contribution is to represent task-conditioned, within-layer co-activation as a weighted network and study its network-level properties. This provides a task-distribution-level view complementary to circuit analysis, which commonly traces a small set of components across layers to explain a particular output. The collaboration graph instead supports comparisons of how within-layer neuron organization changes across domains.
We added two analyses beyond pairwise association. First, collaboration is stochastically transitive: when A--B and B--C are strong, A--C is more likely to be strong. (Presented in revision versio of chapter 4.) Second, the math-only graph does not exhibit the clear modular functional regions expected from biological-network analogies. Across the 32 layers, Louvain modularity of the undirected top-5% collaboration graph averages Q = 0.101, with about 299 communities and 49% of the neurons in the largest one, and the highest value is Q = 0.195 in layer 0. This negative result suggests that task-conditioned collaboration is not organized into simple isolated modules. Whether graphs built from more diverse tasks develop clearer modularity remains open.
To help readers better understand the properties of collaboration graph, we summarize them at the beginning of chapter 4.
2. Graph-structure control
We sampled a symmetric random graph from the fitted distribution of the measured nonzero edge weights and applied the same sparsification and normalization pipeline. Thus, both methods retain the same number of neighbors per row, while the neuron pairs differ. The random and measured graphs give similar new-task code performance, but the measured graph performs substantially better on GSM8K, MMLU, and PubMedQA, raising the overall t2 average from 47.7 to 60.0. This indicates that the identified neuron pairs contribute beyond the edge-weight distribution and mask sparsity.
3. Continual-learning scope and the Mistral result
The two-task, single-seed math-to-code experiment is an application example rather than comprehensive evidence for mitigating catastrophic forgetting. We will narrow this claim and will not treat the aggregate score as an absolute retention advantage.
CollabMask is designed as an extension of Highest. Section 5.1 defines
Collab-S(W)=(1−λ)∣∇W∣+λA(W),
where the first term is the signal used by Highest and the second is the collaboration correction. CollabMask therefore retains the assumption that high-gradient coordinates are generally important. When this assumption fails, CollabMask may also show weaker retention.
This explains the Mistral tendency: Highest performs particularly poorly after code adaptation, while CollabMask also drops from 59.0 to 49.0 on GSM8K. This is not evidence of a general retention advantage. However, the collaboration term prevents the more severe failure of Highest. We therefore use CollabMask versus Highest as the direct test of the graph contribution: CollabMask consistently improves on Highest, while its stronger overall average shows practical value rather than uniform superiority.
We agree that the observed variation across models and mask ratios, together with the single-seed design, does not establish broad experimental stability. The new graph control tests whether the identified connections matter, but does not replace repeated runs or longer task sequences. We will state this limitation explicitly and restrict our conclusions to the evaluated setting.
Modifications are highlighted in the revision version of paper.