- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents an unsupervised pipeline for discovering reusable groups of native MLP neurons. It combines activation-gradient attribution, a frequency-normalized neuron co-occurrence graph, and multi-resolution Leiden community detection. Discovery uses ten million corpus prompts plus targeted data slices on Llama-3.1-8B. Candidates are tested on separate prompts through community necessity, sufficiency within selected layer subsets, and layer necessity. With at most 128 community neurons, 13.9% of evaluation prompts pass the tests for at least one layer; coverage is 11.8% for at least four layers and 4.2% for at least twelve. Examples include grammatical agreement, fixed phrases, copying and arithmetic. Sufficiency is conditional on retaining designated "always on" neurons and the rest of the model outside the tested MLP interventions.
Critical strengths
The method identifies recurring communities of native model neurons across a broad corpus, offering broader insight than isolated case studies. The “an” case study is particularly convincing, and the attribution ablations and structure-preserving shuffled-community baseline further strengthen the empirical evaluation.
Critical weaknesses
Resolution selection reuses the coverage-evaluation data, so an independent evaluation would strengthen the results. The subspan interventions and causal criteria used for baseline comparisons also need clarification.
Questions for authors
Could you select the six resolutions on one set of prompts and measure coverage on a separate set? When reporting results for selected layers, does the necessity test remove only the community neurons in those layers or the entire community? Please also clarify whether all baselines use the same three intervention tests, thresholds, and rules for retaining always-on neurons.
We thank the reviewer for their assessment and time, and we are glad the reviewer found the "an" case study compelling. We also agree with many points the reviewer rasied among the weaknesses of the paper, and we thank the reviewer for strenghtening the paper by providing constructive feedback we can improve on.
First, we want to point out that due to a bug in our data processing pipeline, the original version of the paper overestimates the prompt coverage achieved by our communities. The corrected headline number is 10.3% of prompts can be explained by a sparse neuron community, not 13.9% as reported before.
The resuls have been updated throughout the paper and the rest of the paper's findings remain unaffected. We deeply apologise for this.
We address individual questions and weaknesses below. We have revised the paper and highlight new content in blue.
We agree that resolution selection should be run on an independent set of prompts to avoid overfitting. We repeated the search for the best six-resolution subset using training prompts only and recovered the same set used in the paper: {32, 64, 128, 256, 512, 1024}. In fact, across all resolutions reported in Figure 8, the coverage on train and held-out prompts is almost identical:
This shows that threshold selection is not sensitive to a specific set of prompts, and translates well between train and held-out sets. We have added this comparison below to Appendix C (new content is highlighted in blue).
Only community neurons in those layers. Our apologies, Section 3 should have been more clear on this. We have now updated the necessity definition in Section 3 to state this explicitly.
Thanks for pointing out that this is not clarified in the paper, as it definitely should be. All baselines reported in Appendix B are evaluated in the exact same setup, with the same intervention tests, always-on neurons, as well as data and other hyperparameters. The only difference in the end-to-end pipeline is the attribution method itself.
We have updated the paper to clarify this.
We hope that our revisions adequately address the reviewer’s concerns. If the reviewer agrees, we kindly ask them to consider updating their assessment of the paper.