acceptodds
Under review as a conference paper at ICLR 2027

Large-scale Neuron Community Discovery Reveals Neuron-Sparse Behaviors in Language Models

Abstract

What reusable units of computation can we discover within a language model’s neurons? Prior work has shown that individual forward passes can often be sparsified post-hoc to small groups of neurons, but it has been unclear whether these groups form coherent computational entities reused across inputs. We introduce neuron community discovery, an unsupervised method for finding neuron groups that recur across prompts and are causally necessary and sufficient, within the layers they span, for common computations. The method combines gradient-based attribution, a graph connecting neurons that are repeatedly attributed together, and community detection. Applied to Llama-3.1-8B using ten million prompts, it discovers thousands of candidate communities. We evaluate them on held-out prompts through three causal tests: community necessity, sufficiency within a layer subset, and layer necessity. These tests identify groups whose removal disrupts predictions and whose retention preserves predictions despite ablating the remaining MLP neurons in the tested layers. Discovered communities support functions like grammatical agreement, token-probability calibration, fixed phrases, in-context copying, addition, and multiplication. With communities capped at 128 neurons across all layers, 13.9% of held-out prompts have at least one layer's contribution explained on a neuron-sparse basis. Coverage remains 11.8% when requiring at least four layers and 4.2% when requiring at least twelve. These results show that small groups of existing neurons can provide reusable computational components across prompts, and that such groups can be discovered at scale.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, sparse representations, neural circuits, polysemanticity, community detection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TKy2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents an unsupervised pipeline for discovering reusable groups of native MLP neurons. It combines activation-gradient attribution, a frequency-normalized neuron co-occurrence graph, and multi-resolution Leiden community detection. Discovery uses ten million corpus prompts plus targeted data slices on Llama-3.1-8B. Candidates are tested on separate prompts through community necessity, sufficiency within selected layer subsets, and layer necessity. With at most 128 community neurons, 13.9% of evaluation prompts pass the tests for at least one layer; coverage is 11.8% for at least four layers and 4.2% for at least twelve. Examples include grammatical agreement, fixed phrases, copying and arithmetic. Sufficiency is conditional on retaining designated "always on" neurons and the rest of the model outside the tested MLP interventions.

Critical strengths

The method identifies recurring communities of native model neurons across a broad corpus, offering broader insight than isolated case studies. The “an” case study is particularly convincing, and the attribution ablations and structure-preserving shuffled-community baseline further strengthen the empirical evaluation.

Critical weaknesses

Resolution selection reuses the coverage-evaluation data, so an independent evaluation would strengthen the results. The subspan interventions and causal criteria used for baseline comparisons also need clarification.

Questions for authors

Could you select the six resolutions on one set of prompts and measure coverage on a separate set? When reporting results for selected layers, does the necessity test remove only the community neurons in those layers or the entire community? Please also clarify whether all baselines use the same three intervention tests, thresholds, and rules for retaining always-on neurons.

1 comment

We thank the reviewer for their assessment and time, and we are glad the reviewer found the "an" case study compelling. We also agree with many points the reviewer rasied among the weaknesses of the paper, and we thank the reviewer for strenghtening the paper by providing constructive feedback we can improve on.

First, we want to point out that due to a bug in our data processing pipeline, the original version of the paper overestimates the prompt coverage achieved by our communities. The corrected headline number is 10.3% of prompts can be explained by a sparse neuron community, not 13.9% as reported before.

The resuls have been updated throughout the paper and the rest of the paper's findings remain unaffected. We deeply apologise for this.

We address individual questions and weaknesses below. We have revised the paper and highlight new content in blue.

Could you select the six resolutions on one set of prompts and measure coverage on a separate set?

We agree that resolution selection should be run on an independent set of prompts to avoid overfitting. We repeated the search for the best six-resolution subset using training prompts only and recovered the same set used in the paper: {32, 64, 128, 256, 512, 1024}. In fact, across all resolutions reported in Figure 8, the coverage on train and held-out prompts is almost identical:

γ\gamma valuesTrain coverage (%)Held-out coverage (%)
{256}5.705.74
{128, 512}8.058.13
{128, 256, 512}8.989.03
{...} + 10249.649.66
{...} + 6410.0910.13
{...} + 3210.2110.28
{...} + 1610.2610.33
{...} + 810.2910.36
{...} + 410.3010.36
{...} + 210.3010.37
{...} + 110.3010.37
{...} + 0.510.3010.37
{...} + 0.2510.3010.37
{...} + 204810.3110.37

This shows that threshold selection is not sensitive to a specific set of prompts, and translates well between train and held-out sets. We have added this comparison below to Appendix C (new content is highlighted in blue).

When reporting results for selected layers, does the necessity test remove only the community neurons in those layers or the entire community?

Only community neurons in those layers. Our apologies, Section 3 should have been more clear on this. We have now updated the necessity definition in Section 3 to state this explicitly.

Please also clarify whether all baselines use the same three intervention tests, thresholds, and rules for retaining always-on neurons.

Thanks for pointing out that this is not clarified in the paper, as it definitely should be. All baselines reported in Appendix B are evaluated in the exact same setup, with the same intervention tests, always-on neurons, as well as data and other hyperparameters. The only difference in the end-to-end pipeline is the attribution method itself.

We have updated the paper to clarify this.


We hope that our revisions adequately address the reviewer’s concerns. If the reviewer agrees, we kindly ask them to consider updating their assessment of the paper.

Reviewer_iYemOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a scalable approach to discovering neuron communities through gradient-based attribution, co-occurrence graphs, and community detection. Experiments on Llama-3.1-8B identify reusable sparse neuron groups associated with linguistic and arithmetic behaviors, covering approximately 13.9% of held-out prompts under community necessity, layer-restricted sufficiency, and layer necessity criteria.

Critical strengths
  1. The method extends neuron analysis beyond manually selected behaviors to automated discovery over a large corpus, offering a promising way to identify computational components reused across contexts.

  2. The combination of community necessity, layer-restricted sufficiency, and layer necessity tests provides causal evidence beyond statistical co-occurrence.

  3. The large-scale experiments combine held-out evaluation, detailed case studies, and attribution-method comparisons, providing substantial empirical support for the approach.

Critical weaknesses
  1. All core experiments are conducted on Llama-3.1-8B, leaving generalization across model families and scales unverified. Replicating the main findings on another model would strengthen the conclusions.

  2. Although the method discovers thousands of candidate communities, the characterized behaviors are concentrated in lexical, formatting, and local linguistic patterns, with limited systematic validation of more complex computations. Claims about reusable computational components should therefore be scoped to the demonstrated behaviors and intervention settings.

  3. Attribution and interventions focus exclusively on next-token prediction at the final prompt position, potentially overlooking neuron groups involved primarily at earlier positions, in cross-position interactions, or throughout multi-step generation. The paper should clarify how this discovery bias limits the types and range of communities it can recover.

Questions for authors

see weekness

1 comment

We thank the reviewer for their positive assessment, and we are glad the reviewer recognized the evidence on neccessity and sufficiency of neuron communities. We also agree with many points the reviewer rasied among the weaknesses of the paper, and we thank the reviewer for strenghtening the paper by providing constructive feedback we can improve on.

First, we want to point out that due to a bug in our data processing pipeline, the original version of the paper overestimates the prompt coverage achieved by our communities. The corrected headline number is 10.3% of prompts can be explained by a sparse neuron community, not 13.9% as reported before.

The results have been updated throughout the paper and the rest of the paper's findings remain unaffected. We deeply apologise for this.

We address individual weaknesses below. We have revised the paper and highlight new content in blue.

W1

Replicating the main findings on another model would strengthen the conclusions.

We absolutely agree, and we would be very excited to see future research exploring neuron communities in other models. Recent research on neuron interpretability [1] shows many interpretable neurons exist in Qwen3-32B, for instance, which we take as a promising sign that our method would succeed there as well.

For this work we have focused on Llama-3.1-8B as a safe option, because it was successfully used for neuron-level analysis in prior research. Arora et al. [2] showed that neurons form a good basis for circuit analysis on Llama 3.1-8B-Instruct, and Feucht et al. [3] found 28 neurons in layer 18 of Llama 3.1-8B implementing base-10 addition.

Given the compute intensity of the analysis, we decided to allocate available compute to scale up experiments on Llama up to 10M prompts instead of adding another model to the mix. We hope the reviewer agrees that was a reasonable choice.

W2

the characterized behaviors are concentrated in lexical, formatting, and local linguistic patterns, with limited systematic validation of more complex computations

Indeed, most communities we find represent some form of linguisting pattern, with very few exceptions: in-context copying and arithmetic are the most notable ones. We apologise if some parts of the paper do not make this sufficiently clear, and we appreciate reviewer's suggestion to scope our claims accordingly.

We've clarified the relevant claims across the paper, most notably in the abstract, the introduction and the conclusion. We've also expanded Section 6 (Discussion and Limitations) to explicitly point out the scope of communities we find.

All edits are highlighted in blue in the updated paper pdf.

W3

Attribution and interventions focus exclusively on next-token prediction at the final prompt position

This is a very good point, and we should have explicitly called this out in the limitations section.

We have now updated Section 6 with the following paragraph:

Final token position. We restrict attribution and interventions to the final prompt position only. This will inevitably miss communities that operate across multiple positions. For instance, in the arithmetic example "2+5=", some neurons might read the operands at earlier positions, and other neurons would read it at the final position to perform the addition. The current version of our method would not be able to find these connections. We see this approach as a reasonable first step, which allowed us to ignore the complexities associated with cross-position communities; we believe extending community discovery to cross-position setup is an important direction for future research.


We hope that our revisions adequately address the reviewer’s concerns. If the reviewer agrees, we kindly ask them to consider updating their assessment of the paper.

References

[1] Cammarata, N., 2026. Neural Networks Are Already Clean. Goodfire.

[2] Arora, Aryaman, et al. "Language model circuits are sparse in the neuron basis." arXiv preprint arXiv:2601.22594 (2026).

[3] Feucht, Sheridan, et al. "Arithmetic in the Wild: Llama uses Standard Addition to Reason About Cyclic Concepts." Third Conference on Language Modeling.

Reviewer_HDTsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an unsupervised framework for discovering recurring communities of native MLP neurons in large language models. The proposed pipeline combines gradient-based neuron attribution, a frequency-normalized neuron co-occurrence graph, and multi-resolution Leiden community detection. Candidate communities are subsequently evaluated using causal interventions designed to test community necessity, sufficiency within selected layers, and layer necessity.

Applied to Llama-3.1-8B using ten million discovery prompts, the method identifies thousands of candidate neuron communities. The authors report that communities containing at most 128 neurons explain the contributions of at least one layer on 13.9% of held-out prompts. Coverage remains 11.8% when requiring at least four layers and 4.2% when requiring at least twelve layers.

The paper also presents case studies involving grammatical agreement, token-level calibration, formatting, in-context copying, and arithmetic. A particularly compelling example concerns six neurons associated with vowel-initial word predictions after the article "an", for which the authors provide independent prompt sampling and intervention-based evidence.

Overall, the paper contributes a scalable approach to discovering recurring neuron groups directly in the model's native representation, together with an empirical characterization of the prevalence and diversity of locally neuron-sparse computations.

Critical strengths

S1. Scalable discovery of reusable neuron communities without predefined behavioral categories.

The paper addresses an important limitation of existing mechanistic interpretability approaches, which frequently rely on manually selected tasks or prompt templates. By identifying recurring neuron groups across a large and heterogeneous corpus, the proposed framework offers a systematic way to investigate the organization of computation in a language model.

The combination of neuron attribution, frequency-normalized co-occurrence statistics, and multi-resolution community detection is practically meaningful. Although individual components build on established techniques, their integration into a large-scale discovery and causal evaluation pipeline constitutes a useful contribution.

Importantly, the method operates directly on native MLP neurons rather than requiring an additional learned feature dictionary. The resulting communities and quantitative coverage measurements provide a valuable foundation for further investigations of neuron-level computational structure.

S2. Substantive causal validation and evidence of recurring sparse computations.

A major strength is that the authors do not rely exclusively on correlations between neuron activations. Instead, they evaluate candidate communities through interventions involving neuron ablation, sparse retention, and layer-level ablation.

The "an" + vowel agreement case study is particularly informative. On 8,000 independently sampled prompts, ablating six community neurons produces a median KL divergence of 4.99, whereas retaining these neurons together with four always-on neurons while ablating the remaining MLP neurons in the selected layers yields a median KL divergence of 0.053.

These results provide compelling evidence that a small set of neurons can play a substantial causal role in preserving predictions across multiple inputs, conditional on the remaining network computation.

Furthermore, Appendix B provides comparisons among several attribution methods and a shuffled-community baseline that preserves important structural statistics. These controls strengthen the empirical contribution and support the effectiveness of the proposed discovery procedure.

Taken together, the scale of discovery and the intervention-based validation are the main reasons I lean toward acceptance.

Critical weaknesses

W1. Ambiguity in the causal definitions of community necessity and sufficiency across layer subsets.

My primary concern is that the causal criteria in Section 3 do not appear to operate on exactly the same neuronal subset.

Equation (2) defines community necessity by ablating the entire community CC, whereas Equations (3) and (4) evaluate sufficiency and layer necessity with respect to a selected layer subset SS. The paper then considers a community necessary and sufficient within SS when all three conditions are satisfied.

However, necessity of the full community does not necessarily imply necessity of the neurons belonging to the selected subset, CS={(ℓ,j)∈C:ℓ∈S}C_S=\{(\ell,j)\in C:\ell\in S\}. In principle, the full-community necessity condition could be satisfied primarily because of neurons outside SS, while the retained neurons inside SS are sufficient but redundant with other neurons in the clean model.

Consequently, the three stated conditions do not necessarily establish that the same within-subspan neuronal group is both necessary and sufficient.

This may simply reflect an ambiguity in the mathematical notation rather than an implementation problem. Indeed, the detailed "an" case study explicitly intervenes on its selected six-neuron subspan. Nevertheless, since the aggregate coverage results depend directly on these causal criteria, the precise implementation should be clarified.

A related discrepancy appears in Appendix B (lines 684–701), where coverage is described using necessity and sufficiency, without explicitly mentioning the layer-necessity criterion required in the main experiments. It is therefore unclear whether the attribution comparisons and shuffled-community baseline use exactly the same three-test protocol.

Finally, joint necessity of a selected layer set does not establish that every layer in the set individually makes a substantial causal contribution. The interpretation of the results involving twelve or more layers should therefore be stated carefully.

W2. Limited independence of coverage evaluation and incomplete validation of behavioral generalization.

Appendix C explicitly states that the six Leiden resolution thresholds are selected by maximizing coverage on the same 8,862,662 evaluation prompts used to report the final result of 13.94%.

Although this procedure is transparently disclosed, the reported coverage is therefore measured after configuration selection on the evaluation data. It is not yet an independent estimate of coverage for a previously fixed configuration. A separate validation/test split would strengthen the central quantitative claim.

This issue does not invalidate the intervention results, and the relatively small difference between the selected six-threshold coverage (13.94%) and the full fourteen-threshold coverage (14.11%) is reassuring. Nevertheless, the independence of the final evaluation remains important.

A second concern relates to the interpretation of discovered communities as reusable behavioral mechanisms. Apart from the independently evaluated "an" community, most qualitative examples are drawn from prompts that already satisfy the causal selection criteria. These examples establish recurring intervention effects but provide less direct evidence about how broadly a fixed community generalizes across independently selected instances of its proposed behavior.

The authors appropriately acknowledge limitations in behavior recall and automated interpretation in Section 6. However, additional independent behavior-conditioned evaluations would considerably strengthen the claim that the discovered groups correspond to reusable computational components rather than narrower collections of successful contexts.

These concerns appear addressable through protocol clarification and targeted additional experiments, rather than requiring a fundamental redesign of the method.

Questions for authors

I would appreciate clarification of the following questions. The answers, particularly to Q1 and Q2, could materially affect my assessment.

Q1. Are necessity and sufficiency evaluated on the same within-subspan neuron set?

For a candidate community CC and a selected layer subset SS, is necessity evaluated by ablating the full community CC, as written in Equation (2), or only the restricted community CSC_S?

If the implementation uses the full CC, could the authors report how the main coverage results change when necessity is additionally required for CSC_S?

A short pseudocode description of the three interventions would help resolve this ambiguity.

Q2. Are the causal evaluation criteria consistent across the main experiments and Appendix B?

Section 3 requires community necessity, sufficiency, and layer necessity. However, Appendix B explicitly lists only necessity and sufficiency in its coverage definition.

Is layer necessity also required in all attribution-method comparisons and for the shuffled-community baseline?

If not, how do the comparisons change when all methods are evaluated using the same three-test protocol?

Q3. How robust is the reported coverage under independent resolution selection?

Appendix C selects the six optimal Leiden resolutions using the same evaluation prompts on which coverage is reported.

Could the authors provide coverage after choosing the resolution thresholds on a separate validation partition and evaluating the fixed configuration on an untouched test partition?

It would also be useful to clarify the deduplication and splitting procedures, including how the final evaluation count of 8,862,662 prompts is obtained.

Q4. Can the authors provide additional evidence for behavioral generalization beyond the principal case study?

The independent evaluation of the "an" community is a strong component of the paper.

Could a similar protocol be applied to two or three representative arithmetic, copying, or grammatical communities, where the neuron sets are fixed before testing independently sampled behavior-specific prompts?

Reporting the effects on positive examples, negative controls, and failure cases would help distinguish broadly reusable computational mechanisms from narrower template-specific effects.

Q5. Could the authors clarify the co-occurrence statistic and effective sparsity budget?

Section 3 describes a frequency-normalized co-occurrence z-score but does not provide its exact mathematical definition, including the null model and variance estimate.

Could the authors specify this formula and the relevant graph construction details?

Additionally, since always-on neurons are retained during sufficiency interventions, it would be helpful to report both community size and the total number of retained neurons when interpreting the claimed sparsity budgets.

I view Q1–Q3 as the highest-priority issues. Clear answers and supporting evidence would substantially increase my confidence in the reported findings.

1 comment

We thank the reviewer for their assessment. We are glad the reviewer finds our method compelling, and our results to be convincing. We also thank the reviewer for strenghtening the paper by providing constructive feedback we can improve on.

First, we want to point out that due to a bug in our data processing pipeline, the original version of the paper overestimates the prompt coverage. The corrected headline number is 10.3% of prompts can be explained by a sparse neuron community, not 13.9% as reported before.

The resuls have been updated throughout the paper. The rest of the paper's findings, including case studies remain unaffected. We deeply apologise for this.

We address individual questions and weaknesses below. We have revised the paper and highlight new content in blue.

Q1 (W1)

Are necessity and sufficiency evaluated on the same within-subspan neuron set?

Yes, necessity is evaluated on the same within-subspan neuron set, and only removes community neurons within the subspan layers. Our apologies, Section 3 should have been more clear on this. We have now updated the necessity definition in Section 3 to state this explicitly.

Q2 (W1)

Are the causal evaluation criteria consistent across the main experiments and Appendix B?

Thanks for pointing out that this is not clarified in the paper, as it definitely should be. All baselines reported in Appendix B are evaluated in the exact same setup, with the same intervention tests, always-on neurons, as well as data and other hyperparameters. The only difference in the end-to-end pipeline is the attribution method itself.

We have updated the paper to clarify this.

Q3 (W2)

How robust is the reported coverage under independent resolution selection?

We agree that resolution selection should be run on an independent set of prompts to avoid overfitting. We repeated the search for the best six-resolution subset using training prompts only and recovered the same set used in the paper: {32, 64, 128, 256, 512, 1024}. In fact, across all resolutions reported in Figure 8, the coverage on train and held-out prompts is almost identical:

γ\gamma valuesTrain coverage (%)Held-out coverage (%)
{256}5.705.74
{128, 512}8.058.13
{128, 256, 512}8.989.03
{...} + 10249.649.66
{...} + 6410.0910.13
{...} + 3210.2110.28
{...} + 1610.2610.33
{...} + 810.2910.36
{...} + 410.3010.36
{...} + 210.3010.37
{...} + 110.3010.37
{...} + 0.510.3010.37
{...} + 0.2510.3010.37
{...} + 204810.3110.37

This shows that threshold selection is not sensitive to a specific set of prompts, and translates well between train and held-out sets. We have added this comparison below to Appendix C (new content is highlighted in blue).

It would also be useful to clarify the deduplication and splitting procedures, including how the final evaluation count of 8,862,662 prompts is obtained.

We have added Appendix H describing the sampling and deduplication procedures, including how the final evaluation count is obtained.

Q4 (W2)

Can the authors provide additional evidence for behavioral generalization beyond the principal case study?

The an community was straightforward to evaluate, because it has a simple trigger: the article an. For most other communities, the filtering rule is not clear. Often a community represents model behavior that is hard to predict from the prompt alone (e.g. suppressing tokens or breaking a line).

We have added Appendix F, which applies a similar evaluation to the fixed nine-neuron there is/are community. We sample 1,000 held-out prompts ending in there, followed by is or are in the original text. The overall results are similar to the an experiment: removing the community increases KL, while retaining only the community keeps KL low. However, the effects are less pronounced: only 24% of prompts pass community necessity, and roughly half of those (11.2%) pass all three criteria.

Manual inspection suggests that the model can use many mechanisms to choose between is/are, and or community is only one of the options. For instance, in some prompts "there are" repeated multiple times, and the model relies on copying instead. This gives a more nuanced picture of when the community is needed, and further analysis is needed to thoroughly describe community's role.

Q5 (W2)

Could the authors clarify the co-occurrence statistic and effective sparsity budget?

We have added Appendix G with the exact co-occurrence formula, null model, variance estimate, and graph construction details. It also clarifies how always-on neurons contribute to the effective sparsity budget: the an case study retains 6 + 4 = 10 neurons in total, and the there is/are case study retains 9 + 5 = 14.


We hope that our revisions adequately address the reviewer’s concerns. If the reviewer agrees, we kindly ask them to consider updating their assessment of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.