acceptodds
Under review as a conference paper at ICLR 2027

Causal Core–Periphery Organization and Remodeling in Large Language Models

Abstract

Small subsets of neurons can contribute disproportionately to the behavior of large language models, raising questions about how task-related causal support is organized and remodeled during post-training. We introduce Causal Contribution Profiling, a framework that ranks neurons in the multilayer perceptron (MLP) of Transformer blocks by attribution and measures contribution density by patching activations in progressively larger sets of top-ranked neurons. We study 31 models (0.27B–72B parameters) from five families using a 36-task suite and conduct analyses along three training trajectories. Within the scanned range, we find task-selective core–periphery organization, characterized by a small core with high contribution density and broader support with lower density along the neuron ranking. Core membership is more stable than peripheral membership in 92.0% of 224 eligible task–transition comparisons. Retrospective interventions reveal contributions from future core members before models achieve full accuracy on the evaluated sample groups. These contributions tend to strengthen with behavioral gains. After existing members are excluded, source-stage attribution outperforms layer-matched random baselines in predicting new core and support members in 90.5% and 99.1% of eligible cases, respectively. Together, these findings characterize task support as a contribution-defined organization with differential stability and partially predictable remodeling.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Mechanistic Interpretability, Causal Contribution Profiling, Core–Periphery Organization, Task-Supporting Neurons

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_csz6OpenReview ↗
Rating
4(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Causal Contribution Profiling (CCP), a framework for characterizing how task-related causal contributions are distributed across MLP neurons in large language models and how this organization changes during post-training. The method ranks neurons using attribution patching and progressively expands activation-patching interventions, with size- and layer-matched random controls to estimate contribution density. Experiments cover 31 models from five model families, 36 tasks, and three training trajectories. The results reveal a task-selective core–periphery organization, in which a small subset of neurons accounts for a disproportionately large share of positive causal contributions. The authors further show that core membership is generally more stable than peripheral membership, that future core neurons can contribute before full behavioral success, and that attribution scores from earlier checkpoints can predict newly recruited core and support members. Overall, the paper provides a systematic framework for studying the organization and evolution of task-related causal contributions in LLMs.

Critical strengths
  1. The paper introduces a principled framework that combines attribution-based neuron ranking, progressively expanded activation-patching interventions, and matched random controls. This goes beyond identifying highly ranked neurons by quantifying how causal contribution density varies across the neuron ranking.

  2. The experimental evaluation covers a relatively broad range of model families, model scales, and tasks. The analysis also extends beyond static neuron localization to investigate membership stability, contribution changes, and the predictability of organizational remodeling across training checkpoints.

Critical weaknesses
  1. The related-work discussion appears to miss relevant research on causal identification of critical neurons. In particular, https://arxiv.org/abs/2510.10238 investigates perturbation-based identification of critical neurons and reports highly sparse critical neuron sets whose disruption can cause substantial degradation in language-model performance. This work is closely related to the present paper's investigation of causal neuron contributions and their concentration. The current manuscript should better position its contribution relative to this line of research and clarify what is novel beyond identifying sparse, causally important neurons.

  2. The conclusions about core–periphery organization are based on interventions over a limited scanned range of MLP neurons (approximately 1.6% of the population). In addition, the core and periphery are defined using positive-contribution coverage thresholds. Although this operational definition is reasonable, it remains unclear how robust the organizational findings are to different scan ranges, coverage thresholds, and neuron-ranking choices.

  3. The membership-forecasting results are promising, but their practical predictive value remains somewhat unclear. The evaluation uses the observed size of the future target set to determine the number of neurons selected. Moreover, although early-stage attribution outperforms matched random baselines in most eligible cases, the recovery rates for newly recruited core and support members vary considerably across training trajectories. Stronger comparisons and a clearer assessment of the limitations of this forecasting setup would help establish the significance of the results.

Questions for authors
  1. The authors should conduct sensitivity analyses to assess the robustness of the core–periphery structure. These could vary the coverage thresholds used to define the core and periphery, expand or alter the scanned neuron range, and compare alternative attribution rankings. It would also be useful to examine whether the main stability and forecasting conclusions persist under these choices.

  2. The authors should strengthen the membership-forecasting evaluation by comparing early-stage attribution against more informative baselines, such as rankings based on existing core or peripheral membership and layer-specific statistics. Where feasible, the authors should also evaluate settings in which the future target-set size is not assumed to be known. Reporting recovery distributions and effect sizes across tasks, rather than primarily the proportion of cases exceeding a random baseline, would provide a clearer picture of the method's practical predictive value.

4 comments
AuthorsOpenReview ↗

Response to Reviewer csz6 [Part 1/2]

We sincerely appreciate the reviewer's thoughtful assessment and constructive suggestions, particularly the recognition of CCP's principled design, the breadth of our experiments, and the value of studying contribution dynamics across training.

Q1: What does this study contribute beyond existing work on critical neurons?

We have added The Achilles' Heel of LLMs to Section 6 of the main text and clarified its relationship to our study. Qin et al. demonstrate that perturbing an ultra-sparse set of critical neurons can severely impair language abilities, revealing models' strong dependence on these small sets. From a cross-task perspective, Han et al. establish functional differentiation: neuronal recruitment overlaps more within cognitive domains and is more distinct across domains. We further examine how causal support is distributed within individual tasks and changes during training, revealing a small, high-density core and broader, lower-density support. Core membership is generally more stable than peripheral membership, while remodeling involves strengthening or weakening existing contributions and exchanging high-contribution members.

We appreciate the suggestion to examine how these measurement choices affect our findings. We added scan-range and coverage-threshold sensitivity analyses and compared alternative attribution rankings for prediction.

Q2: How robust are the findings to the scan range, coverage thresholds, and attribution ranking?

(a) Scan range. Extending the scan from 1.6% to 3.2% and 5% preserved contribution concentration: at the 5% limit, 0.109% of all MLP neurons accounted for 50% of the contribution within the scan (Appendix B.4). Across 90 common checkpoint comparisons, mean core stability exceeded peripheral stability in all three trajectories at every limit (Appendix C.1). Recovery rates varied with the target sets redefined at each limit; at 5%, whole-core recovery ranged from 66.33% to 77.88% and new-core recovery from 23.19% to 30.36% across trajectories (Appendix D.3).

(b) Coverage thresholds. We crossed core thresholds of 40%, 50%, and 60% with support thresholds of 80%, 90%, and 95%, yielding nine settings. Across the same 190 primary comparisons, each trajectory retained a positive mean core-minus-periphery stability difference under every setting. Equal-size windows of 20 or 25 neurons also showed greater average stability near the ranking's beginning (Appendix C.1).

(c) Alternative attribution rankings. We adapted EAP (Syed et al., 2024) and EAP-IG (Hanna et al., 2024), originally developed for circuit discovery, to neuron-level scoring for future-member prediction. Using the same checkpoint and samples across seven OLMo2-1B Base–SFT tasks, our method, EAP, and EAP-IG achieved mean average precision (AP) of 31.19%, 29.47%, and 17.61% for new-core prediction, respectively (Appendix D.4). These comparisons evaluate prediction against the same CCP-defined target memberships. The organizational profiles remain conditional on CCP's task-specific ranking.

References

Zixuan Qin, Qingchen Yu, Kunlin Lyu, Zhaoxin Fan, and Yifan Sun. 2026. The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilities. International Conference on Learning Representations (ICLR).

Pengrui Han, Jacob Andreas, Evelina Fedorenko, and Andrea Gregor de Varda. 2026. Modular Cognitive Architecture Emerges in Large Language Models. arXiv:2608.13567.

Aaquib Syed, Can Rager, and Arthur Conmy. 2024. Attribution Patching Outperforms Automated Circuit Discovery. Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 407–416.

Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. 2024. Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms. First Conference on Language Modeling (COLM).

AuthorsOpenReview ↗

Response to Reviewer csz6 [Part 2/2]

We appreciate the reviewer's specific suggestions for evaluating the practical value of membership prediction. We added stronger baselines, evaluations independent of future set size, and task-wise results, and explored a pruning application.

Q3: What is the practical value of membership prediction under stronger baselines, unknown future set sizes, and task-level variation?

(a) Predictive information beyond existing membership. Using the same earlier checkpoint and samples, we compared attribution with a membership-and-layer prior, EAP, and EAP-IG. Results on seven OLMo2-1B Base–SFT tasks show that earlier contributions provide predictive information beyond existing membership and layer statistics, supporting the prioritization of future high-contribution neurons (Table R7; Appendix D.4).

Table R7. Mean AP across Seven Tasks (%).

MethodWhole coreNew coreNew support
Ours68.5731.1925.75
Membership + layer prior55.026.651.99
EAP (Syed et al., 2024)67.2529.4721.82
EAP-IG (Hanna et al., 2024)58.1017.6115.12

(b) Unknown future set size and task-wise results. We added AP and recovery at ten prespecified selection fractions (0.01%–5%), making selection budgets independent of future set size. The mean recovery advantage over EAP-IG is more pronounced at smaller budgets, supporting contribution-based prioritization of future high-contribution neurons. Our method also outperforms EAP-IG on most tasks. For example, new-core and new-support AP improve by 36.81 and 41.87 percentage points on noun agreement. Complete task-wise results are reported in Appendix D.4.

(c) Pruning application. Contribution information also provides a practical basis for model pruning. By preserving the main positive-attribution support and removing weakly attributed channels, we constructed separate OLMo2-1B Instruct variants for three tasks. Without retraining, each variant removes 5.42% of parameters, with BF16 accuracy gains of 2.95–8.77 percentage points and improvements also observed in FP32. All three achieve higher observed accuracy than the full model and the mean of layer-matched random prunings (Appendix E). We hypothesize that removing weakly attributed channels may reduce task-irrelevant interference while preserving the main task support. These results demonstrate the potential of contribution information to reduce model size and improve task performance. We will further investigate this explanation and evaluate contribution-guided pruning across a broader range of tasks and models.

I thank the authors for their detailed rebuttal. Since my concerns have been fully addressed, I have decided to raise my rating.

We sincerely appreciate your positive evaluation. Your valuable suggestions are essential to the improvement of our manuscript.

Reviewer_9UsPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents a useful framework for profiling task-related MLP neurons through attribution-guided activation patching. The static experiments are broad and carefully structured, but several central conclusions depend on choices and analyses that are not sufficiently validated.

Critical strengths
  1. The study covers 31 models, five model families, and 36 tasks, with separate data splits for ranking, profiling, and confirmation.

  2. The paper gives explicit definitions for attribution scores, intervention effects, random controls, contribution density, and membership prediction.

Critical weaknesses
  1. The paper does not test whether the identified core converges as the number of task examples increases. A sample-size scaling experiment should report core overlap, membership probabilities, and convergence across independently sampled example groups.

  2. It is unclear why tracking core membership across training is necessary for the central contribution, what training mechanism causes the observed changes, and how these changes differ across tasks. The analysis mainly reports set overlap and prediction statistics, without explaining the underlying remodeling process.

  3. Core and periphery are defined using 50% and 90% contribution-coverage thresholds, but the paper does not establish that these values correspond to meaningful structural boundaries. Results should be tested under alternative thresholds and compared with fixed-size or continuous concentration measures.

  4. Although the paper extends its analysis to training checkpoints, it does not explain how the discovered core/periphery structure could guide fine-tuning, pruning, targeted intervention, data selection, or other concrete model-improvement procedures.

  5. Several claims are based on broad aggregate statistics, while complete task-level or cross-model results are not shown. For example, task selectivity is described across many models, but the detailed matrix and summary are primarily provided for one model, making it difficult to assess how general the claim is.

Questions for authors

See Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9UsP [Part 1/3]

We appreciate the reviewer's positive assessment of the framework and the constructive suggestions. To provide a clearer logical progression, we have grouped related concerns and adjusted the order of our responses.

Q1: Why is tracking core membership across training necessary, what accounts for the observed remodeling, and how does it vary across tasks?

We address both the motivation for examining neuronal changes during training and the functional basis of remodeling.

(a) Motivation for Studying Neuronal Changes during Training. We compare the contributions of key neurons before and after training to understand how their functional roles change as model capabilities evolve. Specifically, we examine whether task support remains with the same neurons, how their contributions strengthen or weaken, and which neurons become newly involved. These analyses also inform the development of practical methods that use contribution information to guide targeted interventions.

(b) Functional Basis of Remodeling. At the functional level, remodeling reflects changes in the strength and distribution of neuronal causal contributions. Independent interventions show that later-stage cores can contribute before fully correct performance on the evaluated groups, that contributions can strengthen with behavioral improvement, and that membership changes have functional counterparts (Table R3; Appendices C.2–C.4). These results characterize both adjustments to existing support and redistribution of high-contribution roles, with task- and model-specific variation.

Table R3. Temperature Example: Normalized Contribution Changes (×10−3)(\times 10^{-3}).

ModelIncoming coreExiting corePersistent core
OLMo2-1B+6.41-5.37+3.80
OLMo2-7B+2.68-2.76+0.07
Tulu3-8B+1.52-3.24+0.13

Together, these results provide an empirical account of how neuronal task support is maintained and remodeled during training.

Q2: How do the task-level and cross-model results support the generality of the findings, particularly task selectivity?

Appendix B.3 now provides task-selectivity results for each of 30 models, covering 13–36 eligible tasks per model. All models show the same pattern: neurons selected for a task have a stronger average intervention effect on that task than on other tasks. The complete 36×3636\times36 matrix for Qwen2.5-14B illustrates the relationships between individual tasks, while the accompanying index identifies the corresponding results for each model. The training analyses also retain unconfirmed cases, present task-specific functional comparisons, and clarify eligibility for membership prediction (Appendices C.2–C.4 and D.3). These additions make it easier to assess which models and tasks support the reported conclusions.

AuthorsOpenReview ↗

Response to Reviewer 9UsP [Part 2/3]

We appreciate the reviewer's suggestion to demonstrate a concrete application of the contribution-based analysis.

Q3: Can the identified contribution organization guide concrete model improvements, such as structural pruning?

We developed a causally inspired structural-pruning method for OLMo2-1B Instruct. The task-specific variants remove 5.42% of model parameters without retraining and achieve higher observed accuracy on all three evaluated tasks (Appendix E).

(a) Contribution-Guided Pruning. The method is motivated by a simple principle: preserve the channels carrying the main positive support for a task while removing channels with weak attribution. We protect a set covering 90% of positive attribution mass, then prune channels with the smallest absolute attribution outside it. Applied separately to each task, this rule physically removes the selected MLP channels and their associated weights.

(b) Performance after Pruning. All three variants achieve higher observed accuracy than the full Instruct model and the mean of five random prunings with identical per-layer deletion counts, in both BF16 and FP32 (Table R4). The pruning masks are selected using data separate from the test set, and every condition is evaluated on the same complete held-out set.

Table R4. Task-Specific Pruning without Retraining (Paired Accuracy, %).

PrecisionTaskFullRandomPrunedChange (pp)
BF16PROST61.6648.2866.87+5.21
BF16Newton46.0542.2854.82+8.77
BF16Symbolic add/subtract65.4034.9468.35+2.95
FP32PROST61.9647.9866.56+4.60
FP32Newton45.6142.2853.95+8.33
FP32Symbolic add/subtract65.8235.3667.51+1.69

These results demonstrate the potential of contribution-guided pruning to reduce model size while improving task performance. We hypothesize that removing weakly attributed channels may reduce task-irrelevant interference while preserving the main positive support, thereby contributing to the observed gains. We will further investigate this explanation and develop contribution-guided pruning methods.

AuthorsOpenReview ↗

Response to Reviewer 9UsP [Part 3/3]

We appreciate the suggestion to examine how sample size affects core identification.

Q4: Does the identified core converge as the number of task examples increases?

We repeated core identification using resampled data at full and half sample sizes across 258 checkpoint–task profiles from three training trajectories. At each checkpoint, we recomputed neuronal rankings while keeping the original core and support set sizes fixed. Core membership remained substantially reproducible under both conditions (Table R5; Appendix C.1).

Table R5. Mean Core Overlap at Full and Half Sample Sizes (%).

ModelFull-size samplesHalf-size samples
OLMo2-1B83.179.8
OLMo2-7B80.378.5
Tulu3-8B78.576.0

Core membership remains reproducible with half the samples, and all three trajectories show higher mean overlap under the full-size condition. This consistent pattern supports the reliability of core identification under the examined sampling conditions.

Q5: Are the 50% and 90% coverage thresholds meaningful, and do the findings persist under alternative thresholds or fixed-size definitions?

We use 50% and 90% coverage as common reference levels to summarize contribution concentration within the scanned range and compare tasks and checkpoints (Appendix B.2). To assess sensitivity to these choices, we crossed core thresholds of 40%, 50%, and 60% with support thresholds of 80%, 90%, and 95%, yielding nine settings per trajectory. Across the same 190 primary comparisons, each trajectory retained positive mean core-minus-periphery differences in random-corrected membership overlap under all nine settings (Table R6; Appendix C.1). Equal-size windows of 20 or 25 neurons also showed greater average stability near the top of the ranking than in windows beginning at ranks 100 or 200 (Appendix C.1).

Table R6. Core–Periphery Stability Gaps across Coverage Thresholds (pp).

ModelGap at 50%/90%Gap across nine settings (min–max)
OLMo2-1B11.0310.25–18.31
OLMo2-7B15.4512.73–21.21
Tulu3-8B14.7511.99–19.36

Across the tested definitions, the high-contribution region remains more stable on average, providing consistent empirical support for core–periphery organization.

Dear Reviewer 9UsP, thank you again for your time and constructive feedback. As the discussion deadline approaches, we would be grateful if you could review our responses when you have a chance. If any concerns remain, we would be happy to provide further clarification or address any additional questions. Thank you for your consideration.

Reviewer_ScKTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how task-relevant contributions are distributed across MLP neurons and how this organization changes during post-training. Its framework, Causal Contribution Profiling, ranks neurons by attribution and patches progressively larger sets, measuring their effects against size and layer-matched random controls. Across 31 models and 36 tasks, the authors find that contributions are concentrated in a small, task-selective core, with broader, lower-density support. Across three training trajectories, core membership is generally more stable than peripheral membership, and earlier attribution scores help identify neurons that become important later.

Critical strengths

The strongest aspect is the breadth of the intervention study, covering 31 models from five families and a wide range of tasks. The separation of ranking, profile estimation, and confirmation data, together with size and layer-matched controls, makes the concentration finding more convincing than attribution rankings alone.

Critical weaknesses

My main concern is how much the core-periphery interpretation adds beyond a smoothly declining importance ranking. Smooth profiles are favored in most cases, and the core and periphery are defined through coverage thresholds, so their different stability could partly reflect high-scoring neurons being less likely to cross a selection boundary. A control for ranking noise would help establish whether there is additional organizational structure. The concentration results also cover only approximately 1.6% of MLP neurons at the final prompt position. Although the paper states this restriction, it limits conclusions about the distribution of causal support across the full model, particularly when interactions could change the contributions measured along a fixed expansion order.

Questions for authors

How sensitive are the concentration and membership results to extending the scanned range beyond 1.6% or changing the neuron expansion order?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ScKT [Part 1/2]

We appreciate the reviewer's positive assessment of our work and the constructive suggestions for clarifying its interpretation and strengthening the evidence.

Q1: Does core–periphery organization provide insight beyond a smoothly declining importance ranking, and could stability differences partly reflect ranking noise or selection-boundary effects?

We clarify this interpretation and examine its empirical support through complementary reliability, sensitivity, and functional analyses.

(a) Interpretation of Core–Periphery Organization. The core–periphery pattern first describes concentrated task contribution within the scanned range: a small high-density core accounts for a disproportionate share of measured positive contribution, alongside broader, lower-density support. This is compatible with smooth profiles. We further examine how these important neurons change during training. Tracking the same neurons across checkpoints reveals greater average persistence of core than peripheral membership, functional changes accompanying selected membership transitions, and early signals predictive of new members. “Organization” therefore characterizes empirical relationships between contribution levels, membership persistence, and functional dynamics across training.

(b) Ranking Noise and Membership Stability. We added repeated-identification and matched-input calibration experiments covering 190 task-level comparisons across three trajectories. The results showed that core membership was more reproducible. We therefore compared cross-checkpoint overlap on identical input groups against a within-group checkpoint-label-swap reference, retaining the original region sizes. The mean core-minus-periphery overlap gap exceeded this conditional reference in all three trajectories, with individual 95% confidence intervals for the excess above zero (Table R1; Appendix C.1).

Table R1. Conditional Noise Calibration of Core–Periphery Membership Stability.

Training trajectoryObserved gapLabel-swap referenceExcess gap
OLMo2-1B8.644.594.05
OLMo2-7B13.858.835.02
Tulu3-8B12.287.284.99

(c) Threshold Sensitivity and Functional Evidence. We crossed core coverage thresholds of 40%, 50%, and 60% with support coverage thresholds of 80%, 90%, and 95%, defining the periphery as support outside the core. All nine settings yielded positive mean core-minus-periphery overlap gaps in each trajectory. Equal-size windows of 20 or 25 neurons also showed greater average stability near the top of the ranking (Appendix C.1). Independent D3D_3 interventions on the same selected sets at both checkpoints showed increased normalized contributions in all six incoming-core sets and decreases in all five evaluable exiting sets on Temperature and Emotion across three models. Persistent core sets showed changes in both directions (Appendix C.4).

Together, these findings support empirical links between contribution levels, membership persistence, and functional change under the examined conditions.

AuthorsOpenReview ↗

Response to Reviewer ScKT [Part 2/2]

Q2: How sensitive are the concentration and membership findings to the scanned range, intervention position, and neuron expansion order?

We address this concern through extended-range analyses and clarification of the intervention mechanism.

(a) Contribution Concentration and Stability under Extended Scans. We extended the scan range on the same 21 OLMo2-1B Instruct tasks. The task-averaged D3D_3 curves continued to show strong contribution concentration, with coverage defined within each scan (Table R2; Appendix B.4). Across 90 common longitudinal comparisons from three trajectories, the mean core-minus-periphery overlap difference remained positive at all three scan limits, supporting greater average core stability under the expanded ranges (Appendix C.1).

Table R2. Contribution Coverage under Extended Scan Ranges. (% of all MLP neurons)

Scan limitNeurons for 50% coverageNeurons for 90% coverage
1.60.041201.20010
3.20.072481.99966
5.00.109103.19977

(b) Intervention Position and Expansion Order. Changing the expansion order can alter the intermediate neuron sets, and consequently the contribution profile and coverage-defined memberships. We use task-specific attribution ranking to test whether neurons prioritized on D1D_1 also carry concentrated causal contributions on independent D2/D3D_2/D_3 data. Joint interventions include neuronal interactions, so density describes conditional increments along this ranking. The final prompt position provides a common, answer-relevant location for comparing tasks and checkpoints. Within this protocol, we identify task-selective support, functional changes accompanying selected membership transitions, and early signals predictive of new members. These findings extend static contribution localization to a longitudinal characterization of how task support is maintained and remodeled during training.

Together, these results support concentrated task contributions and greater average core stability over the tested scan ranges, providing an empirical basis for studying task-support organization and remodeling under the specified intervention protocol.

Reviewer_ScKTOpenReview ↗

Response

I thank the authors for their thorough and thoughtful rebuttal. My concerns have been satisfactorily addressed, and I have therefore decided to increase my overall rating.

Thank you very much for your recognition of this work. Your constructive comments have greatly helped improve this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.