acceptodds
Withdrawn from ICLR 2027

Pre-carved Niches: The Formation Dynamics of Modular Task Partitions in Early LLM Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models exhibit task-related modular organization. To explain why such organization is already visible before training, we examine its origins and subsequent evolution across multiple model families and parameter scales, using tasks spanning language, mathematics, and physics. Neuron groups are identified by their estimated contributions to task predictions, and their patterns of sharing and separation are tracked during training. Reproducible patterns are present before any parameter update. Controlled experiments further show that changing input structure or prediction targets alters the associated neuron groups and their overlap while model parameters remain fixed. Task characteristics thus already shape how neurons contribute to predictions in an untrained network, producing structured patterns of sharing and separation across tasks. During subsequent training, these relationships evolve in ways that differ across models. These findings establish the role of task characteristics in initial task organization and explain why modular structure can be visible before learning begins.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Mechanistic Interpretability, Training Dynamics, Functional Modularity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wNshOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This submission studies how the initialization of the network and the training affect the task-related neuron organization in large language models (LLMs). The authors first study the networks with initial weights, while manipulating the words in the input frame or the candidate answers. It was reported that both input context and prediction targets influence the task-related neuron organization. The authors also studied these factors affect neuron organization early in the training.

Overall, the paper studies a meaningful question and some of the observations reported seemed promising. However, the results are too preliminary and the conclusions are premature. The writing is messing at the moment. The study is not as well motivated, and not well grounded on prior work. Some heavy-lifting work would be needed for turning this work into publishable quality.

Critical strengths

None.

Critical weaknesses
  1. While I find that the topic is be relevant and some of the analyses are interesting, the results are too preliminary and the results are not carefully interpreted. Additional control analyses would be needed to better understand these results. The study only concerns the initial state of the network and early in training, what happens towards later phases of the training and how that relates to the organization in early training wasn’t clear.

  2. The writing needs major improvements. Right now, the submission reads like a preliminary technical report.

Questions for authors

Can the authors provide a formal argument for the similarity of the patch attribution and loss gradient shown in Fig. 1?

What was the interpretation for plots in the Fig. 4? It appeared that the traces were basically flat.

Reviewer_XFSrOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper claims that task-dependent neuron organization can emerge from the interaction between a randomly initialized network, its input context, and the prediction being evaluated. And these initial groups, defined as pre-carved niches, do not necessarily represent learned task-specific circuits.

  • The authors examine feed-forward neurons using attribution scores that estimate their contributions to predictions. Their primary technique combines activation differences between clean and corrupted inputs with gradients of an output probability contrast. For each task condition, they select the top 1% of positively attributed neurons and compare overlap between the resulting groups. Their experiments cover a detailed initialization analysis of Pythia 2.8B, comparisons across seven model configurations, and a longitudinal analysis of Pythia 410M across 165 checkpoints.

  • Even a randomly initialized model exhibits structured differences in which neurons receive high attribution scores. This suggest, initialization has some interesting influence on the training trajectory of the model. The paper claims that input context affects neuron selection, since replacing three words in the input frame reduces the overlap with the original neuron group.

  • The prediction targets have a particularly large effect since changing candidate answers while preserving the exact prompt reduces group overlap when the answer vocabularies are disjoint.

  • Also the authors find that continuous neuron attribution vectors can evolve smoothly, even when the identities of top-ranked neurons change significantly. So the training trajectory reshapes the initial organization.

Critical strengths
  • The method of holding the prompt fixed while changing the answers freezes the activation-difference term at the final position. These experiments demonstrate that neuron attribution depends strongly on which prediction is evaluated, even when the model's weights and forward activations remain unchanged.
  • The authors reproduce the qualitative distinction between input-context and prediction-target changes across seven model configurations, including Pythia, OLMo-2, BLOOM, and OpenELM.
Critical weaknesses
  • I think the central findings are confounded by the defined neural attribution method. The attribution is defined as the activation difference times the gradient of the prediction score. So, since both of these components are task-dependent, wouldn't it mean that perturbing any of these components affects the attribution score? It remains unclear whether the observed changes in neuron groups reflect meaningful task-specific organization or simply arise from the attribution method itself. I suggest including additional baseline comparisons to establish whether the observed patterns differ meaningfully from those expected under the attribution method.

  • Also, the paper calls the observed structures modular task partitions, but the principal evidence comes from identifying highly ranked neuron-attribution groups. The experiments do not establish that these neurons causally implement corresponding tasks. For example, the authors do not demonstrate that selectively suppressing a task-attributed neuron group produces a greater performance decrease on that task than on unrelated tasks.

  • The paper omits closely related work on structure in untrained transformers, such as Heap et al. and Francazi et al. Also Adebayo et al. can be looked at for randomization-based sanity checks to distinguish meaningful explanations from artifacts of attribution methods. I suggest incorporating these studies into the related work section to better contextualize the findings and clarify the paper's contributions relative to existing research.

    • Heap et al. (2026). Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers. ICLR 2026

    • Francazi et al. (2024). Initial guessing bias: how untrained networks favor some classes. ICML 2024

    • Adebayo et al. (2018). Sanity Checks for Saliency Maps. NeurIPS 2018

Questions for authors
  1. Since neuron attribution directly depends on task-specific activations and gradients, what evidence establishes that the observed neuron groups reflect meaningful organization rather than the expected behavior of the attribution method?

  2. What evidence supports interpreting the top-ranked neuron groups as modular task partitions, given that the reported causal interventions do not establish task-specific functionality?

  3. How were the initialization checkpoints obtained across architectures, and what explains the 165 Pythia checkpoints reported in the paper compared with the standard public release?

Reviewer_Ry9aOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies task-conditioned neuron organization before learning and its evolution during early training. It measures attribution vectors and top-1% neuron-group overlap, separately changing input context and candidate answers at fixed weights. Changing answer vocabulary produces substantially lower overlap than modifying the surrounding words, with the same ordering across seven model configurations. In Pythia 2.8B, continuous contributions respond nearly linearly along the first effective update while thresholded membership changes less regularly. A separate Pythia 410M trajectory, evaluated with independent items, shows selective weakening of a number-sequence direction rather than demonstrated functional transfer.

Critical strengths
  1. The paired manipulations provide a clear way to separate input-context dependence from prediction-target dependence. In particular, changing candidate answers while preserving matched clean and corrupted prompts keeps the activation difference fixed, isolating changes associated with output sensitivity. This is more informative than comparing unrelated tasks and supports the paper's careful distinction between measured neuron organization and learned functional specialization.

  2. Combining continuous vectors with thresholded membership gives a useful account of apparent instability. Along the first update, attribution cosines follow the endpoint direction for 116 of 136 condition pairs, versus only 47 for top-1% overlap. The longer-term analysis evaluates fixed, previously selected directions on independent items using paired bootstrap intervals. These choices reduce the risk of mistaking threshold effects or item-specific variation for functional change.

Critical weaknesses
  1. The practical importance of the spectral component remains limited by its modest gains. Relative to Quality-only and Global Cosine, strict joint success improves by 4.0 and 2.7 percentage points, while mean worst-task gain increases by 0.12 and 0.07 percentage points, respectively (Appendix D.1). The results do not yet establish a substantial benefit over simpler selection under the same budget; a repeated random-pair control would help assess this.

  2. The contribution of spectral information among similarly scored pairs remains unclear. In Equations (6)–(7), the bounded penalty 0.5Ω can alter relative scores by at most 0.5 percentage points, concentrating its effect on close or tied quality scores. Appendix D's paired outcomes and weight sensitivity are helpful, but do not show how often spectral ranking changes the shortlist or whether random tie-breaking achieves similar gains. Section 4 explicitly does not derive the spectral score from its interaction analysis, leaving its empirical role insufficiently explained.

Questions for authors
  1. Relative to Pochinkov et al. (2024) and Zhang et al. (2023), what is the main new finding? Please identify the uncertainty resolved by the paired manipulations and the empirical patterns that go beyond the input and target dependence already implied by the attribution definition.

  2. Why do the Pythia 2.8B overlaps differ between Table 2 (77.02%, 14.34%) and Table 3 (78.0%, 14.0%)? Were initialization and stimuli identical?

  3. Is corresponding training-dynamics evidence available for another model family or a larger model? Please distinguish conclusions supported only by the two Pythia configurations from those supported across architectures, and state the evidence for any claimed generalization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.