acceptodds
Under review as a conference paper at ICLR 2027

Decodable But Not Detachable: Training Data Granularity Determines Parametric Modularity in Large Language Models

Abstract

Do large language models contain domain-specific parametric shells: concentrated, causally necessary neuron populations whose removal selectively degrades a target domain while sparing others? We apply a uniform causal methodology across two domain granularities, three model families (1.5B to 7B parameters), and eight domains. At the academic subject level, zero neurons exceed 60% domain selectivity across 939,008 combined FFN neurons and causal damage matrices are flat, despite domain identity being linearly decodable above 85% accuracy. At the language and modality level, 0.65–1.14% of neurons exceed 60% selectivity, damage matrices are near-perfectly diagonal (ratios up to 595:1), and shell neuron sets are essentially disjoint (IoU < 0.003). Masking code-selective neurons reduces mathematical reasoning accuracy by 16–24 percentage points across all models; chain-of-thought prompting recovers up to 75% of this damage, revealing that the code shell carries both an answer- structuring function and an irreducible reasoning function. Masking Spanish or Chinese neurons leaves mathematical reasoning at or below random. Shell causal strength increases monotonically with model scale, shells are spatially interleaved in a pattern that precludes group-level selective quantization, and amplification of shell neurons degrades performance monotonically, confirming that shells operate at a precisely calibrated functional point. Parametric shells form where and only where training data was modular at the token level.

Primary area:
interpretability and explainable AI
Keywords:
large language models, parametric modularity, language-specific neurons, domain selectivity, causal intervention, multilingual representations, model interpretability, FFN neurons, knowledge localization, training data distribution, chain-of-thought reasoning, model compression, granularity, neuron masking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PdZmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

I think (but I'm not sure) that the paper aims to measure whether the capabilities of an LLM (across different domains or languages) are tied to specific populations of neurons of their MLP blocks, in the sense of these neurons being necessary for only 1 domain at a time (meaning that they are necessary for that domain and not others).

The contribution/novelty (which is not stated clearly) seems to be to conduct such an analysis more thoroughly than prior work.

Critical strengths

(S1) The experiments cover multiple dimensions: 3 models from different families, 2 granularities of domains (subjects/languages).

(S2) The findings complement those of prior smaller-scale studies of the topic.

Critical weaknesses

(W1) The paper is generally poorly written and organized. The motivation for the research questions is not even clearly established. The first paragraph of the introduction launches into a full-on survey of the prior work without explaining why we even care about what-neuron-does-what. The term "parametric shells" appears out of nowhere and I'm still not sure what it means exactly (what's a "shell" here?).

(W1b) The language used in important technical descriptions is also (seemingly) unnecessarily complicated, to the point that I'm not sure I understand what the authors mean. For example: the "concentrated, causally necessary neuron populations whose removal selectively degrades a target domain while sparing others". Does this just mean "neurons necessary to only one domain*"?

(W2) The methodology seems to be the same as in prior work (see citations L80). Two pages (Section 3) describe the methodology, which is fine to make the paper self-contained, but it's not clear if any of it is novel. If notm it should be more clearly labeled as background/review material.

(W3) The paper claims to "assess when parametric shells exist and when they do not". I guess that the culmination of this quest is the final section titled "The Granularity Principle"? The story about "non-overlapping regions of the token distribution" and "pushing the gradients" is very hand-wavy. I'm not saying that the findings are not valid, but the observations made on pretrained/instruction-tuned models do not seem sufficient to really establish the conditions under which the observed organization of neurons can appear.


Minor comments:

  • The term "modalitiy" to refer to language vs. GSM-8k: I don't think this is a good qualified. A different "modality" usually refers to a different type of input like images in vision-and-language models. The contents of GSM-8k are full English sentences.

  • Wouldn't Section 4.7 be easier to follow with plots rather than listing numbers in the text?

  • The word "null" is used multiple times in a way I don't understand. E.g. "the subject level null", "without a null condition", etc. Do you mean a "null hypothesis"? Either way this is one example among many things that make the paper hard to follow.

  • The writing mostly grammatically fine on the surface, but the underlying structure and organization of ideas is really messy. I don't know if this is the reason, but in my experience this is what usually happens in LLM-generated text (rather than LLM-edited text).

Questions for authors

I don't have precise questions and I don't anticipate to raise my evaluation of the paper.

The paper, in both subject and form is at the level of a workshop paper at best.

This could work can be part of a future good paper, but much more work would be necessary. As a constructive comment, I would suggest the authors to first reflect on why do we even care about measuring the "parametric modularity" in LLMs? The answers to this question could guide the future work that could be done around the current experiments to expand them into a future heftier paper.

Reviewer_68aWOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether large language models contain domain-specific parametric shells, i.e., concentrated populations of FFN neurons whose removal selectively degrades performance in a target domain while largely preserving other domains. The authors evaluate three instruction-tuned models ranging from 1.5B to 7B parameters across academic subject domains and language/modality domains. They identify domain-selective neurons based on activation statistics and assess their functional necessity through causal masking with count-matched random baselines. The results reveal a notable difference between the two granularities: no neurons exceed the 60% selectivity threshold for subject-level domains, despite domain identity being linearly decodable with over 85% accuracy, whereas 0.65–1.14% of neurons exceed this threshold for language and modality domains. Masking language-specific neuron sets produces strongly domain-selective degradation, and masking code-selective neurons reduces mathematical reasoning accuracy by 16–24 percentage points across the evaluated models. Based on these findings, the authors propose that parametric modularity emerges primarily when training data is partitioned at the token level.

Critical strengths
  1. The paper provides a systematic comparison between subject-level and language-level modularity under a relatively consistent experimental framework. The negative result at the subject level is particularly interesting, as it distinguishes the presence of decodable domain information in representations from the existence of functionally specialized parameter subsets.

  2. The authors go beyond activation-based neuron identification by conducting causal masking experiments with count-matched random baselines and cross-domain damage matrices. The additional analyses of neuron-set overlap, mask-size sensitivity, model scale, and mathematical reasoning provide useful evidence about the functional roles and organization of the identified neuron populations.

Critical weaknesses
  1. The related-work discussion is incomplete. In particular, https://arxiv.org/abs/2510.10238 studies the causal identification of critical neurons and reports that perturbing a small subset of neurons can cause severe performance degradation, with additional analyses of their layer-wise distribution and scaling behavior. This work is closely related to the present paper's investigation of causally important neurons and their functional organization, but it is not discussed. The authors should clarify how their definition of domain-specific parametric shells, neuron identification procedure, and causal evaluation differ from prior work on critical neurons, and explain more explicitly what methodological or conceptual advances this paper contributes beyond existing findings.

  2. The central claim that parametric shells form specifically when training data is modular at the token level is stronger than what the current experiments directly establish. The comparison between subject-level and language-level domains involves different datasets, input formats, evaluation metrics, and potentially different training-data distributions. These factors may contribute to the observed differences in selectivity and causal effects. Although the results are consistent with the proposed granularity principle, they do not directly isolate training-data granularity as the causal factor. The conclusion should therefore be qualified, or supported by more controlled experiments.

  3. The robustness and generality of the findings remain unclear. Neuron identification relies primarily on mean activation magnitudes, a fixed selectivity threshold, and a particular shell-selection procedure. Although the authors report results across several mask fractions, it remains unclear how sensitive the main conclusions are to alternative neuron attribution methods and selectivity criteria. Moreover, the scale analysis compares three different model families, making model size difficult to disentangle from architecture, tokenizer, and pretraining-data differences. The quantization analysis also does not establish practical selective-compression benefits, as the proposed group-level selective conditions yield no qualifying groups under the chosen configuration.

Questions for authors
  1. The authors should provide stronger evidence for the proposed granularity principle. One promising direction would be to conduct controlled experiments in which training or fine-tuning data is partitioned at different granularities while keeping the model architecture and other experimental factors as consistent as possible. Alternatively, the authors could use more closely matched datasets and evaluation settings to reduce potential confounds between subject-level and language-level domains. These experiments would help determine whether token-level data modularity is the underlying cause of the observed parametric structure rather than a correlate of other differences between the evaluated domains.

  2. The authors should conduct additional robustness and scaling analyses. These could include varying the selectivity threshold, comparing activation-based identification with gradient- or relevance-based attribution methods, and evaluating additional mask fractions and random controls. To strengthen the scaling claim, models from the same family at multiple parameter scales would be preferable. For the quantization analysis, evaluating alternative group sizes or neuron-reordering strategies would help clarify whether spatial interleaving fundamentally limits selective quantization or merely limits the specific group-level configuration considered in this paper.

Reviewer_4iAiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks when domain information in an LLM is supported by selectively necessary neuron populations, rather than merely being decodable from its activations. The authors compare academic subjects with language and code domains across three instruction-tuned models. They identify neurons using activation shares, measure the effects of masking them against count-matched random masks, and examine mask size, mathematical reasoning, and quantization. The main claim is that selective parametric structure appears at the language/modality level but not at the academic-subject level.

Critical strengths

The distinction between decodability and selective necessity is worth studying, and the language-level interventions provide a concrete result. Figure 2 shows a consistent concentration of damage on the targeted domain across all three models, rather than relying on activation preferences alone. The code-mask experiment on GSM8K is also useful: it checks whether the selected population matters outside the corpus used to identify it. These findings give the paper substance even without accepting the proposed explanation of how the populations formed.

Critical weaknesses

I do not think Figure 6 supports the stated subject-level null. The biomedical diagonal is 0.135, 0.189, and 0.189 across the three models, while the other entries in those rows are much smaller. That looks different from the description of flat damage matrices or comparable diagonal and off-diagonal effects. It does not by itself establish a biomedical module, but it needs to be explained before concluding that subject-specific functional structure is absent. The absence of neurons above the 0.60 activation-share threshold does not resolve this discrepancy.

Furthermore, The training-data explanation remains a hypothesis. These are interventions on already trained models; neither training-data granularity nor token overlap is manipulated. The subject/language comparison also changes the source corpora and evaluation measure. The experiments can establish differences in the organization of the tested models, but they do not identify training-data partitioning as the cause. This distinction matters because “determines” and “precisely where” are central to the paper’s contribution, not incidental wording.

Questions for authors
  • How should the biomedical row in Figure 6 be reconciled with the subject-level null? Please provide the underlying accuracies and uncertainty over items and random-mask draws.
  • Is the training-data account intended as an explanatory hypothesis or an experimentally established result? What evidence distinguishes it from differences in the domain corpora and evaluation tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.