acceptodds
Under review as a conference paper at ICLR 2027

Sparse Cross-Layer Modules: A Mesoscopic View of Language Model Organization

14 views

Abstract

Interpreting a language model requires connecting individual response patterns to the organization of representations across the network. We study sparse cross-layer modules as a mesoscopic unit for this purpose. Joint sparse coding associates each within-document token response with a decoder direction across the layer stack. In Pythia-410M, the resulting population exhibits depth-dependent differentiation: salient decoder coordinates concentrate in adjacent-layer bands, and response breadth, lexical coherence, and description scores vary across the resulting depth groups. Responses remain heterogeneous within groups, while response similarities extend across modules with little salient-coordinate overlap. Across text domains, the same dictionary exhibits shared usage with distinct preferences, and independently fitted dictionaries retain partial response correspondence. Dictionaries fitted at successive training checkpoints show an early increase in best-match response similarity and a more gradual increase in salient-coordinate overlap. Across three Pythia sizes, matched response subsets retain substantial similarity at comparable standardized-space reconstruction quality. Automated descriptions, activating examples, and selected-module interventions connect these population measurements to inspectable patterns and local output effects. Together, the analyses characterize language-model organization through the distribution of individual responses, their relationships within a population, and their recurrence across inputs and model conditions.

Primary area:
interpretability and explainable AI
Keywords:
sparse cross-layer module, language models, representations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SAPVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper learns a sparse dictionary over activations from all layers of an LM and studies responses across depth, text domains, training stages, and model sizes. Most analyses use document-standardised Pythia activations and describe variation within complete documents. The authors check depth patterns in other model families and use selected interventions to examine local output effects.

Critical strengths
  1. Coarse depth profiles recur across normalisation and decomposition choices (§3.2). Repeated fits recover similar layer allocations on the smaller, 100000-token fitting panel. Individual responses are dictionary-dependent, with only partial correspondence on held-out documents (Appendix M, Table 12).

  2. I support weak acceptance based on the descriptive findings and repeated-fit results, independently of the interventions. Shallow modules have narrower lexical responses, while deeper modules have broader responses (Fig. 3). During training, correspondence to the final checkpoint increases even with 1-to-1 matching and equal candidate counts (Table 23).

Critical weaknesses
  1. Missing or unclear protocol details limit the weight I can give to cross-scale correspondence and module-specific local output effects. Appendix AA compares matched dense responses at similar standardised-space reconstruction quality, but does not specify the matching/evaluation split, shuffle construction or pairwise results. Appendix AF reports separate discovery/evaluation documents, discovery-defined signatures, norm matching and same-depth controls. Key intervention, selection and statistical details remain unclear. The interpretation supplement adds no protocol details.

  2. The supplied records show lower Fuzzing-scores for deep modules. The description-score summaries cannot be checked exactly, because the interpretation supplement contains 927 unique module records and appendix Q reports 921. Some labels also differ from the scored labels in appendix F, for example, FM40.

Questions for authors
  1. Which documents were used to choose matches and which were used to test them in appendix AA? How were the shuffled controls made? For each model pair please report the mean correlations and the fraction of modules covered by the matches.
  2. Please specify the intervention operation and location, signature statistic, target/control selection, statistical test and per-target effect sizes, so the strength and specificity of local output effects can be measured.
  3. Which module IDs, group labels and scores were used for appendixes F, Q, R? Please explain the differences from the supplied interpretation collection so I can check the description-score summaries.
5 comments

W1a Appendix AA compares matched dense responses at similar standardised-space reconstruction quality ...

We used one set of documents to find modules with similar responses in two models, then tested those pairs on a different set of documents. In the supplementary experiment, all three Pythia models read the same 64 documents. On these documents, we selected dictionaries with comparable reconstruction quality and matched modules one to one for each model pair according to how strongly they responded at different positions. We then fixed the matches and tested them on another 64 documents. Both sets were separate from the 32 documents used to train the dictionaries.

We sampled 32 token positions per test document. In the aligned comparison, we compared the first sampled position in one model with the first in the other, the second with the second, and so on. For the control, we shifted all module responses in the second model together by 1 to 31 sampled positions within each document, wrapping around at the end. The paired modules and documents stayed the same, while the token positions were offset. We ran 99 such controls.

Model pairMean response correlation at aligned positionsMean response correlation after shiftingModules matched / total in each model
70M–410M0.2749−0.00161,024/1,024 (100%)
70M–1.4B0.2171−0.00081,024/1,024 (100%)
410M–1.4B0.2786−0.00151,024/1,024 (100%)

All three model pairs showed the same pattern. Even on new documents, some cross-layer modules responded more strongly or weakly at similar positions in both models. Shifting the positions brought the mean correlation close to zero. All three model pairs had higher response correlations at aligned positions than in the shifted control, supporting the conclusion that some cross-layer responses to the input recur across model sizes.

W1b Appendix AF reports separate discovery/evaluation documents ...

Thank you for pointing out that the intervention details were insufficiently described. We have added the operations, selection rules, statistical definitions, and per-module results so readers can check the magnitude of prediction changes associated with module-related activity and whether these changes recur on new documents.

To address the signature question, we added a 24-module analysis. We randomly sampled eight modules with sufficient activating documents from each of the shallow, middle, and deep groups. At a token where a module was active, we zeroed the activations at coordinates with prominent decoder weights in the raw outputs of the associated layers. Four random controls modified the same number of other coordinates at the same layers and token position, matching the L2 norm of the actual perturbation. These controls were used to estimate the additional output change caused by the target intervention relative to generic perturbations.

The “signature” records how each candidate token’s log probability rises or falls in this additional change. We used four discovery documents per module to determine its average change direction, then conducted 93 evaluations on 88 documents disjoint from discovery. We compared each new change direction with the module’s own signature and with the signatures of the other seven modules in the same depth group. For 23 of the 24 modules, mean similarity was higher to their own signature, with a module-equal mean difference in cosine similarity of 0.1328. Randomly reassigning signature labels within depth groups 1,999 times yielded lower mean differences in every case, providing an exploratory mismatching reference. These results support cross-document recurrence in some of the prediction changes elicited by perturbing module-related activity.

A separate 96-module experiment directly measured prediction loss for the correct next token. In addition to the 24 modules above, we randomly sampled 24 more modules from each depth group. For 128-token passages where a module was active, we subtracted the activity component it represented at a preselected layer at the final token position. The selected layer had the largest sum of squared decoder weights. The component was computed from the current activation strength and decoder weights. For the final layer, we used the output of the final layer normalization. Four random-direction controls matched the actual perturbation magnitude at the same layer and token position.

For each module, the effect size was the additional prediction-loss increase under the target intervention relative to the mean of the four controls, averaged across that module’s samples. Across 1,488 paired comparisons from 661 documents, we first averaged within each module and then averaged equally across the 96 modules. In this sample, removing the activity component represented by a module at the selected layer increased prediction loss for the correct next token by 0.0251 nats more, on average, than a random perturbation of the same magnitude; the difference was statistically significant (two-way clustered test, two-sided p=0.0083). Per-module results for the two experiments are listed separately in “24-Module Signature Analysis Details” and “96-Module Prediction-Loss Intervention Details,” which report the intervention sites, sample counts, target and control results, and differences.

W2 The supplied records show lower Fuzzing-scores for deep modules ...

We apologize for the inconsistency. Multiple versions were maintained during our multi-author collaboration, and we did not consistently reconcile the interpretation supplement with the score tables when assembling the materials. We have now unified the descriptions and scores using the original scoring outputs and aligned the group labels with the original clustering results.

FM40 belongs to the Deep group, with Detection 0.73 and Fuzzing 0.72. We have restored the six scored modules omitted from the previous score table and corrected the relevant labels, descriptions, and scores. The unified records contain 927 scored modules: 923 belong to the three main shallow, middle, and deep groups, while four belong to the residual group in the original clustering and are listed separately.

Recomputing the summaries produces only small changes in the three groups’ mean Fuzzing scores. Deep remains below Middle and Shallow, preserving the previously reported score pattern.

Depth groupPrevious score summaryUnified summary
Deep0.58890.5907
Middle0.62370.6244
Shallow0.63060.6307

The unified record table provides each module’s description, group, and both scores. The supplement and summaries use this common record table, allowing each entry to be checked.

Q1 Which documents were used to choose matches and which were used to test them in appendix AA ...

The document splits, construction of shifted controls, mean response correlations for each model pair, and matched-module coverage are specified in our response to W1a. For all three model pairs, correlations at aligned positions on independent test documents exceed those of the shifted controls, with all 1,024 of 1,024 modules from each model included in matching (100% coverage). The numerical results are provided in the table in W1a.

Q2 Please specify the intervention operation and location ...

The intervention operations and locations, signature definition, selection of modules and controls, effect-size calculations, and statistical analyses are described in W1b. The 24-module signature analysis supports cross-document recurrence in some module-related prediction changes. The 96-module experiment measures the change in correct-next-token prediction loss after subtracting the activity component represented by a module at the selected layer.

Q3 Which module IDs, group labels and scores were used for appendixes F, Q, R ...

The record differences and corrections are described in W2. The unified records are linked by FM ID throughout. Group labels follow the original clustering, and descriptions and scores come from the same original scoring output.

AppendixRecords used
FIDs and depth labels of the 925 modules in the original clustering’s main groups.
QDetection and Fuzzing scores for 927 complete scoring records. Comparisons among the shallow, middle, and deep groups use 923 of these modules; the other four are listed separately.
RThe same 923 main-group modules, joined to activation frequency by ID, using the same labels and scores as Q for the adjustment analysis.

FM221 and FM644 in F have no complete scoring records and are therefore excluded from score summaries. The other 923 main-group modules have consistent labels across F, Q, and R. The unified records list each module’s group, description, both scores, and activation frequency for checking.

Reviewer_b8vWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies "sparse cross-layer modules" as a mesoscopic unit for interpreting language models. Residual-stream states from all layers are concatenated per token (24,576 dimensions for Pythia-410M), standardized within each document, and decomposed with a tied TopK sparse dictionary (M = 1024, K = 32), so that each module pairs one sparse response with a full-stack decoder direction. Modules are characterized by depth profiles (counts of salient decoder coordinates per layer), response correlations, salient-coordinate overlap, and automated descriptions scored by Detection and Fuzzing. The authors report that modules cluster into shallow, middle, and deep templates with differing response breadth and lexical coherence, that response similarity extends across modules with little coordinate overlap, and that related modules recur across 19 Pile domains, independently fitted domain dictionaries, 35 training checkpoints, and Pythia models from 14M to 12B. Selected-module interventions on six modules are used to connect modules to local output effects.

Critical strengths
  1. Concatenating all layers into one sparse code is a well-motivated design for the residual stream (Sections 2, 3.1). Because each layer adds an incremental update to the residual stream, representations change slowly across depth (adjacent-layer CKA averages 0.897), and the same information is spread over many layers. Layer-local SAEs must re-learn and then chain such features, and the paper shows this chaining breaks down (adjacent-layer code correspondence r = .405, but only r = .0085 after chaining layers 1 to 24). Fitting one dictionary on the full stack instead learns each feature directly as a cross-depth direction, which is a sensible unit for interpretability.

  2. The coarse depth organization is robust to many analysis choices (Section 3.2, Appendices H–K). The shallow/middle/deep ordering recurs across a 144-setting sweep of width, sparsity, saliency threshold, and cluster count, under PCA, under energy- and norm-based profiles, and in four auxiliary model families. This makes the depth templates unlikely to be an artifact of one fitting configuration.

Critical weaknesses
  1. The paper is descriptive, with no testable claim, and its novelty over existing cross-layer methods is unclear (Sections 1, 5). The contributions are phrased as "characterizing" organization, and none of the analyses tests a hypothesis or shows something that existing tools could not. The method is closely related to Sparse Crosscoders (cited as [7]), differing mainly in full-stack concatenation and per-document standardization, yet there is no direct comparison with crosscoders, RouteSAE, or chained layer-local SAEs on any task. Several findings (depth gradients, adjacent-layer similarity, domain-dependent feature usage) are also expected from prior work. This matters because, without a clear question or baseline, it is hard to tell what the reader learns about language models that was not already known.

  2. The links to model behavior and the cross-scale claims rest on weak evidence (Sections 3.4, 4.3, Table 2). The intervention covers only six selected modules, and next-token logit and log-probability changes are not statistically significant. Interpretability is assessed only by automated LLM scoring, with Detection/Fuzzing medians around 0.58–0.63. For scale, the 6.9B and 12B dictionaries explain only about a quarter of standardized variation (EV .2825/.2446), so cross-scale correspondences describe a small fraction of each model's representation. This limits the claim that modules reveal how language models "distribute, differentiate, and reuse" their representations.

Questions for authors
  1. What is the main finding about language models that this framework reveals and that crosscoders or chained layer-local SAEs could not? Can you provide a direct comparison (e.g., reconstruction, interpretability scores, or intervention effects) against a standard crosscoder on the same data?
  2. Can the intervention be extended beyond six hand-selected modules, for example to a random sample of modules from each depth group, with effects on next-token predictions reported against the matched random controls?
  3. Given that the 6.9B and 12B dictionaries explain only about a quarter of standardized variance, how should the cross-scale results be interpreted? Do the conclusions change with larger M for these models?
  4. The paper relies heavily on appendices (A–AI) and on descriptive phrasing. Stating two or three explicit claims in the introduction, each tied to one experiment and a baseline, would make the contribution much easier to assess.
4 comments

W1 The paper is descriptive, with no testable claim ...

We apologize that the original manuscript did not clearly explain the purpose and contribution of this study. We seek to understand how the joint activity of a language model’s layers reflects functional differentiation and relationships among responses as it processes language. Examining individual neurons or features can tell us which inputs each is sensitive to, but it remains difficult to understand how the model as a whole processes language from a large collection of local descriptions. Our contribution is to use cross-layer functional modules as units of interpretation, allowing a particular language response to be understood within the whole network through its linguistic preference, its distribution across layers, and its relationships to other responses. We build on Lindsey et al.’s cross-layer sparse coding to carry out this study. Method comparisons on the same data examine which text responses remain recognizable when a different analysis tool is used.

The established sensitivity of early layers to lexical information provides a reference for assessing this perspective. Module-level analysis further reveals differentiation within that sensitivity. An individual shallow-biased module often favors a group of related words, while different modules have different preferences (Fig. 3). As the model reads a passage, these modules also exhibit relationships among their responses. Some share few salient coordinates, yet their responses rise and fall together (Fig. 4). This lets us understand responses to particular words and phrases as cross-layer activity with differentiated preferences and related responses. Looking back over training, responses to text approach those of the final model relatively early, while salient-coordinate overlap increases over a longer portion of training (Fig. 9), further revealing how this organization develops. The interpretive contribution of this work is therefore to connect concrete language content with the model’s overall functional organization through these inspectable responses.

W2 The links to model behavior and the cross-scale claims rest on weak evidence ...

Thank you for this suggestion. To test the connection between cross-layer functional modules and actual predictions more directly, we redesigned the intervention experiment. We selected 32 modules from each of the shallow, middle, and deep groups in Pythia-410M. Using each module’s response strength in the current text and its decoder direction, we calculated its activity component at a preselected layer, then subtracted this component and measured the change in the model’s prediction of the correct next token. Random controls applied perturbations of the same magnitude to the same text, token position, and layer.

Across 1,488 paired comparisons involving 661 documents, we first averaged within each module and then averaged equally across the 96 modules:

InterventionMean increase in prediction loss for the correct next token
Remove the module’s activity component at the selected layer0.0328 nat
Matched random perturbation0.0076 nat
Difference0.0251 nat

An increase in prediction loss following removal of the module’s activity component at the selected layer means that the model assigns a lower probability to the correct token. In this sample, removing the activity component represented by a module at the selected layer increased prediction loss for the correct next token by 0.0251 nats more, on average, than a random perturbation of the same magnitude; the difference was statistically significant (two-way clustered test, two-sided p=0.0083). This connects module responses recognizable in text to the predictions the model actually makes.

The modules’ linguistic preferences are supported by the passages that activate them and by lexical-coherence analysis. Detection and Fuzzing further test whether we can describe these preferences accurately in words. Checking the automated judgments against actual activation records makes the interpretation of what a module prefers open to verification.

To test whether the descriptions apply to new text, we generated descriptions for 96 modules using 64 documents and evaluated them on another 128 documents, with 32 test passages per module. In the central-token tasks, with equal numbers of active and inactive examples, Detection and center-marked Fuzzing achieved accuracies of 54.75% and 64.06%, respectively, both above the 50% constant-prediction reference. This shows that some of the modules’ linguistic preferences can be summarized in words and used to predict responses to new text. Reading the descriptions alongside the passages that actually activate the modules allows readers to understand their different linguistic preferences and check the evidence behind those interpretations.

The cross-scale analysis further examines whether these input responses recur in models of different sizes. EV measures the standardized activity variance covered by a dictionary; we compare the response patterns recovered by those dictionaries. For 6.9B and 12B, we conducted paired width comparisons with equal training exposure, increasing dictionary width from 1,024 to 2,048. All three initializations used the same set of 64 training, 8 validation, and 32 test documents, with 512 updates per fit. Increasing width raised explained standardized test variance by approximately 0.025 for 6.9B and 0.023 for 12B.

Dictionary widthMean test response correlation between matched 6.9B and 12B modules (range across three initializations)
1,0240.356–0.358
2,0480.312–0.315

At both widths, response correspondence exceeded the reference obtained by shifting token positions. Thus, some input responses recur across model sizes at different dictionary capacities. These shared responses provide comparable clues for understanding models of different sizes.

Together, these results connect linguistic preferences in concrete text, internal activity components, and their effects on predictions within the same module-level perspective, while examining whether some responses recur at other model sizes. They support the use of cross-layer modules to understand functional differentiation and response relationships in language models.

Q1 What is the main finding about language models ...

We apologize that the original manuscript did not clearly explain the relationship between our research purpose and the tools we use. We build on existing cross-layer sparse coding. Our contribution is to use cross-layer functional modules as units of interpretation to understand functional differentiation and response relationships within the model. The findings discussed above reveal distinct linguistic preferences within the same depth region, as well as response relationships spanning different depth regions.

We compare decomposition methods to examine whether other methods recover similar responses and distributions, and to help select a suitable tool for this interpretability study. We performed direct comparisons using the same model activations, document splits, and dictionary width. Module correspondences were determined on the validation set and then evaluated on independent test text:

Representation compared with our modulesMean response correlation of matched modulesMean layer-profile similarity of matched modules (cosine)
TopK-SAE with independent encoder and decoder weights0.59190.9745
ReLU crosscoder (supplementary fit)0.46580.9540

Response correlations are computed from module scores before TopK or ReLU is applied, measuring whether they vary similarly over the text; layer-profile similarity compares the proportion of squared decoder-weight energy in each layer, measuring whether the modules have similar distributions across the network. These comparisons show that other decompositions recover some similar text responses and their cross-layer distributions, providing a check on the regularities we observe.

We also chose TopK-SAE for the directness of its presentation: activity at each token is represented by a small set of active modules. This makes it straightforward to return to particular words and phrases, explain what these cross-layer activities prefer and how their responses relate, and use them to study the model’s functional organization.

Q2 Can the intervention be extended beyond ...

Following your suggestion, we randomly sampled 24 additional modules from each of the shallow, middle, and deep groups. As reported in our response to W2, across the expanded sample of 96 modules, subtracting a module’s activity component at the selected layer increased next-token prediction loss more, on average, than matched random perturbations did.

Q3 Given that the 6.9B and 12B dictionaries explain only ...

The functional organization of models of different sizes has shared aspects that can be identified through their responses to language. Even when their layer counts and widths differ, we can relate responses to words and phrases identified in one model to cross-layer activity in another. This gives our interpretation of one model concrete leads that can be checked in other models. As reported in our response to W2, wider dictionaries describe more activity, while this cross-scale commonality remains observable. Thus, the use of cross-layer modules to understand some shared language responses is supported at both dictionary capacities. The degree of correspondence varies with dictionary capacity, and this interpretation concerns the responses extracted by the dictionaries.

Q4 The paper relies heavily on appendices (A–AI) ...

Thank you for the suggestion. In the introduction, we will make the paper’s central question explicit: starting from the joint activity of the model’s layers, we use cross-layer functional modules to understand functional differentiation and response relationships within the model. We will then connect the main findings directly to the corresponding experiments and controls.

Main finding stated in the introductionCorresponding experiments and controls
Modules have different linguistic preferences, and response relationships also span different depths.Activating passages and lexical coherence characterize the preferences. Equal numbers of candidate modules are used to compare response relationships within and across depth groups.
Different domains use many shared modules, with different emphases.We compare the use of a fixed dictionary across domains. Response correspondence between independently fitted domain dictionaries is assessed relative to repeated fits within the same domain.
Response patterns develop over training, and some responses also recur across model sizes.Training comparisons control the number of candidate modules. Scale comparisons include token-position shifts and paired dictionary-width controls.

We will bring the key evidence and controls together in the main text and point to the corresponding results from the introduction, so readers can see how the experiments jointly address the model’s internal functional organization.

Dear Reviewer b8vW

Thank you for your careful review and helpful suggestions. As the discussion period draws to a close, we wanted to briefly follow up and ask whether you have any remaining questions or concerns. We would be happy to provide further clarification. If our responses have resolved your concerns, we would be grateful if you could consider revisiting your assessment and score when convenient. Thank you for your time and consideration.

We sincerely look forward to your feedback, and thank you for your time and effort throughout this process.

Best regards,

Authors of submission 56577

Reviewer_RxWzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper uses an interpretability method called sparse cross-layer modules to examine how feature representations are organized in LLMs. This method looks for interpretable, high-level features in the model's activations, not just in a single neuron or layer, but across all layers.

Having extracted a set of 1024 such high-level features (which they call "modules"), the paper shows that they cluster into three groups according to which layers of the network they are most associated with, the early, middle, or late layers. They then interpret the semantic meaning or role of each module with the help of an LLM.

It then presents a few experiments about these modules, seeking to address questions like

  • Are certain modules associated with particular text domains (code vs Chines)?
  • How does the number of active modules change during training, or between models of different sizes?
  • Do models of different sizes use some semantically similar features
Critical strengths

It sure is thorough. There are lots of different experiments and most of them come with ablations (e.g. the sweep in Appendix H). That's why there are 35 appendix sections.

Critical weaknesses

It's murky what main problem the paper is trying to solve or what main question it is answering. It does not seem to be claiming a novel method exactly, and for all the experiments I can't say what the headline finding is. It doesn't seem to add up to a single coherent argument.

The presentation is quite opaque. Even after reading for many hours, whole paragraphs are lost on me. Key parts of the methodology ("Detection and Fuzzing") are simply not described in the main text, and even in the appendix the description is too terse for me to understand. Results are reported ("Detection and Fuzzing correlate at r = 0.5603") without interpretations or implications (does this mean these two scores are very consistent with each other? So what?) Some findings don't seem very significaint or consistent ("The distributions overlap substantially" on line 253 but there is an "association between depth group and description predictability" on line 262?) At times, it reads more like a log of experiments tried, rather than a curated report with an argument aimed at an external reader.

Questions for authors
  1. If a guy in the elevator asked you what you learned from doing this project, how would you answer in a couple sentences?

  2. If each of your functional modules is indeed associated with either early, middle, or late layers, what does that tell us? Is this simply because activations do not change much from one layer to the next due to residual connections? If PCA of the activations recovers similar results (line 202), then what new perspective do the cross-layer modules give?

  3. Many of the results attempt to show how modules associated with shallow layers are capturing different kinds of information from those of the deep layers. Does this simply reflect the known fact that initial layers deal with local, low-level grammar and semantics while later layers process more abstract concepts?

Additional questions

  • Is the methodology of spares cross-layer modules meant to be meaningfully different from, e.g., Lindsey et al.? Is the method part of this paper's contribution, or just the findings?
  • 111: you "select the upper tail", that is, indices for which zm,j>1.65z_{m,j} > 1.65. what about those whose z score is zm,j<−1.65z_{m,j} < -1.65? It looks like Table 9 addresses this but I'm not quite sure how
  • 113: this methodology is definitely reasonable, but also pretty ad hoc. to find the "depth profile" of a module (how much it affects each layer), why not simply loop over the test set and measure, on average, how much each module actually did affect each layer? see also minor questions
  • What conclusions should I draw from the figures on pages 36 to 50?
11 comments

W1a It's murky what main problem ...

What internal functional organization supports language processing in a large language model? Examining individual neurons or features can tell us which inputs each is sensitive to. Yet it is difficult to understand how the model works as a whole from a large collection of such local descriptions. We therefore start from the joint activity of the model’s layers and take cross-layer functional modules as our units of interpretation. We ask how these modules differ in their functional roles during language processing and how their responses relate to one another.

W1b It does not seem to be claiming a novel method exactly ...

Our main finding is that neuronal activity distributed across layers repeatedly exhibits recognizable, structured cross-layer activation patterns. Taking cross-layer functional modules as units of interpretation lets us understand a particular language response within the model as a whole, through the functional differentiation it reflects, its relationships to other responses, and the conditions under which these regularities recur.

  1. The model exhibits detailed functional differentiation and response relationships that span different depths. Shallow layers' sensitivity to vocabulary is an established observation. Cross-layer modules reveal differentiation within this sensitivity. An individual shallow module often responds strongly to a set of related words, while different modules favor different word groups. Across the model, some modules share few salient coordinates yet show similar response fluctuations over the same text.

  2. Different domains reuse many of the same modules, with different emphases. Across 19 text domains, the relative usage of deep modules is comparatively stable. Deep modules that are frequently active in one domain also tend to be frequently active in others. Which middle and shallow modules participate more often varies more with the text domain.

  3. The distinct response patterns seen in the final model develop gradually during learning. Early in Pythia-410M training, module responses most closely resemble a relatively small subset of the final model's responses. As training progresses, more of the final model's distinct response patterns emerge. This connects the response organization observed after training to the learning process. The two aspects also develop on different timescales. Responses to text approach later-training patterns relatively early, while the distributions of salient coordinates across independently fitted checkpoint populations become closer over a longer period of training.

  4. Some response regularities recur despite differences in model depth and width, providing common reference points for understanding models of different sizes. Across ten Pythia sizes from 14M to 12B, some similar cross-layer input responses can be identified. At the same dictionary capacity, larger models have fewer active modules overall, but these modules appear at more token positions on average, showing more frequent pattern reuse. Thus, common response regularities connect networks of different sizes, while their module usage shows different emphases. Additional comparisons at similar reconstruction quality across three model sizes further support this cross-model commonality.

W2a Key parts of the methodology ...

Thank you for pointing this out. We will clarify both tasks in the main text and appendix. Following Paulo et al., we use Detection and Fuzzing to check whether a module's verbal description accurately captures its input preferences. Paulo et al., 2025

Consider a hypothetical example. A module is described as “responding to Arabic numerals.” Suppose its recorded activations show a response at “3” in “Buy 3 apples,” and no response in “Buy a few apples.”

TaskWhat the scoring model judgesCorrect judgments in this example
DetectionGiven the description, predict whether the module activates anywhere in the unmarked passage.Activation in “Buy 3 apples”; no activation in “Buy a few apples.”
FuzzingGiven the description, judge whether the marked content fits it.The mark in “Buy <<3>> apples” fits the numeral description; the mark in “Buy a <<few>> apples” does not.

For each module, both tasks use the same 200 passages, comprising 100 with activation and 100 with no activation anywhere in the passage. Fuzzing marks all genuinely activated positions in an active passage and one randomly chosen position in an inactive passage. We compare the scoring model's judgments with the activation records. Each score is the number of correctly classified passages divided by 200.

Detection checks whether the description identifies the kinds of text that elicit a module's response; Fuzzing further checks whether it accounts for the specific content at the activated positions. In our study, this allows descriptions such as “responds to numerals, pronouns, or a particular language structure” to be tested against actual responses, providing evidence for understanding cross-layer modules' input preferences and functional differentiation.

W2b Results are reported ("Detection and Fuzzing correlate at r = 0.5603") ...

Across all modules with completed scores, r=0.5603r=0.5603 indicates a moderate positive correlation between the two scores. Descriptions that better help judge which texts activate a module (higher Detection scores) also tend to better capture what the module responds to within those texts (higher Fuzzing scores).

Consider the description “this module prefers numerals.” Frequent activation in sentences containing numerals gives us a clue for proposing this interpretation. If the module also repeatedly activates at the numerals themselves, we have more reason to identify numerals as part of its input preference. The positive correlation between the two scores indicates that, for many modules, these two kinds of evidence tend to agree.

W2c Some findings don't seem very significaint or consistent ...

Yes. Many modules in different depth groups have similar scores, and the groups can still have different medians.

Median scoreDeepMiddleShallow
Detection0.6000.62250.585
Fuzzing0.5750.6200.6275

For Fuzzing, the median is 5.25 percentage points lower in the deep group than in the shallow group. This difference describes the typical score in each group. Each group contains both higher- and lower-scoring modules, and many modules from the two groups fall in the same score range, so the distributions still overlap substantially.

W2d At times, it reads more like a log of experiments tried ...

Thank you for pointing this out. In the introduction, we will make clear how the paper's argument starts from the joint activity of the model's layers and uses cross-layer functional modules to understand how language responses differ within the model and how they relate to one another. Section 3 will begin with a few individual modules. For each one, we will show the words and phrases it responds to alongside its distribution across the network. We will then compare how these modules respond to the same passage to show what each favors and which modules have related responses. At the end of Section 3, we will use this account of functional organization to motivate Section 4. Once we have described the division of labor in a trained model, the domain analysis can show how the model draws on these modules when it reads different kinds of text. Looking back over training reveals how these response patterns develop. Comparing models of different sizes shows which patterns recur in other models.

Q1 If a guy in the elevator asked you what you learned from doing this project ...

I will say "Sir, in the vast neural activity of a large language model, we found recurring functional modules that span multiple layers. These modules have their own preferences for words and phrases, and their responses can rise and fall together as the model reads a passage—that lets us connect local activity across many layers and see more clearly how the model organizes its responses to language!"

Q2a If each of your functional modules is indeed associated with either early, middle, or late layers ...

Where a module is concentrated gives us a clue to the range of language it responds to. Modules concentrated in early layers tend to favor small groups of similar words, while those concentrated in later layers respond to a wider variety of words (Fig. 3). Modules can also have related responses across depths. Some are concentrated in different parts of the network, yet their responses rise and fall together as the model reads a passage (Fig. 4).

Take the early layers. Their sensitivity to lexical information is well established. The modules show us how that sensitivity is divided up within the early layers. In Fig. 6, two modules are both concentrated in early layers. One often responds to pronouns such as we and she, while the other favors markers that refer to sources or figures. We can thus say which kinds of words or textual cues each cross-layer pattern responds to.

Q2b Is this simply because activations do not change much from one layer to the next ...

Only in part. Residual connections do make adjacent layers more similar, but modules still concentrate in neighboring layers when we look only at what each layer adds.

For Transformer layers 2–23, we subtracted each layer’s input from its output to remove the activity carried over directly from the preceding layer. Across 42 test documents for Pythia-410M, the mean similarity (centered linear CKA) between a layer’s input and its full output was 0.937. The mean similarity between the same input and the layer’s update was 0.341.

We then fitted a new set of modules using only the updates from the same 22 layers. For each module, we found the four consecutive layers containing the largest share of its salient coordinates. That share was 29.6% for modules fitted to the full outputs and 30.1% for modules fitted to the updates (the median across modules, averaged over three fits). Removing the activity carried over directly from the preceding layer therefore reduced adjacent-layer similarity substantially, while modules remained about as concentrated in neighboring layers.

Q2c If PCA of the activations recovers similar results ...

The similar structure recovered by PCA gives us greater confidence that the shallow, middle, and deep distribution patterns can be observed using different decomposition methods. We compare methods to determine which tools are better suited to studying the model’s functional organization.

We use TopK-SAE because of its widespread use in model interpretability research, while also considering how directly its results can be examined. It represents activity at each token using a small set of active modules. This lets us return to the text to explain which words and phrases elicit which cross-layer responses, understand the modules’ linguistic preferences and their relationships, and use these modules to uncover and explain further patterns in the model’s functional organization.

Q3 Many of the results attempt to show how modules associated with shallow layers ...

The familiar distinction between early layers processing local information and later layers processing abstract concepts captures only part of what we found. Recovering a known pattern of specialization across layers serves as a check on our interpretation. It shows that this perspective can capture established functional differences.

Cross-layer functional modules help us understand a particular response to language in the context of the whole network. As the model reads a particular kind of word or phrase, activity distributed across layers forms a recognizable response pattern. Within the same depth region, we can distinguish several patterns with different linguistic preferences. Patterns concentrated at different depths can also rise and fall together as the model reads a passage (Figs. 4 and 6).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful feedback. We seek to understand how the joint activity of a large language model’s layers reflects functional differentiation and relationships among responses as it processes language. Examining individual neurons or features can tell us which inputs each is sensitive to, but it remains difficult to understand how the model as a whole processes language from a large collection of local descriptions. We therefore start from the joint activity of the model’s layers, build on existing cross-layer sparse coding, and take cross-layer functional modules as our units of interpretation.

This perspective reveals detailed functional differentiation and response relationships within the model. Neuronal activity distributed across layers repeatedly exhibits recognizable, structured cross-layer activation patterns:

  • Functional differentiation and response relationships. Modules concentrated in the same depth region can favor different language content, while modules concentrated at different depths can show similar response fluctuations over the same text.
  • Domains. Different domains reuse many of the same modules with different emphases.
  • Training. A wider range of the final model’s responses becomes recognizable during learning.
  • Model sizes. Some language responses recur despite changes in network depth and width.

Cross-layer modules thus connect responses to particular words and phrases with their place in the network’s overall functional organization. The domain, training and model-size analyses extend this account of organization, showing how it is used, how it develops, and which aspects are shared across models.

To address the reviewers’ questions about the evidence for these interpretations and their connection to actual predictions, we strengthened two complementary lines of evidence:

  • Evidence for the interpretations. Activating passages and lexical coherence characterize modules’ input preferences. Detection, Fuzzing and evaluation on held-out documents test whether verbal descriptions capture these preferences and predict responses to new text.
  • Connection to actual predictions. Two local interventions further examine the relationship between these activities and model outputs. After perturbing salient coordinates, some patterns of candidate-token probability changes recur in new documents. In the tested sample, removing a module’s represented component at a selected layer makes the observed next token less likely on average than equal-magnitude random perturbations.

These sources of evidence allow module interpretations to be checked against both input responses and actual predictions.

Method comparisons and controls serve the same scientific purpose: examining which regularities in functional organization can be observed repeatedly and under what conditions these interpretations apply.

  • Tests on held-out documents. Cross-model tests examine whether shared responses recur on new text.
  • Comparisons before and after increasing dictionary width. Partial commonality remains observable at both dictionary capacities, while average response correspondence changes with capacity.

The revised manuscript organizes the main findings and their evidence around these questions, distinguishing input preferences, cross-layer distributions, response relationships and local effects on prediction. Our contribution is to use cross-layer functional modules as units of interpretation to provide a coherent account of concrete language content within the model’s overall functional organization, and to reveal its functional differentiation, response relationships, use and development. We thank the reviewers for helping us clarify this contribution and its supporting evidence, and would be happy to address any remaining questions.

We thank the reviewers for their careful feedback and the Area Chair for considering the discussion. Both rounds asked what we learn beyond known layer hierarchies and how well verifiable evidence supports it.

We ask how joint activity across a language model's layers reflects functional differentiation and response relationships. Local neuron and feature descriptions reveal preferences but leave network organization unclear. Using established cross-layer sparse coding and cross-layer functional modules as units of interpretation, we test response similarity, its relation to depth, and recurrence during learning and across model sizes.

Insights clarified in the discussion

  1. Functional differentiation is organized by depth, with response relationships across depths. Modules at similar depths can favor different language content. At equal candidate counts, mean best response correlation is highest within each of the three groups, reflecting relationships among text responses. Some modules at different depths rise and fall together over a passage, placing individual preferences among related activity. Domains show how these preferences are used: deep-module usage is most consistent across domains in natural text; shallow-module usage is most consistent on shared tokens at matched frequencies. Vocabulary partly explains domains' different use of existing preferences.

  2. Recognizable responses differentiate during training; early similarity does not mean settled organization. Despite similar active counts, early modules' nearest matches cover 241 distinct final modules, versus 802 late; early resemblance is weak. The recognizable range keeps expanding. By step 2,000, text-response correspondence has made about half its total gain, versus one quarter for salient-coordinate overlap; both increase afterward. “Early” means faster agreement about which text positions elicit stronger responses, while response differentiation and cross-layer distributions keep developing.

  3. Cross-scale commonality extends to distinguishing occurrences of the same token. Favoring the same token need not mean favoring the same occurrence. On independent documents, fixed, preselected pairs score two occurrences of the same token in one 256-token window. We ask whether both models give the same occurrence a higher continuous score before sparse selection. Agreement is 52.9%–54.6% for three smaller Pythia pairs and 55.5%–56.1% for 6.9B–12B; other partners at exactly the same text positions give 50.0%–50.1%. This modest advantage on new documents shows common preferences in distinguishing occurrences, checkable across model depths and widths.

Addressing reviewers' concerns

ReviewerConcernResponse
RxWzBeyond known hierarchies? Shared input alone?Analyses clarify links to depth and early versus continuing development. Fixed positions with different partners test particular pairs' shared preferences. RxWz confirmed the residual-update answer addressed that question. PCA as suggested shows modest neighboring-layer concentration in updates, without three clear major groups.
b8vWBeyond existing coding? Links to prediction? Coverage limits?Other decompositions partly recover similar responses and layer profiles, checking regularities. Detection and Fuzzing on independent documents show some verbal descriptions predict responses to new text. In the 96-module sample, single-layer component removal adds 0.0251 nat more correct-next-token loss than equal-magnitude random perturbations on average, linking activity to prediction. Wider large-model dictionaries cover more activity; partial cross-scale correspondence persists for the recovered responses.
SAPVVerifiable protocols and consistent scoring records?Matching/test documents are separate; shifted-position controls, pairwise results and coverage are specified. 24-module coordinate suppression tests recurrence of prediction changes across documents; 96-module component removal measures correct-next-token loss. Operations, selection, controls, statistical definitions and per-module effects are specified separately. Unified labels and scores make summaries checkable.

The value is cumulative, testable insight into functional organization. A response to particular words and phrases gains a wider network context, supporting judgments about functional differentiation and relationships with other activity, testable on independent text and controls. These empirical insights link concrete language content to overall functional organization through cross-layer modules. We believe this interpretability contribution merits acceptance.

These clarifications and analyses are in the revision. We thank the reviewers for helping clarify our question, interpret results and strengthen the evidence. We hope these contributions and responses inform the final assessment and welcome further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.