- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper learns a sparse dictionary over activations from all layers of an LM and studies responses across depth, text domains, training stages, and model sizes. Most analyses use document-standardised Pythia activations and describe variation within complete documents. The authors check depth patterns in other model families and use selected interventions to examine local output effects.
Critical strengths
-
Coarse depth profiles recur across normalisation and decomposition choices (§3.2). Repeated fits recover similar layer allocations on the smaller, 100000-token fitting panel. Individual responses are dictionary-dependent, with only partial correspondence on held-out documents (Appendix M, Table 12).
-
I support weak acceptance based on the descriptive findings and repeated-fit results, independently of the interventions. Shallow modules have narrower lexical responses, while deeper modules have broader responses (Fig. 3). During training, correspondence to the final checkpoint increases even with 1-to-1 matching and equal candidate counts (Table 23).
Critical weaknesses
-
Missing or unclear protocol details limit the weight I can give to cross-scale correspondence and module-specific local output effects. Appendix AA compares matched dense responses at similar standardised-space reconstruction quality, but does not specify the matching/evaluation split, shuffle construction or pairwise results. Appendix AF reports separate discovery/evaluation documents, discovery-defined signatures, norm matching and same-depth controls. Key intervention, selection and statistical details remain unclear. The interpretation supplement adds no protocol details.
-
The supplied records show lower Fuzzing-scores for deep modules. The description-score summaries cannot be checked exactly, because the interpretation supplement contains 927 unique module records and appendix Q reports 921. Some labels also differ from the scored labels in appendix F, for example, FM40.
Questions for authors
- Which documents were used to choose matches and which were used to test them in appendix AA? How were the shuffled controls made? For each model pair please report the mean correlations and the fraction of modules covered by the matches.
- Please specify the intervention operation and location, signature statistic, target/control selection, statistical test and per-target effect sizes, so the strength and specificity of local output effects can be measured.
- Which module IDs, group labels and scores were used for appendixes F, Q, R? Please explain the differences from the supplied interpretation collection so I can check the description-score summaries.
5 comments
W1b Appendix AF reports separate discovery/evaluation documents ...
Thank you for pointing out that the intervention details were insufficiently described. We have added the operations, selection rules, statistical definitions, and per-module results so readers can check the magnitude of prediction changes associated with module-related activity and whether these changes recur on new documents.
To address the signature question, we added a 24-module analysis. We randomly sampled eight modules with sufficient activating documents from each of the shallow, middle, and deep groups. At a token where a module was active, we zeroed the activations at coordinates with prominent decoder weights in the raw outputs of the associated layers. Four random controls modified the same number of other coordinates at the same layers and token position, matching the L2 norm of the actual perturbation. These controls were used to estimate the additional output change caused by the target intervention relative to generic perturbations.
The “signature” records how each candidate token’s log probability rises or falls in this additional change. We used four discovery documents per module to determine its average change direction, then conducted 93 evaluations on 88 documents disjoint from discovery. We compared each new change direction with the module’s own signature and with the signatures of the other seven modules in the same depth group. For 23 of the 24 modules, mean similarity was higher to their own signature, with a module-equal mean difference in cosine similarity of 0.1328. Randomly reassigning signature labels within depth groups 1,999 times yielded lower mean differences in every case, providing an exploratory mismatching reference. These results support cross-document recurrence in some of the prediction changes elicited by perturbing module-related activity.
A separate 96-module experiment directly measured prediction loss for the correct next token. In addition to the 24 modules above, we randomly sampled 24 more modules from each depth group. For 128-token passages where a module was active, we subtracted the activity component it represented at a preselected layer at the final token position. The selected layer had the largest sum of squared decoder weights. The component was computed from the current activation strength and decoder weights. For the final layer, we used the output of the final layer normalization. Four random-direction controls matched the actual perturbation magnitude at the same layer and token position.
For each module, the effect size was the additional prediction-loss increase under the target intervention relative to the mean of the four controls, averaged across that module’s samples. Across 1,488 paired comparisons from 661 documents, we first averaged within each module and then averaged equally across the 96 modules. In this sample, removing the activity component represented by a module at the selected layer increased prediction loss for the correct next token by 0.0251 nats more, on average, than a random perturbation of the same magnitude; the difference was statistically significant (two-way clustered test, two-sided p=0.0083). Per-module results for the two experiments are listed separately in “24-Module Signature Analysis Details” and “96-Module Prediction-Loss Intervention Details,” which report the intervention sites, sample counts, target and control results, and differences.
W2 The supplied records show lower Fuzzing-scores for deep modules ...
We apologize for the inconsistency. Multiple versions were maintained during our multi-author collaboration, and we did not consistently reconcile the interpretation supplement with the score tables when assembling the materials. We have now unified the descriptions and scores using the original scoring outputs and aligned the group labels with the original clustering results.
FM40 belongs to the Deep group, with Detection 0.73 and Fuzzing 0.72. We have restored the six scored modules omitted from the previous score table and corrected the relevant labels, descriptions, and scores. The unified records contain 927 scored modules: 923 belong to the three main shallow, middle, and deep groups, while four belong to the residual group in the original clustering and are listed separately.
Recomputing the summaries produces only small changes in the three groups’ mean Fuzzing scores. Deep remains below Middle and Shallow, preserving the previously reported score pattern.
| Depth group | Previous score summary | Unified summary |
|---|---|---|
| Deep | 0.5889 | 0.5907 |
| Middle | 0.6237 | 0.6244 |
| Shallow | 0.6306 | 0.6307 |
The unified record table provides each module’s description, group, and both scores. The supplement and summaries use this common record table, allowing each entry to be checked.
Q1 Which documents were used to choose matches and which were used to test them in appendix AA ...
The document splits, construction of shifted controls, mean response correlations for each model pair, and matched-module coverage are specified in our response to W1a. For all three model pairs, correlations at aligned positions on independent test documents exceed those of the shifted controls, with all 1,024 of 1,024 modules from each model included in matching (100% coverage). The numerical results are provided in the table in W1a.
Q2 Please specify the intervention operation and location ...
The intervention operations and locations, signature definition, selection of modules and controls, effect-size calculations, and statistical analyses are described in W1b. The 24-module signature analysis supports cross-document recurrence in some module-related prediction changes. The 96-module experiment measures the change in correct-next-token prediction loss after subtracting the activity component represented by a module at the selected layer.
Q3 Which module IDs, group labels and scores were used for appendixes F, Q, R ...
The record differences and corrections are described in W2. The unified records are linked by FM ID throughout. Group labels follow the original clustering, and descriptions and scores come from the same original scoring output.
| Appendix | Records used |
|---|---|
| F | IDs and depth labels of the 925 modules in the original clustering’s main groups. |
| Q | Detection and Fuzzing scores for 927 complete scoring records. Comparisons among the shallow, middle, and deep groups use 923 of these modules; the other four are listed separately. |
| R | The same 923 main-group modules, joined to activation frequency by ID, using the same labels and scores as Q for the adjustment analysis. |
FM221 and FM644 in F have no complete scoring records and are therefore excluded from score summaries. The other 923 main-group modules have consistent labels across F, Q, and R. The unified records list each module’s group, description, both scores, and activation frequency for checking.
We used one set of documents to find modules with similar responses in two models, then tested those pairs on a different set of documents. In the supplementary experiment, all three Pythia models read the same 64 documents. On these documents, we selected dictionaries with comparable reconstruction quality and matched modules one to one for each model pair according to how strongly they responded at different positions. We then fixed the matches and tested them on another 64 documents. Both sets were separate from the 32 documents used to train the dictionaries.
We sampled 32 token positions per test document. In the aligned comparison, we compared the first sampled position in one model with the first in the other, the second with the second, and so on. For the control, we shifted all module responses in the second model together by 1 to 31 sampled positions within each document, wrapping around at the end. The paired modules and documents stayed the same, while the token positions were offset. We ran 99 such controls.
All three model pairs showed the same pattern. Even on new documents, some cross-layer modules responded more strongly or weakly at similar positions in both models. Shifting the positions brought the mean correlation close to zero. All three model pairs had higher response correlations at aligned positions than in the shifted control, supporting the conclusion that some cross-layer responses to the input recur across model sizes.