- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies how the importance of a single layer in a transformer is for next-token predictions changes over training checkpoints. They test most layers across various small open-source models at 3-5 checkpoints in the training, measuring the difference in prediction scores across general/math/code datasets. They find that the rankings of most to least important layers is strongly correlated between adjacent checkpoints and correlated between the earliest and latest checkpoints. Furthermore, most (65%) of the change occurs in 20% of the blocks, and patterns transfer across domains. Finally, they find that skipping a layer makes a bigger difference later in training partly because that layer performs a larger update to the residual stream, not necessarily because the model is more sensitive to that layer.
Critical strengths
None
Critical weaknesses
- The significance of the results is under-motivated. Nothing about the experiments reveals what the blocks compute or why the block rankings change. Furthermore, the reason they are experimenting on bypassing blocks in the first place is not clear; there is no further interpretation of the revealed block ordering in terms of improving model interpretability or capabilities. They do not try to characterize the blocks or explain the reasoning behind their effect on the outputs.
- Many parts of the paper, including figures, seem AI-generated with little human iteration. For example, parts of the abstract are cryptic and difficult to read ("single-block identity bypass on fixed teacher-forced contexts", "longitudinal layer sensitivity is structured but not static"). Furthermore, Figure 1, which should typically summarize the main result or contribution of the paper in a visually appealing way, is completely unintelligible to me: it is not clear from the figure or the caption why the information is being organized in these blocks with arrows between them, what "follow the checkpoint x depth response field" means, what 14k/72k/143k refer to, etc. These presentation issues, which continue throughout the rest of the paper, make it very difficult for a human to understand and assess the contribution of the paper. Therefore, the paper's current presentation does not meet the threshold for ICLR and should receive a clear rejection.
Questions for authors
Can the authors please explain, either from their existing results or from additional experiments, what can be said about how the model performs computation or develops capabilities? Is there any interpretability insight we can gather from these results?
2 comments
Thank you for the thoughtful response and revisions. I appreciate the efforts to improve the presentation and clarify the practical motivation, and will raise my score from 1 to 2. However, my main concern remains: the results provide limited insight into what the blocks compute or how capabilities develop, leaving the broader interpretability significance insufficiently established.
Response to Reviewer kf2Z
Thank you for the direct and detailed feedback. We agree with both concerns and have revised the paper accordingly.
Presentation. We agree that the submitted version made the contribution hard to assess. The phrases you quoted were unclear, and Figure 1 did not communicate its content. We rewrote the main text. The abstract and introduction now state the question and findings in plain language, every term is defined at first use, and Figure 1 is a new figure with a schematic and three data panels in which training progress is labeled explicitly. We would be grateful if you could look at the revised version.
Significance: why bypass blocks, and what we learn. The motivation is practical. Layer importance from skipping layers is widely used to decide which layers to prune or skip, and it is almost always measured on a single checkpoint. We ask whether such conclusions survive continued training. The revision makes this concrete (Section 4.6): in SmolLM2, removing the two blocks that an early checkpoint ranked least important hurt the final model far more than removing the two selected at the final checkpoint (perplexity 25.3 vs. 15.6), while in OLMo-2 1B the early choice was better. Rankings from one checkpoint are thus a fair guide to coarse depth organization, but they should be re-evaluated before specific layers are removed.
Q1: Interpretability insights. We do not claim to identify circuits, but the results bear on how ablation evidence should be read:
We also tested one simple explanation of persistence and found it unsupported in Pythia-160M, which the revision reports explicitly.
Thank you again for pushing us to make the contribution clear.