- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper explains that the performance of Vocabulary Projection Methods (VPM) is degraded due to the 'final-layer jump' symptom that occurs within Transformer layers. To improve the interpretability of vocabulary-space readouts from hidden layers, the authors propose adding an additional linear layer before LM head at every transformer layer. The paper empirically shows that the extra layer effectively mitigates the final-layer jump, resulting in making the hidden state better aligned to the output of the final layer. Several ablation studies further support the effectiveness of the proposed architectural change.
Critical strengths
-
This study finds out the causal relationship between the final-layer jump symptom and Transformer architecture. Through heavy empirical study, the authors argue that the symptom is generally found in many Transformer-based LLMs.
-
Thorough and extensive ablation studies provide useful insights. Appendix also includes a wide range of experimental results.
-
The proposed solution is simple and can be easily applied to any kinds of Transformer models.
Critical weaknesses
Although I see useful empirical results and interesting explanations of the observed results, I also see several non-negligible limitations as follows.
-
[major] First of all, although it is interesting that the final-layer jumps are the main reason for the performance drop in VPM, this cannot be considered as a significantly impactful finding. In Section 3, the authors comprehensively investigate what makes the final-layer jumps occur and where they are often observed (final linear transformation layers at each block). This symptom becomes problematic in a very specific situation where the hidden state representation is used to extract the output tokens (e.g., LogitLens). Therefore, I consider the contribution is not sufficiently strong to provide valuable insights to the ICLR readership.
-
[major] Inserting a buffer layer into Transformer blocks may relocate the layer where the final-layer jump occurs, however, it is a simple (as the authors stated in the paper) trick rather than a meaningful architectural advancement. In Table 18, the Commonsense suite performance is consistently degraded though the drop is marginal. Why would inserting a single buffer layer right before LM head be the optimal solution? Why not adding more layers with smaller number of weights (e.g., LoRA-like bottleneck layers)? This simple trick should be more rigorously substantiated.
-
[major] While the proposed buffer layer relocates the final-layer jump, the underlying mechanism responsible for this effect remains unclear. The generality of this effect is still not theoretically grounded. The empirical study relies only on a limited set of benchmarks and a few models. Thus, the proposed approach should be viewed as an empirical observation rather than remedy to a certain problem.
-
[major] In Section 3.3, the authors state that no layers other than the reported ones have larger than 3.0. However, Figure 2(d) shows for the ReLU variant, which means the jump occurs in both of the last two layers for this model. The authors should correct this statement and explain why the ReLU variant also exhibits a jump.
-
[minor] The font size in most figures should be increased. It is hard to read the text, especially Figure 6.
-
[minor] The text in Section 3.3 lists the jump scores 9.26, 6.13, 16.00, and 5.49 without indicating which model each value belongs to. I recommend explicitly mapping those scores with corresponding models (methods) in the main text so that the paper becomes self-contained.
Questions for authors
My questions and concerns are all summarized in the above weakness section. Please carefully address them.
Thank you for the response and additional experiments. I now understand the role of the buffer much better: it absorbs the jump without increasing the expressive capacity of the readout, yet substantially changes the interpretation obtained from the standard LM-head-only lens. This addresses my main concern.
However, I do not have enough context to judge how important or necessary this analysis and the proposed intervention are, so I think it will be important for the authors to convince the other reviewers on this point.
That said, I find the empirical observation that the jump location is strongly tied to the pair of linear transformations closest to the output quite interesting. I will raise my score to 3.