- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work analyzes transformers using their residual stream (i.e., the embedding, multi-head attention, MLP, RMSNorm, projection, and logits), where the loss is first rewritten as a sum of bias (the expected loss with respect to the true logits) and variance (the expected loss with respect to the predicted logits). The work aims to find the relationship between MSE, dias, and diversity. Experiments with different datasets and models are provided to explore this relationship. Overall, I found the writing and organization of the paper very confusing to fully understand the claims and experiments in the paper.
Critical strengths
- The paper analyzes the MSE of logits in transformer residual streams (i.e., the embedding, multi-head attention, MLP, RMSNorm, projection, and logits) by decomposing the MSE loss between bias and diversity and then analyzes the relationship between diversity and MSE. The paper also provides some information theoretic analysis on the benefits of adding layers in transformers
- The paper provides experiments to analyze the relationship between MSE and diversity.
Critical weaknesses
- The paper needs major rewriting to clearly write the problem being addressed, the proposed solution and what the experiments want to validate. Currently the writing does not do a good job. More details follow: a) in Abstract, the authors directly discuss the “residual stream components” without describing what it is, then the authors describe “departures from diminishing returns” without describing what it means in this context; b) Section 3.1.1 goes directly into bias-diversity decomposition, then in Section 3.1.2, the authors separately describe the weighted case, instead of describing them together and describing before; c) Section 3.2 suddenly introduces information-theoretic bounds without fully defining the terms or motivating this section; d) In Section 5.1, there is no clear description of the experiments performed (not adding even the basic experimental details in main content is not easy to read); e) Fig. 4 is very difficult to read or interpret. Unless there is a major reorganization of the paper, I am unable to understand the paper or provide a good review.
Questions for authors
Please address the concerns raised in the weakness section.
Reply to AI use in this review
Dear Authors and AC,
Because the system flagged that my review comments were written by AI: to be completely honest, my native language is not Chinese.
However, rewriting the review will take time. If you have already completed your rebuttal or are currently working on it, please continue with your rebuttal. I will make sure my rewritten review covers my current core concerns.
However, this will take time, and I hope to finish the rewrite before the deadline and carefully consider both your rebuttal and the paper.
Best regards,
Reviewer