- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors study the interaction between the attention output and the MLP in deep Transformers. They consider the residual stream with , where is an attention, is an MLP, and is the depth. In a standard sequential Transformer, .
The authors find that setting with and preserves stability (i.e. no blowups) of the interaction between the attention output and the MLP, both at initialization and after one gradient update. Specifically, choosing as in CompleteP results in the interaction magnitude of at initialization and after one update, while choosing as in Depth-P results in a magnitude of in both cases. Furthermore, if we set as in the standard practice, these interaction magnitudes vanish as depth increases, provided that .
Numerical experiments suggest that performance may be improved by setting gates based on each , rather than using the fixed constant .
Critical strengths
I believe it is an important finding that, in deep Transformers, while the interaction vanishes under the standard setting of , setting and may preserve the interaction after one gradient update. Although the main theoretical results address only the case of one update, experimental results support the same conclusion for multiple updates.
Critical weaknesses
The most of the theoretical results only derive the upper bounds. Therefore, as the authors clarify in the paper, making does not automatically imply that the interaction is also .
However, I note that the experimental result presented in Figure 1(c) addresses this concern, demonstrating that the interaction does not decay under the experimental setting in this paper.
Questions for authors
None
We sincerely thank the reviewer for taking the time to read our paper carefully and for their positive assessment. We particularly appreciate the reviewer's attention to the distinction between our theoretical one-update upper bound and how it is complemented by the empirical evidence in Figure 1(c).