- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper shows a phenomenon of the first-word attention aggregation in the decoder Transformer and proposes the relation invariance bottleneck as an explanation. The paper firstly proves that under the constraints of causality and row randomness, concentrating all attention on the first element is the only operator that maintains the difference between word elements before and after residual update for any Value matrix. Furthermore, it establishes a boundary between the relationship output and the distance of this operator. Based on this, the paper proposes Adaptive Diagonal Target Editing (ADTE), which combines learnable diagonal cancellation and word-wise amplification to provide another path for achieving relation invariance.
Critical strengths
-
The author focuses on why attention usually converges on the first word element and attempts to provide an explanation based on the structural constraints of causal attention. I believe this issue has research value and also offers a meaningful perspective for understanding the attention sink.
-
The design of ADTE is simple and the additional parameter overhead is small. The paper reports consistent improvement in perplexity under different model scales and normalized architectures, indicating that this method has certain practical value.
Critical weaknesses
- Firstly, the title and abstract of the manuscript have affected the reading experience. The title on the front page still uses the format of a conference submission description, which does not match the content of the paper. The multiple key symbols in the abstract formula also lack necessary explanations, and readers need to refer to the main text to understand the main conclusions.
- Secondly, the design basis of the word-by-word amplifier is not yet fully established. The author independently determined the amplification coefficient, but did not conduct experimental comparisons and directly adopted the relatively intuitive scheme of to restore the unit row and equal comparison. The existing ablation studies indicate that the amplifier is effective, but do not fully explain the necessity of this specific design.
- Finally, the current experiments still do not provide sufficient support for the research motivation of the paper. The experiments demonstrated the reduction of aggregation and the improvement of perplexity, but the verification of preserving the relationships between word elements was not direct enough. It is suggested to supplement the comparison of the differences in word element representations before and after the update of the baseline and ADTE, in order to more directly support the explanation of the invariance of the relationships.
Questions for authors
- Author should revise the title of the main page (pdf file) and clarify the key symbols in the equations presented in the abstract.
- Compared with alternatives such as directly enforcing unit row sums, what advantages does the design of a separately learned scaling factor offer? Could you provide corresponding ablation studies to support this?
- By comparing token representations before and after the ADTE update with those from the baseline, can we obtain more direct evidence to support the invariance interpretation?
- Could you include downstream evaluation results on 1B or even larger models to verify the practical gains and scalability of ADTE?