- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a modulated transformer by modulating the amplitude of softmax attention map to handle the problem of attention sink and attention noise. Compared with vanilla transformer, this proposed method achieves model size and token efficiency.
Critical strengths
- The paper is easy to follow. The idea is simple but seems to be effective. The design is well motivated.
- Ablation studies are conducted and effectively demonstrate the choice of design.
- The AI use statement is clear. This is a model disclosure of AI involvement.
Critical weaknesses
- The claimed value of “model size and token efficiency” is not fully demonstrated. Actually, from the results, this cannot be derived.
- The central motivation, alleviating attention sink and attention noise, are not sufficiently and quantitively evaluated. In the qualitative evaluation in Figure 6, top and bottom rows are not head-to-head comparison.
- Figure 3 is not evidence that modulated scores are better, just showing the scores are lower.
- Modulated is not shown to be superior to Differential although it is built based on differential transformer.
- Table 2 does not have results for differential transformer.
Questions for authors
see weakness