- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a simple but effective approach by temporarily reducing the learning rate of upper-layer query and key projections to handle the premature upper-layer attention specialization problem in language model pretraining.
Critical strengths
- The idea is simple but sounds effective at different scales.
- Experiments are relatively comprehensive.
Critical weaknesses
- The learning-rate-control scheme itself is not novel, and the paper's novelty claim rests on the diagnosis — which is only established through the intervention
- The "best scheme" ranking is not established. Only the magnitude and timing of one intervention are explored; the target is fixed by assumption.
Questions for authors
check the weakness