- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper under review studies the length generalization problem in transformers. The authors study this problem from the lens of the optimizer essentially seeking to understand whether the optimizer itself brings an implicit bias to how the transformer performs on length generalization. On a particular transformer model the authors show that both GD and Adam bring a different bias for length generalization.
Critical strengths
The authors study an interesting problem for transformers from a very nice lens, namely optimizer bias. I do feel this is a very novel lens to try and understand how transformers perform on length generalization and how the optimizer affects that generalization performance.
Critical weaknesses
I did find the paper quite difficult to read especially to understand the point the authors were giving with their paper. Given that Adam and GD bring a different bias to this problem, which I think people working in the field would have guessed, what does that say about how we should be optimizing such models for problems like length generalization? No real effort was made in bringing deeper insight. While I admire the honesty of the authors in their AI statement in saying that Codex and Claude helped with various aspect of the work, I think it is therefore imperative that the authors put significant time into expositing the work well. I feel this was not the case.
Questions for authors
-
Does your work tell us anything about how we should design optimizers for such models for good performance for problems like length generalization? What is the key insight this tells us about how we should optimizer transformers for such problems.
-
I understand the approach you took was to understand the situation for a 2-layer transformer. Is it possible to carry out some experiments for deeper transformers and at least show empirically what is happening in that case?
I thank the authors for taking the time to respond to my reviewer and appreciate the clarity. After reading the paper several more times and their rebuttal I am still convinced this paper is not at the level of ICLR. While the authors have put effort in showing how optimizer bias affects length generalization I don't believe their insights are at the level of a top tier publication. I do thank the authors for doing deeper experiments and that does add to their paper, I would encourage them to include such results in future iterations. I am thus keeping my score.