- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents new generalization results for transformers along two different axes: in-distribution generalization and out-of-distribution generalization in terms of length. The paper provides a very novel proof for length generalization for transformers and combine it with statistical proofs for in-distribution generalization bounds to obtain novel bounds for length as well as in-distribution generalization. In particular, the paper shows that in compute budget constrained settings, length generalization comes when the transformer has seen a certain minimum length of data, and in-distribution generalization comes when we add more data. The paper provides the proofs in the setting where a student transformer learns from a teacher transformer model, and it provides experiments validating the conclusions from the bound that training on longer traces helps generalization in compute constrained setting.
Critical strengths
- This paper provides a novel generalization bound across two axes of in-distribution generalization and length generalization. The bounds are in the setting of distillation from teacher to student, where in the length generalization bound it is shown that if the student and teacher agree on all possible words of certain finite length C (computed in the paper), then they also agree on all words of any length. This bound combined with the in-distribution generalization gives a novel generalization bound that accounts for both in-distribution and out-of-distribution data.
- The bounds lead to the conclusion that in compute constrained settings using longer training data helps length-generalization to certain length of data, beyond which more samples help generalization. This is very insightful and likely to be useful for training of models.
- The paper provides experiments with small transformer models that validate the claim of generalization for length generalization in compute constrained setting.
Critical weaknesses
- Although the theory in the paper is very novel and useful, the experiments are very weak and needs further work. In particular, the experiments done are with transformers of number of layers 1 or 2 for the student transformer, which is very small compared to regularly used transformers.
- The paper assumes in Remark 2.1 that the dimensions of the student are bigger than the teacher. It is unclear if this is a necessary assumption and if so, is it a practical one where the student is bigger than the teacher?
Questions for authors
- Please address the concerns in the weaknesses.
We thank the reviewer for the close reading and the positive assessment.
We'd like to point out that, for the Alternating languages, we have substantially deeper teacher transformers (up to 6 layers -- Appendix H.1.1, H.3.1) and deeper student transformers (up to 20 layers -- Appendix H.3.2). These Alternating languages in particular are noteworthy for provably requiring increasing depth (Yang et al, NeurIPS 2025, cited in the paper); their inclusion is thus especially well-suited to understanding the role of depth.
Importantly, the student architecture needs to be able to represent the teacher’s function. The componentwise size inequalities in Remark 2.1 provide sufficient conditions for this through our explicit embedding construction. These conditions might not be necessary, as other realizations may trade off depth, width, and number of heads.