- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the periodicity generalization ability of transformers. In controlled experiments, the paper trains transformers and other language models to learn periodic functions and their compositions. The paper distinguishes two kinds of generalization modes: (i) Hollow, where the model can generalize out-of-distribution through interpolating trained examples, and (ii) Extrapolation, where the test examples lie outside the support of the training distribution.
The results compare various architectures of language models, and find that they mostly generalize via the hollow mode, while struggle to extrapolate.
Critical strengths
-
Clear motivation and theory-inspired synthetic task. The main task Coper is a valuable task to study compositional generalization of language models. In particular, the results can be visually inspected as in Figure 1, thus providing a good way of understanding the generalization ability of language models.
-
Informative generalization notions. I like the classification of OOD generalization into interpolative generalization and extrapolative generalization. The distinction is helpful to understand the benefits and limitation of current model architectures and training algorithms.
Critical weaknesses
- The main issue is training is not well implemented. Model training seems to be insufficient or incomplete, which raises the concern that models may be not correctly trained. For example, in Figure 5, the loss curves are still decreasing, which means that training is not complete. The authors should consider increasing the training steps to a much large number.
Since the main claim of the paper is about generalization, not optimization speed, training is only complete if the ID loss decreases to a sufficiently low level. In Table 1, however, the three models other than Transformer are not sufficiently trained. Giving the confounding effects of incomplete training, the claims about OOD generalization are not well supported.
If the authors want to make a claim about optimization (like difficulty in jumping out of local minima or saddle points), the experiments should be designed very differently, and the paper would need to sweep across hyperparameters and control for training steps / time.
- The scale of experiments is also small. For example, the number of layers in Table 7 is no larger than 7. A better model scaling experiment would need to sweep the model size to a wider range. The small scale would prevent the paper from convincingly draw conclusions about model capacity of memorization. In addition, it would informative to report the variability of results across several independent runs, or analyze the robustness to key model hyperparameters.
Questions for authors
See my comments in "Critical Weaknesses"
2 comments
Thank you again for taking the time to review our responses. As the discussion period is coming to an end, we wanted to check if there are any remaining questions or concerns we can clarify. Please feel free to let us know.
Weakness 1 & 2 (Scaling, several independent runs and hyperparameters)
We thank the reviewer for this point and re-ran the main experiment so that every model is trained until its ID loss is below 0.1; epochs differ across models, and results are mean ± std over three seeds (which also reports the number of training tokens).
Training to convergence. The original learning rate (1e-5) kept Transformer-type models on a long loss plateau. After a learning-rate search on held-out training data (no OOD data used; Table R2), Transformer and FANformer reach held-out ID loss ≤ 0.004 and ID accuracy ≥ 99.9% within ~20 epochs (100–170M tokens). Mamba reaches a training loss of 0.071.
The OOD failure is not caused by under-training. With ID fully fitted, the 10-layer Transformer reaches 29.5 ± 3.9 on Hollow and 20.5 ± 1.7 on Extrapolation, and FANformer 30.7 ± 10.1 and 17.5 ± 3.4, while the OOD loss rises to 7–8.
Larger models. We scaled to 10 layers (48–54M trainable parameters) and to 12 layers with hidden size 2048 (~0.3B: Transformer 302M, FANformer 340M). The 0.3B models also fit ID perfectly (100.0 / 99.9) but generalize worse (Hollow 18.3 / 17.6, Extrapolation 17.0 / 15.0). More capacity speeds up ID fitting but does not improve periodicity generalization.
We will include these additional experiments and results in the revised manuscript.
If you have any further questions, please do not hesitate to contact us.