- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the scaling laws of model growth, recursion, and boundary operators in a looped transformer. With compute budget of 10^18 to 10^20 flops in 30 runs, the authors show that model growth and boundary operators provide compute-efficiency gains that increase with scale and increasing the loop count with scale in the data-constrained setting is better than scaling model sizes or tuning weight decay. A main insight is that architectural changes can, despite claim in other works, yield improvements to scaling exponents rather than just the constants of the scaling laws
Critical strengths
- This work covers a range of architectures of looped transformers and shows the improvements caused by architectures
- The experiment recipe tuning covers the major aspects of scaling hyperparameters
- The insights gained are important, and showcase the importance of considering architectural details when considering the scaling laws of models.
Critical weaknesses
-
The discussion of overfitting in the variants of looped transformers is not convincing enough. Appendix D provides supporting evidence through experiments with data repetition, weight decay, and weight sharing. But these mainly show final validation losses, so it is still unclear whether looping reduces overfitting or simply helps optimization. Showing training and validation loss over time would help distinguish the two.
-
The authors are presenting the plots using a linear scale in the y axis. This is different than most other works in this space. Is there are a reason for this choice? It makes reasoning about their work needlessly challenging compared to the rest of the literature.
Questions for authors
- Could you please make the plots in the log-log scale? If not, what is the reason of using linear scale for the y axis?
- Could you please provide evidence that overfitting is avoided? [See the weakness point 1 above]
- What are the irreducible floors E for each corpus in your fitting results? Are they consistent in the fitting between variants of different architectures?
2 comments
Dear Reviewer 6yAd,
Thank you again for your feedback. We made a strong effort in our response, including new analyses and paper edits. As the discussion phase approaches its end, we are happy to answer any additional questions you may have or provide further clarifications. We would greatly appreciate if you would increase your score in light of our response.
Author Response to Reviewer 6yAd
Thank you for your thoughtful questions. Inspired by your comments, we conducted additional analyses and report the results below.
Overfitting effects
Motivated by your questions, we examine how model depth and looping affect the generalization gap and training loss.
Table 1: Generalization gap (validation loss minus training loss) across depths and loops. Models are trained on 100M unique tokens for 10 epochs with WD = 0.05. Results are similar when WD=0.8.
Table 2: Training loss across model depths and loops under the same training conditions as Table 1.
Increasing depth lowers training loss but widens the generalization gap. Increasing loops, by contrast, generally narrows the gap. At larger depths, looping slightly increases training loss while reducing the generalization gap. In particular, at depth 14, increasing looping from 1 to 8 raises training loss from 1.045 to 1.122 while reducing the generalization gap from 3.953 to 3.679. As a result, the overall validation loss decreases from 4.998 to 4.801. This supports a regularizing effect of looping and helps explain its benefits in the data-constrained setting. We will incorporate this analysis into the revision.
Logarithmic Y-Axis
Thank you for pointing this out. The current plots use a linear y-axis. We will switch to a logarithmic y-axis in the revision. Because the plotted losses span a narrow range, this change has only a modest effect on their visual appearance.
For the discussion of irreducible loss, please see “Exponent Analysis” in our response to Reviewer Hvrs.
Thank you again for your supportive and constructive suggestions. We hope these clarifications and additional analyses, inspired by your comments, help answer your questions. We would appreciate if you could consider raising your score in light of our response.