- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how performance of depth recurrent models changes with model size and number of times the recurrent block is iterated, fitting scaling laws. The initial setup is quite clean, the authors follow the Huginn architecture, and study 3 tasks modular arithmetic, deduction and permutation. The authors motivate well that it is actually circuit depth that is a dial for difficulty for the number of iterations required to solve a problem, and that the circuit depth is a distribution in the data. The authors note different models have different accuracies and gains from iterations and combine this with the circuit depth observation to motivate a closed form. The authors train models at different sizes on a Chinchilla optimal number of tokens. The authors study models from 19 to 121 million parameters, showing experiments where the law extrapolates the 121M parameter models. The authors also use the larger Huginn model to show their experiments generalise and also vary the number of iterations of the recurrent block by fine-tuning their models. The authors also compare with Coconut, showing that this gives a much smaller return that recurrent depth for solving state tracking tasks. A neat finding is that the accuracy over iteration distribution is often aligned with the difficulty distribution of the benchmark, although the definition of difficulty is defined as the number of lines in the solution. The authors also use a branching task to show that their circuit depth observation is correct, and it is not just the size of the problem that impacts number of iterations required.
Critical strengths
- This is a well motivated work studying inference scaling laws. It could be highlighted a little more how this paper is not a regular scaling law, emphasising more all the merits the paper does have.
- The function form is well motivated and generalises to larger models.
- The majority of the paper is well written and explained.
Critical weaknesses
- The related works section “Scaling laws for depth-recurrent models.” Is missing several related works which study scaling laws for recurrent models, for example [1]. There is also prior work studying recurrent models for arithmetic [2]
- The last section of the paper “How should inference budget be split between size and depth” is a little rushed and could do with more work and space.
- There is a lot of results presented in the text, it would be nice to have these in figures and tables so they’re easy to pick out.
The scoring system is very coarse I am very close to awarding a 4. Cleaning up the “How should inference budget be split between size and depth” is most likely to move my score. I will also take into account the other reviews also in my updated score, so this is not a guarantee of score movement. Score moved 3 -> 4 during rebuttal
[1] Prairie, H., Novack, Z., Berg-Kirkpatrick, T. and Fu, D.Y., 2026. Parcae: Scaling laws for stable looped language models. arXiv preprint arXiv:2604.12946. [2] McLeish, S., Bansal, A., Stein, A., Jain, N., Kirchenbauer, J., Bartoldson, B.R., Kailkhura, B., Bhatele, A., Geiping, J., Schwarzschild, A. and Goldstein, T., 2024. Transformers can do arithmetic with the right embeddings. Advances in Neural Information Processing Systems, 37, pp.108012-108041.
Questions for authors
- In Table 1 the largest prediction gap is at T=1, is this due to the functional form caring too much about the larger T values? Perhaps this could suggest an incorrectness in the functional form?
- Why doesn’t the number of layers in the prelude and coda need to be taken into account in the functional form? E.g. a model with a huge prelude and coda could see very different uses recurrent block
2 comments
Thank you for your positive assessment and for recognizing the contribution of our work. We sincerely appreciate your time and consideration.
Rebuttal Response
Thank you for your prompt response. It addresses my concerns. I believe the paper addresses a gap in the literature well and should therefore be accepted.