- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the in-context learning (ICL) ability of two variant model architectures, depth-grown language models (LMs) and looped LMs. The main finding is that looped LMs and depth-grown LMs, compared to dense transformers, exhibit strong ICL abilities and rely more on the context.
The paper presents various experiments to support and further analyze their ICL abilities. For the main claim, the paper compares three models with matched layers, and NLL, NLL difference, and few-shot accuracy show that looped LMs and depth-grown LMs rely more on the context. Then, controlled experiments show that the two model variants trade IWL accuracy for ICL accuracy, and induction head emerges early. Head ablation further confirms the effects of induction head for ICL in looped LMs and depth-grown LMs.
Critical strengths
-
I find this topic very timely, and writing is clear and easy to follow. Considering that model variants are increasingly used by production LLMs (e.g., GPT-6), the findings and analysis provide critical insights into the mechanism of newer language model architectures.
-
The main claim is well supported by different experiments (pretrained LMs or controlled training) and a variety of metrics (NLL change, few-shot accuracy, TVD, head ablation). The analysis looks solid to me, though I didn't check details carefully.
Critical weaknesses
- Novelty. The main claim is not particularly surprising to me: looped LMs share parameters across layers, and depth-grown LMs partially share training trajectory of model parameters. In a sense, the "model capacity" is constrained compared with dense models, so they are trading memorization abilities for ICL (generalization) abilities. Also, the effects of induction head on ICL are already analyzed in [1]. It is assuring but not surprising to see that induction heads have stronger effects on looped LMs and depth-grown LMs.
The (more or less) anticipated empirical finding does not necessarily mean lack of novelty, but this paper would benefit greatly from a better framing: how does the study of the two model variants change our understanding about memorization vs generalization tradeoff? Is there a intrinsic notion of model capacity? These are fundamental questions recurring through machine learning and LLM research. For transformers, there is already an abundance of research on such tradeoff, analysis, phase transitions, etc; for example, see [2,3,4,5,6]. Making connections to the established literature would help readers better understand the mechanism of looped LMs and depth-grown LMs.
I think this weakness is fixable: making connections to the established literature, more analysis of the memorization abilities, more insights into tradeoff between IWL and ICL model capacity, etc. I will raise my score if the authors address this weakness.
- Olsson, Catherine, et al., In-context learning and induction heads, 2022.
- Park, Core Francisco, Ekdeep Singh Lubana, and Hidenori Tanaka. "Competition dynamics shape algorithmic phases of in-context learning." International Conference on Learning Representations. Vol. 2025. 2025.
- Song, Jiajun, Zhuoyan Xu, and Yiqiao Zhong. "Out-of-distribution generalization via composition: a lens through induction heads in transformers." Proceedings of the National Academy of Sciences 122.6 (2025): e2417182122.
- Yan, Hao, Haolin Yang, and Yiqiao Zhong. "Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers." arXiv preprint arXiv:2605.03780 (2026).
- Carroll, Liam, et al. "Dynamics of transient structure in in-context linear regression transformers." arXiv preprint arXiv:2501.17745 (2025).
- Wurgaft, Daniel, et al. "In-context learning strategies emerge rationally." Advances in Neural Information Processing Systems 38 (2026): 118425-118485.
Questions for authors
In Figure 3, there is a substantial difference between dense 4L and dense 12L. When controlling for independently initialized parameters, dense model performs comparably to the two variants. Does this mean layer count is not a good measure of model capacity? Perhaps the memorization ability is largely affected by the number of independently initialized parameters? Clarifying this point is useful for understanding what matched comparison means.
Final Summary
We thank all reviewers and the Area Chair for the time and care they invested in evaluating our work. We are glad that the reviewers recognized the clear presentation and figures, the timely insights into depth-grown and looped language models, and the complementary analyses across pretrained models and controlled tasks.
This study systematically examines how reusing computations across depth through depth growing and looping shapes in-context learning (ICL) across pretrained language models up to 3B parameters and controlled tasks. Beyond establishing stronger context adaptation, we separate ICL acquisition from retention against in-weights learning (IWL), trace how layer duplication propagates early induction heads across depth, and show through ablations that grown and looped models rely more strongly on these heads. During the rebuttal, we addressed the reviewers' concerns through the following experiments and manuscript revisions:
These experiments and manuscript updates resolve all questions and suggestions raised in the reviews, leaving no open concerns. We appreciate the reviewers' constructive feedback, which strengthened the paper, and hope these additions provide a clear basis for the Area Chair's final decision.