acceptodds
Under review as a conference paper at ICLR 2027

How Depth Growth and Looping Shape In-Context Learning in Language Models

Abstract

Looped and depth-grown language models reuse computations across depth and have attracted interest for their reasoning capabilities. Building on prior suggestive evidence of improved in-context-learning (ICL), we investigate how such reuse favors ICL, including how it shapes the acquisition, retention, and mechanisms of this ability. Our analysis spans pretrained language models up to 3B parameters and controlled tasks. The experiments provide evidence for a shared inductive bias toward learning from context: grown and looped models rely more on information in context and, compared with a full-depth dense reference, acquire ICL earlier in controlled tasks and retain it more strongly when learning fixed associations in the weights is also possible. Tracing attention heads through training reveals that depth growth propagates early induction heads across layers. Ablations show that grown and looped models rely more strongly on induction-ranked heads for both few-shot performance and later-token prediction. Together, these findings provide evidence for a shared inductive bias toward learning from context and suggest that (soft) parameter reuse shapes how this ability develops and which attention mechanisms support it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context learning, depth growing, looped Transformers, induction heads, function vectors, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_zivwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the in-context learning (ICL) ability of two variant model architectures, depth-grown language models (LMs) and looped LMs. The main finding is that looped LMs and depth-grown LMs, compared to dense transformers, exhibit strong ICL abilities and rely more on the context.

The paper presents various experiments to support and further analyze their ICL abilities. For the main claim, the paper compares three models with matched layers, and NLL, NLL difference, and few-shot accuracy show that looped LMs and depth-grown LMs rely more on the context. Then, controlled experiments show that the two model variants trade IWL accuracy for ICL accuracy, and induction head emerges early. Head ablation further confirms the effects of induction head for ICL in looped LMs and depth-grown LMs.

Critical strengths
  1. I find this topic very timely, and writing is clear and easy to follow. Considering that model variants are increasingly used by production LLMs (e.g., GPT-6), the findings and analysis provide critical insights into the mechanism of newer language model architectures.

  2. The main claim is well supported by different experiments (pretrained LMs or controlled training) and a variety of metrics (NLL change, few-shot accuracy, TVD, head ablation). The analysis looks solid to me, though I didn't check details carefully.

Critical weaknesses
  1. Novelty. The main claim is not particularly surprising to me: looped LMs share parameters across layers, and depth-grown LMs partially share training trajectory of model parameters. In a sense, the "model capacity" is constrained compared with dense models, so they are trading memorization abilities for ICL (generalization) abilities. Also, the effects of induction head on ICL are already analyzed in [1]. It is assuring but not surprising to see that induction heads have stronger effects on looped LMs and depth-grown LMs.

The (more or less) anticipated empirical finding does not necessarily mean lack of novelty, but this paper would benefit greatly from a better framing: how does the study of the two model variants change our understanding about memorization vs generalization tradeoff? Is there a intrinsic notion of model capacity? These are fundamental questions recurring through machine learning and LLM research. For transformers, there is already an abundance of research on such tradeoff, analysis, phase transitions, etc; for example, see [2,3,4,5,6]. Making connections to the established literature would help readers better understand the mechanism of looped LMs and depth-grown LMs.

I think this weakness is fixable: making connections to the established literature, more analysis of the memorization abilities, more insights into tradeoff between IWL and ICL model capacity, etc. I will raise my score if the authors address this weakness.

  1. Olsson, Catherine, et al., In-context learning and induction heads, 2022.
  2. Park, Core Francisco, Ekdeep Singh Lubana, and Hidenori Tanaka. "Competition dynamics shape algorithmic phases of in-context learning." International Conference on Learning Representations. Vol. 2025. 2025.
  3. Song, Jiajun, Zhuoyan Xu, and Yiqiao Zhong. "Out-of-distribution generalization via composition: a lens through induction heads in transformers." Proceedings of the National Academy of Sciences 122.6 (2025): e2417182122.
  4. Yan, Hao, Haolin Yang, and Yiqiao Zhong. "Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers." arXiv preprint arXiv:2605.03780 (2026).
  5. Carroll, Liam, et al. "Dynamics of transient structure in in-context linear regression transformers." arXiv preprint arXiv:2501.17745 (2025).
  6. Wurgaft, Daniel, et al. "In-context learning strategies emerge rationally." Advances in Neural Information Processing Systems 38 (2026): 118425-118485.
Questions for authors

In Figure 3, there is a substantial difference between dense 4L and dense 12L. When controlling for independently initialized parameters, dense model performs comparably to the two variants. Does this mean layer count is not a good measure of model capacity? Perhaps the memorization ability is largely affected by the number of independently initialized parameters? Clarifying this point is useful for understanding what matched comparison means.

Reviewer_sUdCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies ICL in dense vs. grown vs. looped transformers (controlling for the effective number of layers). First, they train 3B versions of each model architecture on 100B tokens with otherwise matched settings and evaluate ICL performance as both in terms of NLL loss as the context grows on FineWeb, and GSM8K acc. scaling with the number of shots. The find that rate of improvement for grown and looped transformers is better than that of dense models. They also find they adapt better to unseen formal languages.

Next, they evaluate their performance on ICL vs. in-weight learning (IWL) across training via an Omniglot image-classification task. They test ICL on unseen classes with reassigned labels, and IWL on training classes whose query class is absent from context. They find that grown and looped models retain higher ICL performance but achieve lower IWL performance than the dense model. In a separate setting with labels reassigned in every sequence, grown, looped, and shallow dense models acquire ICL faster than the dense model.

They then investigate attention heads using prefix-matching (PS) and function-vector (FV) scores. They find that in grown models these heads emerge from just two early heads because of the duplication of layers and stay relatively consistent. Finally, they ablate these heads selectively and find that removing PS heads hurts few-shot performance and per-token ICL loss more in grown and looped models than in dense models, suggesting that these heads contribute differently to the abilities in these models.

Critical strengths
  • The paper presents multiple complimentary analyses demonstrating that grown and looped transformers appear to be more competent at ICL than dense models and acquire these capabilities earlier. They also cleanly separate ICL and IWL via the Omniglot task. These analyses appear to be well executed.
Critical weaknesses
  • The paper identifies induction heads purely based on their prefix-matching score, and not also w.r.t to the usually associated copying behaviour (Olssen et al., 2022, In-context learning and induction heads). The authors should verify this copying behaviour before interpreting the results as evidence about induction heads. Otherwise, the claims should refer more narrowly to heads with high prefix-matching scores.
Questions for authors
  • Omniglot ICL accuracy (fixed labels) for the grown model appears to be high at step 0. Could you add a log-scaled plot or zoomed in version to show when it actually develops?
  • Could the ablation differences reflect how a behaviour is distributed across heads? For example, there might be fewer, more important prefix-matching heads, rather than more reliance on induction in general.
Reviewer_LzEUOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Looping and depth growth are increasing important in LLMs, and even the fact that it is possible to do either at all is in my mind yet another striking phenomenon in these models. An interesting question is then how these two forms of depth modification change the resulting abilities or "inclinations" of the resulting trained models. Prior work has suggested that depth growth and looping improve in-context learning (ICL), and this paper expands on this analysis and suggests that the improvement in ICL comes via the propagation of early induction heads.

Critical strengths

I found the writing, presentation, and figures to be clear and compelling. The authors had a clear and interesting hypothesis (based on prior work) and demonstrated the improvement in ICL in a number of convincing ways.

Critical weaknesses

The largest concern that is likely to arise with this paper is the question of degree of advance relative to other papers that have found improved ICL performance (Saunchi et al, Kapl et al). This is not my field, but I took a look at these papers and I think that the present paper is significantly different and makes a nice contribution to the field. I therefore don't see a critical weakness that supports rejection.

Questions for authors

Would it be possible for the authors to more directly demonstrate that the copied heads are causal for the improved ICL? In particular, could the authors perturb the copied induction head and show a larger ICL effect as compared to perturbing a random head?

In some of the figures I was left wondering about statistical significance. I understand not wanting to make the figures ugly (they are currently quite nice), but I think that the style that the authors used in Fig 2b,c (colored region to indicated standard errors) could be used in a variety of other cases. I first thought about this in Fig 1, but I again became concerned about it in Figure 4 (the signal is quite small).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary

We thank all reviewers and the Area Chair for the time and care they invested in evaluating our work. We are glad that the reviewers recognized the clear presentation and figures, the timely insights into depth-grown and looped language models, and the complementary analyses across pretrained models and controlled tasks.

This study systematically examines how reusing computations across depth through depth growing and looping shapes in-context learning (ICL) across pretrained language models up to 3B parameters and controlled tasks. Beyond establishing stronger context adaptation, we separate ICL acquisition from retention against in-weights learning (IWL), trace how layer duplication propagates early induction heads across depth, and show through ablations that grown and looped models rely more strongly on these heads. During the rebuttal, we addressed the reviewers' concerns through the following experiments and manuscript revisions:

  • Memorization vs. generalization trade-off and model capacity. We improved the connection to the established literature on competing ICL and IWL strategies (Sections 2 and 3.4) and clarified how total layer count, unique parameters, and independently initialized parameters shape strategy preference in Figure 3. We also trained all four Omniglot configurations in a new IWL-only control setting where context cannot solve the query, confirming that every model achieves nearly 100% IWL accuracy and that the differences in Figure 3 reflect strategy preference rather than a limit on memorization capacity in this task.
  • Robustness and causal role of induction heads. We verified via logit attribution that prefix-matching heads also exhibit clear token-copying behavior across all models (Table 3, Figure 16), and added all-head score distributions showing that grown and looped models rely on a broader set of strong induction heads rather than concentrating behavior into fewer heads (Figure 17). Building on the appendix random-head baselines, we included random-head controls directly in Figure 6 in the main paper and ran a targeted ablation of the copied induction-head lineage in 3B LIDAS, confirming a substantially larger causal impact on ICL than random or non-copied high-scoring heads.
  • Statistical significance across datasets and scales. We added standard-error bands to the token-loss, GSM8K, and induction-head trajectory plots (Figures 1, 4, 18) and extended our evaluation to three additional datasets across all three model sizes (Figures 8-10), confirming consistent context-adaptation gains over the baseline.

These experiments and manuscript updates resolve all questions and suggestions raised in the reviews, leaving no open concerns. We appreciate the reviewers' constructive feedback, which strengthened the paper, and hope these additions provide a clear basis for the Area Chair's final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.