acceptodds
Under review as a conference paper at ICLR 2027

LayerLoop: Stage Granularity in Recurrent-Depth Language Models

Abstract

Recurrent-depth Transformers reuse stored blocks to increase executed depth, but the number of blocks repeated together is another architectural choice. We introduce LayerLoop, which divides the recurrent middle into consecutive multi-block stages and repeats each stage before advancing.This stage-partitioned design spans the space between repeating each block before advancing (blockwise recurrence) and repeating the entire middle region as one unit (whole-middle recurrence).We train and compare stage groupings at matched stored and executed depths, repeat count, and processed tokens. In the four-repeat comparison, four-block stages score 71.2 on a 24-component diagnostic suite of reference-relative percentile ranks, versus 55.4 for blockwise and 57.5 for whole-middle recurrence. Blockwise recurrence instead has the numerically lowest measured next-token prediction loss, showing that stage granularity changes the trained model's quality profile even at fixed depths. At similar parameter storage and processed tokens, the four-block-stage model also leads the tested dense and recurrent baselines on the suite. Across two tested widths, its stored parameters participate more broadly than those of matched dense models. Without predefined roles, its stages develop distinct, repeatable responses when their repeated blocks are bypassed. LayerLoop thus turns stage granularity into an explicit design choice, with trained-model behavior that stored and executed depth counts alone do not describe.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recurrent-depth Transformers, parameter sharing, stage granularity, looped language models, efficient language modeling, model architecture

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CjtfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LayerLoop, a variant of depth-recurrent transformers that partitions the recurrent layers into multi-block stages and repeatedly executes each stage. The authors study the effects of stage granularity while controlling stored depth and executed depth. The proposed configuration achieves improvements on a custom diagnostic suite (Bsuite), and the paper further analyzes parameter participation and functional differences across stages.

Critical strengths
  1. This paper investigates an architectural design choice of depth-recurrent transformers: how does stage granularity affect the trained model when stored and executed depths are fixed?
  2. This paper provides controlled comparisons across different stage granularities and includes comparisons with several existing recurrent architectures.
Critical weaknesses
  1. Limited methodological or theoretical novelty or insights. The core contribution appears to be a relatively straightforward exploration of the grouping granularity of existing recurrent transformer architectures. Stage granularity effectively acts as an architectural hyperparameter. The paper does not establish a principle for selecting this hyperparameter, nor does it provide a theoretical explanation. As a result, the contribution appears largely empirical and incremental.
  2. The main results are based on a nonstandard evaluation metric. The main performance improvements are measured by Bsuite, a nonstandard, complex metric based on percentile rankings across 24 diagnostic components and a reference pool of 58 checkpoints, which is hard to reproduce. Moreover, the authors do not clearly demonstrate the relationship between this metric and standard metrics. This raises concerns about whether Bsuite captures practically meaningful architectural advantages.
  3. Insufficient evidence of performance improvements. While the paper reports comparisons with several recurrent transformer baselines (Table 11), the results of LayerLoop do not consistently show performance gains. The other experiments also do not show why LayerLoop should be preferred over existing architectures.
Questions for authors
  1. What motivates the construction of Bsuite? More importantly, what evidence supports its validity as a measure of meaningful model capability?
  2. What's the advantage of LayerLoop over existing recurrent transformer architectures?
3 comments
AuthorsOpenReview ↗

Clarification of Contributions, Evaluation Metrics, and Empirical Results

W1: Contribution and insight. We propose a straightforward but effective stage-partitioned recurrent architecture: instead of repeating a single block or the entire middle stack, LayerLoop repeats consecutive multi-block stages before advancing. Section 4.2 (Figure 2) shows that this organization matters: at the same stored and executed depths, an intermediate grouping substantially outperforms the tested blockwise and whole-middle endpoints on the diagnostic suite.

We also investigate why it may help. Section 4.5 (Figure 5) finds less concentrated loss sensitivity across stored parameters and fewer rarely active MLP units, consistent with broader parameter participation. Section 4.6 (Figures 13–15) finds repeatable, stage-specific intervention responses and increased loss when complete trained stages are swapped, suggesting that different stages learn distinguishable, non-interchangeable behavior. Together, these experiments offer possible explanations for the observed benefits beyond the simplicity of the architecture, without claiming that they establish causality.

W2 & Q1: Standard metrics and the meaning of BsuiteB_{\mathrm{suite}}. We report conventional measures alongside the aggregate: held-out next-token NLL (Table 1), raw task accuracies (Appendix Table 10), and zero-shot accuracy on seven established benchmark families (Appendix Table 11). These are separate evaluations, not hidden within the aggregate.

BsuiteB_{\mathrm{suite}} summarizes 24 pre-training diagnostics, assigning each component a direction-aligned percentile against the same frozen 58-checkpoint reference pool and averaging the resulting percentiles (Section 4.1). Appendix B (Table 4) specifies what each component measures and which direction is better; the accompanying per-component table gives raw values for the main-table baselines, the main LayerLoop checkpoint, and the new Dense control. Thus the aggregate gives a compact overview of specified diagnostics, while the conventional results remain available for direct task-level assessment.

W3 & Q2: Advantage relative to existing architectures. Table 1 and Figure 3 show that LayerLoop achieves the highest BsuiteB_{\mathrm{suite}} among the completed storage- and token-matched Dense and recurrent baselines, with gains across several diagnostic domains. Figure 4 additionally shows positive suite gaps against Dense across five tested stored depths:

ModelBsuiteB_{\mathrm{suite}} ↑
Dense46.73
Huginn33.19
Mixture-of-Recursions45.73
Ouro48.81
Parcae51.15
LayerLoop71.23

We also report the full standard-benchmark comparison in Appendix Table 11. Its winners vary by task; on zero-shot MMLU, all eight compact base checkpoints score near chance, so that particular evaluation does not clearly separate them. Our claim is a substantial advantage on the specified pre-training diagnostics, not uniform superiority on every downstream benchmark.

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

Thank you for clarifying the definition of Bsuite and the availability of conventional metrics. However, my concern is not whether these metrics are reported, but whether the substantial gains in Bsuite reflect meaningful improvements in model capability. In the main width-608 comparison, LayerLoop (L=4) achieves the highest Bsuite but not the best NLL. More importantly, Appendix Table 11 shows no consistent superiority on standard benchmarks. Thus, the validity of Bsuite as evidence for the proposed architecture's practical advantages remains unclear.

Reviewer_VxRrOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose LayerLoop, a new loop Transformer architecture that groups consecutive blocks into stages and repeats each stage before advancing. Experiments on small-scale models demonstrate its effectiveness.

Critical strengths

The architecture is clearly structured and easy to understand, with a thoughtful combination of block-level and stage-boundary residuals. Experiments show that LayerLoop outperforms Huginn, Mixture-of-Recursions, Ouro, and Parcae on the paper’s diagnostic suite.

Critical weaknesses

Scalability is not sufficiently established. The experiments remain small-scale, with the largest models at approximately 460M parameters. At this scale, model depth may still be insufficient, leaving it unclear whether the architecture retains its advantages when the underlying model already has sufficient depth. The explanation of effectiveness and scalability remains insufficient. The paper needs a more fine-grained analysis of the model’s internal behavior and performance to explain why the architecture works and why it should scale. The contribution of additional computation remains unclear. The approach increases computation through repeated execution without substantially increasing stored parameters. At matched parameter counts and training tokens, additional computation can itself improve performance. The comparisons do not sufficiently distinguish architectural benefits from gains attributable to a larger compute budget.

Questions for authors

At matched training FLOPs, does LayerLoop retain its reported performance advantage over Dense and the recurrent baselines? Please also see Critical Weaknesses.

5 comments
AuthorsOpenReview ↗

Scalability, Internal Mechanisms, and Compute-Matched Comparisons

1. Tested-scale behavior. Section 4.4 (Figure 4) evaluates five Dense–LayerLoop pairs spanning approximately 150M–434M parameters, with LayerLoop achieving higher diagnostic-suite scores in all five. A separate comparison at approximately 456M parameters also favors LayerLoop on both the diagnostic suite and held-out NLL (Table 1, Figure 8).

We agree that larger-scale validation would be valuable. However, a meaningful scaling study requires more than training a single larger LayerLoop model: it also requires corresponding Dense and recurrent baselines, as well as comparisons across stage configurations. Under our current training protocol, even a single 1B-parameter LayerLoop run is estimated to require approximately 1,500 GPU-hours on comparable hardware. A systematic comparison at this scale could require a five-figure budget in USD for cloud GPU resources, depending on the configurations and repetitions. Such a budget exceeds our available computational resources. We therefore focus on controlled experiments at accessible scales, while recognizing billion-parameter validation as an important direction for future work.

2. Internal behavior and possible explanation. Section 4.5 (Figure 5) shows two related observations at widths 608 and 1024: LayerLoop has less concentrated first-order loss sensitivity across its stored parameters and fewer rarely active MLP units than the storage-matched Dense models. Both are consistent with more of the stored parameter budget participating in computation.

Section 4.6 (Figures 13–15) examines the stages individually. Bypassing a given stage produces a response profile that remains recognizable across evaluation samples, while responses of different stages are distinguishable; replacing one complete trained stage with another increases prediction loss in all 12 tested swaps. These observations suggest that staged recurrence may encourage broader parameter use and distinct learned processing across stages. They are empirical clues to effectiveness, rather than a proven causal mechanism or evidence of large-scale behavior.

3. Additional computation. The original cross-architecture comparisons match parameter storage and approximately 10 processed tokens per parameter, rather than training FLOPs. We have now trained an additional FLOP-matched Dense control. LayerLoop is better on 17 of 24 raw diagnostics, ties on 5, and is worse on 2. The persistence of these advantages suggests that they are not merely an effect of additional training FLOPs. The full raw comparison is below (direction follows Appendix Table 4; better values are bold):

Raw component (direction)FLOP-matched DenseLayerLoop
analogy_symbol_acc ↑0.33330.3333
choice_pmi_ece ↓0.14450.1163
code_ece ↓0.46100.2570
controlled_long_acc ↑0.89100.9019
long_context_gain_loss ↑0.23850.2661
copy_induction_ece ↓0.24900.2143
counterfactual_acc ↑0.50000.5833
definition_recall_acc ↑0.55560.5556
few_shot_pmi_acc ↑0.40000.4667
max_repeated_token_run ↓1.62501.3750
knowledge_pmi_acc ↑0.66670.7500
order_ece ↓0.35900.3300
robust_clean_worst_delta ↓0.10000.0667
raw_bits_per_byte ↓1.04101.0348
raw_p95_token_loss ↓8.95838.9167
raw_p99_token_loss ↓11.794811.7844
sampled_temp07_rep ↓0.55210.3880
text_answer_loss ↓6.01005.6099
text_exact ↑0.01560.0625
reading_coreference_acc ↑0.66670.6667
needle_long_acc ↑0.83330.8333
counterfactual_pair_consistency ↑0.33330.1667
needle_acc ↑0.77780.6667
truthfulness_acc ↑0.58330.5833

The other recurrent baselines remain storage- and token-matched rather than FLOP-matched.

Thank you for your response. Could you provide the configuration of the FLOP-matched Dense control, including its depth, width, parameter count, training tokens, and total training FLOPs, and clarify how the FLOPs were matched to LayerLoop?

AuthorsOpenReview ↗

FLOP-Matched Dense Control: Configuration and Compute Accounting

Thank you for the follow-up. The FLOP-matched Dense control continues pretraining from the original Dense checkpoint without changing its architecture. The configurations and compute budgets are summarized below.

ConfigurationDense (FLOP-matched)LayerLoop
Hidden width608608
Stored / executed depth68 / 6868 / 260
Parameters434.11M434.16M
Cumulative training tokens16.015B4.342B
Tokens per parameter (TPP)36.8910.00
Counted training FLOPs~6.157×10196.157\times10^{19}~6.157×10196.157\times10^{19}

FLOP matching. We use the same operator-counting convention for both models, including the forward and backward matrix-multiplication and FlashAttention operations, but excluding gradient-checkpoint recomputation. Under this convention, LayerLoop requires approximately 3.69× the training FLOPs per token of Dense. We therefore extend Dense training from approximately 10 TPP to a target of 36.89 TPP, matching the cumulative counted training FLOPs while keeping its architecture unchanged.

The Dense token count and total FLOPs shown above are the calculated matching targets. We are verifying the exact continuation endpoint and will report the confirmed values.

Reviewer_k2RCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose LayerLoop, a looping variant that parameterizes a looped transformer through L, the number of blocks per looped component (called stage granularity), K, the number of distinct looped parameter sets, and R, the number of loops performed for each parameter set, K. The authors then study pre-training decoder-only transformers and ablate the effects of K, R, and L compared to a number of baselines from the literature. Their main focus is to compare the granularity of L. At the model sizes tried, the authors find that varying the granularity leads to stronger results a higher granularities.

Critical strengths
  • The paper is well written and easy to understand.
  • The Idea of ablating design choices for looping architectures is an interesting research topic in current times.
  • The authors consider a reasonable transformer architecture for pre-training
Critical weaknesses

I have a couple main concerns

  1. It seems to me that the FLOP budget across the different methods is not matched. Therefore, the paper compares methods with wildly different computational costs, making it hard to discern which method is most efficient.
  2. The authors mention that they used a fixed set of default hyperparameters for all methods. It is well known that the optimal hyperparameters for a given method change based on the depth and width of the network [1,2]. Unfortunately, in deep learning, performance can be very sensitive to optimization hyperparameters so making an effort to control for this is essential for a sound study.

References

  1. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
  2. Don't be lazy: CompleteP enables compute-efficient deep transformers
Questions for authors
  • Are the experiments FLOP-matched?
  • What is the impact of looping on inference-time latency
2 comments
AuthorsOpenReview ↗

Clarification of Training Compute, Optimization Settings, and Inference Cost

Response to Reviewer k2RC

W1 & Q1: Training FLOPs. The original Table 1 comparisons were not training-FLOP-matched. Instead, they approximately match stored parameters (around 434M) and training exposure (around 10 processed tokens per parameter), providing a common model-size and data budget.

To directly address your question, we additionally trained a Dense control to approximately matched total training FLOPs. Against this new control, LayerLoop performs better on 17 of 24 raw diagnostic components, ties on 5, and performs worse on 2. The advantage therefore persists on many diagnostics even after matching training computation. The comparison below uses the metric directions in Appendix Table 4 (the more favorable value is bold):

Raw component (direction)FLOP-matched DenseLayerLoop
analogy_symbol_acc ↑0.33330.3333
choice_pmi_ece ↓0.14450.1163
code_ece ↓0.46100.2570
controlled_long_acc ↑0.89100.9019
long_context_gain_loss ↑0.23850.2661
copy_induction_ece ↓0.24900.2143
counterfactual_acc ↑0.50000.5833
definition_recall_acc ↑0.55560.5556
few_shot_pmi_acc ↑0.40000.4667
max_repeated_token_run ↓1.62501.3750
knowledge_pmi_acc ↑0.66670.7500
order_ece ↓0.35900.3300
robust_clean_worst_delta ↓0.10000.0667
raw_bits_per_byte ↓1.04101.0348
raw_p95_token_loss ↓8.95838.9167
raw_p99_token_loss ↓11.794811.7844
sampled_temp07_rep ↓0.55210.3880
text_answer_loss ↓6.01005.6099
text_exact ↑0.01560.0625
reading_coreference_acc ↑0.66670.6667
needle_long_acc ↑0.83330.8333
counterfactual_pair_consistency ↑0.33330.1667
needle_acc ↑0.77780.6667
truthfulness_acc ↑0.58330.5833

W2: Shared optimization settings. We agree that optimal optimization hyperparameters can depend on architecture and scale. To clarify, we shared the AdamW optimization recipe, not every architectural hyperparameter: model width, stored depth, executed depth, and recurrence structure follow the configurations in Appendix Table 8. The common optimizer and learning-rate schedule are described in Appendix A; we did not run separate optimizer hyperparameter searches for individual methods or scales.

Under this shared training recipe, LayerLoop still leads the compact-model suite comparison (Table 1), despite receiving no architecture-specific tuning advantage.

Q2: Inference latency. Appendix A (Inference timing protocol) and Figure 6 report this cost. On an NVIDIA L40 at batch size one, LayerLoop's prefill/decode latency is 3.86×/4.01× that of the storage-matched Dense model at width 608 and 1.80×/1.82× at width 1024. These are uncached, full-context-recomputation measurements; the overhead is consistent with the increased number of serial block applications.

Thank you for the reply.

Unfortunately, given the lack of tuning hyperparameters for each method, it's not possible to distinguish improvements in methodology from poor hyperparameter selection. Your method may improve performance, or it may not; it is not currently possible to tell from the results. I encourage the authors to conduct HP sweeps across methods or at least for the baseline.

I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.