- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LayerLoop, a variant of depth-recurrent transformers that partitions the recurrent layers into multi-block stages and repeatedly executes each stage. The authors study the effects of stage granularity while controlling stored depth and executed depth. The proposed configuration achieves improvements on a custom diagnostic suite (Bsuite), and the paper further analyzes parameter participation and functional differences across stages.
Critical strengths
- This paper investigates an architectural design choice of depth-recurrent transformers: how does stage granularity affect the trained model when stored and executed depths are fixed?
- This paper provides controlled comparisons across different stage granularities and includes comparisons with several existing recurrent architectures.
Critical weaknesses
- Limited methodological or theoretical novelty or insights. The core contribution appears to be a relatively straightforward exploration of the grouping granularity of existing recurrent transformer architectures. Stage granularity effectively acts as an architectural hyperparameter. The paper does not establish a principle for selecting this hyperparameter, nor does it provide a theoretical explanation. As a result, the contribution appears largely empirical and incremental.
- The main results are based on a nonstandard evaluation metric. The main performance improvements are measured by Bsuite, a nonstandard, complex metric based on percentile rankings across 24 diagnostic components and a reference pool of 58 checkpoints, which is hard to reproduce. Moreover, the authors do not clearly demonstrate the relationship between this metric and standard metrics. This raises concerns about whether Bsuite captures practically meaningful architectural advantages.
- Insufficient evidence of performance improvements. While the paper reports comparisons with several recurrent transformer baselines (Table 11), the results of LayerLoop do not consistently show performance gains. The other experiments also do not show why LayerLoop should be preferred over existing architectures.
Questions for authors
- What motivates the construction of Bsuite? More importantly, what evidence supports its validity as a measure of meaningful model capability?
- What's the advantage of LayerLoop over existing recurrent transformer architectures?
3 comments
As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.
Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.
Thank you all for your time and constructive engagement.
Thank you for clarifying the definition of Bsuite and the availability of conventional metrics. However, my concern is not whether these metrics are reported, but whether the substantial gains in Bsuite reflect meaningful improvements in model capability. In the main width-608 comparison, LayerLoop (L=4) achieves the highest Bsuite but not the best NLL. More importantly, Appendix Table 11 shows no consistent superiority on standard benchmarks. Thus, the validity of Bsuite as evidence for the proposed architecture's practical advantages remains unclear.
Clarification of Contributions, Evaluation Metrics, and Empirical Results
W1: Contribution and insight. We propose a straightforward but effective stage-partitioned recurrent architecture: instead of repeating a single block or the entire middle stack, LayerLoop repeats consecutive multi-block stages before advancing. Section 4.2 (Figure 2) shows that this organization matters: at the same stored and executed depths, an intermediate grouping substantially outperforms the tested blockwise and whole-middle endpoints on the diagnostic suite.
We also investigate why it may help. Section 4.5 (Figure 5) finds less concentrated loss sensitivity across stored parameters and fewer rarely active MLP units, consistent with broader parameter participation. Section 4.6 (Figures 13–15) finds repeatable, stage-specific intervention responses and increased loss when complete trained stages are swapped, suggesting that different stages learn distinguishable, non-interchangeable behavior. Together, these experiments offer possible explanations for the observed benefits beyond the simplicity of the architecture, without claiming that they establish causality.
W2 & Q1: Standard metrics and the meaning of Bsuite. We report conventional measures alongside the aggregate: held-out next-token NLL (Table 1), raw task accuracies (Appendix Table 10), and zero-shot accuracy on seven established benchmark families (Appendix Table 11). These are separate evaluations, not hidden within the aggregate.
Bsuite summarizes 24 pre-training diagnostics, assigning each component a direction-aligned percentile against the same frozen 58-checkpoint reference pool and averaging the resulting percentiles (Section 4.1). Appendix B (Table 4) specifies what each component measures and which direction is better; the accompanying per-component table gives raw values for the main-table baselines, the main LayerLoop checkpoint, and the new Dense control. Thus the aggregate gives a compact overview of specified diagnostics, while the conventional results remain available for direct task-level assessment.
W3 & Q2: Advantage relative to existing architectures. Table 1 and Figure 3 show that LayerLoop achieves the highest Bsuite among the completed storage- and token-matched Dense and recurrent baselines, with gains across several diagnostic domains. Figure 4 additionally shows positive suite gaps against Dense across five tested stored depths:
We also report the full standard-benchmark comparison in Appendix Table 11. Its winners vary by task; on zero-shot MMLU, all eight compact base checkpoints score near chance, so that particular evaluation does not clearly separate them. Our claim is a substantial advantage on the specified pre-training diagnostics, not uniform superiority on every downstream benchmark.