acceptodds
Under review as a conference paper at ICLR 2027

DepthBench: Measuring How Residual Connections Enable More Computational Depth

Abstract

Depth is a natural way to increase the computational capacity in Transformers, yet the contribution of deeper layers can diminish as depth grows larger. Recent approaches enhance normalization (e.g., LayerNorm Scaling) or residual connections (e.g., mHC, AttnRes) to enable better information flow and depth utilization. However, it remains unclear whether they truly translate increased architectural depth into effective computational depth, and whether their reported gains stem from better access to information across depth, or unaccounted-for confounding factors. In this paper, we introduce DepthBench, a controlled benchmark for studying computational depth across various architectures. We systematically vary the width-depth aspect ratio () from shallow-wide to deep-narrow shapes, while keeping the model size and pre-training recipe fixed. Across 10 representative architectures, we find that the benefit of allocating more capacity to depth is strongly architecture-dependent. Standard Pre-LN and most of its norm- and scaling-based variants provide little benefit and can even degrade performance as models become deeper and narrower, whereas HC and Full AttnRes improve consistently even at extreme deep shapes. These gains extend beyond pre-training loss and consistently translate into improved domain-specific performance and effective computation. Controlled layer-level analyses further show that the gains of HC and Full AttnRes are associated with more effective utilization of additional layers, revealing distinct mechanisms of computational depth across architectures. Overall, our results identify residual connection design as a key determinant of whether depth can serve as a meaningful scaling axis by enabling additional architectural depth to translate into effective computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformer architecture, residual connections, pre-training, depth scaling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_k3prOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper compares ten normalization and residual designs under a controlled width-to-depth reallocation at fixed parameter budget. This paper report that Pre-LN and most norm variants gain nothing from deeper, narrower shapes, while HC and Full AttnRes keep improving in validation loss and downstream NLL. Layer-level analyses (causal and permutation scores, pruning, LogitLens) are used to argue these two use extra layers more effectively.

Critical strengths
  • All methods share one recipe and backbone to isolate the effect of more depth.
  • A range of diagnostics (causal and permutation scores, pruning, LogitLens) is provided beyond validation loss.
Critical weaknesses
  1. The paper does not explain which designs benefit from depth. HC and Full AttnRes improve with depth, while mHC, Block AttnRes and MoDA do not. The paper's own taxonomy in Table 1 doesn't separate these groups. Block AttnRes and MoDA are dynamic and cross-layer, like Full AttnRes, and mHC is dynamic and multi-stream, like HC. So the benchmark reports outcomes without a principle that predicts them, and a practitioner cannot tell whether a new residual design will scale with depth.

  2. Limited novelty relative to prior work. The headline result, that HC and AttnRes-style designs use depth better than Pre-LN, follows the motivation of those papers. AttnRes already showed that its optimum shifts toward deeper, narrower shapes, and it already compared against mHC. Then, this paper's contribution is a cleaner and wider comparison, with no new method, metric or design principle. One finding is that Block AttnRes fails to benefit from depth despite the original paper claim that about 8 blocks recover most of Full AttnRes's gain. This result receives little analysis.

Questions for authors

Is there a property of the residual pathway that predicts whether a design benefits from depth? Table 1's taxonomy does not separate the successful designs from the unsuccessful ones.

5 comments
AuthorsOpenReview ↗

Response to Reviewer k3pr [1/3]

Thank you for your valuable feedback. We’ll address your concerns below.

W1: The paper does not explain which designs benefit from depth. HC and Full AttnRes improve with depth, while mHC, Block AttnRes and MoDA do not. The paper's own taxonomy in Table 1 doesn't separate these groups. Block AttnRes and MoDA are dynamic and cross-layer, like Full AttnRes, and mHC is dynamic and multi-stream, like HC. So the benchmark reports outcomes without a principle that predicts them, and a practitioner cannot tell whether a new residual design will scale with depth.

Q: Is there a property of the residual pathway that predicts whether a design benefits from depth? Table 1's taxonomy does not separate the successful designs from the unsuccessful ones.

We respectfully disagree that our paper does not explain which designs benefit from depth. Our results demonstrate that the design principles of HC and Full AttnRes enable more effective depth scaling, while efficiency/stability-oriented modifications in mHC and Block AttnRes can compromise these benefits, and their underlying reasons. This distinction is a key finding of our study, rather than a contradiction.

We would like to clarify three points:

  • Table 1 serves as a preliminary architectural overview, introducing key structural properties to help readers understand the architectures and experimental setup, rather than serving as a predictive taxonomy.
  • Our results affirm that design principles of HC and Full AttnRes are well-suited for depth scaling, but it would be misleading to assume that simply sharing their high-level properties guarantees the same benefits. Our results show that these benefits are not automatically preserved by their efficiency/stability-oriented variants, mHC and Block AttnRes. This is a key finding of our study, rather than a contradiction, revealing the potential cost of engineering optimizations at scale.
  • Our layerwise analyses provide further mechanistic insights. A common characteristic of effective designs is that additional layers continue to contribute meaningful sequential transformations as depth increases. Specifically, Figures 10 and 11 in Appendix G show that mHC exhibits weakened sequential transformations in late layers relative to HC, whereas Block AttnRes exhibits similar weakening in early layers relative to Full AttnRes. We provide analysis to explain why those variants do not scale depth well (section 4.3, Appendix D) and we refer W2 (Part 2) below for more clarifications. These observations help explain their different depth-scaling behaviors and provide useful diagnostics and predictions beyond model performance.

Taken together, our study identifies not only which designs benefit from depth, but also when and why these benefits can be compromised. Rather than assuming depth-scaling benefits from architectural labels alone, we provide controlled empirical evidence and layerwise diagnostics to understand their underlying mechanisms.

AuthorsOpenReview ↗

Response to Reviewer k3pr [2/3]

W2 (Part 1): Limited novelty relative to prior work. The headline result, that HC and AttnRes-style designs use depth better than Pre-LN, follows the motivation of those papers. AttnRes already showed that its optimum shifts toward deeper, narrower shapes, and it already compared against mHC. Then, this paper's contribution is a cleaner and wider comparison, with no new method, metric or design principle.

We thank the reviewer for bringing relevant prior works. We acknowledge that prior works have investigated optimum shapes and reported promising results. However, Our goal is not merely to demonstrate better performance or marginal shifts in optimal depth–width configurations, but to systematically compare the depth-scaling capabilities of different architectural designs and understand the underlying mechanisms that make additional depth computationally effective.

  • We agree that prior works have studied depth utilization and demonstrated architectural improvements. However, these methods are often evaluated under different configurations, training recipes, and depth–width regimes, making direct comparisons challenging. For example, KEEL [2] focuses on extremely deep, narrow models (aspect ratios 1–16), whereas AttnRes [3] primarily works o much shallower, wider models (aspect ratios around 74). A systematic, apples-to-apples comparison of depth scaling across diverse architectural designs remains lacking.
  • DepthBench addresses this gap through controlled comparisons of 10 architectures across a wide range of depth–width ratios under matched model sizes and training recipe, complemented by layerwise diagnostics. Importantly, our study reveals several findings that extend beyond prior observations: (1) Despite their reported architectural improvements, normalization and residual-scaling variants [1,2] do not consistently translate additional depth into performance gains. (2) While AttnRes [3] observed a modest shift toward deeper, narrower optima for Block AttnRes (aspect ratio: 60 → 45), we find that HC and Full AttnRes continue benefiting from suprisingly extreme deep configurations (aspect ratio ≈ 9), revealing greater depth-scaling potential than previously demonstrated. (3) Our layerwise analyses further reveal how residual mechanisms influence computation across depth: HC and Full AttnRes preserve heterogeneous, layer-specific transformations, whereas Pre-LN exhibits increasingly homogeneous representations. These findings provide additional insight into the mechanisms underlying effective depth utilization.
  • Notably, strong overall performance does not necessarily imply favorable depth scaling. Our comparisons reveal that Block AttnRes and mHC, despite achieving competitive performance at certain depth–width ratios, do not exhibit the same favorable depth-scaling trends as Full AttnRes and HC, respectively. This distinction highlights why evaluating architecture quality at individual configurations is insufficient to characterize its depth-scaling behavior.
  • As depth scaling receives growing attention, understanding which architectural designs actually benefit from increased depth is increasingly important. By combining controlled comparisons with layerwise analyses, our work provides empirical evidence that extends prior findings and helps clarify the architectural principles underlying effective depth scaling.
AuthorsOpenReview ↗

Response to Reviewer k3pr [3/3]

W2 (Part 2): One finding is that Block AttnRes fails to benefit from depth despite the original paper claim that about 8 blocks recover most of Full AttnRes's gain. This result receives little analysis.

For Block AttnRes, our paper clarifies when it preserves Full AttnRes’s benefits, when it fails to do so, and analyzes its potential hidden cost (Section 3 (Fig. 3); Section 4.3; Appendix D).

The original AttnRes paper’s experiments use relatively shallow-wide shapes, with aspect ratios of approximately 74 (refer Table 2 in [3]). Our results are consistent with this observation (Fig. 3): at shallow–wide shapes with aspect ratios of 56–76, Block AttnRes likewise matches or even outperforms Full AttnRes. The distinction emerges as models become deeper and narrower when aspect ratios are below 42, where Block AttnRes no longer retains Full AttnRes’s favorable scaling trend.

We analyze the underlying cause for this result in Section 4.3 and Appendix D (Fig. 7). With 8 blocks fixed, increasing depth places more sublayers within each block. Fig. 7 shows that, as depth increases, more early layers give almost no weight to previous outputs within the same block. This can reduce the number of effective transformations in these early layers. This provides a possible explanation for why Block AttnRes fails to retain its favorable depth-scaling trend.

[1] Sun W, Song X, Li P, et al. The curse of depth in large language models[J]. Advances in Neural Information Processing Systems, 2026, 38: 163104-163136.

[2] Chen C, Wei L. Post-layernorm is back: Stable, expressive, and deep[J]. arXiv preprint arXiv:2601.19895, 2026.

[3] Team K, Chen G, Zhang Y, et al. Attention residuals[J]. arXiv preprint arXiv:2603.15031, 2026.

Reviewer_D3nhOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper conducts a detailed benchmark study of the role and effect of depth in modern transformer architectures. They evaluate 10 different architectures across four families of different normalizations and residual update techniques, varying the width/depth aspect ratio from shallow and wide to narrow and deep.

The motivating question they are focused on is: if you hold the parameters and training tokens fixed, which of their normalizaiton or residual techniques allow you to trade width for more depth.

The main takeaways are that

  1. most normalization or scaling techniques have little benefit as models get deeper and narrower
  2. however, residual connection designs do improve even at extreme depths
Critical strengths

The authors conduct a very rigorous and careful experiment setup. For example, they are careful to use a fixed backbone so that for their fixed parameter width-vs-depth analysis the parameter count doesn't unintentionally shift parameters to the backbone. This allows them to justify the two main claims above: that conventional residual connections (like Pre-LN and layerNorm variants) don't benefit from depth; while other residual connections (like cross-layer residuals) do benefit from depth.

Critical weaknesses

I didn't notice any glaring or major weaknesses on my first read. After discussing the paper details with AI, various potential weaknesses were proposed. The most convincing is using fixed learning rates and initializations across different width and depth ratios and model sizes.

The authors tune peak learning rates at only a single depth (L=24) for the 400M models and hardcode a single fixed LR (5e-4) across all 1.6B models without adjusting for depth and width. But one of the main claims of the paper is that at that 1.6B scale hyper-connections fail to improve with depth. Is that really the problem or is it because of an un-optimized learning rate? How would the depth-scaling curves for Pre-LN change if you did re-tune the LR at extreme depth/widht ratios? Or apply Depth-Aware Hyperparameter Transfer rules?

Questions for authors

The AI pass yielded two very minor typos:

  • Footnote 4 (Page 8): "an sum of the four residual streams" "a sum".
  • Section 5 (Page 9, Line 471): "Our work compare those designs" "compares".
3 comments
AuthorsOpenReview ↗

Response to Reviewer D3nh

We appreciate your careful review and are encouraged that you found our work interesting and robust. We answer your questions below.

W: The authors tune peak learning rates at only a single depth (L=24) for the 400M models and hardcode a single fixed LR (5e-4) across all 1.6B models without adjusting for depth and width. But one of the main claims of the paper is that at that 1.6B scale hyper-connections fail to improve with depth. Is that really the problem or is it because of an un-optimized learning rate? How would the depth-scaling curves for Pre-LN change if you did re-tune the LR at extreme depth/widht ratios? Or apply Depth-Aware Hyperparameter Transfer rules?

We thank the reviewer for raising this important concern. We would like to clarify that we indeed have carefully tuned the learning rates for all depths and methods, including the extreme-depth configurations and 1.6B models. We found that the optimal learning rate remains consistent across different depth–width ratios within each architecture. Therefore, our findings cannot be simply attributed to insufficient hyperparameter tuning.

Regarding depth-aware hyperparameter transfer, we acknowledge that existing approaches such as depth-μP [1] and CompleteP [2] may improve the performance of deeper Pre-LN models. However, all these transfer rules were developed for Pre-LN and do not necessarily generalize to architectures with fundamentally different residual connections, such as HC and AttnRes. Directly applying depth-aware hyperparameter transfer would introduce an unfair comparison to these new techniques. Nevertheless, exploring those transfering techniques to HC and AttnRes are indeed very interesting for future plan.

At the 1.6B scale, the main challenge for HC is its numerical instability rather than simply a lack of learning-rate tuning. This instability is a known limitation of HC and a key motivation behind subsequent work such as mHC [3]. While more extensive architecture-specific tuning may further improve its performance, the optimization difficulty itself represents an important practical challenge for scaling HC to larger models.

Q: The AI pass yielded two very minor typos

We thank reviewers' warm reminders. We will revise them in next version.

[1] Yang G, Yu D, Zhu C, et al. Tensor programs VI: Feature learning in infinite depth neural networks[C]//International Conference on Learning Representations. 2024, 2024: 55099-55150.

[2] Dey N, Zhang B, Noci L, et al. Don't be lazy: CompleteP enables compute-efficient deep transformers[J]. Advances in Neural Information Processing Systems, 2026, 38: 137707-137739.

[3] Xie Z, Wei Y, Cao H, et al. mhc: Manifold-constrained hyper-connections[J]. arXiv preprint arXiv:2512.24880, 2025.

Thank you for your satisfying explanations. I therefore maintain my score.

AuthorsOpenReview ↗

Thank you for your support!

Thank you for your positive feedback and for engaging thoughtfully with our work. We are glad that our explanations addressed your concerns. If you find the paper’s contributions valuable, we would greatly appreciate your support in championing it during the reviewer discussion.

Reviewer_7mjYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a controlled benchmark for studying when architectural depth transaltes into effective computational depth in Transformers. It studies this problem from three aspects: 1. fix overall budget and training recipe, and test different architecture's performance under varying width / depth ratios. 2. layer wise analysis about how adding additional layer contributes to the representation learning of different architectures. 3. computational latency and hardware utilization analysis of the architecture. Baased on extensive experimental results, the paper shows that HC and Full AttnRes consistently benefit from deeper and narrower model comparing to Pre-LN and most normalization-based variants.

Critical strengths
  1. The experimental methods conducted to verify the efficacy of adding depth is comprehensive and rigorous. Specifically, by setting fixed computational budget and training recipe, and varying width / depth ratio; In addition, layer wise analysis further gives a fine grained perspective of how adding layers contribute to the representation learning.
Critical weaknesses
  1. Although the experiments are comprehensive and sound, they serve to only verify what's already been stated in existing papers, i.e., that by attending previous layer with selective weights bring better learning results comparing to traditional residual connections. The paper doesn't bring any new insights about the compared architectures.
Questions for authors

please see weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7mjY

Thank you for your review. We’ll address your concerns below.

W: Although the experiments are comprehensive and sound, they serve to only verify what's already been stated in existing papers, i.e., that by attending previous layer with selective weights bring better learning results comparing to traditional residual connections. The paper doesn't bring any new insights about the compared architectures.

We sincerely appreciate the reviewer’s comments. We respectfully disagree that our work merely verifies existing findings. Our work differs from prior studies in its motivation, methodology, and findings, offering contributions that extend beyond the scope of those studies.

1. Our motivation is different: Prior works primarily propose architectural improvements and demonstrate performance gains at particular model configurations. In contrast, we ask a fundamentally different question: which residual connection designs can enable depth to serve as an effective scaling axis by translating additional architectural depth into meaningful computation in a iso-parameter setting? More importantly, better performance at the standard depth–width ratio does not necessarily imply better depth scaling. Therefore, we target a wide range of aspect ratios. This distinction has been systematically ignored in prior works.

2. Our methodology is different: Controlled evaluation of computational depth along with comprehensive layerwise analysis. Existing methods are evaluated under vastly different architectures, training recipes, and depth–width regimes. For example, KEEL [2] focuses on extremely deep, narrow models (aspect ratios 1–16), whereas AttnRes [3] studies much shallower, wider models (aspect ratios around 74). Their reported gains therefore cannot be cleanly attributed to better depth utilization rather than confounding factors. DepthBench introduces a systematic, apples-to-apples evaluation of 10 architectures across a wide range of depth–width ratios under matched model size and training recipe, complemented by layerwise diagnostics (causal score, permutation score, logitlens, layer pruning, etc.) that directly probe effective computation across depth. This methodology goes beyond the evaluation scope of the original papers.

3. Our findings are new: Revealing residual connection design as a key determinant of whether depth can serve as a meaningful scaling axis. Our study uncovers several previously unestablished insights:

  • Depth-scaling benefits are not universal. Contrary to the motivations of several prior methods [1,2], normalization and residual-scaling variants fail to consistently benefit from increased depth.
  • Residual design can unlock depth as a new scaling axis. While AttnRes [3] reported only a modest shift toward deeper optima for Block AttnRes (aspect ratio: 60 → 45), we show that HC and Full AttnRes surprisingly continue benefiting from substantially deeper configurations (aspect ratio ≈ 9), revealing previously unexplored depth-scaling potential.
  • Different residual mechanisms fundamentally change how computation evolves across depth. Our layerwise analyses reveal that HC and Full AttnRes maintain more heterogeneous, layer-specific transformations, unlike the increasingly homogeneous representations observed in Pre-LN. These findings provide new insights into how residual designs affect effective computational depth.

In summary, our work is not simply another performance comparison of existing architectures. It introduces a distinct research question, establishes a controlled methodology to investigate it, and reveals previously unknown depth-scaling behaviors and computational mechanisms. As depth scaling attracts increasing attention, rigorously establishing which architectural principles actually make additional depth effective is both a timely and important contribution.

[1] Sun W, Song X, Li P, et al. The curse of depth in large language models[J]. Advances in Neural Information Processing Systems, 2026, 38: 163104-163136.

[2] Chen C, Wei L. Post-layernorm is back: Stable, expressive, and deep[J]. arXiv preprint arXiv:2601.19895, 2026.

[3] Team K, Chen G, Zhang Y, et al. Attention residuals[J]. arXiv preprint arXiv:2603.15031, 2026.

Thanks the authors for the detailed response. I agree that the non-universal depth scaling benefits across different residual connections is one of the main findings of the paper. In terms of insights, however, I'm still with the impression that the paper seems to be a re confirmation of the representation characteristics of the compared methods. i.e., the layerwise analysis confirms Full AttnRes maintain more heterogenous, layer specific transformations, which is the motivation of the design stated clearly in the Attention Residual paper. So overall, the main contribution of the paper tends to be empirical findings in cleaner experimental setup, with limited new insights. Therefore I tend to maintain my rating

AuthorsOpenReview ↗

Thank you for acknowledging the non-universal depth-scaling benefits across residual connections as a main finding.

We respectfully disagree with this characterization. The finding about Full AttnRes’s representation characteristics is only a small part of our paper; it does not capture the scope of our research question or our main contribution. Focusing on this observation reduces a broader investigation of depth scaling across residual connections to a confirmation of one method’s design motivation.

Our central question is how residual connections affect the benefits of increasing depth. As you acknowledge, a main finding is that these benefits are not universal across residual designs. Confirming that Full AttnRes maintains heterogeneous, layer-specific transformations does not answer this broader question. That observation may be consistent with Attention Residuals’ motivation, but such consistency does not make our comparative findings a reconfirmation of that motivation.

The distinction matters for assessing novelty: a design rationale for one method and an empirical finding about the limits of generalization across methods are different contributions. Our layerwise analysis is one component of this broader investigation, not the entirety of its insight. Likewise, the controlled experimental setup is what enables a meaningful assessment of these differences, rather than merely a cleaner reproduction of an existing result.

We therefore ask that the novelty assessment reflect the paper’s full scope and its findings across residual designs. The overlap of one observation with Attention Residuals’ stated motivation is not sufficient grounds to characterize the overall contribution as offering limited new insights.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.