- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper compares ten normalization and residual designs under a controlled width-to-depth reallocation at fixed parameter budget. This paper report that Pre-LN and most norm variants gain nothing from deeper, narrower shapes, while HC and Full AttnRes keep improving in validation loss and downstream NLL. Layer-level analyses (causal and permutation scores, pruning, LogitLens) are used to argue these two use extra layers more effectively.
Critical strengths
- All methods share one recipe and backbone to isolate the effect of more depth.
- A range of diagnostics (causal and permutation scores, pruning, LogitLens) is provided beyond validation loss.
Critical weaknesses
-
The paper does not explain which designs benefit from depth. HC and Full AttnRes improve with depth, while mHC, Block AttnRes and MoDA do not. The paper's own taxonomy in Table 1 doesn't separate these groups. Block AttnRes and MoDA are dynamic and cross-layer, like Full AttnRes, and mHC is dynamic and multi-stream, like HC. So the benchmark reports outcomes without a principle that predicts them, and a practitioner cannot tell whether a new residual design will scale with depth.
-
Limited novelty relative to prior work. The headline result, that HC and AttnRes-style designs use depth better than Pre-LN, follows the motivation of those papers. AttnRes already showed that its optimum shifts toward deeper, narrower shapes, and it already compared against mHC. Then, this paper's contribution is a cleaner and wider comparison, with no new method, metric or design principle. One finding is that Block AttnRes fails to benefit from depth despite the original paper claim that about 8 blocks recover most of Full AttnRes's gain. This result receives little analysis.
Questions for authors
Is there a property of the residual pathway that predicts whether a design benefits from depth? Table 1's taxonomy does not separate the successful designs from the unsuccessful ones.
5 comments
Response to Reviewer k3pr [2/3]
W2 (Part 1): Limited novelty relative to prior work. The headline result, that HC and AttnRes-style designs use depth better than Pre-LN, follows the motivation of those papers. AttnRes already showed that its optimum shifts toward deeper, narrower shapes, and it already compared against mHC. Then, this paper's contribution is a cleaner and wider comparison, with no new method, metric or design principle.
We thank the reviewer for bringing relevant prior works. We acknowledge that prior works have investigated optimum shapes and reported promising results. However, Our goal is not merely to demonstrate better performance or marginal shifts in optimal depth–width configurations, but to systematically compare the depth-scaling capabilities of different architectural designs and understand the underlying mechanisms that make additional depth computationally effective.
- We agree that prior works have studied depth utilization and demonstrated architectural improvements. However, these methods are often evaluated under different configurations, training recipes, and depth–width regimes, making direct comparisons challenging. For example, KEEL [2] focuses on extremely deep, narrow models (aspect ratios 1–16), whereas AttnRes [3] primarily works o much shallower, wider models (aspect ratios around 74). A systematic, apples-to-apples comparison of depth scaling across diverse architectural designs remains lacking.
- DepthBench addresses this gap through controlled comparisons of 10 architectures across a wide range of depth–width ratios under matched model sizes and training recipe, complemented by layerwise diagnostics. Importantly, our study reveals several findings that extend beyond prior observations: (1) Despite their reported architectural improvements, normalization and residual-scaling variants [1,2] do not consistently translate additional depth into performance gains. (2) While AttnRes [3] observed a modest shift toward deeper, narrower optima for Block AttnRes (aspect ratio: 60 → 45), we find that HC and Full AttnRes continue benefiting from suprisingly extreme deep configurations (aspect ratio ≈ 9), revealing greater depth-scaling potential than previously demonstrated. (3) Our layerwise analyses further reveal how residual mechanisms influence computation across depth: HC and Full AttnRes preserve heterogeneous, layer-specific transformations, whereas Pre-LN exhibits increasingly homogeneous representations. These findings provide additional insight into the mechanisms underlying effective depth utilization.
- Notably, strong overall performance does not necessarily imply favorable depth scaling. Our comparisons reveal that Block AttnRes and mHC, despite achieving competitive performance at certain depth–width ratios, do not exhibit the same favorable depth-scaling trends as Full AttnRes and HC, respectively. This distinction highlights why evaluating architecture quality at individual configurations is insufficient to characterize its depth-scaling behavior.
- As depth scaling receives growing attention, understanding which architectural designs actually benefit from increased depth is increasingly important. By combining controlled comparisons with layerwise analyses, our work provides empirical evidence that extends prior findings and helps clarify the architectural principles underlying effective depth scaling.
Response to Reviewer k3pr [3/3]
W2 (Part 2): One finding is that Block AttnRes fails to benefit from depth despite the original paper claim that about 8 blocks recover most of Full AttnRes's gain. This result receives little analysis.
For Block AttnRes, our paper clarifies when it preserves Full AttnRes’s benefits, when it fails to do so, and analyzes its potential hidden cost (Section 3 (Fig. 3); Section 4.3; Appendix D).
The original AttnRes paper’s experiments use relatively shallow-wide shapes, with aspect ratios of approximately 74 (refer Table 2 in [3]). Our results are consistent with this observation (Fig. 3): at shallow–wide shapes with aspect ratios of 56–76, Block AttnRes likewise matches or even outperforms Full AttnRes. The distinction emerges as models become deeper and narrower when aspect ratios are below 42, where Block AttnRes no longer retains Full AttnRes’s favorable scaling trend.
We analyze the underlying cause for this result in Section 4.3 and Appendix D (Fig. 7). With 8 blocks fixed, increasing depth places more sublayers within each block. Fig. 7 shows that, as depth increases, more early layers give almost no weight to previous outputs within the same block. This can reduce the number of effective transformations in these early layers. This provides a possible explanation for why Block AttnRes fails to retain its favorable depth-scaling trend.
[1] Sun W, Song X, Li P, et al. The curse of depth in large language models[J]. Advances in Neural Information Processing Systems, 2026, 38: 163104-163136.
[2] Chen C, Wei L. Post-layernorm is back: Stable, expressive, and deep[J]. arXiv preprint arXiv:2601.19895, 2026.
[3] Team K, Chen G, Zhang Y, et al. Attention residuals[J]. arXiv preprint arXiv:2603.15031, 2026.
Response to Reviewer k3pr [1/3]
Thank you for your valuable feedback. We’ll address your concerns below.
We respectfully disagree that our paper does not explain which designs benefit from depth. Our results demonstrate that the design principles of HC and Full AttnRes enable more effective depth scaling, while efficiency/stability-oriented modifications in mHC and Block AttnRes can compromise these benefits, and their underlying reasons. This distinction is a key finding of our study, rather than a contradiction.
We would like to clarify three points:
Taken together, our study identifies not only which designs benefit from depth, but also when and why these benefits can be compromised. Rather than assuming depth-scaling benefits from architectural labels alone, we provide controlled empirical evidence and layerwise diagnostics to understand their underlying mechanisms.